Skip to content

Зависание Celery Worker

Если во время релиза, были изменения которые приводят к перезупуску Redis, возможно зависание Celery Worker https://opendistro.prozorro.sale/goto/bac6fffb62c42d24c6428a474569d613?security_tenant=global

Jul 13, 2023 @ 20:32:12.106	[2023-07-13 20:32:12,102: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 336, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 726, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python3.9/site-packages/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/usr/local/lib/python3.9/site-packages/kombu/asynchronous/hub.py", line 373, in create_loop
    cb(*cbargs)
  File "/usr/local/lib/python3.9/site-packages/kombu/transport/redis.py", line 1336, in on_readable
    self.cycle.on_readable(fileno)
  File "/usr/local/lib/python3.9/site-packages/kombu/transport/redis.py", line 566, in on_readable
    chan.handlers[type]()
  File "/usr/local/lib/python3.9/site-packages/kombu/transport/redis.py", line 910, in _receive
    ret.append(self._receive_one(c))
  File "/usr/local/lib/python3.9/site-packages/kombu/transport/redis.py", line 920, in _receive_one
    response = c.parse_response()
  File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 3505, in parse_response
    response = self._execute(conn, conn.read_response)
  File "/usr/local/lib/python3.9/site-packages/redis/client.py", line 3479, in _execute
    return command(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 470, in read_response
    self.read_from_socket()
  File "/usr/local/lib/python3.9/site-packages/redis/connection.py", line 429, in read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.	celery-worker-index
	Jul 13, 2023 @ 20:32:12.107	[2023-07-13 20:32:12,107: WARNING/MainProcess] /usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:386: CPendingDeprecationWarning: 
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered
back to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss
setting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.

  warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)
	celery-worker-index
	Jul 13, 2023 @ 20:32:12.107	[2023-07-13 20:32:12,107: WARNING/MainProcess] /usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:386: CPendingDeprecationWarning: 
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered
back to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss
setting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.

  warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)
	celery-worker-index
	Jul 13, 2023 @ 20:32:12.107	[2023-07-13 20:32:12,107: WARNING/MainProcess] /usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:386: CPendingDeprecationWarning: 
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered
back to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss
setting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.

  warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)
	celery-worker-index
	Jul 13, 2023 @ 20:32:12.111	[2023-07-13 20:32:12,110: WARNING/MainProcess] /usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:498: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(
	celery-worker-index
	Jul 13, 2023 @ 20:32:12.111	[2023-07-13 20:32:12,111: WARNING/MainProcess] /usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:498: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(
	celery-worker-index
	Jul 13, 2023 @ 20:32:12.111	[2023-07-13 20:32:12,111: WARNING/MainProcess] /usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py:498: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(
	celery-worker-index
	Jul 13, 2023 @ 20:32:12.113	[2023-07-13 20:32:12,113: ERROR/MainProcess] consumer: Cannot connect to redis://marketplace-portal-prod-redis-master:6379/1: Error 111 connecting to marketplace-portal-prod-redis-master:6379. Connection refused..
Trying again in 2.00 seconds... (1/100)
	celery-worker-index
	Jul 13, 2023 @ 20:32:12.113	[2023-07-13 20:32:12,112: ERROR/MainProcess] consumer: Cannot connect to redis://marketplace-portal-prod-redis-master:6379/1: Error 111 connecting to marketplace-portal-prod-redis-master:6379. Connection refused..
Trying again in 2.00 seconds... (1/100)
	celery-worker-index
	Jul 13, 2023 @ 20:32:12.114	[2023-07-13 20:32:12,114: ERROR/MainProcess] consumer: Cannot connect to redis://marketplace-portal-prod-redis-master:6379/1: Error 111 connecting to marketplace-portal-prod-redis-master:6379. Connection refused..
Trying again in 2.00 seconds... (1/100)