mardi 29 avril 2014

Django - céleri ne tâches pas l'exécution sur Heroku utilisant RabbitMQ - Stack Overflow


Our scheduling stack consists of a Django application with djCelery 3.1.9 and RabbitMQ, and hosted on Heroku with 3 Celery worker dynos and one scheduler. We have scheduled tasks that are grouped to run together at different times. We're finding that some tasks are not executing with the group.


Example:



  1. From a celery inspect schedule this task is scheduled to run on April 3, 2014 at 19:40 UTC:



    • {u'eta': u'2014-04-03T19:40:00+00:00', u'priority': 6, u'request': {u'name': u'api.tasks.close_event', u'time_start': None, u'id': u'c97813af-b070-4b67-8604-da572478e43e', u'kwargs': u'{}', u'args': u'[1002304]', u'delivery_info': {u'exchange': u'celery', u'redelivered': True, u'routing_key': u'celery', u'priority': None}, u'acknowledged': False, u'hostname': u'celery@3508e5c8-c6f7-46cb-be52-79405e9a5112', u'worker_pid': None}}


  2. There are 82 other tasks scheduled for the same time.


  3. Each task takes about 1 second to run.

  4. Out of the 83 tasks, 29 did not run.

  5. A search of the celery inspect schedule output at 19:59 shows that task is no longer in the schedule.

  6. The heroku log shows the following (local timezone is PST so 16:26:13 is 23:26:13 UTC).


Apr 02 16:26:13 ogs-network app/worker.3: [2014-04-02 23:26:13,658: INFO/MainProcess] Received task: api.tasks.close_event[c97813af-b070-4b67-8604-da572478e43e] eta:[2014-04-03 19:40:00+00:00]



  1. So the task is scheduled but did not run at the expected time.

  2. At 14:07:00 PST (21:07:00 UTC), the worker was restarted and the task executed. From the heroku log:


Apr 03 14:07:09 ogs-network app/worker.2: [2014-04-03 21:07:09,596: INFO/MainProcess] Received task: api.tasks.close_event[c97813af-b070-4b67-8604-da572478e43e] eta:[2014-04-03 19:40:00+00:00] Apr 03 14:07:10 ogs-network app/worker.2: [2014-04-03 21:07:10,490: INFO/MainProcess] Task api.tasks.close_event[c97813af-b070-4b67-8604-da572478e43e] succeeded in 0.372298751958s: None



We feel the task is getting scheduled and accepted but not running at the expected time.


Some other settings that we use:


CELERY_ACKS_LATE = FALSE (default) CELERYD_PREFETCH_MULTIPLIER = 4 (default) CELERY_RESULT_BACKEND = "database"


Looking for any and all assistance. Thanks for any help in advance.



Our scheduling stack consists of a Django application with djCelery 3.1.9 and RabbitMQ, and hosted on Heroku with 3 Celery worker dynos and one scheduler. We have scheduled tasks that are grouped to run together at different times. We're finding that some tasks are not executing with the group.


Example:



  1. From a celery inspect schedule this task is scheduled to run on April 3, 2014 at 19:40 UTC:



    • {u'eta': u'2014-04-03T19:40:00+00:00', u'priority': 6, u'request': {u'name': u'api.tasks.close_event', u'time_start': None, u'id': u'c97813af-b070-4b67-8604-da572478e43e', u'kwargs': u'{}', u'args': u'[1002304]', u'delivery_info': {u'exchange': u'celery', u'redelivered': True, u'routing_key': u'celery', u'priority': None}, u'acknowledged': False, u'hostname': u'celery@3508e5c8-c6f7-46cb-be52-79405e9a5112', u'worker_pid': None}}


  2. There are 82 other tasks scheduled for the same time.


  3. Each task takes about 1 second to run.

  4. Out of the 83 tasks, 29 did not run.

  5. A search of the celery inspect schedule output at 19:59 shows that task is no longer in the schedule.

  6. The heroku log shows the following (local timezone is PST so 16:26:13 is 23:26:13 UTC).


Apr 02 16:26:13 ogs-network app/worker.3: [2014-04-02 23:26:13,658: INFO/MainProcess] Received task: api.tasks.close_event[c97813af-b070-4b67-8604-da572478e43e] eta:[2014-04-03 19:40:00+00:00]



  1. So the task is scheduled but did not run at the expected time.

  2. At 14:07:00 PST (21:07:00 UTC), the worker was restarted and the task executed. From the heroku log:


Apr 03 14:07:09 ogs-network app/worker.2: [2014-04-03 21:07:09,596: INFO/MainProcess] Received task: api.tasks.close_event[c97813af-b070-4b67-8604-da572478e43e] eta:[2014-04-03 19:40:00+00:00] Apr 03 14:07:10 ogs-network app/worker.2: [2014-04-03 21:07:10,490: INFO/MainProcess] Task api.tasks.close_event[c97813af-b070-4b67-8604-da572478e43e] succeeded in 0.372298751958s: None



We feel the task is getting scheduled and accepted but not running at the expected time.


Some other settings that we use:


CELERY_ACKS_LATE = FALSE (default) CELERYD_PREFETCH_MULTIPLIER = 4 (default) CELERY_RESULT_BACKEND = "database"


Looking for any and all assistance. Thanks for any help in advance.


0 commentaires:

Enregistrer un commentaire