Celery in python ecosystem. Key concepts cheatsheet

According to official documentation celery is distributed task queue. Let's get familiar with its key concepts to understand it better.

Queue

First thing we are going to look into is queue. Queue is a place where we store messages. A message is an item which holds some kind information like task name, its arguments, date, etc.

Celery worker and tasks

Task is a unit of execution described in python code. Tasks are executed by workers. Worker is a working process with child processes which consume messages from queue and execute tasks described in them.

Distributed

Distributed means that you can distribute celery workers across several machines. This way you can create a cluster of machines and delegate execution across all of them in order to speed up tasks execution.

Brokers

Broker is a special store where our queues are located. There are several brokers available to be used with celery: redis, rabbitmq, sqs, etc. Each broker has its own unique features. Messages inside different brokers are stored and retrieved in different way.

Celery-Beat

Celery beat allows to schedule tasks executions to be called in certain time like crontab.  For example, you can schedule you task to be executed at 12:00 to collect some statistics. In that time celery-beat instance will send another message into broker which will be picked up by celery workers.