What is Celery?

Celery is designed to handle asynchronous workloads and can be used for executing long-running tasks, scheduled jobs, or any task that can be performed in the background. It supports various message brokers, such as RabbitMQ and Redis, to manage the task queue and can be integrated with web frameworks like Flask and Django.

Key Features of Celery

  • Asynchronous Task Execution: Allows tasks to run in the background without blocking the main application.
  • Task Scheduling: Supports periodic tasks and scheduling with the built-in Celery Beat.
  • Result Storage: Can store the results of tasks in various backends, such as Redis or a database.
  • Extensible: Easily integrates with existing Python applications and frameworks.

Setting Up Celery

To get started with Celery, you need to install it along with a message broker. In this example, we will use Redis as our message broker. You can install Celery and Redis using pip:

pip install celery redis

Make sure you have Redis installed and running on your machine. You can download it from the official Redis website or use a package manager.

Basic Configuration

Create a new Python file, tasks.py, and set up a simple Celery application:

from celery import Celery

# Create a Celery instance
app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    return x + y

In this example, we create a Celery instance named tasks, specifying Redis as our broker. We define a simple task called add that takes two arguments and returns their sum.

Running the Celery Worker

To execute tasks asynchronously, you need to start a Celery worker. Open a terminal and run the following command in the directory where tasks.py is located:

celery -A tasks worker --loglevel=info

This command starts a worker that listens for tasks sent to the queue. The --loglevel=info option allows you to see detailed logs of task execution.

Sending Tasks to the Queue

Now that the worker is running, you can send tasks to the queue. You can do this from a Python shell or another script. Here’s how to call the add task:

from tasks import add

# Send a task to the queue
result = add.delay(4, 6)

# Check the result
print(f'Task sent! Task ID: {result.id}')

The delay method is a shortcut for sending tasks to the queue. The task will be executed by the worker asynchronously, and you can retrieve the result using the task ID.

Retrieving Task Results

To retrieve the result of a task, you can use the AsyncResult class provided by Celery:

from celery.result import AsyncResult

# Get the result using the task ID
result = AsyncResult(result.id)

if result.ready():
    print(f'Task result: {result.result}')
else:
    print('Task is still processing...')

The ready() method checks if the task has finished processing, and if so, you can access the result through the result attribute.

Periodic Tasks with Celery Beat

Celery also supports periodic tasks using a scheduler called Celery Beat. To define a periodic task, you need to configure the beat_schedule in your Celery application. Here’s an example of a task that runs every 10 seconds:

from celery.schedules import crontab

app.conf.beat_schedule = {
    'add-every-10-seconds': {
        'task': 'tasks.add',
        'schedule': 10.0,
        'args': (16, 16),
    },
}

To run the Celery Beat scheduler, execute the following command in a new terminal window:

celery -A tasks beat --loglevel=info

This command will schedule the add task to run every 10 seconds with the specified arguments.

Best Practices

  1. Use a Dedicated Message Broker: For production, consider using a dedicated RabbitMQ or Redis server instead of the default localhost setup.
  2. Configure Timeouts: Set timeouts for tasks to prevent them from running indefinitely.
  3. Monitor Task Performance: Use tools like Flower or Celery's built-in monitoring features to keep track of task execution and performance.
  4. Error Handling: Implement error handling and retries for tasks to manage failures gracefully.

Conclusion

Celery is a powerful tool for managing asynchronous tasks in Python applications. By following the steps outlined in this tutorial, you can easily set up Celery, create tasks, and run them in the background. This enables your applications to remain responsive and efficient, even when handling long-running processes.

Learn more with useful resources: