Redis with Celery
Use Redis as the message broker and result backend for Celery task queues.
Celery is a distributed task queue for Python. Your application sends tasks to a broker, and a pool of workers runs them. Redis can act as the broker, which holds the queued tasks, and as the result backend, which stores task results. See the job queue use case for when to choose this pattern.
Requirements
The examples on this page were run with Celery 5.7 and Kombu 5.7 (the messaging library that Celery's Redis transport uses) against Redis 7.2. Install Celery with its Redis dependencies:
pip install "celery[redis]"
You also need a running Redis server. You can run one locally with Redis Open Source, use Docker, or connect to Redis Cloud.
Configure Redis as the broker and result backend
Create a Celery application and point broker and backend at your Redis server. Use
different logical databases if you want to keep queued
tasks and results apart:
from celery import Celery
app = Celery(
"tasks",
broker="redis://<your-redis-host>:6379/0",
backend="redis://<your-redis-host>:6379/1",
)
app.conf.result_expires = 3600 # seconds
Define and call tasks
Decorate a function with @app.task to make it a task. Call delay() to queue it,
or apply_async() to set options such as a delay. The returned result object lets you
wait for the value:
@app.task
def add(x, y):
return x + y
@app.task(bind=True, max_retries=3, default_retry_delay=1)
def flaky(self):
if self.request.retries < 2:
raise self.retry(exc=RuntimeError("smtp down"))
return f"ok after {self.request.retries} retries"
result = add.delay(2, 3)
print(result.get(timeout=10)) # 5
late = add.apply_async((10, 5), countdown=2) # run in about 2 seconds
print(late.get(timeout=10)) # 15
Start a worker in a separate terminal:
celery -A tasks worker --loglevel=INFO --concurrency=2
In this example, calling flaky.delay() fails twice and succeeds on the third attempt, so
the result is ok after 2 retries.
What Celery stores in Redis
With the configuration above, Redis holds the following:
| Key | Type | Contents |
|---|---|---|
celery |
list | Messages for tasks that are waiting for a worker (the default queue). The key exists only while the queue isn't empty. |
_kombu.binding.celery and similar |
set | Routing information that Kombu keeps for each queue. |
celery-task-meta-<task-id> |
string | The task result as JSON, with a time to live (TTL) equal to result_expires. |
With result_expires = 3600, a result key has a TTL of about 3600 seconds.
Keep in-flight tasks safe
By default, a worker acknowledges a task before it runs it, so a worker that crashes
mid-task loses that task. Set task_acks_late so the worker acknowledges the task after
it finishes:
app.conf.task_acks_late = True
With late acknowledgment, Kombu holds each running task in a hash named unacked, with a
sorted set named unacked_index that tracks when to give up on it. Both are empty again
when the task completes. Your tasks must be
idempotent, because a task can run more
than once if a worker dies before acknowledging it.
For Redis, Celery redelivers a task that stays unacknowledged for longer than the
visibility timeout. Set it with broker_transport_options, and make it longer than your
longest task or the longest countdown or eta you use. See the
Celery Redis documentation
for details and caveats:
app.conf.broker_transport_options = {"visibility_timeout": 3600} # seconds