Redis with Dramatiq

Run Python background tasks with Dramatiq and Redis.

Dramatiq is a Python library for background task processing. You define actors, which are functions that run in worker processes, and send them messages. Dramatiq supports Redis and RabbitMQ as brokers. See the job queue use case for when to choose this pattern.

Requirements

The examples on this page were run with Dramatiq 2.2 and redis-py against Redis 7.2. Dramatiq 2.2 requires Python 3.10 or later. Install it with its Redis dependencies:

pip install "dramatiq[redis]"

You also need a running Redis server. You can run one locally with Redis Open Source, use Docker, or connect to Redis Cloud.

Configure the Redis broker

Create a RedisBroker and register it with Dramatiq. To store the return values of actors, add the Results middleware with a Redis backend. This example puts results in a different logical database from the queues:

# tasks.py
import dramatiq
from dramatiq.brokers.redis import RedisBroker
from dramatiq.results import Results
from dramatiq.results.backends import RedisBackend

broker = RedisBroker(url="redis://<your-redis-host>:6379/0")
broker.add_middleware(Results(backend=RedisBackend(url="redis://<your-redis-host>:6379/1")))
dramatiq.set_broker(broker)

Define actors and send messages

Decorate a function with @dramatiq.actor. Set store_results=True to keep its return value, and max_retries with a backoff range in milliseconds to retry failures:

@dramatiq.actor(store_results=True)
def add(x, y):
    return x + y

@dramatiq.actor(max_retries=3, min_backoff=500, max_backoff=2000)
def send_email():
    ...  # raise an exception to trigger a retry

Call send() to queue a message, and send_with_options() to set a delay in milliseconds. A message object lets you wait for the stored result:

message = add.send(2, 3)
print(message.get_result(block=True, timeout=10000))  # 5

late = add.send_with_options(args=(10, 5), delay=2000)
print(late.get_result(block=True, timeout=15000))     # 15, after about 3 seconds

Start a worker in a separate terminal, in the directory that contains tasks.py:

dramatiq tasks --processes 1 --threads 2

In a test run, an actor that raised an error twice logged Retrying message for each failure and succeeded on the third attempt.

What Dramatiq stores in Redis

Key Type Contents
dramatiq:default list Internal IDs of the messages that are waiting for a worker (default is the queue name).
dramatiq:default.msgs hash The message payloads, keyed by those internal IDs.
dramatiq:__heartbeats__ sorted set The last heartbeat time of each worker process.
dramatiq:__acks__.<worker-id>.default set The internal IDs of messages that a worker has taken but not yet acknowledged.

Results go to the database that you gave the results backend, one key for each message, with a time to live (TTL) of about 600 seconds by default.

What happens to a message when a worker is killed

A worker takes a message, runs the actor, and acknowledges the message only afterward, so a message that a worker was running when it died stays in that worker's __acks__ set. Another worker puts it back on the queue when two conditions hold. The dead worker's heartbeat must be older than the broker's heartbeat_timeout (60 seconds by default), and a worker must run the broker's maintenance, which Dramatiq triggers for a small random fraction of the broker's Redis commands (maintenance_chance, 1,000 in a million by default).

The test results show how much those defaults matter. After a worker running a 100-second job was stopped with kill -9:

  • With the defaults, a new worker did not run the job again in 204 seconds.
  • With RedisBroker(..., heartbeat_timeout=5000, maintenance_chance=1000000), a new worker started the job 6 seconds after the kill.

Plan for this delay and for double execution: write actors so that running one twice is safe, and set heartbeat_timeout to a value that suits how quickly you need recovery.

More information

RATE THIS PAGE
Back to top ↑