Redis with Sidekiq
Run Ruby background jobs with Sidekiq and Redis.
Sidekiq runs background jobs in Ruby, using threads to process many jobs at once. Redis is Sidekiq's only data store: queues, schedules, retries, and statistics all live there. See the job queue use case for when to choose this pattern.
Requirements
| Dependency | Supported versions |
|---|---|
| Sidekiq | 8.x (the examples were run with 8.1.7) |
| Redis | 7.0 or later (Sidekiq checks this at startup and refuses to run on an older server) |
| Ruby | 3.2 or later (the examples were run with Ruby 4.0.6) |
Install the gem:
bundle add sidekiq
You also need a running Redis server. You can run one locally with Redis Open Source, use Docker, or connect to Redis Cloud.
Define a job
Include Sidekiq::Job in a class and write a perform method. Point both the server
(worker) and the client (the code that enqueues jobs) at your Redis server:
require "sidekiq"
Sidekiq.configure_server { |config| config.redis = { url: "redis://<your-redis-host>:6379/0" } }
Sidekiq.configure_client { |config| config.redis = { url: "redis://<your-redis-host>:6379/0" } }
class WelcomeEmailJob
include Sidekiq::Job
sidekiq_options queue: "default", retry: 3
def perform(user_id, greeting)
puts "Sending welcome to #{user_id}: #{greeting}"
end
end
Arguments must be simple JSON types, because Sidekiq stores the job as JSON in Redis.
Enqueue and run jobs
Call perform_async to run a job as soon as a worker is free, and perform_in to run it
later:
WelcomeEmailJob.perform_async(42, "hello")
WelcomeEmailJob.perform_in(60, 43, "a minute later")
Start the worker process in a separate terminal. The -r option loads the file that
defines your jobs, and -c sets the number of threads:
bundle exec sidekiq -r ./worker.rb -c 3
Sidekiq checks for scheduled jobs with a poller that runs about every 5 seconds on average.
On a freshly started worker, a job scheduled 2 seconds ahead ran 8 to 10 seconds after it was
enqueued. Don't rely on perform_in for sub-second or few-second accuracy.
What Sidekiq stores in Redis
| Key | Type | Contents |
|---|---|---|
queue:<name> |
list | Jobs waiting to run, as JSON. |
schedule |
sorted set | Jobs scheduled with perform_in, scored by run time. |
retry |
sorted set | Jobs that failed and will be retried, scored by the next attempt time. |
dead |
sorted set | Jobs that failed all their retries. |
queues and processes |
set | The queue names and the running worker processes. |
A job that raises an error moves to the retry set with a retry_count and the error
message. In the examples, a job that always raises smtp down appeared in the retry set
after its first failure, with retry_count 0 and the next attempt about 11 seconds
away. Use Sidekiq::RetrySet and the other classes in sidekiq/api to inspect these sets
from Ruby.
What happens to a job when a worker stops
Sidekiq removes a job from its queue with BRPOP when a
worker takes it. The job then exists only in the worker's memory, so the result depends on
how the worker stops:
- Graceful shutdown (
TERM): Sidekiq waits for running jobs up to the shutdown timeout (-t), then pushes any that are still running back to Redis. With-t 2and a 30-second job, the log showsPushed 1 jobs back to Redis, and the job is back inqueue:default. - Hard kill (
kill -9, or a crash): the running job is gone. After akill -9of a worker that was running a 30-second job, the queue,retry, anddeadwere all empty.
Write jobs so that running one twice is safe, and give your process manager a stop timeout longer than your Sidekiq shutdown timeout.