Redis with Sidekiq

Run Ruby background jobs with Sidekiq and Redis.

Sidekiq runs background jobs in Ruby, using threads to process many jobs at once. Redis is Sidekiq's only data store: queues, schedules, retries, and statistics all live there. See the job queue use case for when to choose this pattern.

Requirements

Dependency Supported versions
Sidekiq 8.x (the examples were run with 8.1.7)
Redis 7.0 or later (Sidekiq checks this at startup and refuses to run on an older server)
Ruby 3.2 or later (the examples were run with Ruby 4.0.6)

Install the gem:

bundle add sidekiq

You also need a running Redis server. You can run one locally with Redis Open Source, use Docker, or connect to Redis Cloud.

Define a job

Include Sidekiq::Job in a class and write a perform method. Point both the server (worker) and the client (the code that enqueues jobs) at your Redis server:

require "sidekiq"

Sidekiq.configure_server { |config| config.redis = { url: "redis://<your-redis-host>:6379/0" } }
Sidekiq.configure_client { |config| config.redis = { url: "redis://<your-redis-host>:6379/0" } }

class WelcomeEmailJob
  include Sidekiq::Job
  sidekiq_options queue: "default", retry: 3

  def perform(user_id, greeting)
    puts "Sending welcome to #{user_id}: #{greeting}"
  end
end

Arguments must be simple JSON types, because Sidekiq stores the job as JSON in Redis.

Enqueue and run jobs

Call perform_async to run a job as soon as a worker is free, and perform_in to run it later:

WelcomeEmailJob.perform_async(42, "hello")
WelcomeEmailJob.perform_in(60, 43, "a minute later")

Start the worker process in a separate terminal. The -r option loads the file that defines your jobs, and -c sets the number of threads:

bundle exec sidekiq -r ./worker.rb -c 3
Note:

Sidekiq checks for scheduled jobs with a poller that runs about every 5 seconds on average. On a freshly started worker, a job scheduled 2 seconds ahead ran 8 to 10 seconds after it was enqueued. Don't rely on perform_in for sub-second or few-second accuracy.

What Sidekiq stores in Redis

Key Type Contents
queue:<name> list Jobs waiting to run, as JSON.
schedule sorted set Jobs scheduled with perform_in, scored by run time.
retry sorted set Jobs that failed and will be retried, scored by the next attempt time.
dead sorted set Jobs that failed all their retries.
queues and processes set The queue names and the running worker processes.

A job that raises an error moves to the retry set with a retry_count and the error message. In the examples, a job that always raises smtp down appeared in the retry set after its first failure, with retry_count 0 and the next attempt about 11 seconds away. Use Sidekiq::RetrySet and the other classes in sidekiq/api to inspect these sets from Ruby.

What happens to a job when a worker stops

Sidekiq removes a job from its queue with BRPOP when a worker takes it. The job then exists only in the worker's memory, so the result depends on how the worker stops:

  • Graceful shutdown (TERM): Sidekiq waits for running jobs up to the shutdown timeout (-t), then pushes any that are still running back to Redis. With -t 2 and a 30-second job, the log shows Pushed 1 jobs back to Redis, and the job is back in queue:default.
  • Hard kill (kill -9, or a crash): the running job is gone. After a kill -9 of a worker that was running a 30-second job, the queue, retry, and dead were all empty.

Write jobs so that running one twice is safe, and give your process manager a stop timeout longer than your Sidekiq shutdown timeout.

More information

RATE THIS PAGE
Back to top ↑