Redis with Resque
Run Ruby background jobs with Resque and Redis.
Resque is a Ruby library for creating background jobs, placing them on queues, and processing them later with workers. Redis is its only data store. See the job queue use case for when to choose this pattern.
Requirements
The examples on this page were run with Resque 3.2 on Ruby 4.0 against Redis 7.2. Resque 3.2 uses the redis-rb client. Install the gem:
bundle add resque
You also need a running Redis server. You can run one locally with Redis Open Source, use Docker, or connect to Redis Cloud.
Define and enqueue a job
A Resque job is a Ruby class with a perform class method and a @queue instance variable
that names its queue. Set the Redis connection with Resque.redis:
# jobs.rb
require "resque"
Resque.redis = "<your-redis-host>:6379"
class WelcomeEmail
@queue = :default
def self.perform(user_id, greeting)
puts "Sending welcome to #{user_id}: #{greeting}"
end
end
Call Resque.enqueue with the class and its arguments. Resque stores the arguments as JSON,
so use only simple types:
Resque.enqueue(WelcomeEmail, 42, "hello")
Run a worker
Resque workers run through a rake task. Add a Rakefile that loads your jobs and the Resque
tasks:
# Rakefile
require_relative "jobs"
require "resque/tasks"
Then start a worker for the queue:
QUEUE=default INTERVAL=1 bundle exec rake resque:work
The resque command-line tool doesn't start workers. It lists them (resque list) and
removes or kills a registered worker.
What Resque stores in Redis
| Key | Type | Contents |
|---|---|---|
resque:queue:<name> |
list | Waiting jobs, as JSON with the class and args. |
resque:queues |
set | The names of the queues. |
resque:workers |
set | The registered workers. |
resque:worker:<id> |
string | The job that the worker is processing now, while it works. |
resque:stat:processed |
string | A counter of processed jobs. |
resque:failed |
list | One JSON entry for each failed job. |
Failed jobs
When a job raises an error, Resque doesn't retry it. It adds an entry to resque:failed
with the error, the exception class, the backtrace, the failed_at time, the queue,
the worker, and the original payload. In a test run, a job that raised smtp down
produced an entry with payload FlakyJob and the error message smtp down, and the job did
not run again.
What happens to a job when a worker is killed
Resque forks a child process for each job. After a worker and its child were stopped with
kill -9 partway through a job:
- The queue was empty and
resque:failedstayed empty for the 30 seconds that the test ran, and the job did not run again. The dead worker remained registered. - Resque finds dead workers by their heartbeat. A worker started afterward unregistered the
dead one and moved its job to
resque:failedasResque::DirtyExit("Job still being processed") once the dead worker's heartbeat was older than the prune interval.
The defaults are a heartbeat every 60 seconds and a prune interval of 300 seconds, so by
default a killed worker's job fails after about five minutes. With
Resque.heartbeat_interval = 2 and Resque.prune_interval = 6, the job was in
resque:failed 5 seconds after the kill. Either way, Resque doesn't re-run the job for you,
so write jobs that are safe to run twice and decide what to do with entries in
resque:failed.