Redis with Asynq
Run Go background tasks with Asynq and Redis.
Asynq is a Go library for queueing tasks and processing them asynchronously with workers. It stores everything in Redis through the go-redis client. See the job queue use case for when to choose this pattern.
Requirements
The examples on this page were run with Asynq 0.26 and Go 1.27 against Redis 7.2. Add the module to your project:
go get github.com/hibiken/asynq
You also need a running Redis server. You can run one locally with Redis Open Source, use Docker, or connect to Redis Cloud.
Enqueue tasks
Create a client with your Redis address, build a task with a type name and a payload, and
enqueue it. Options such as MaxRetry and ProcessIn control how it runs:
import (
"encoding/json"
"time"
"github.com/hibiken/asynq"
)
redisOpt := asynq.RedisClientOpt{Addr: "<your-redis-host>:6379"}
client := asynq.NewClient(redisOpt)
defer client.Close()
payload, _ := json.Marshal(map[string]any{"UserID": 42, "Greeting": "hello"})
// Run as soon as a worker is free.
info, err := client.Enqueue(asynq.NewTask("email:welcome", payload))
// Retry up to 3 times if the handler returns an error.
client.Enqueue(asynq.NewTask("email:flaky", nil), asynq.MaxRetry(3))
// Run no sooner than 3 seconds from now.
client.Enqueue(asynq.NewTask("email:welcome", payload), asynq.ProcessIn(3*time.Second))
Process tasks
Create a server, register a handler for each task type on a mux, and run it. A handler that
returns an error causes a retry. asynq.GetRetryCount tells the handler which attempt this is:
func handleWelcome(ctx context.Context, t *asynq.Task) error {
var p struct {
UserID int
Greeting string
}
if err := json.Unmarshal(t.Payload(), &p); err != nil {
return err
}
fmt.Printf("Sending welcome to %d: %s\n", p.UserID, p.Greeting)
return nil
}
srv := asynq.NewServer(redisOpt, asynq.Config{Concurrency: 2})
mux := asynq.NewServeMux()
mux.HandleFunc("email:welcome", handleWelcome)
if err := srv.Run(mux); err != nil {
log.Fatal(err)
}
Asynq moves scheduled and retried tasks to the queue with a poller that runs about every
5 seconds. In a test, a task scheduled 3 seconds ahead ran after 4.8 seconds, and retries
with a 1-second delay ran 4.2 to 4.6 seconds apart. Don't rely on ProcessIn or on the
retry delay for sub-5-second accuracy.
What Asynq stores in Redis
Asynq keys include the queue name in braces:
| Key | Type | Contents |
|---|---|---|
asynq:{default}:pending |
list | Tasks that are ready to run. |
asynq:{default}:scheduled |
sorted set | Tasks enqueued with ProcessIn, scored by run time. |
asynq:{default}:retry |
sorted set | Tasks whose handler returned an error and that will be retried. |
asynq:{default}:active |
list | Tasks that a worker is processing now. |
asynq:{default}:lease |
sorted set | The lease expiry time of each active task. |
asynq:{default}:t:<task-id> |
hash | The task: msg, state, and pending_since. |
asynq:queues |
set | The names of the queues. |
What happens to a task when a server is killed
A server holds a lease on each active task and renews it while it works. After a server
running a 60-second task was stopped with kill -9, the task stayed in the active state
for about a minute, because its lease had to expire. Then it moved to retry, then to
pending, and a new server ran it again about 73 seconds after the kill. The task went
through the retry state, so Asynq treats the lost run as a failed attempt.
Asynq therefore re-runs a task whose server dies, but only after a delay and at least once. Write handlers so that running one twice is safe.