Process Redis streams with Spring

Add entries to a Redis stream and consume them with a consumer group in a Spring application.

Spring Data Redis supports Redis streams through the template (opsForStream()) and through a stream message listener container that reads entries in the background. See the streaming use case for when to choose a stream.

Add entries

Use StreamRecords to build a record and add it to the stream with the template:

template.opsForStream().add(
        StreamRecords.string(Map.of("sku", "a1")).withStreamKey("orders"));

Consume with a consumer group

A consumer group shares the entries of a stream between several consumers, and tracks which entries each consumer has processed. The following configuration creates the container, creates the group, and registers a listener that acknowledges each entry after processing it:

@Configuration
public class StreamsConfig {
    static final String STREAM = "orders";
    static final String GROUP = "order-workers";

    @Bean(initMethod = "start", destroyMethod = "stop")
    StreamMessageListenerContainer<String, MapRecord<String, String, String>> streamContainer(
            RedisConnectionFactory connectionFactory) {
        var options = StreamMessageListenerContainerOptions.builder()
                .pollTimeout(Duration.ofSeconds(1))
                .build();
        return StreamMessageListenerContainer.create(connectionFactory, options);
    }

    @Bean
    Subscription orderSubscription(
            StreamMessageListenerContainer<String, MapRecord<String, String, String>> container,
            StringRedisTemplate template) {
        // Creates the stream if needed. Throws if the group already exists.
        try {
            template.opsForStream().createGroup(STREAM, ReadOffset.from("0"), GROUP);
        } catch (Exception e) {
            // The group already exists.
        }

        return container.receive(
                Consumer.from(GROUP, "worker-1"),
                StreamOffset.create(STREAM, ReadOffset.lastConsumed()),
                message -> {
                    process(message.getValue());
                    template.opsForStream().acknowledge(STREAM, GROUP, message.getId());
                });
    }
}

The createGroup call creates the stream if it doesn't exist, and the container must find the group before it starts reading. ReadOffset.lastConsumed() asks for entries that no consumer in the group has received yet.

Find and claim pending entries

An entry stays in the group's pending list until a consumer acknowledges it. If a consumer stops before acknowledging, its entries stay pending. List them with pending() and take them over with claim():

PendingMessages pending = template.opsForStream()
        .pending(STREAM, GROUP, Range.unbounded(), 10);

for (PendingMessage entry : pending) {
    List<MapRecord<String, Object, Object>> claimed = template.opsForStream()
            .claim(STREAM, GROUP, "rescuer", Duration.ofMinutes(1), entry.getId());
    // Process the claimed entries, then acknowledge them.
}

The claim call transfers only the entries that have been idle for at least the duration you give, so a consumer that is still working isn't interrupted.

For a reactive alternative to the listener container, see Redis Streams in the Spring Data Redis documentation.

RATE THIS PAGE
Back to top ↑