{ "id": "php", "title": "Redis streaming with Predis", "url": "https://redis.io/docs/latest/develop/use-cases/streaming/php/", "summary": "Implement a Redis event-streaming pipeline in PHP with Predis", "tags": [ "docs", "develop", "stack", "oss", "rs", "rc" ], "last_updated": "2026-05-26T09:29:27-05:00", "children": [], "page_type": "content", "content_hash": "df3d2fc1f594e72d51716180c80762701aa25309b30dee4dff380721e302cac7", "sections": [ { "id": "overview", "title": "Overview", "role": "overview", "text": "This guide shows you how to build a Redis-backed event-streaming pipeline in PHP with [Predis](https://github.com/predis/predis). It includes a small local web server built on PHP's built-in dev server so you can produce events into a single Redis Stream, watch two independent consumer groups read it at their own pace, and recover stuck deliveries with `XAUTOCLAIM` after simulating a consumer crash." }, { "id": "overview", "title": "Overview", "role": "overview", "text": "A Redis Stream is an append-only log of field/value entries with auto-generated, time-ordered IDs. Producers append with [`XADD`](https://redis.io/docs/latest/commands/xadd); consumers belong to *consumer groups* and read with [`XREADGROUP`](https://redis.io/docs/latest/commands/xreadgroup). The group as a whole tracks a single `last-delivered-id` cursor, and each consumer gets its own pending-entries list (PEL) of messages it has been handed but not yet acknowledged. Once a consumer has processed an entry it calls [`XACK`](https://redis.io/docs/latest/commands/xack) to clear the entry from its PEL; entries left unacknowledged past an idle threshold can be reassigned to a healthy consumer with [`XAUTOCLAIM`](https://redis.io/docs/latest/commands/xautoclaim).\n\nThat gives you:\n\n* Ordered, durable history that many independent consumer groups can read at their own pace\n* At-least-once delivery, with per-consumer pending lists and automatic recovery of crashed consumers\n* Horizontal scaling within a group — add a consumer and Redis automatically splits the work\n* Replay of any range with [`XRANGE`](https://redis.io/docs/latest/commands/xrange), independent of consumer-group state\n* Bounded retention through [`XADD MAXLEN ~`](https://redis.io/docs/latest/commands/xadd) or\n [`XTRIM MINID ~`](https://redis.io/docs/latest/commands/xtrim), without a separate cleanup job\n\nIn this example, producers append order events (`order.placed`, `order.paid`, `order.shipped`, `order.cancelled`) to a single stream at `demo:events:orders`. Two consumer groups read the same stream:\n\n* **`notifications`** — two consumers (`worker-a`, `worker-b`) sharing the work, modelling a fan-out worker pool.\n* **`analytics`** — one consumer (`worker-c`) processing the full event flow on its own.\n\nThis port is **structurally different from the other clients in this use case**: every other client runs its consumers as in-process threads or async tasks, but PHP's built-in `php -S` development server runs each HTTP request in a brand-new short-lived process, so an in-process consumer would die as soon as the request that started it returned. The helper sidesteps that by spawning each consumer as a detached OS process and keeping every piece of cross-request state in Redis. See [Production usage](#production-usage) for the longer story." }, { "id": "how-it-works", "title": "How it works", "role": "content", "text": "The flow looks like this:\n\n1. The application calls `$stream->produce($eventType, $payload)` which runs [`XADD`](https://redis.io/docs/latest/commands/xadd) with an approximate [`MAXLEN ~`](https://redis.io/docs/latest/commands/xadd) cap. Redis assigns an auto-generated time-ordered ID.\n2. Each consumer process loops on [`XREADGROUP`](https://redis.io/docs/latest/commands/xreadgroup) with the special ID `>` (meaning \"deliver entries this group has not yet delivered to anyone\") and a short block timeout.\n3. After processing each entry, the consumer calls [`XACK`](https://redis.io/docs/latest/commands/xack) so Redis can drop it from the group's pending list.\n4. If a consumer is killed (or crashes) before acking, its entries sit in the group's PEL. A periodic [`XAUTOCLAIM`](https://redis.io/docs/latest/commands/xautoclaim) sweep reassigns idle entries to a healthy consumer.\n5. Anyone — including code outside the consumer groups — can read history with [`XRANGE`](https://redis.io/docs/latest/commands/xrange) without affecting any group's cursor.\n\nEach consumer group has its own cursor (`last-delivered-id`) and its own pending list, so the two groups in this demo process the same events without coordinating with each other." }, { "id": "the-event-stream-helper", "title": "The event-stream helper", "role": "content", "text": "The `EventStream` class wraps the stream operations\n([source](https://github.com/redis/docs/blob/main/content/develop/use-cases/streaming/php/EventStream.php)):\n\n[code example]" }, { "id": "data-model", "title": "Data model", "role": "content", "text": "Each event is a single stream entry — a flat dict of field/value strings — with an auto-generated time-ordered ID:\n\n[code example]\n\nThe ID is `{milliseconds}-{sequence}`, monotonically increasing within the stream, so you can range-query by approximate wall-clock time without an extra index. (IDs are ordered within a stream, not across streams — two events appended to different streams at the same millisecond can produce the same ID.)\n\nThe PHP port also keeps a small per-process bookkeeping keyspace under `demo:streaming:*` so every fresh `php -S` request can find the running consumers:\n\n[code example]\n\nThe implementation uses:\n\n* [`XADD ... MAXLEN ~ n`](https://redis.io/docs/latest/commands/xadd), pipelined, for batch production with a retention cap\n* [`XREADGROUP`](https://redis.io/docs/latest/commands/xreadgroup) with the special ID `>` for fresh deliveries to a consumer\n* [`XACK`](https://redis.io/docs/latest/commands/xack) on every processed entry\n* [`XAUTOCLAIM`](https://redis.io/docs/latest/commands/xautoclaim) for sweeping idle pending entries to a healthy consumer\n* [`XCLAIM`](https://redis.io/docs/latest/commands/xclaim) for handing one consumer's PEL over to a peer before removing it\n* [`XRANGE`](https://redis.io/docs/latest/commands/xrange) for replay and audit\n* [`XPENDING`](https://redis.io/docs/latest/commands/xpending) for inspecting the per-group pending list\n* [`XINFO STREAM`](https://redis.io/docs/latest/commands/xinfo-stream),\n [`XINFO GROUPS`](https://redis.io/docs/latest/commands/xinfo-groups), and\n [`XINFO CONSUMERS`](https://redis.io/docs/latest/commands/xinfo-consumers) for surface-level observability\n* [`XTRIM`](https://redis.io/docs/latest/commands/xtrim) for explicit retention enforcement" }, { "id": "producing-events", "title": "Producing events", "role": "content", "text": "`produceBatch()` pipelines `XADD` calls in a single round trip. Each call carries an approximate `MAXLEN ~` cap so the stream stays bounded as it rolls forward:\n\n[code example]\n\nThe `~` flavour of `MAXLEN` lets Redis trim at a macro-node boundary, which is much cheaper than exact trimming and is what you want when the cap is a retention *guardrail*, not a hard size constraint. With 300 events produced and `MAXLEN ~ 50`, you might end up with 100 entries left — Redis released the oldest whole macro-node and stopped. The next `XADD` will keep length stable.\n\nIf you genuinely need an exact cap (rare), drop the `~` from the `trim` array. The performance difference is significant on busy streams.\n\nPredis 3.x's `xadd()` takes the fields as an **associative array** in the second argument and the trim options as an `['trim' => ['MAXLEN', '~', n]]` entry in the fourth — a different shape from `hset()`, which takes its fields variadically. Skim the [Predis stream tests](https://github.com/predis/predis/tree/main/tests/Predis/Command/Redis) if you need to confirm the signature for a command this guide doesn't show." }, { "id": "reading-with-a-consumer-group", "title": "Reading with a consumer group", "role": "content", "text": "Each consumer runs the same `XREADGROUP` loop. The special ID `>` means \"deliver entries this group has not yet delivered to *anyone*\":\n\n[code example]\n\n`blockMs` makes the call efficient even when the stream is idle: the client parks on the server until either an entry arrives or the timeout expires, so consumers don't busy-loop.\n\nReading with an explicit ID like `0-0` instead of `>` does something different — it replays entries already delivered to *this* consumer name (its private PEL). That is the canonical recovery path when the same consumer restarts: catch up on its own pending entries first, then resume reading new ones. The helper exposes that as `consumeOwnPel()`." }, { "id": "acknowledging-entries", "title": "Acknowledging entries", "role": "content", "text": "Once the consumer has processed an entry, `XACK` tells Redis it can drop the entry from the group's pending list. Predis 3.x's `xack()` takes the IDs **variadically** rather than as a single array, so the helper unpacks the list with `...`:\n\n[code example]\n\nThis is the linchpin of at-least-once delivery: an entry that is never acked stays in the PEL until a claim moves it elsewhere. If your consumer process crashes between processing and ack, the next claim sweep picks the entry back up. The one caveat is retention: `XADD MAXLEN ~` and `XTRIM` can release the entry's *payload* even while its ID is still in the PEL. The next `XAUTOCLAIM` returns those IDs in its `deletedIds` list and removes them from the PEL inside the same command — the entry cannot be retried, so the caller should log it and route to a dead-letter store for audit.\n\nThe trade-off is the opposite of pub/sub: a slow or crashed consumer doesn't lose messages, but it does mean your downstream system must be idempotent. If you process an order twice because the first attempt died after the side effect but before the ack, the second attempt must be safe." }, { "id": "multiple-consumer-groups-one-stream", "title": "Multiple consumer groups, one stream", "role": "content", "text": "The big difference between Redis Streams and a job queue is that any number of independent consumer groups can read the same stream. The demo sets up two groups on `demo:events:orders`:\n\n[code example]\n\nEach group has its own cursor. Producing 5 events results in `notifications` and `analytics` each receiving all 5, with no coordination between them. Within `notifications`, the work is split across `worker-a` and `worker-b`: Redis hands each `XREADGROUP` call whatever entries are not yet delivered to anyone in the group, so adding a second worker doubles throughput without any rebalance logic.\n\nThe `'0-0'` argument means \"deliver everything in the stream from the beginning\" — useful in a demo and for fresh groups bootstrapping from history. In production, a brand-new group reading a long-existing stream usually starts at `$` (\"only events after this point\") and uses [`XRANGE`](https://redis.io/docs/latest/commands/xrange) explicitly if it needs history." }, { "id": "recovering-crashed-consumers-with-xautoclaim", "title": "Recovering crashed consumers with XAUTOCLAIM", "role": "content", "text": "The demo's \"Crash next 3\" button tells a chosen consumer to drop its next three deliveries on the floor without acking them — the same effect as a worker process dying mid-message. Those entries stay in the group's PEL with their delivery counter incremented. Once they have been idle for at least `claimMinIdleMs`, any healthy consumer in the group can rescue them by calling `XAUTOCLAIM` *with itself as the target*. `ConsumerWorker::reapIdlePel()` wraps that pattern:\n\n[code example]\n\nThe underlying `$stream->autoclaim()` helper pages through the group's PEL with `XAUTOCLAIM`'s continuation cursor:\n\n[code example]\n\nA single `XAUTOCLAIM` call scans up to `pageCount` PEL entries starting at `startId`, reassigns the ones idle for at least `claimMinIdleMs` to the named consumer, and returns a continuation cursor in the first slot of the reply. For a full sweep, loop until the cursor returns to `0-0` (with a `maxPages` safety net so one call cannot monopolise a very large PEL). The delivery counter is incremented on every claim — after a few cycles you can use it to spot a *poison-pill* message that crashes every consumer that touches it, and route it to a dead-letter stream so the bad entry stops cycling. (New entries keep flowing past the poison pill — `XREADGROUP >` still delivers fresh work — but the bad entry's repeated reclaim wastes consumer time and keeps the PEL larger than it needs to be.)\n\nThe `deletedIds` list contains PEL entry IDs whose stream payload was already trimmed by the time the claim ran (typically because `MAXLEN ~` retention outran a slow consumer). `XAUTOCLAIM` removes those dangling slots from the PEL itself, so the caller does *not* need to `XACK` them — but the entries cannot be retried either, so log and route them to a dead-letter store for offline inspection. Redis 7.0 introduced this third return element; the example requires Redis 7.0+ for that reason.\n\n`reapIdlePel` is the right primitive for the recovery path because it claims and processes in one step: every entry the call returned is now in *this* consumer's PEL, so the same consumer is responsible for processing and acking it. In production each consumer process runs `reapIdlePel` periodically (every few seconds, on a timer) so a crashed peer's entries never sit invisibly. The demo exposes it as a manual button so you can trigger the reap after waiting for the idle threshold.\n\n`XCLAIM` (singular, no auto) does the same thing for a specific list of entry IDs you already have in hand — useful when you want to take ownership of one known stuck entry, or when you need to move a specific consumer's PEL to a peer (the case the demo's \"Remove consumer\" button handles via `handoverPending()`). `XAUTOCLAIM` cannot filter by source consumer, so it cannot be used for a per-consumer handover." }, { "id": "replay-with-xrange", "title": "Replay with XRANGE", "role": "content", "text": "`XRANGE` reads a slice of history. It is completely independent of any consumer group — no cursors move, no acks happen — so it is safe to call any number of times, from any process:\n\n[code example]\n\nThe special IDs `-` and `+` mean \"from the very beginning\" and \"to the very end\". You can also pass real IDs (`1716998413541-0`) or just the millisecond part (`1716998413541`, which Redis interprets as \"any entry with this timestamp\").\n\nTypical uses:\n\n* **Bootstrapping a new projection** — read the entire stream from `-` and build a derived view in another store (a search index, a SQL table, a different cache). Doing this against a consumer group would consume the entries; `XRANGE` lets you do it without disrupting live consumers.\n* **Auditing recent activity** — read the last few minutes by ID range without touching any group cursor.\n* **Debugging** — fetch one specific entry by its ID, or a tight range around an incident timestamp, to see exactly what producers wrote." }, { "id": "the-consumer-worker-process", "title": "The consumer worker process", "role": "content", "text": "`ConsumerWorker` wraps the `XREADGROUP` → process → `XACK` loop and is intended to run as a **separate CLI process**\n([source](https://github.com/redis/docs/blob/main/content/develop/use-cases/streaming/php/ConsumerWorker.php)):\n\n[code example]\n\n`handleEntry()` either acks (the normal path) or, when the demo's `crash_next` counter is `> 0`, drops the entry on the floor and increments the per-consumer `crashed_drops` count. The crash check uses a tiny Lua script (`GET` + conditional `DECR`) so two simultaneous deliveries can't both undercount the counter past zero.\n\nRecovery of stuck PEL entries — this consumer's, after a restart, or another consumer's, after a crash — runs through `reapIdlePel()` rather than the read loop. That method calls `XAUTOCLAIM` with this consumer as the target, then processes whatever was claimed in the same flow as new entries. This is the textbook Streams pattern: each consumer is its own reaper, running `XAUTOCLAIM(self)` periodically (or on demand) so a crashed peer's entries never sit invisibly in the PEL. The demo's \"XAUTOCLAIM to selected\" button calls `reapIdlePel()` on the chosen consumer; in production you would run it from a timer every few seconds.\n\nNote that the worker's main read loop deliberately does *not* call `XREADGROUP 0` to drain its own PEL on every iteration. That would re-deliver every pending entry continuously and *reset its idle counter to zero* each time, which would keep crashed entries below the `XAUTOCLAIM` threshold forever. Using `XAUTOCLAIM(self)` as the recovery primitive — which only fires for entries idle longer than `min_idle_time` — avoids that whole class of bug.\n\nThe pause and crash levers exist only for the demo. A real consumer is just the read-process-ack loop — everything else in this class is instrumentation." }, { "id": "prerequisites", "title": "Prerequisites", "role": "content", "text": "* Redis 7.0 or later. `XAUTOCLAIM` was added in Redis 6.2, but its reply gained a third\n element (the list of deleted IDs) in 7.0; the example relies on that shape.\n* PHP 8.1 or later, with the `pcntl` and `posix` extensions enabled (both ship with the\n official PHP binary on macOS and most Linux distros).\n* The Predis client (3.x). Install it with [Composer](https://getcomposer.org/):\n\n [code example]\n\nIf your Redis server is running elsewhere, start the demo with `REDIS_HOST=...` and `REDIS_PORT=...` (see [Start the demo server](#start-the-demo-server) below)." }, { "id": "running-the-demo", "title": "Running the demo", "role": "content", "text": "" }, { "id": "get-the-source-files", "title": "Get the source files", "role": "content", "text": "The demo consists of four files plus the Composer manifest. Download them from the [`php` source folder](https://github.com/redis/docs/tree/main/content/develop/use-cases/streaming/php) on GitHub, or grab them with `curl`:\n\n[code example]\n\nThen install dependencies:\n\n[code example]" }, { "id": "start-the-demo-server", "title": "Start the demo server", "role": "content", "text": "From that directory:\n\n[code example]\n\nYou should see:\n\n[code example]\n\nBy default the demo wipes the configured stream key on startup so each run starts from a clean state. The Composer-built-in `php -S` doesn't accept user CLI flags through to the script, so the demo uses environment variables for the equivalent overrides:\n\n| Env var | CLI equivalent | Default | Meaning |\n|----------------------|----------------------|------------------------|-----------------------------------------------------------------------------------------------|\n| `REDIS_HOST` | `--redis-host` | `127.0.0.1` | Redis host the demo server and every worker connect to. |\n| `REDIS_PORT` | `--redis-port` | `6379` | Redis port. |\n| `STREAM_KEY` | `--stream-key` | `demo:events:orders` | The Redis Stream key the demo writes to and reads from. |\n| `MAXLEN` | `--maxlen` | `2000` | Approximate `MAXLEN ~` cap on every `XADD`. |\n| `CLAIM_IDLE_MS` | `--claim-idle-ms` | `5000` | Minimum idle time before `XAUTOCLAIM` may reassign a pending entry. |\n| `NO_RESET` | `--no-reset` | (reset on first request) | Set to `1` to keep any existing data at `STREAM_KEY` instead of dropping it on first request. |\n| `PROCESS_LATENCY_MS` | — | `25` | Per-entry processing latency the workers simulate (purely for visualisation). |\n\nFor example, to point the demo at a different stream and tighten the autoclaim window:\n\n[code example]\n\nOpen [http://127.0.0.1:8083](http://127.0.0.1:8083) in a browser. You can:\n\n* **Produce** any number of events of a chosen type (or random types). Watch the stream length grow and the tail update.\n* See each **consumer group**: its `last-delivered-id`, the size of its pending list, and the consumers in it. Each consumer shows its processed count, pending count, and idle time.\n* **Add or remove** consumers within a group at runtime to see Redis split the work across the new shape.\n* Click **Crash next 3** on a consumer to drop its next three deliveries — the same effect as a worker process dying after `XREADGROUP` but before `XACK`. Watch the **Pending entries (XPENDING)** panel fill up.\n* Wait until the idle time exceeds the threshold (default 5000 ms), pick a healthy target consumer, and click **XAUTOCLAIM to selected** — the stuck entries are reassigned and the delivery counter increments.\n* **Replay (XRANGE)** any range to confirm the full history is independent of consumer-group state.\n* **XTRIM** with an approximate `MAXLEN` to bound retention. Note that an approximate trim only releases whole macro-nodes — `MAXLEN ~ 50` on a small stream may not delete anything; on a 300-entry stream it typically lands at around 100.\n* Click **Reset demo** to drop the stream, kill every worker, and re-seed the default groups." }, { "id": "stopping-the-demo-cleanly", "title": "Stopping the demo cleanly", "role": "content", "text": "`php -S` doesn't run a shutdown handler when you Ctrl-C out of it, and the consumer worker processes — which are *intentionally* detached so they survive request boundaries — will outlive the demo server unless you stop them first. Before stopping the server, click **Reset demo** in the UI (which kills every worker), or run:\n\n[code example]\n\nIf you forgot, clean up by name:\n\n[code example]" }, { "id": "production-usage", "title": "Production usage", "role": "content", "text": "" }, { "id": "why-this-php-port-differs-from-the-others", "title": "Why this PHP port differs from the others", "role": "content", "text": "Every other client in this use case keeps consumers as in-process objects with a background thread (Python's `threading.Thread`, .NET's task pool, Node's event loop, Go's goroutines, etc.). That works because those runtimes have a long-lived server process that owns the consumer's connection, callback, and dispatch loop.\n\nPHP's traditional one-process-per-request model — used by `php -S`, mod_php, PHP-FPM with the default `pm` setting — fundamentally doesn't fit that shape. A consumer created inside an HTTP handler dies the moment the handler returns. Even if you used a long-running PHP daemon (Roadrunner, Swoole, ReactPHP), you'd still need separate worker processes if you wanted multiple independent consumers, because Predis's blocking `XREADGROUP` call blocks the calling process.\n\nThis port therefore keeps each consumer as a **separate OS process**, with its full state (PID, processed/reaped/dropped counts, recent buffer) persisted in Redis. Every HTTP request reconstructs its view of the consumer registry from those keys. The pattern is closer to how a real production PHP application would run stream consumers: a `supervisord`, `systemd`, or container orchestrator drives N copies of `ConsumerWorker.php`, each owning one logical consumer in one logical group, and the web tier never tries to host a consumer itself. The demo just inlines the supervision (via `proc_open` + `posix_kill`) so a single `php -S` command is enough to play with the pattern end-to-end.\n\nTwo cross-process subtleties are worth calling out, because both will bite anyone who tries to copy this pattern naively:\n\n* **Capture the worker's real PID, not a wrapper's.** A `proc_open(['setsid', '-f', $args...])` call returns the *wrapper's* PID — `setsid -f` forks, exec's the worker as the new session leader, and the wrapper exits. A subsequent `posix_kill($recordedPid, SIGTERM)` then signals a dead PID and the worker survives. The fix is a shell-wrapped `& echo $!` pattern that backgrounds the worker and echoes its real PID back through the wrapper's stdout pipe, which is what `spawnWorker()` in `demo_server.php` does on both Linux and macOS.\n* **Pause/resume across processes uses Redis flags, not in-process events.** The reference Python port uses a `threading.Event` to park a consumer while the demo server hands its PEL off to a peer; this port uses two Redis keys per worker (`paused` and `idle`). The demo server `SET`s `paused=1`, waits for the worker to write `idle=1` (the worker checks the flag at the top of every loop iteration with a 20 ms spin-wait), runs the surgical operation, then `DEL`s both keys. That's what makes the \"Remove consumer\" handover safe even though the demo server can't touch the worker's in-memory state directly." }, { "id": "pick-retention-by-length-or-by-minimum-id", "title": "Pick retention by length or by minimum ID", "role": "content", "text": "The demo uses `MAXLEN ~` on every `XADD`. Two alternatives are worth considering:\n\n* `MINID ~ ` — keep only entries newer than an ID. If you want \"the last 24 hours\", compute the wall-clock cutoff and pass `XTRIM MINID ~ -0`. This is the right pattern when retention is time-bounded.\n* No cap on `XADD` plus a periodic `XTRIM` job — useful if your producer is hot and the per-`XADD` work has to stay minimal, or if retention rules are complex (a separate process can also factor in consumer-group lag).\n\nIn all three cases the trimming is approximate by default. Use exact trimming (`MAXLEN n` or `MINID id` without `~`) only when you genuinely need an exact count." }, { "id": "don-t-let-consumer-group-lag-silently-grow", "title": "Don't let consumer-group lag silently grow", "role": "content", "text": "`XINFO GROUPS` reports each group's `lag` (entries the group has not yet read) and `pending` (entries delivered but not acked). In production, alert on either of these crossing a threshold — a steadily growing pending count usually means consumers are crashing without `XAUTOCLAIM` running, and a growing lag means consumers can't keep up with producers.\n\nThe same applies inside a group: `XINFO CONSUMERS` reports per-consumer pending counts and idle times, so you can spot one slow consumer holding entries that the rest of the group is waiting on." }, { "id": "make-consumer-logic-idempotent", "title": "Make consumer logic idempotent", "role": "content", "text": "`XAUTOCLAIM` can re-deliver an entry to a different consumer after a crash. If your processing has side effects (sending email, charging a card, updating a downstream store), make sure the same entry processed twice gives the same result — use an idempotency key, an upsert with conditional check, or a once-per-id guard table. Redis Streams cannot give you exactly-once semantics on its own." }, { "id": "bound-the-delivery-counter-as-a-poison-pill-signal", "title": "Bound the delivery counter as a poison-pill signal", "role": "content", "text": "`XPENDING` returns each entry's delivery count, incremented on every claim. If an entry has been delivered (and dropped) several times, the next consumer is unlikely to fare better. After some threshold — `deliveries >= 5`, say — route the entry to a *dead-letter stream*, ack it on the original group, and alert. New entries keep flowing past a poison pill (`XREADGROUP >` still delivers fresh work), but the bad entry's repeated reclaim wastes consumer time and keeps the PEL bigger than it needs to be — without a DLQ threshold it can also slowly trip retention/lag alerts." }, { "id": "partition-by-tenant-or-entity-for-scale", "title": "Partition by tenant or entity for scale", "role": "content", "text": "A single Redis Stream is a single key, and on a Redis Cluster a single key lives on a single shard. If your throughput exceeds what one shard can handle, partition the stream — for example by tenant ID (`events:orders:{tenant_a}`, `events:orders:{tenant_b}`) — so different tenants land on different shards. Hash-tags (`{tenant_a}`) keep all related streams on the same shard if you need to multi-stream atomically.\n\nPer-entity partitioning (`events:order:{order_id}`) is the canonical pattern when you treat each entity's stream as the event-sourcing log for that entity: every state change for one order goes on its own stream, which is also bounded in size by the entity's lifetime." }, { "id": "use-a-separate-consumer-pool-per-group", "title": "Use a separate consumer pool per group", "role": "content", "text": "The demo runs every consumer alongside one demo server. In production each consumer group is usually its own deployment — its own pool of pods or VMs — so a slow projection in `analytics` cannot pull `notifications` workers off their stream. Each pod runs one consumer process per CPU core, with `XAUTOCLAIM` either embedded in the consumer loop (every N reads, claim idle entries to self) or run by a separate reaper. `supervisord`, `systemd`, or a container orchestrator owns the process lifecycle, not your web tier." }, { "id": "don-t-read-with-xread-no-group-and-then-try-to-ack", "title": "Don't read with XREAD (no group) and then try to ack", "role": "content", "text": "`XREAD` and `XREADGROUP` are different mechanisms. `XREAD` is a tail-the-log read with no consumer-group state — entries are not added to any PEL, and you cannot `XACK` them. If you want at-least-once delivery and crash recovery, you must read through a consumer group.\n\n`XREAD` is still useful for read-only tail clients (a UI streaming events, a debugger, a `tail -f`-style command-line tool). It's just not part of the at-least-once path." }, { "id": "inspect-the-stream-directly-with-redis-cli", "title": "Inspect the stream directly with redis-cli", "role": "content", "text": "When testing or troubleshooting, inspect the stream directly to confirm the consumer state is what you expect:\n\n[code example]\n\nIf a group's `lag` is growing while consumers' `idle` times are short, consumers are healthy but producers are outpacing them — add more consumers. If `pending` is growing while `lag` is small, consumers are *receiving* entries but not *acking* them — either they are crashing mid-message or your acking logic has a bug." }, { "id": "learn-more", "title": "Learn more", "role": "related", "text": "This example uses the following Redis commands:\n\n* [`XADD`](https://redis.io/docs/latest/commands/xadd) to append an event with an approximate `MAXLEN` cap.\n* [`XREADGROUP`](https://redis.io/docs/latest/commands/xreadgroup) to read new entries for a consumer in a group.\n* [`XACK`](https://redis.io/docs/latest/commands/xack) to acknowledge a processed entry.\n* [`XAUTOCLAIM`](https://redis.io/docs/latest/commands/xautoclaim) to reassign idle pending entries to a healthy consumer.\n* [`XCLAIM`](https://redis.io/docs/latest/commands/xclaim) to reassign specific entry IDs (used here to hand a consumer's PEL over to a peer before deletion).\n* [`XRANGE`](https://redis.io/docs/latest/commands/xrange) for replay and audit, independent of consumer-group state.\n* [`XPENDING`](https://redis.io/docs/latest/commands/xpending) to inspect the per-group pending list with idle times and delivery counts.\n* [`XTRIM`](https://redis.io/docs/latest/commands/xtrim) for explicit retention enforcement.\n* [`XGROUP CREATE`](https://redis.io/docs/latest/commands/xgroup-create) and\n [`XGROUP DELCONSUMER`](https://redis.io/docs/latest/commands/xgroup-delconsumer) to manage groups and consumers.\n* [`XINFO STREAM`](https://redis.io/docs/latest/commands/xinfo-stream),\n [`XINFO GROUPS`](https://redis.io/docs/latest/commands/xinfo-groups), and\n [`XINFO CONSUMERS`](https://redis.io/docs/latest/commands/xinfo-consumers) for observability.\n\nSee the [Predis README](https://github.com/predis/predis) for the full client reference, and the [Streams overview](https://redis.io/docs/latest/develop/data-types/streams) for the deeper conceptual model — consumer groups, the PEL, claim semantics, capped streams, and the differences with Kafka partitions." } ], "examples": [ { "id": "the-event-stream-helper-ex0", "language": "php", "code": "require __DIR__ . '/vendor/autoload.php';\nrequire __DIR__ . '/EventStream.php';\n\nuse Predis\\Client as PredisClient;\n\n$redis = new PredisClient(['host' => '127.0.0.1', 'port' => 6379]);\n$stream = new EventStream(\n $redis,\n 'demo:events:orders',\n 2000, // retention guardrail (approximate MAXLEN)\n 5000 // XAUTOCLAIM idle threshold (ms)\n);\n\n// Producer\n$streamId = $stream->produce('order.placed', [\n 'order_id' => 'o-1234',\n 'customer' => 'alice',\n 'amount' => '49.50',\n]);\n\n// Consumer group + one consumer\n$stream->ensureGroup('notifications', '0-0');\n$entries = $stream->consume('notifications', 'worker-a', count: 10, blockMs: 500);\nforeach ($entries as [$entryId, $fields]) {\n handle($fields); // your processing\n $stream->ack('notifications', [$entryId]); // XACK\n}\n\n// Recover stuck PEL entries by reaping them into a healthy consumer.\n// The textbook pattern: each consumer periodically calls XAUTOCLAIM\n// with itself as the target and processes whatever it claimed.\n// `ConsumerWorker::reapIdlePel()` wraps that flow; the low-level\n// helper `$stream->autoclaim($group, $target)` is also available if\n// you want to drive XAUTOCLAIM directly.\n$result = $worker->reapIdlePel();\n// $result == ['claimed' => N, 'processed' => M, 'deleted_ids' => [...]]\n// deleted_ids are PEL entries whose payload was already trimmed.\n// Redis 7+ has already removed those slots from the PEL, so no XACK\n// is needed — log them and route to a dead-letter store for audit.\n\n// Replay history (independent of any group's cursor)\nforeach ($stream->replay('-', '+', 50) as [$entryId, $fields]) {\n print \"$entryId \" . json_encode($fields) . \"\\n\";\n}", "section_id": "the-event-stream-helper" }, { "id": "data-model-ex0", "language": "text", "code": "demo:events:orders\n 1716998413541-0 type=order.placed order_id=o-1234 customer=alice amount=49.50 ts_ms=...\n 1716998413542-0 type=order.paid order_id=o-1234 customer=alice amount=49.50 ts_ms=...\n 1716998413542-1 type=order.shipped order_id=o-1235 customer=bob amount=12.00 ts_ms=...\n ...", "section_id": "data-model" }, { "id": "data-model-ex1", "language": "text", "code": "demo:streaming:stats (hash) produced_total, acked_total, claimed_total\ndemo:streaming:workers (set) \"{group}/{name}\" entries for every spawned worker\ndemo:streaming:worker:{group}:{name}:pid (string) worker process PID\ndemo:streaming:worker:{group}:{name}:processed (string) per-consumer ack count\ndemo:streaming:worker:{group}:{name}:reaped (string) per-consumer XAUTOCLAIM-claimed count\ndemo:streaming:worker:{group}:{name}:crashed_drops (string) per-consumer drop count\ndemo:streaming:worker:{group}:{name}:crash_next (string) integer counter for the crash lever\ndemo:streaming:worker:{group}:{name}:paused (string) \"1\" while paused, deleted otherwise\ndemo:streaming:worker:{group}:{name}:idle (string) \"1\" while the worker has acknowledged the pause\ndemo:streaming:worker:{group}:{name}:recent (list) last N processed entries for the UI", "section_id": "data-model" }, { "id": "producing-events-ex0", "language": "php", "code": "public function produceBatch(array $events): array\n{\n $stream = $this->streamKey;\n $maxlen = $this->maxlenApprox;\n $results = $this->redis->pipeline(function ($pipe) use ($events, $stream, $maxlen) {\n foreach ($events as [$eventType, $payload]) {\n $fields = self::encodeFields($eventType, $payload);\n // XADD {trim => [MAXLEN, ~, n]}\n $pipe->xadd($stream, $fields, '*', ['trim' => ['MAXLEN', '~', $maxlen]]);\n }\n });\n // ...\n}", "section_id": "producing-events" }, { "id": "reading-with-a-consumer-group-ex0", "language": "php", "code": "public function consume(string $group, string $consumer, int $count = 10, int $blockMs = 500): array\n{\n // Predis xreadgroup signature:\n // xreadgroup(group, consumer, count, block, noack, key, id)\n $raw = $this->redis->xreadgroup(\n $group, $consumer, $count, $blockMs, false,\n $this->streamKey, '>'\n );\n return self::flattenReadGroup($raw);\n}", "section_id": "reading-with-a-consumer-group" }, { "id": "acknowledging-entries-ex0", "language": "php", "code": "public function ack(string $group, array $ids): int\n{\n if (empty($ids)) {\n return 0;\n }\n $n = (int) $this->redis->xack($this->streamKey, $group, ...$ids);\n if ($n > 0) {\n $this->redis->hincrby(self::STATS_KEY, 'acked_total', $n);\n }\n return $n;\n}", "section_id": "acknowledging-entries" }, { "id": "multiple-consumer-groups-one-stream-ex0", "language": "php", "code": "$stream->ensureGroup('notifications', '0-0');\n$stream->ensureGroup('analytics', '0-0');", "section_id": "multiple-consumer-groups-one-stream" }, { "id": "recovering-crashed-consumers-with-xautoclaim-ex0", "language": "php", "code": "public function reapIdlePel(): array\n{\n $result = $this->stream->autoclaim($this->group, $this->name, 100, '0-0', 10);\n $claimed = $result['claimed'];\n $deletedIds = $result['deletedIds'];\n\n $processed = 0;\n foreach ($claimed as [$entryId, $fields]) {\n try {\n $this->handleEntry($entryId, $fields, /*viaReap*/ true);\n $processed++;\n } catch (\\Throwable $exc) {\n fwrite(STDERR, \"[{$this->group}/{$this->name}] reap failed on {$entryId}: \" . $exc->getMessage() . \"\\n\");\n }\n }\n return [\n 'claimed' => count($claimed),\n 'processed' => $processed,\n 'deleted_ids' => $deletedIds,\n ];\n}", "section_id": "recovering-crashed-consumers-with-xautoclaim" }, { "id": "recovering-crashed-consumers-with-xautoclaim-ex1", "language": "php", "code": "public function autoclaim(\n string $group, string $consumer,\n int $pageCount = 100, string $startId = '0-0', int $maxPages = 10\n): array {\n $claimedAll = []; $deletedAll = []; $cursor = $startId;\n for ($i = 0; $i < $maxPages; $i++) {\n $reply = $this->redis->xautoclaim(\n $this->streamKey, $group, $consumer,\n $this->claimMinIdleMs, $cursor, $pageCount\n );\n // Reply shape: [nextCursor, [[id, [k,v,k,v,...]], ...], [deletedIds...]]\n $nextCursor = (string) $reply[0];\n foreach (($reply[1] ?? []) as $entry) {\n $claimedAll[] = [(string) $entry[0], self::pairsToDict($entry[1] ?? [])];\n }\n foreach (($reply[2] ?? []) as $id) { $deletedAll[] = (string) $id; }\n if ($nextCursor === '0-0') break;\n $cursor = $nextCursor;\n }\n return ['claimed' => $claimedAll, 'deletedIds' => $deletedAll];\n}", "section_id": "recovering-crashed-consumers-with-xautoclaim" }, { "id": "replay-with-xrange-ex0", "language": "php", "code": "public function replay(string $startId = '-', string $endId = '+', int $count = 100): array\n{\n $raw = $this->redis->xrange($this->streamKey, $startId, $endId, $count);\n $out = [];\n foreach ($raw as $id => $fields) {\n $out[] = [(string) $id, is_array($fields) ? $fields : []];\n }\n return $out;\n}", "section_id": "replay-with-xrange" }, { "id": "the-consumer-worker-process-ex0", "language": "php", "code": "public function run(): void\n{\n // SIGTERM handler so the demo server's posix_kill gives the\n // worker a chance to leave the loop cleanly.\n $stop = false;\n pcntl_async_signals(true);\n pcntl_signal(SIGTERM, function () use (&$stop) { $stop = true; });\n\n while (!$stop) {\n // Cross-process pause flag (see Production usage).\n if ((string) $this->redis->get($pausedKey) === '1') {\n $this->redis->set($idleKey, '1');\n usleep(20 * 1000);\n continue;\n }\n $entries = $this->stream->consume($this->group, $this->name, 10, 500);\n foreach ($entries as [$entryId, $fields]) {\n usleep($this->processLatencyMs * 1000);\n $this->handleEntry($entryId, $fields, /*viaReap*/ false);\n }\n }\n}", "section_id": "the-consumer-worker-process" }, { "id": "prerequisites-ex0", "language": "bash", "code": "composer require \"predis/predis:^3.0\"", "section_id": "prerequisites" }, { "id": "get-the-source-files-ex0", "language": "bash", "code": "mkdir streaming-demo && cd streaming-demo\nBASE=https://raw.githubusercontent.com/redis/docs/main/content/develop/use-cases/streaming/php\ncurl -O $BASE/EventStream.php\ncurl -O $BASE/ConsumerWorker.php\ncurl -O $BASE/demo_server.php\ncurl -O $BASE/composer.json", "section_id": "get-the-source-files" }, { "id": "get-the-source-files-ex1", "language": "bash", "code": "composer install", "section_id": "get-the-source-files" }, { "id": "start-the-demo-server-ex0", "language": "bash", "code": "php -S 127.0.0.1:8083 demo_server.php", "section_id": "start-the-demo-server" }, { "id": "start-the-demo-server-ex1", "language": "text", "code": "[...] PHP 8.4.6 Development Server (http://127.0.0.1:8083) started", "section_id": "start-the-demo-server" }, { "id": "start-the-demo-server-ex2", "language": "bash", "code": "STREAM_KEY=demo:events:orders-php CLAIM_IDLE_MS=500 php -S 127.0.0.1:8083 demo_server.php", "section_id": "start-the-demo-server" }, { "id": "stopping-the-demo-cleanly-ex0", "language": "bash", "code": "curl -X POST http://127.0.0.1:8083/reset", "section_id": "stopping-the-demo-cleanly" }, { "id": "stopping-the-demo-cleanly-ex1", "language": "bash", "code": "pgrep -f ConsumerWorker.php | xargs kill\nredis-cli --scan --pattern 'demo:streaming:*' | xargs redis-cli del", "section_id": "stopping-the-demo-cleanly" }, { "id": "inspect-the-stream-directly-with-redis-cli-ex0", "language": "bash", "code": "# Stream summary\nredis-cli XLEN demo:events:orders\nredis-cli XINFO STREAM demo:events:orders\n\n# Group cursors and pending counts\nredis-cli XINFO GROUPS demo:events:orders\n\n# Consumers within a group\nredis-cli XINFO CONSUMERS demo:events:orders notifications\n\n# Pending entries with idle time and delivery count\nredis-cli XPENDING demo:events:orders notifications - + 20\n\n# Tail the stream live (no consumer-group state — like tail -f)\nredis-cli XREAD BLOCK 0 STREAMS demo:events:orders '$'\n\n# Replay a range\nredis-cli XRANGE demo:events:orders - + COUNT 50\n\n# The PHP port's own bookkeeping keys\nredis-cli --scan --pattern 'demo:streaming:*'", "section_id": "inspect-the-stream-directly-with-redis-cli" } ] }