Redis Streams and Consumer Groups

The idea: an append-only log inside one key

A Redis stream is a log stored under one key. Producers append entries with XADD; each entry gets an ID and holds a few field-value pairs. Entries are never changed, and nothing is removed by reading: readers keep their own position. A consumer group adds a work queue on top: it hands each entry to one of its consumers and remembers it until that consumer acknowledges it.

The page has one stream, orders, a producer P, a plain reader R and two consumers, c1 and c2, in group g. Pick a demo and press Run Demo, or pick a consumer and a command and press Send. Times and sizes are scaled down: the clock starts at 1000 ms, a node holds 4 entries instead of 100, and a pending entry counts as stuck after 50 ms instead of the minute a real app would use.

Entry IDs and XADD

An ID has two parts, <ms>-<seq>: a Unix time in milliseconds and a sequence number for entries in the same millisecond. With XADD orders * item A the * asks Redis to make the ID. It takes the server clock; if the clock is not ahead of the last ID's milliseconds, Redis keeps those milliseconds and adds 1 to the sequence. So IDs always grow, even when NTP steps the clock back (Demo 1: after the clock drops from 1002 to 999 ms, the next IDs are 1001-1, 1001-2, 1001-3). A reader can rely on "every ID after X" meaning "everything added after X".

Clock vs last ID	New ID	Example (last 1001-0)
clock ahead	clock-0	clock 1002 → 1002-0
same millisecond	ms-(seq+1)	clock 1001 → 1001-1
clock behind	ms-(seq+1)	clock 999 → 1001-1

Storage: a radix tree of listpacks

Inside, the stream is a rax, Redis's radix tree, keyed by entry ID (16 bytes, big-endian, so the tree order is the ID order). Each rax key points to a listpack, a compact byte array holding many entries. A node takes new entries until it reaches stream-node-max-entries (default 100) or stream-node-max-bytes (default 4096); then the next XADD opens a new node. On the page a node holds 4 entries.

The first entry written into a node is its master entry: its ID is the rax key, and its field names are stored once. Every entry after it stores only the difference of its ID from the master (Δ ms, Δ seq, small numbers) and, when its fields are the same as the master's, only the values. That is why a stream of millions of similar events takes little memory. (Demo 1: XADD grows the rax and listpacks)

Reading: XRANGE and XREAD BLOCK

XRANGE orders - + reads a range by ID, like a sorted log. XREAD STREAMS orders 1001-0 returns entries after an ID, and with BLOCK 5000 it waits up to 5 s when there are none yet. The special ID $ means "the last ID in the stream right now": reader R in Demo 1 blocks with $ on an empty stream and wakes up with the first new entry. To keep reading it must pass the last ID it got, not $ again, or it misses entries added between two calls. Every XREAD reader sees every entry; Redis keeps no state for it.

Consumer groups, last_delivered_id and the PEL

XGROUP CREATE orders g 0 makes a group with a starting last_delivered_id (0: the whole stream is new; $: only future entries). XREADGROUP GROUP g c1 COUNT 2 STREAMS orders > asks for entries never delivered to anyone in the group: Redis takes the next 2 after last_delivered_id, moves last_delivered_id forward, and records each entry in the pending entries list (PEL) with the consumer's name, the delivery time and a delivery count of 1. Each consumer also has its own PEL pointing at the same records. Consumers that ask in turn get different entries, so the work is split (Demo 2: two consumers split the entries).

XACK and XPENDING

When a consumer has processed an entry, XACK orders g 1000-0 1001-0 removes it from the group PEL and from the consumer's PEL. The entry itself stays in the stream; only trimming or XDEL removes it. XPENDING orders g reports how many entries are pending and per consumer; the long form XPENDING orders g - + 10 lists each one with its idle time and delivery count (Demo 3: XACK clears the PEL). This is at-least-once delivery: an entry is handed out again until someone acknowledges it, so handlers should be idempotent.

Crashes: re-read with 0, or XAUTOCLAIM

Redis does not track whether a consumer is alive. When c1 crashes, its entries simply stay pending under c1 and their idle time grows. Two ways out:

• c1 comes back. A restarted consumer first calls XREADGROUP GROUP g c1 STREAMS orders 0: an ID instead of > returns its own pending entries (not other consumers'), with the delivery count + 1. When that comes back empty it switches to > (Demo 5: restart, re-read the PEL with 0).
• c1 does not come back. Another consumer runs XAUTOCLAIM orders g c2 50 0-0 COUNT 10: it scans the PEL from 0-0 and takes every entry idle for at least 50 ms (the min-idle-time), whoever owns it. Each one changes owner, its delivery count goes up by 1 and its idle time restarts. The reply starts with a cursor for the next call (0-0 when the scan is done) and ends with IDs that were trimmed meanwhile; those leave the PEL. XCLAIM does the same for IDs you name, usually ones found with XPENDING (Demo 4: crash, then XAUTOCLAIM).

The min-idle-time also keeps two consumers from claiming the same entry: after the first claim the entry is no longer idle.

Poison messages and a dead-letter stream

Some entries can never be processed: a bad payload makes every handler throw. Without a limit such an entry is claimed, fails and is claimed again forever. The usual app pattern uses the delivery count: a recovery loop runs XPENDING orders g IDLE 50 - + 10, and for an entry delivered 3 or more times (the app's MAX_DELIVERIES) it copies the entry to another stream with XADD orders:dlq * … and then calls XACK on the original. Redis has no built-in dead-letter queue; it is just a second stream. (Demo 7: poison message → dead letter)

Trimming: MAXLEN ~

A stream grows until it is trimmed. XTRIM orders MAXLEN ~ 5 (or XADD orders MAXLEN ~ 1000 * …) keeps about the newest entries. With ~ Redis only removes whole nodes from the head, while the rest still holds at least the asked number; removing a node is cheap (free one listpack, delete one rax key). So the stream may keep a few more entries than asked: in Demo 6, 11 entries in nodes of 4, 4 and 3 become 7, not 5. An exact MAXLEN 5 would also have to delete entries inside a node, which only marks them deleted and is slower. MINID trims by ID instead of by count. Trimming ignores consumer groups: an entry can be trimmed while still pending.

Redis Streams vs Kafka consumer groups

	Kafka consumer group	Redis Streams consumer group
Unit of work	a partition, owned by one consumer	a single entry, any consumer
Progress	one committed offset per partition	last_delivered_id + a PEL of unacknowledged entries
Acknowledging	commit offset N: everything before N is done	XACK each entry
Slow or failed record	blocks its partition (or is skipped by committing past it)	others go on; XAUTOCLAIM moves just that entry
Ordering	per partition, kept by its single owner	one order of IDs, but processing order across consumers is not kept
Consumer failure	rebalance reassigns partitions	no rebalance; pending entries wait for a claim
Storage	log segments on disk, replicated	in memory, persisted by RDB/AOF

See how Kafka works for partitions and offsets and Kafka consumer group rebalancing for what happens when a consumer leaves.

See also how Redis serves many connections with one thread (blocking XREAD uses the same event loop), Redis Cluster (a stream is one key, so it lives on one shard), the other Redis internals pages: the Redis dict, sorted sets, compact encodings such as listpack, expiry and eviction, RDB and AOF persistence, replication and MULTI/EXEC transactions; and radix trees, Kafka log segments and the outbox and inbox patterns for idempotent consumers.

What the page leaves out

Several fields per entry and fields that differ from the master entry; the exact listpack byte layout, entry flags and stream-node-max-bytes; XDEL and deleted-entry counters; explicit IDs and XADD … NOMKSTREAM; XRANGE/XREVRANGE and XINFO; several streams in one XREAD; XREADGROUP BLOCK and NOACK; several groups on one stream; XGROUP SETID, CREATECONSUMER and DELCONSUMER; consumer seen-time and active-time; XCLAIM options (JUSTID, FORCE); MINID and LIMIT trimming; how the PEL is persisted and replicated; and Redis 8.2's XDELEX/XACKDEL.

References

Redis documentation: Redis Streams (IDs, XREAD, consumer groups, XAUTOCLAIM)

Redis command reference: XAUTOCLAIM

Redis source: t_stream.c (streamAppendItem, XREADGROUP, the PEL)

Redis source: stream.h (stream, streamCG, streamNACK)