Kafka Log Segments: How a Partition Is Stored on Disk

The idea: a partition is a folder of segment files

A Kafka partition is an append-only log, but on disk it is not one big file. Each partition is a directory on the broker, named after the topic and the partition number: orders-0/. Inside are segments. Each segment is a group of three files, all named after the segment's base offset, the offset of its first record, padded to 20 digits:

• 00000000000000000006.log: the records themselves, one after another;
• 00000000000000000006.index: the offset index, which finds a record's byte position from its offset;
• 00000000000000000006.timeindex: the time index, which finds an offset from a timestamp.

Splitting the log into segments makes three things cheap: finding the right file for an offset (by name), deleting old data (remove whole files), and keeping each index small. The page draws partition orders-0 on broker B1 with room for four segments. All sizes are scaled down so that a segment fits on the screen:

Setting	On the page	Real default
segment.bytes	300 B	1 GiB
segment.ms	30 s	7 days
index.interval.bytes	80 B	4096 B
retention.ms	60 s	7 days
retention.bytes	-1 (no limit) or 600 B	-1
max.partition.fetch.bytes	150 B	1 MiB

Records are 30 to 70 B each and arrive one every 2 s. A record cell's width is its size in bytes, and the number under it is the byte position where it starts in the .log.

Appending and rolling

Only the last segment is active. Every append goes to the end of its .log; all other segments are closed and never change again. Before each append the broker checks whether the active segment must roll:

• by size: the record would make the segment bigger than segment.bytes;
• by time: the segment's first record is older than segment.ms, even if the segment is small.

Rolling closes the active segment and creates a new one whose base offset is the next offset to be written. In Demo: append and roll, segment 0 holds offsets 0 to 5 (270 B); offset 6 is 70 B, and 270 + 70 = 340 B > 300 B, so segment 6 starts. Demo: retention by time shows a roll by time: after a quiet 40 s, the next append rolls a segment that holds only 140 B. Rolling by time matters for retention and compaction, which only ever work on closed segments: a quiet topic would otherwise keep old data in its active segment for ever.

The sparse offset index and the time index

The offset index does not have an entry for every record. It is sparse: the broker counts the bytes written since the last entry, and when that count is above index.interval.bytes, the next record gets an entry and the count restarts. An entry is two small numbers: the relative offset (offset minus the segment's base offset, so it fits in 4 bytes) and the record's byte position in the .log. On the page, entry 2 → 115 in segment 6 says "offset 8 starts at byte 115".

The time index gets an entry at the same moments: timestamp → relative offset, here t=18 → 2. Both index files are small, sorted and memory-mapped, so a lookup is a binary search in memory.

Reading at an offset

A consumer fetches "partition orders-0 from offset N". The broker finds the record in three steps (Demo: find offset 9):

1. Pick the segment. The segments are kept in a sorted map keyed by base offset. The floor lookup (the largest base offset ≤ N) gives the segment: for N = 9 and bases 0, 6, 11, 17 it is segment 6.
2. Binary-search the .index for the largest entry ≤ N − base. For 9 − 6 = 3 that is entry 2 → 115.
3. Scan the .log forward from that byte, record by record, until offset N. Offset 8 is at byte 115; skip its 35 B; offset 9 is at byte 150.

The scan is short because there is an index entry every index.interval.bytes. A smaller interval means a bigger index and shorter scans. The broker then sends records from offset 9 on, up to max.partition.fetch.bytes, often straight from the page cache to the socket (see Kafka Zero-Copy).

Reading by timestamp

consumer.offsetsForTimes() (the ListOffsets request with a timestamp) answers "what is the first offset whose timestamp is ≥ T?". Tools use it to replay a topic from a point in time, for example kafka-consumer-groups --reset-offsets --to-datetime. The broker:

1. picks the first segment whose newest timestamp is ≥ T;
2. binary-searches its .timeindex for the largest entry with time ≤ T, which gives an offset to start from;
3. looks that offset up in the .index to get a byte position;
4. scans the .log from there to the first record with timestamp ≥ T.

In Demo: find by timestamp (t=31 s), segments 0 and 6 are too old, segment 11 has t=30 → 3, the offset index sends the scan to byte 140 (offset 14, t=30), and the answer is offset 15 (t=32).

Retention deletes whole segments

Kafka does not delete records when they are read. A background task, every log.retention.check.interval.ms (5 minutes), deletes old data, but only whole closed segments, oldest first:

• retention.ms: a segment goes when its newest record is older than retention.ms. A segment therefore lives until its last record expires, so data can stay up to retention.ms + segment.ms (Demo: retention by time).
• retention.bytes: per partition. While the partition is bigger than retention.bytes by at least the size of the oldest segment, that segment goes. The partition can stay above the limit by up to one segment (Demo: retention by size (600 B): 950 B shrinks to 680 B, not 600 B).

The check stops at the first segment it keeps. The active segment is not deleted while it takes writes; if it has expired too, Kafka first rolls a new, empty active segment and then deletes the old ones. Deleted files are first renamed with a .deleted suffix and removed a minute later (file.delete.delay.ms), so that reads in progress can finish.

A consumer behind the log start offset

The first offset still on disk is the log start offset: the base offset of the oldest segment. A consumer group that was stopped longer than the retention period comes back with a committed offset that no longer exists. The broker answers its fetch with OFFSET_OUT_OF_RANGE, and the consumer's auto.offset.reset decides what happens (Demo: consumer offset deleted):

auto.offset.reset	Restarts at	Effect
earliest	the log start offset (6)	reads everything still on disk; records 3 to 5 are lost for it
latest (default)	the log end offset (16)	skips all stored records and only sees new ones
none	nowhere	the consumer gets an exception, and the application decides

The same rule applies to a new group with no committed offset at all. Monitoring consumer lag against the retention period catches this before data is skipped.

See also How Kafka Moves a Message (producers, replication, consumer groups), Kafka Consumer Groups: Rebalancing, Duplicates and Exactly-Once, and the other Kafka internals pages: Kafka Log Compaction, which rewrites closed segments to keep only the latest record per key, Kafka Replication, Kafka KRaft and Kafka Zero-Copy. Other append-only storage: LSM Tree vs B-Tree and Write-Ahead Log Recovery; for the binary search itself, Binary Search vs Linear Search.

What the page leaves out

Record batches (the broker indexes and scans whole batches, not single records), compression, the batch header and CRC, the page cache and fsync (flush.messages, flush.ms: Kafka relies on replication rather than flushing), preallocated index files and their truncation when a segment closes, the extra time-index entry written when a segment rolls, the leader epoch checkpoint and producer snapshot files in the partition directory, log recovery after an unclean shutdown, the high watermark limit on what retention may delete, log.retention.check.interval.ms jitter, tiered storage (KIP-405) and log compaction (cleanup.policy=compact).

References

Apache Kafka documentation: design, persistence and the log

Topic configuration (segment.bytes, segment.ms, index.interval.bytes, retention.ms, retention.bytes)

KIP-33: Add a time based log index

Confluent documentation: Kafka design