Kafka Zero Copy Visualization | sendfile, Page Cache & DMA

The idea: serve a fetch without touching the bytes

A Kafka broker spends most of its time doing one thing: sending bytes that are already in a log file to consumers. The broker does not need to look at those bytes. It stores record batches in exactly the format the consumer will receive, so a fetch is just "send bytes X to Y of this file to that socket". Zero copy means letting the kernel do exactly that, without the data ever passing through the JVM.

Left: read() plus write(). Disk to page cache by DMA (1), page cache to heap buffer by the CPU (2), heap buffer to socket buffer by the CPU (3), socket buffer to NIC by DMA (4): 2 DMA plus 2 CPU copies and 4 context switches. Right: sendfile(). Disk to page cache by DMA (1), only a pointer goes to the socket buffer, and the NIC reads the page cache by gather DMA (2): 2 DMA copies, 0 CPU copies, 2 context switches — With sendfile() the data goes from the page cache straight to the NIC; the CPU copies nothing and the broker's JVM never holds the bytes.

The canvas splits the machine into three regions. User space is the broker JVM: a network thread and a heap buffer, plus an SSLEngine used only on an SSL listener. The kernel holds the page cache and the socket buffers. The hardware is the disk and the network card (NIC); both move data with DMA (direct memory access), so the CPU does not copy those bytes. The yellow CPU box shows whether the CPU is running user code or kernel code. The scoreboard counts the copies, the context switches and the CPU time of the last request and of all requests so far. Sizes are scaled: one cell is one 4 KB page, a fetch moves one or two pages, and CPU time is in made-up units (1 per mode switch, 2 per page copied by the CPU, 4 per page encrypted).

Four copies, four switches: read() + write()

The plain way to send a file is a loop of read(file, buf) and write(socket, buf). Each call is a system call, which switches the CPU from user mode to kernel mode and back. (These are mode switches, not switches between threads; the page counts each one as a context switch, as the classic zero-copy articles do.)

On a page cache miss the disk copies the page into the page cache by DMA.
read(): the CPU copies it from the page cache into the user buffer.
write(): the CPU copies it from the user buffer into the socket buffer.
The NIC copies it from the socket buffer to the card by DMA.

That is 4 copies (2 of them by the CPU) and 4 switches per chunk. The two CPU copies also push useful data out of the CPU caches, and the user buffer is one more place the same bytes take up memory (Demo: read() + write(), 4 copies). Kafka does not serve fetches this way; it is the baseline.

sendfile() and FileChannel.transferTo

Kafka sends log data with FileChannel.transferTo(), which the JDK turns into the Linux sendfile() system call. The kernel finds the pages in the page cache and appends only pointers to them (page, offset, length) to the socket buffer. A NIC with scatter-gather DMA then reads the bytes directly from the page cache. The result: 2 DMA copies, 0 CPU copies and 2 switches (Demo: sendfile(), 0 CPU copies).

Path	DMA copies	CPU copies	Switches
read() + write(), page on disk	2	2	4
read() + write(), page in cache	1	2	4
sendfile(), page on disk	2	0	2
sendfile(), page in cache	1	0	2
SSL listener, page in cache	1	2 + encryption	4

This only works because the broker never changes the data. The producer compresses and batches records, the broker stores the batches as they are, and the consumer decompresses them. A broker that had to convert message formats for old clients lost zero copy for those clients.

The page cache is Kafka's cache

Kafka keeps no cache of its own in the JVM heap. It relies on the operating system's page cache: free RAM that holds recently read and written file pages. A tailing consumer, one that reads just behind the producer, asks for pages that were written a moment ago, so they are still in RAM and the fetch causes no disk I/O at all (Demo: tailing consumer hits the cache). This is why Kafka brokers run with a small heap and lots of memory left to the OS.

A lagging consumer (a new consumer group, a replay from an old offset, a consumer that was down) reads old pages that are only on disk. Each miss blocks the thread on a disk read, and the pages it brings in push other pages out. The page cache evicts the least recently used page first, so a big catch-up read can evict exactly the recent pages the tailing consumers need, and they start missing too (Demo: lagging consumer reads cold).

Four rows of a 4-page page cache. Start: p2 p3 p4 p5, the tailing consumer's p5 is in RAM. The lagging consumer reads p0 and p1, evicting p2 and p3, then p2 and p3, evicting p4 and p5. The tailing consumer's read of p5 now misses and goes to disk, evicting p0 — A lagging consumer's catch-up read fills the page cache with old pages and evicts the tail, so the tailing consumer now reads from disk too.

Real Linux uses two LRU lists (active and inactive), which protects pages read twice, but a large scan still hurts. Kafka 3.6 added tiered storage partly for this: old segments are read from remote storage instead of the broker's disk and page cache.

read() vs mmap() on the same page cache, with page faults, dirty pages and msync(), is on The Page Cache: read() vs mmap().

TLS: why the copies come back

On an SSL listener the bytes on the wire are not the bytes in the file: they are encrypted, with a key for this one connection. The JVM's SSLEngine does the encryption, in user space, so Kafka must read() the data into the JVM, encrypt it into a new buffer and write() the result. All the copies and switches of the plain path come back, plus the encryption itself (Demo: TLS, the copies come back). This is the main reason TLS costs Kafka brokers noticeable CPU.

kTLS (kernel TLS) moves the record encryption into the Linux kernel, or even onto the NIC, after a user-space handshake; then sendfile() works again with encryption. The Java broker does not use kTLS (the JDK's TLS stack does not support it), so on an SSL listener Kafka copies.

The write path: page cache, no fsync

A produce request is the other direction. The NIC copies it into the socket receive buffer by DMA, the network thread reads it into the JVM (the broker must look at it: it checks the CRC and assigns offsets), and the log appends it to the segment file with FileChannel.write(). That write only copies the bytes into a page of the page cache and marks it dirty. Nothing goes to the disk yet, and the broker answers the producer at once.

Kafka does not call fsync per message: log.flush.interval.messages and log.flush.interval.ms are unset (Long.MAX_VALUE) by default. The kernel's flusher threads write dirty pages back later, after vm.dirty_expire_centisecs (30 s by default) or when dirty memory passes vm.dirty_background_ratio (Demo: produce, then background flush). Durability comes from replication instead: with acks=all, a record is acknowledged once the in-sync replicas on other brokers have it, so losing the page cache of one machine loses nothing. The fresh dirty page is also exactly what a tailing consumer reads next, straight from RAM.

What the page leaves out

Real fetches move up to max.partition.fetch.bytes (1 MB) per partition and many partitions per request; the page moves one or two 4 KB pages. The request and response headers, the fetch session, and the copies of the small request itself are not drawn. Kafka's network threads hand requests to separate I/O threads; the page uses one thread. sendfile() is not quite free: the kernel still maps and pins the pages, and without scatter-gather NIC support it copies once inside the kernel. On a cold read the page cache may also read ahead the next pages. The segment index files, mmap of the offset index, splice() and io_uring are left out, and so is the follower fetch path, which uses the same zero-copy send.