The idea: memory has an address and a place

A server with two or more CPU sockets does not have one pool of memory. Each socket has its own memory controller (IMC) and its own DRAM. A socket, its cores and its memory form a NUMA node (Non-Uniform Memory Access). Every core can still read every byte, but memory on its own node is local and memory on another node is remote: the load has to cross the link between the sockets (Intel UPI, AMD Infinity Fabric), wait for the other socket's controller, and come back. That costs time, roughly 80 ns local against 140 ns remote on the machine drawn here.

Left: UMA, four cores share one memory bus and one memory, every load costs the same. Right: NUMA, two nodes, each with four cores, its own memory controller and its own DRAM, joined by the UPI link; a local load costs 80 ns, a load that crosses UPI to the other node costs 140 ns
With one memory every load costs the same; with NUMA the cost depends on which node holds the page.

So on a NUMA machine it matters where a page lives, not only whether it is in memory. The canvas shows node 0 and node 1 side by side, each with four CPUs, its IMC and twelve 4 KiB frames, and the UPI link between them. Below them is the process app: four threads, two on each node, and its page table for an array of 8 pages, A0 to A7. Every load is drawn as a box that travels from the core through the controllers to the frame; red boxes are remote. Caches are left out: every load here misses the caches and goes to DRAM.

Nodes and distances

Linux reads the layout from the firmware (the ACPI SRAT and SLIT tables) and prints it with numactl --hardware:

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3
node 0 size: 48 KB
node 1 cpus: 4 5 6 7
node 1 size: 48 KB
node distances:
node   0   1
  0:  10  21
  1:  21  10

The distance is relative: 10 means local, 21 means about twice as far. Measured remote latency is usually 1.5 to 2 times the local latency, and the bandwidth of the link is lower than the bandwidth of a node's own memory. lscpu shows the same split as "NUMA node0 CPU(s)". (Demo 1: local vs remote load)

First touch decides

malloc() does not place memory. It only reserves addresses. A page gets a frame at its first touch: the first write causes a page fault, and with the default policy the kernel takes a frame from the node of the CPU that faulted. The page then stays on that node, whoever uses it later.

That makes a common pattern slow. The main thread allocates a big array and clears it with memset, then starts worker threads on all nodes. Every page was touched by the main thread, so every page sits on its node. The workers on the other node make only remote loads, and all the traffic goes through one memory controller, which becomes a hot spot. The fix is to let each worker write its own part of the data first (parallel init, or #pragma omp parallel for around the initialisation in OpenMP).

Before: T0 on node 0 initialises A0 to A7, so all 8 pages are on node 0; T2 and T3 on node 1 load only remote pages and node 0's controller has a queue, 1 200 ns in total. After: each thread initialises its own two pages, 4 pages per node, every load is local, 640 ns
Who writes a page first decides its node: a serial init puts everything on one node, a parallel init puts each page next to its user.

Demo 2: first-touch trap: T0 writes all 8 pages, so numa_maps says N0=8. In the run, T2 and T3 make only remote loads and wait behind T0 and T1 at IMC0: 440 ns each against 160 ns, 1 200 ns in total. Demo 3: parallel init: 4 pages per node, every load local, 640 ns.

Memory policies

A process can choose where its pages go with a memory policy, set with numactl before the program starts, or from inside the program with set_mempolicy() (whole process) and mbind() (one address range).

PolicynumactlWhere a new page goesWhen that node is full
default (local)nonenode of the CPU that touches itanother node; counted as numa_miss
interleave--interleave=allround-robin over the nodes, by page indexthe next node
bind--membind=1only the listed nodesreclaim, then the OOM killer, even if another node has free memory
preferred--preferred=1the preferred nodeanother node; numa_miss and numa_foreign

Interleave is for data that every thread uses, such as a shared hash table or a database buffer pool. Half the loads become remote, but the load is spread over both memory controllers instead of piling up on one. Demo 4: shared table: with T0's first touch all 32 loads go to IMC0 and T2, T3 always wait, 4 800 ns; with --interleave=all each slot sends two loads to each controller, nobody waits, 3 520 ns, although 16 of the 32 loads are remote.

Bind is strict. Demo 6 runs numactl --membind=1 while another process holds half of node 1: six pages fit, the seventh has nowhere to go, and the kernel kills app with constraint=CONSTRAINT_MEMORY_POLICY while node 0 has ten free frames. With --preferred=1 the same program puts six pages on node 1 and two on node 0.

Latency is half of it: bandwidth

A memory controller can only serve so many requests at a time. The page models that as two loads per slot; a third waits. On real machines the effect is a bandwidth limit: a node's memory channels give a fixed number of GB/s, and the UPI link gives less than that. A program whose data sits on one node can use only that node's bandwidth, no matter how many sockets it runs on. That is why spreading pages (by parallel init or interleave) helps even when it makes some loads remote.

Automatic NUMA balancing

The scheduler moves threads between CPUs to balance load, and sometimes moves one to another node. Its pages do not follow. Since Linux 3.8 the kernel can fix that by itself when kernel.numa_balancing = 1 (the default on most distributions):

  1. Every scan period (numa_balancing_scan_period_min_ms, starting at 1 000 ms) task_numa_work() marks a window of the task's page-table entries PROT_NONE (numa_balancing_scan_size_mb, 256 MB at a time). The pages stay in memory.
  2. The next load to such a page traps: a NUMA hinting fault. It tells the kernel which node uses the page right now.
  3. If the page is on another node, the kernel checks the page's last fault. It moves the page only if that fault came from the same node, or there was none. A page used from both nodes gives alternating faults and stays put, so shared pages do not bounce between nodes.
  4. migrate_misplaced_folio() copies the page to a free frame on the right node and changes the page-table entry.
Time flows down. T1 moves from CPU 1 on node 0 to CPU 6 on node 1; its page A2 stays on node 0 and every load is remote, 140 ns. The scan marks A2's page-table entry PROT_NONE. T1's next load traps as a NUMA hinting fault; A2 is on node 0, used from node 1, and has no earlier fault from another node, so the kernel copies it to a frame on node 1 over UPI. The following loads are local, 80 ns
NUMA balancing learns where a page is used from hinting faults, then moves the page next to its thread.

The hinting faults and copies cost time (here 500 ns per fault and 2 000 ns per copy), so balancing pays off only for data that is used for a while. The kernel also does the opposite: it can move a thread to the node that holds most of its memory. Demo 5: thread moved + NUMA balancing: after T1 moves to CPU 6 its two loads take 280 ns; one scan later both pages migrate, and the next run is all local again. It takes 800 ns, not 640: three threads now use node 1's memory controller, so T3 waits once per slot. That is why the kernel also weighs moving the thread back instead of the memory.

Seeing it: numastat and numa_maps

numastat counts page allocations per node. numa_hit: the page came from the node that was asked for. numa_miss: the page came from this node although another was asked for. numa_foreign: this node was asked for but the page came from another. local_node and other_node: whether the allocating CPU was on this node. numastat -p <pid> and /proc/<pid>/numa_maps show where one process's pages are: a line like default anon=8 N0=4 N1=4. /proc/vmstat counts the balancing work: numa_pte_updates, numa_hint_faults, numa_pages_migrated. Remote loads themselves are counted by the CPU's performance counters (perf stat, perf mem).

In practice

  • Pin a service and its memory to one node when it fits: numactl --cpunodebind=0 --membind=0, or taskset plus the default policy.
  • Databases with a huge shared buffer pool are often started with numactl --interleave=all (MySQL and MongoDB document it), so one node does not fill up and swap while the other is free.
  • The JVM has -XX:+UseNUMA: each thread allocates new objects in the part of the heap on its own node.
  • vm.zone_reclaim_mode = 1 makes the kernel reclaim page cache on the local node before using another node. It is off by default because it often hurts more than remote memory does.

See also CPU Cache (what happens before a load reaches DRAM), Linux Virtual Memory (page tables and page faults), CPU Scheduling, the Buddy Allocator (each node has its own free lists) and Java threads to cores.

What the page leaves out

The caches and the coherence traffic between sockets (a write to a line another socket holds is also remote), machines with 4 or 8 nodes and sub-NUMA clustering inside one socket, CXL memory that appears as a node without CPUs, huge pages, the kernel's real scan timing and how NUMA balancing also groups tasks that share memory (numa_group), memory tiering, and real bandwidth numbers. Twelve frames per node and four threads stand in for gigabytes and hundreds of threads.