The idea: one instruction, 32 threads

A GPU runs a kernel, a small function, on thousands of threads at once. It does not give each thread its own core. It groups the threads 32 at a time into a warp, and a warp has a single program counter. The warp scheduler fetches and decodes one instruction and sends it to all 32 lanes. Each lane holds one thread's registers and computes on its own data. NVIDIA calls this SIMT: single instruction, multiple threads.

A block of 128 threads is cut into warps 0 to 3 of 32 threads each. Warp 0 goes to an SM sub-core, where the warp scheduler fetches and decodes one instruction, y = x * 2, and sends it to 32 lanes numbered 0 to 31. Each lane has its own registers. The other warps keep their registers on the SM, ready to run next.
One fetch and decode feeds 32 lanes; that is why a GPU can spend its chip area on arithmetic instead of control.

This gives two rules that decide how fast GPU code runs. The 32 threads of a warp should take the same path through the code. And there should be many warps, so that one warp's wait for memory is filled with another warp's work. The page shows both.

Warps, lanes and the active mask

A kernel launch kernel<<<blocks, threads>>> creates thread blocks. The GPU cuts each block into warps of consecutive threads: threads 0–31 are warp 0, 32–63 warp 1, and so on. On the page thread i runs on lane i and starts with x = i.

Every warp carries a 32-bit active mask, one bit per lane. An instruction only changes the registers of the lanes whose bit is 1. The others still "see" the instruction go by, but do nothing: that lane's ALU is idle for the cycle. With all lanes on, the mask is 0xFFFFFFFF (Demo 1: all 32 lanes agree).

Branch divergence and reconvergence

At an if, every lane computes the condition. When all 32 agree, the branch is warp-uniform and costs nothing extra: the warp goes one way and never issues the other path. When they disagree, the warp diverges. It cannot be at two PCs at once, so it runs the paths one after the other:

  1. It pushes the reconvergence point (the first line after the if/else, with the full mask) and the else path (with the mask of the false lanes) on a small stack.
  2. It runs the then path with the mask of the true lanes, for example 0x0000FFFF for i < 16.
  3. It pops the stack: PC jumps to the else path and the mask flips to the false lanes.
  4. It pops again at the reconvergence point. All 32 lanes run together from there.
Two grids of issued instructions, one row per instruction and 32 cells per row for the lanes. Uniform branch if i is less than 32: 5 rows, all green, 160 of 160 lane slots used, 100 percent. Divergent branch if i is less than 16: 7 rows; the then rows have lanes 0 to 15 blue and 16 to 31 grey, the else rows have lanes 16 to 31 orange and 0 to 15 grey; 160 of 224 slots used, 71 percent.
A divergent warp issues both paths; every grey cell is a lane that waited.

The SIMT efficiency is the share of lane slots that did work: lane slots used / (32 × instructions issued). On the page the uniform branch gives 5 issues and 100 %. Every divergent split gives 7 issues and 160 / 224 = 71 %, whether it is half and half (Demo 2), every other lane (Demo 3) or one lane alone (Demo 4). The pattern does not matter, only whether the lanes of one warp agree. With long branch bodies, a 2-way split can halve the speed; a switch with 32 different cases can make it 32 times slower.

ConditionActive mask of the then pathIssuedSIMT efficiency
i < 320xFFFFFFFF5100 %
i < 160x0000FFFF771 %
i % 2 == 00x55555555771 %
i == 00x00000001771 %

So GPU programmers try to make conditions depend on the warp, not the lane: if ((i / 32) % 2 == 0) splits work between warps and costs nothing. Short branches are often compiled into predicated instructions (both sides run, each lane keeps one result) with no jump at all.

Latency hiding with many warps

A load from GPU memory takes hundreds of cycles. A CPU fights this with big caches and out-of-order execution. A GPU instead keeps many warps resident on each SM, each with its registers already in the register file. Switching warps costs nothing, so every cycle the scheduler can issue from any warp that is ready.

On the page each warp runs ld, mul, add, st, and the load takes 8 cycles. One warp keeps the issue slot busy 4 of 11 cycles (Demo 5). With 4 warps the loads overlap: 16 instructions in 20 cycles, 80 % (Demo 6). With 8 warps there is always a ready warp: 32 instructions in 32 cycles, 100 %, against 88 cycles if the warps ran one after another (Demo 7). The load is still 8 cycles; the scheduler hides the latency, it does not shorten it.

Issue timelines. One warp: ld at cycle 0, waits 7 cycles, then mul, add, st; the issue slot is idle 7 of 11 cycles. Four warps: W0 to W3 issue their loads at cycles 0 to 3, wait, then take turns with mul, add and st; the issue slot is idle only at cycles 4 to 7, busy 16 of 20 cycles.
While one warp waits for memory, the scheduler issues from another; enough warps fill every slot.

The number of resident warps is the occupancy. It is limited by what each warp needs: registers per thread, shared memory per block, and a hardware cap (for example 64 warps per SM on recent NVIDIA GPUs). A kernel that uses many registers per thread fits fewer warps and hides less latency. A rule of thumb: warps needed ≈ load latency / instructions each warp can issue before it must wait. On the page that is 8 / 1 = 8 warps.

SIMT, SIMD and CPU threads

CPU SIMD (AVX, NEON)GPU SIMT warpCPU threads
Width4–16 lanes in one vector register32 threads (64 on AMD wavefronts)1 thread per core or SMT slot
Programmer writesVector code or intrinsics; the compiler vectorizes loopsScalar code for one threadScalar code per thread
BranchesMasks written by hand or by the compilerHardware active mask and reconvergenceEach thread branches freely
Hiding memory latencyCaches, out-of-order executionSwitching between many resident warpsCaches, OoO, a few SMT threads

See also SIMD, multithreading and multicore for the CPU side of the same ideas, branch prediction for how a CPU core handles branches instead, the MIPS pipeline for issue slots and stalls in one core, and the memory hierarchy for why a load takes so long.

What the page leaves out

The page shows one warp scheduler issuing one instruction per cycle; a real SM has four, each with its own lanes, and executes a warp's 32 lanes over one or two cycles. Real load latency is about 400–800 cycles and ALU latency about 4, so real kernels need more warps and independent instructions per warp. Since Volta (2017) NVIDIA GPUs have independent thread scheduling: each thread has its own PC, so diverged paths can interleave and threads can wait on each other, but the warp still issues one path at a time. Memory coalescing, shared memory, bank conflicts, tensor cores, nested branches, loops with different trip counts and AMD's 64-wide wavefronts are left out.