Inside a Goroutine: the g Struct, its Stack and a Switch

The idea: a goroutine is a struct and a small stack

An OS thread comes with a stack the kernel sets aside for it (8 MB of address space by default on Linux), and switching between threads is a trip into the kernel. A goroutine is only data in the Go heap: a g struct of a few hundred bytes and a stack that starts at 2 KB. The Go runtime switches goroutines by saving and loading a few registers, in user space. That is why one program can run a million of them.

This page zooms into one goroutine. Where goroutines wait (run queues, work stealing, the netpoller) is on the Go scheduler page; here there is one thread M0 with one processor P0. The canvas runs a small program: main (G1) starts worker (G2), waits for it on a channel, and starts another worker at the end. On the left is the code, the middle shows each goroutine's g struct and its stack to scale, and the right shows the CPU registers and M0's g0 stack. Press Next to run one phase, or pick a demo; Restart starts over.

The g struct

The runtime describes every goroutine with a g (runtime/runtime2.go). The fields the page shows:

field	what it holds
goid	the goroutine's number; never reused
atomicstatus	_Gidle, _Grunnable (ready), _Grunning (on a CPU), _Gwaiting (blocked), _Gdead (unused), and the short-lived _Gcopystack, _Gpreempted, _Gsyscall
stack.lo, stack.hi	the stack's bounds; it grows down from hi
stackguard0	stack.lo + 928 B (StackGuard on linux/amd64); every function compares SP with it
sched	a gobuf: the saved sp, pc, g and a few more words, filled in when the goroutine stops running

A running goroutine's real SP and PC are in the CPU. On amd64 the register R14 always holds the current g, so the runtime finds "who am I" in one instruction.

go f(): newproc

The statement go worker(ch) compiles to a call to runtime.newproc. It switches to the thread's g0 stack and runs newproc1, which first asks gfget for a dead g on the P's gFree list. If there is none, malg allocates a new g with a stack of _StackMin = 2048 bytes from the runtime's stack allocator, which takes its memory from the Go heap. Then comes a small trick: newproc1 writes the address of goexit at the top of the new stack, as if goexit had called worker, and sets sched.pc to worker. So when worker returns, it returns into goexit, which cleans up. Last, the g gets a new goid, becomes _Grunnable and goes into the P's runnext slot (Demo 1).

Switching goroutines: gopark, mcall, schedule, gogo

When G1 blocks on v := <-ch, the channel code puts it on the channel's wait queue and calls gopark. What follows is the whole goroutine switch:

1. mcall(park_m) saves G1's SP and PC in G1.sched, loads the SP of M0's g0 stack and calls park_m there.
2. park_m sets G1 to _Gwaiting and calls schedule(), which finds G2 in runnext.
3. execute sets G2 to _Grunning and gogo(&G2.sched) loads G2's SP and PC: the CPU is now running G2 (Demo 2).

Every M has a g0, a stack that belongs to the thread rather than to a goroutine. The scheduler runs there because it cannot use the stack of the goroutine it is leaving: that goroutine may be woken and run on another thread while schedule() is still busy. The g0 of the main thread is the OS stack; for threads the runtime creates it is a separate 16 KB stack (or the OS one with cgo).

The switch saves so little because gopark is an ordinary function call: the Go calling convention treats every register as clobbered by a call, so the compiled code has already put the live values on the stack. SP, PC, g and a couple of other words are all that is left to save. No system call, no change of address space, no TLB flush: about 100–200 ns, against 1–10 µs for an OS thread context switch. (A goroutine stopped by a signal for asynchronous preemption is different: then all registers are saved.)

Growing the stack: morestack and copystack

2 KB is small, so a stack must be able to grow. Every Go function that needs a frame starts with a prologue that compares the stack pointer with g.stackguard0. For sum, whose frame is 512 bytes, the amd64 code is roughly:

main.sum:
    LEAQ  -384(SP), R12     // SP - (512 - 128 bytes of slack)
    CMPQ  R12, 16(R14)      // R14 = g, 16(R14) = g.stackguard0
    JLS   grow              // too close to the end: grow first
    SUBQ  $512, SP          // room for the frame
    ...
grow:
    CALL  runtime.morestack_noctxt
    JMP   main.sum          // run the prologue again, on the new stack

The page uses the simpler rule "SP − frame size < stackguard0". In the demo, sum(3) still fits, but sum(2) would reach below stackguard0. So it calls morestack, which saves G2's SP and PC and switches to g0. There newstack allocates a stack twice as big (4 KB), and copystack copies the used bytes to the top of it, so every frame keeps its distance from hi (Demo 3).

Copying is only safe because the runtime knows where every pointer into the stack is. The compiler emits stack maps that say which words of each frame are pointers, and escape analysis guarantees that the heap never points into a stack: a variable whose address could outlive its frame is put on the heap. So adjustpointers walks the frames and moves every pointer that falls between the old lo and hi. In the demo, acc goes from 0xc0000627f0 to 0xc000070ff0; ch points to the heap and stays. The few runtime structures that do point into stacks, such as a channel's waiting sudog, are adjusted separately. The old stack goes back to the stack pool, and the function's prologue runs again, now with plenty of room.

Shrinking, exiting and reuse

A stack that grew for one deep call would otherwise keep its size forever. When the garbage collector scans a goroutine's stack and finds less than a quarter of it in use, shrinkstack copies it to a stack half as big (Demo 4: 256 of 4096 bytes used → 2 KB). This only happens when the goroutine is stopped at a safe point.

When worker returns, its return address leads into goexit, which calls mcall(goexit0). On g0, goexit0 marks the g _Gdead, clears it and puts it on the P's gFree list with gfput; a stack of the standard 2 KB stays attached, a bigger one is freed. Then schedule() runs the next goroutine, here G1, which the channel send had already made runnable with goready (Demo 5). The next go statement takes that dead g with gfget, stack and all, and gives it a new goid (Demo 6: G3 lives in G2's old g). A program that starts many short goroutines mostly recycles structs and stacks.

Compared with OS threads and segmented stacks

	OS thread (Linux pthread)	goroutine
stack	8 MB reserved (ulimit -s), fixed size, a guard page at the end	2 KB, grows and shrinks by copying
overflow	the guard page faults: the program crashes	the prologue check calls morestack; the limit is 1 GB (debug.SetMaxStack)
where the state is saved	in the kernel, on a switch	in the g's gobuf, by the runtime
switch	a kernel entry and exit, 1–10 µs	a function call into the runtime, about 100–200 ns
create	clone(), tens of µs	newproc, often a reused g, under a µs

Before Go 1.3 goroutine stacks were segmented: when a stack ran out, the runtime linked a new segment to it and freed the segment when the function returned. A loop that calls a function right at a segment boundary allocated and freed a segment on every call, the hot split problem, and its cost was hard to predict. Go 1.3 switched to contiguous stacks that are copied when they grow, and Go 1.4 lowered the starting size from 8 KB to 2 KB. A copy costs time in proportion to the stack's size, but doubling means a stack is copied only a few times in its life.

See also how a Java thread reaches a CPU core (one OS thread per Java thread, and virtual threads, which copy their stack frames to the heap when they park), and coroutines for stackful coroutines like goroutines against stackless ones like async/await.

To compare a goroutine with an OS thread (2 KB vs 1 MB stack, gogo vs a kernel context switch), see OS threads vs goroutines.

What the page leaves out

Real frame sizes (the drawing rounds them up to 256 and 512 bytes) and the 128-byte slack of the real prologue check; more Ps and threads, run queues and work stealing (see the scheduler page); the global gFree lists and stack caches per P; adaptive starting stack sizes (since Go 1.19 the runtime can start stacks at the average size it has seen); _Gscan states and how the GC stops a goroutine; asynchronous preemption by signal; system calls (_Gsyscall); and goroutines locked to a thread.

References

Go source: runtime/runtime2.go (g, gobuf, goroutine statuses)

Go source: runtime/stack.go (newstack, copystack, shrinkstack)

Go source: runtime/asm_amd64.s (mcall, gogo, morestack)

Go 1.3 release notes: contiguous stacks