The idea: predict the next token, then do it again

A large language model (LLM) such as GPT does one thing. It reads the text so far and gives a probability for every token in its vocabulary to come next. To write a sentence, it picks one token, appends it, and runs again. Everything else on this page is how that one prediction is computed.

The pipeline: text is split into tokens, tokens become vectors, a stack of transformer blocks (attention and MLP) updates them, the last vector gives logits, softmax gives probabilities, one token is picked and appended, and the loop runs again.
One forward pass gives one token; generation is that pass in a loop.

The canvas runs a real but tiny model. It has the shape of GPT-2 and was trained on 12 short sentences such as the cat sat on the mat. and the dog ate the bone. Every number you see is computed in your browser from its weights. (Demo 1: one token, end to end)

Tokens and BPE

The model never sees letters. A tokenizer cuts the text into pieces from a fixed vocabulary and gives each piece an id. GPT models use byte-pair encoding (BPE): start with single characters, then apply a learned list of merges in order (h+e, t+he, ·+the, …). Common words end up as one token. A word usually keeps its leading space, drawn here as ·, so the and ·the are different tokens.

A word the tokenizer never saw is not a problem. It is built from smaller pieces: cats becomes ·cat + s. This page has 58 tokens. GPT-2 has 50,257, and newer models have 100,000 or more. (Demo 4: tokenization of an unseen word)

Embeddings and positions

Each token id selects one row of the token embedding table wte: a vector of d_model numbers (8 here, 768 in GPT-2 small). Attention by itself does not know word order, so a position embedding wpe[pos] is added. The result x = wte[id] + wpe[pos] is the token's starting vector, drawn as a strip of cells (orange positive, blue negative).

Self-attention: query, key, value

Attention is how tokens use each other. Every token makes three vectors from its own: a query q (what it is looking for), a key k (what it contains) and a value v (what it passes on).

  1. Score: compare the query with each earlier token's key, q·k / √d_head.
  2. Mask: later tokens are set to −∞. A token may only look back. That is the causal mask.
  3. Softmax: turn the scores into weights between 0 and 1 that add up to 1.
  4. Mix: the output is the weighted sum of the values, Σ weight × v.
Head 2 for the prompt the cat sat on the: the query of the last token is compared with every key, softmax gives weight 1.00 to cat and about 0 to the rest, and the output is the value of cat.
In "the cat sat on the", head 2 puts all its weight on ·cat, so the subject's information reaches the last token.

This is why the model can finish the dog sat on the with ·rug but the cat sat on the with ·mat. The last token is ·the in both cases. Only attention brings the subject's information to it. (Demo 2: attention decides)

Heads, MLP, residuals and layers

A layer runs several attention heads side by side, each with its own Wq, Wk, Wv. Each head learns to look for something different. In the trained model on this page, head 1 looks at the words just before (·sat ·on: "a place comes next") and head 2 looks back at the subject ("which animal"). Their outputs are joined and mixed by Wo. (Demo 6: two heads)

Next comes an MLP that works on each token alone: 8 → 32 → 8 numbers with a GELU in between. Both parts add their result to the token's vector instead of replacing it. This is the residual stream. LayerNorm keeps the numbers in a good range before each part. Attention + MLP is one transformer block. This page has 1 block; GPT-2 small has 12; very large models have around 100.

From logits to a token: temperature, top-k, top-p

The last vector is compared with every row of the embedding table. That gives one logit per token, and softmax turns the logits into probabilities. How to pick one is a separate choice:

SettingWhat it doesEffect
temperature 0 (greedy)always take the highest probabilitysame prompt → same text
temperature Tsoftmax(logit / T), then a random drawT < 1 sharper, T > 1 flatter and more surprising
top-kkeep only the k most likely tokenscuts rare tokens
top-p (nucleus)keep the most likely tokens until their sum reaches pthe cut adapts to how sure the model is

After the cat, the model gives ·saw 0.53, ·sat 0.24 and ·ate 0.23. These reflect the training sentences. Greedy always writes "saw"; sampling sometimes writes the others. (Demo 5: temperature)

The generation loop, prefill and the KV cache

To generate, the model appends the picked token and predicts again. The keys and values of earlier tokens do not change when a token is added, so they are kept in a KV cache. The first pass over the prompt is the prefill: all prompt tokens are processed at once, in parallel. Each later step is a decode step: it computes K and V for one new token and attends over the cache. Generation stops at an end token (<|endoftext|>) or after max_new_tokens. (Demo 3: write a sentence)

Left: without a cache each step recomputes K and V for every token, 2 then 3 then 4 then 5 rows. Right: with a cache each step adds one row.
Without a cache the work grows with the square of the length; with it, each new token costs one new K,V row.

This is why the first token of a reply can take a while (prefill) and later tokens stream at a steady rate (decode). The cache is also why long chats use a lot of GPU memory.

The context window

A model has a maximum number of positions, the context window n_ctx: 8 here, 1,024 in GPT-2, and hundreds of thousands in current models. Tokens before the window are simply not there for the model. They are not remembered in any other way. When the window slides, every token's position changes. With learned position embeddings like here, the cached keys and values change too, so the cache has to be rebuilt. (Demo 7: context window full)

Training: next-token loss

Training uses the same forward pass on known text. Every position predicts its next token at the same time, and the causal mask makes sure no position can peek at the answer. The loss at each position is cross-entropy, −log p(correct token). Backpropagation gives the gradient of the mean loss for every weight, and gradient descent (here Adam) makes the loss a little smaller. (Demo 8: training view)

Real assistants are trained in stages. Pretraining on a huge amount of text gives a model that continues text. Instruction tuning on example conversations teaches it to answer. Preference training (RLHF and similar) shapes which answers it prefers. All of them still train the same next-token predictor.

This page vs a real model

ParameterThis pageGPT-2 small
vocab_size5850,257
n_ctx81,024
d_model8768
n_layer112
n_head × d_head2 × 412 × 64
MLP hidden323,072
parametersabout 1,400124 million

The mechanism is the same. More layers and wider vectors let a model store far more patterns, but each step is still "attention, MLP, next-token probabilities".

See also: Byte-Pair Encoding for how the merge list is learned, Self-Attention for every step of one attention layer worked out, Neural Network Forward Pass for the MLP inside each block, and Recurrent Networks: RNN and LSTM for how language models read text before transformers.

See also LoRA and Quantization: how such a model is fine-tuned cheaply with low-rank adapters and stored in 8 or 4 bits.

See also Decoding Strategies, where greedy, beam search and top-p sampling continue the same prompt side by side.

See also The KV Cache: both lanes side by side, the memory per token and layer, and PagedAttention blocks.

See also Mixture of Experts: many models (Mixtral, DeepSeek-V3) replace the MLP of each block with several expert MLPs and a router that picks the top-k per token.

After pretraining, models are aligned with human preference pairs; see RLHF vs DPO for the reward model + PPO road and the direct DPO loss.

Each generated token costs one full forward pass; Speculative Decoding shows how a small draft model lets one pass produce several tokens.

What the page leaves out

Many layers, batching many requests on a GPU, rotary position embeddings (RoPE) used by newer models, mixture-of-experts MLPs, quantization, chat templates and system prompts, tool use, and speculative decoding. Byte-level BPE (GPT-2 starts from 256 bytes, not letters) is simplified to letters.