The idea: one stack reads, the other writes

The original transformer from 2017 was built for translation. It has two parts. The encoder reads the whole source sentence at once. The decoder writes the target sentence one word at a time. They are joined by cross-attention: at every step the decoder asks a question (a query) and looks up the answer in the encoder's output.

The encoder turns the source words the, black, cat and end token into keys K and values V with self-attention that has no mask. The decoder reads the words written so far (start token, le, chat, noir) with masked self-attention, then cross-attention takes its query from the decoder and K and V from the encoder, and picks the next word.
The encoder leaves K and V behind once; every decoder step reads them through cross-attention.

The canvas translates the black cat into le chat noir with vectors of only 4 and 6 numbers, so every score fits on screen. (Demo 1: encode once, then one decoder step)

Self-attention vs cross-attention

Both use the same formula, softmax(Q·Kᵀ / √d_k) · V. The difference is where Q, K and V come from.

Left, self-attention: Q, K and V are all made from the same sentence. Right, cross-attention: Q is made from the decoder (French) and K and V from the encoder (English). Both feed softmax of Q times K transpose over the square root of d_k, times V.
Same formula, different sources: cross-attention takes Q from one sequence and K, V from another.
LayerQ fromK and V fromMask
encoder self-attentionsourcesourcenone: every word sees every word
decoder self-attentiontarget so fartarget so farcausal: only earlier words
cross-attentiondecoderencoder outputnone: the whole source

A real decoder layer has all three steps in this order: masked self-attention, then cross-attention, then an MLP. The canvas shows the first one as a single box and opens up the cross-attention.

The encoder runs once

The source sentence does not change while the decoder writes, so the encoder runs a single time. Its output is turned into keys and values, often called the memory, and every decoder step reads the same K and V. After Translate the counters read 1 encoder pass and 4 decoder steps. (Demo 2: translate the whole sentence)

A decoder-only model such as GPT has no encoder: the prompt and the answer are one sequence and only self-attention with a causal mask is used. Its KV cache plays the role of the memory, but it grows by one row per token instead of being fixed.

Keys say what a word is, values say what it means

Here each key is the role of a source word: det, adj, noun or end. Each value is the meaning: one 1 at the French word it translates to. The decoder's query asks for a role, for example q = [0 1 2 0] after le: "I need a noun, and a bit of the adjective".

  1. Scores: q·kⱼ for the 4 source words = [0 2 4 0].
  2. Scale by √d_k = 2: [0 1 2 0].
  3. Softmax: [0.08 0.22 0.61 0.08], mostly on cat.
  4. Context Σ wⱼ·vⱼ puts 0.61 on chat, so greedy decoding writes "chat".

Change the source to the white dog and run the same decoder: the queries are identical, but the values are different, so it writes le chien blanc. The decoder does not know the source words; it only knows what it can read through cross-attention. (Demo 3: another source, same decoder)

The weights are an alignment

Each row of the bottom grid is one French word and shows where the decoder looked when it wrote it. In French the colour comes after the noun, so "chat" looks at cat (column 3) and "noir" looks at black (column 2): the alignment crosses. Before attention, an RNN encoder had to squeeze the whole sentence into one vector; attention lets each output word pick its own source words.

Encoder–decoder vs decoder-only

Encoder–decoderDecoder-only
examplesoriginal Transformer, T5, BART, Whisper, MarianGPT, Llama, Claude
input is read bythe encoder, no maskthe same causal stack
attention kindsself, masked self, crossmasked self only
good fitinput → output tasks: translation, speech to text, summariesone model for every task through a prompt

See also Self-Attention for the score, scale, mask and softmax steps on one sentence, How a Large Language Model Works for a whole decoder-only model, and Decoding Strategies for beam search, which real translators use instead of greedy decoding.

What the page leaves out

The numbers are hand-picked so they can be read: real keys and values are dense learned vectors of 64 or more numbers, made by multiplying the encoder output by W_K and W_V. The page uses one layer and one head; real models stack 6 to 24 layers with 8 to 16 heads, and every decoder layer has its own cross-attention to the top encoder layer. Also left out: position encodings, residual connections and layer norm, the MLP, the output softmax over a vocabulary of tens of thousands of subword tokens (BPE), padding masks for batches, beam search, and training with teacher forcing.