The idea: same data, three kinds of feedback
Machine learning is usually split into three families. The page gives all three the same 12 points, in the same order, and changes only one thing: what the learner is told about each point. In supervised learning every point comes with the right answer, a label. In unsupervised learning it comes with nothing at all. In reinforcement learning (RL) the learner must first act, here by guessing a class, and then gets a reward: +1 if the guess was right, −1 if it was wrong. Each Step shows one point to all three lanes, so you can count what each one needs.
Supervised: labels → a classifier
The left lane is a nearest class mean classifier (a nearest-centroid classifier, a very small cousin of k-nearest neighbors). Each point arrives with its label, takes its class colour and pulls the mean of its class towards it. The background shows the answer it would give anywhere in the plane: the class of the nearest mean. After 12 steps it is done, because every step carried the full answer. The price is that someone had to label all 12 points; in real projects labelling is often the most expensive part.
Unsupervised: no labels → groups
The middle lane is k-means with k = the number of classes. It never sees a label. One step assigns one point to its nearest centroid; after a full pass, one step moves every centroid to the mean of its points; it stops when a whole pass changes nothing. What it finds are clusters, numbered 1, 2, 3 — not A, B, C. To score it on the test points, each cluster is named afterwards with the majority class of its points, which is what a person looking at the clusters would do.
Clusters are the structure that is in the data, and that need not be the structure you care about. In Demo: labels cut across the groups the points form a left and a right column, but the labels split them by height. k-means finds the columns, every cluster is half A and half B, and it gets 2 of 4 test points right. Nothing in its input could tell it which split was wanted. It can also stop at a poor answer from a poor start (Demo: k-means starts badly): two centroids begin in group A, split it in two, and the third has to cover both B and C.
Reinforcement: rewards → a policy
The right lane cuts the plane into a 5 × 5 grid of states. For each point it sees which cell the point is in, picks an action (a class), and gets the reward. It keeps a value Q(s, a) for each cell and action and updates it with
Q(s, a) ← Q(s, a) + α · (r − Q(s, a)) α = 0.5, r = +1 or −1, Q starts at 0
Its policy is the action with the highest Q in each cell. With probability ε = 0.1 it explores, trying a random class even when it has a favourite, so a lucky early answer cannot lock it in. A cell turns grey once a guess there was punished and colours in once a guess earned +1.
A reward is evaluative, not instructive: "your guess A was wrong" rules out A but does not say whether the answer is B or C. So RL needs more tries than supervised learning needs labels, and it pays for learning with wrong guesses. In Demo: three clear groups it makes 12 wrong guesses in 36 tries before its policy is right everywhere it was tested. With only 12 tries (Demo: only 12 tries for RL) it makes 10 wrong guesses, and a cell where its only try was wrong still has two classes to choose from: 2 of the 4 test points come out wrong.
The demos in numbers
| demo | supervised | k-means | RL |
|---|---|---|---|
| three clear groups | 4/4, 12 labels, 12 steps | 4/4, 25 steps | 4/4, 36 rewards, 12 wrong guesses |
| labels cut across the groups | 4/4 | ✗ 2/4 | 4/4 |
| only 12 tries for RL | 4/4 | 4/4 | ✗ 2/4, 10 wrong guesses in 12 |
| k-means starts badly | 4/4 | ✗ 3/4 | 4/4 |
When to use which
| supervised | unsupervised | reinforcement | |
|---|---|---|---|
| feedback per example | the right answer (a label) | none | a reward for the action taken |
| what it learns | a function from input to answer | structure: groups, directions, density | a policy: which action to take in each state |
| main cost | labelling the data | deciding what the groups mean | many tries, and mistakes along the way |
| goes wrong when | labels are wrong, too few or biased | the structure is not the one you care about | rewards are rare, late or reward the wrong thing |
| typical uses | spam filters, image classes, price prediction | customer segments, anomaly detection, compression | games, robot control, recommendations, tuning LLMs from feedback |
Rewards that come late
The page's RL lane is the simplest kind, a contextual bandit: one action, then its reward right away. Full reinforcement learning is about sequences of actions where the reward comes at the end, like a maze where only reaching the exit pays. The learner then has to work out which of the earlier moves deserve the credit, which is what the discount γ and the max Q(s′, ·) term of Q-learning are for.
See also K-Nearest Neighbors, K-Means Clustering and Q-Learning in a Grid World, each of which shows one of the three lanes in depth.
What the page leaves out
The data has 2 features and 12 points, and every learner is about the simplest of its family. The page leaves out semi-supervised learning (a few labels plus many unlabelled points), self-supervised learning (labels made from the data itself, such as the next word, which is how large language models are pre-trained), regression (numeric answers instead of classes), overfitting and validation sets, other unsupervised methods such as PCA and density estimation, and the sequential, delayed-reward side of RL with discounting, value functions and policy gradients.