Draw a Digit Visualization | Template Matching

The idea: compare your drawing with the average of each digit

The simplest way to read a handwritten digit needs no neural network. Take all the training images of a 3 and average them. The result is a blurry template of a typical 3. Do the same for every digit, so there are ten templates. To read a new drawing, find the template it is closest to. This is template matching, also called the nearest centroid classifier. Draw a digit on the pad and press Recognize. The page shows every number it computes, in your browser, in plain JavaScript.

A drawn 3, as a 28 by 28 image x, is compared with ten blurry templates m0 to m9, the average training image of each digit. Under each template is its distance d. Template 3 has the smallest distance, 31.4, so the answer, argmin of d, is 3 — The drawing x is compared with all ten templates. The template with the smallest distance d_k is the answer.

From a drawing to 784 numbers

The templates come from MNIST, a set of 70,000 handwritten digits, each a 28 × 28 grey image. In MNIST each digit was scaled to fit a 20 × 20 box and moved so its centre of mass is in the middle. Your drawing gets the same treatment: crop it to the ink, scale the longer side to 20 pixels, then shift it so its centre of mass sits at the centre. Reading the 784 pixels row by row gives a vector x = (x₀, …, x₇₈₃), with x_i = 0 for white and 1 for full ink. The left panel lists each step. The details are on Draw a Digit: Neural Network Math, which uses the same code.

The templates: one average image per digit

MNIST has 60,000 training images. N_k of them show digit k, about 6,000 each (5,421 fives, 6,742 ones). Template k is their pixel-by-pixel average:

mₖ = (1 / Nₖ) · Σ (the Nₖ training images of digit k)
mₖᵢ = average ink of pixel i over those images, 0 … 1

A pixel of m_k is dark where most people put ink when they write k. It is grey where only some do. MNIST images are already 20 pixels tall and centred, so they were averaged as they are. The ten templates were computed once, offline, and stored on the page as one byte per pixel (0 … 255).

The distance, and the answer

To compare x with template m_k, subtract them pixel by pixel, square each difference and add the squares up. This is the squared Euclidean distance:

dₖ = Σᵢ (xᵢ − mₖᵢ)²         i = 0 … 783
answer = argminₖ dₖ          the template with the smallest distance

Show Distance draws the three stages for one template. First x − m_k: blue where you drew ink the template lacks (extra ink), red where the template has ink you did not draw (missing ink). Then the squares, which are all positive. Then their sum, split into the blue part and the red part. A pixel where x and m_k agree adds 0. Squaring makes a big difference count much more than a small one: 0.9² = 0.81 but 0.3² = 0.09 (Demo: a clean 3, read correctly).

The dot product, and why it favours fat templates

An even simpler score is the dot product: multiply matching pixels and add them up.

sₖ = x·mₖ = Σᵢ xᵢ·mₖᵢ        answer = argmaxₖ sₖ

It is big when your ink lies where the template has ink. But it never punishes the template for ink in places where you drew nothing. So a template with ink everywhere scores high on every drawing. Expanding the square shows how the two scores are related:

dₖ = Σᵢ (xᵢ − mₖᵢ)² = Σᵢ xᵢ² − 2·Σᵢ xᵢ·mₖᵢ + Σᵢ mₖᵢ²
   = |x|² − 2·x·mₖ + |mₖ|²

|x|² is the same for every k, so it cannot change the answer. Minimising d_k is the same as maximising x·m_k − ½|m_k|². The distance is the dot product with a penalty for the template's own ink, |m_k|². The fattest templates are 0 (|m₀|² = 68.44) and 8 (55.01); the thinnest is 1 (28.85). The table on the canvas has all three columns, so you can check the sum.

A slanted 1 compared with templates 1 and 8. The dot product with 8, 23.25, is bigger than with 1, 20.94, because template 8 has almost twice as much ink: 55.01 against 28.85. The distance adds that ink back: d1 = 25.36 is the smallest and d8 = 46.92 is bigger, so the answer is 1 — The dot product says 8 for a 1, because template 8 has twice as much ink. The distance adds |m_k|² back and says 1.

On the 10,000 MNIST test digits the plain dot product answers 0 or 8 for 48 % of them, and never answers 5. It reads only 63.11 % correctly (Demo: a 1, and why the dot product says 8).

How well it works

The ten templates were tested offline on the 10,000 MNIST test digits, which were not used to build them:

Method	Test digits read correctly
Guess by total ink ("average darkness", Nielsen ch. 1)	22.25 %
Dot product, argmax x·m_k	63.11 %
Template distance, argmin d_k (this page)	82.05 %
The 784-16-16-10 network of Draw a Digit: Neural Network Math	94.95 %

The rule does best on 1 (96.2 %), whose writing varies least, and worst on 5 (68.5 %). The most common mistakes are a 5 read as 3 (119 times) and a 4 read as 9 (114 times). Those pairs share most of their ink.

Why it fails

Template matching compares pixel i with pixel i, and nothing else. It has two weaknesses.

No invariance. A digit that is slanted, thicker or moved by a few pixels has its ink at different pixel indices. The shape is the same to us, but the distance grows. The page's slanted 7 is closer to template 8 than to template 7 (Demo: a slanted 7 read as 8). A small drawing scaled up gets thick strokes, so every distance is large. Moving that 3 by 3 pixels after the centring makes it an 8 (Demo: a small 3 in a corner, then moved 3 px). Centring by the centre of mass removes only the shift, and only roughly.
One template per digit. People write 4 open or closed, 7 with or without a bar, 1 as a stroke or with a flag and a base. An average of several styles is a blur that matches none of them well. The page's 5 is read as 3: its lower half matches template 3 better than the blurred average of all the 5s (Demo: a 5 read as 3).

The network of Draw a Digit: Neural Network Math reads the slanted 7 and the 5 correctly.

From templates to learned weights

Write the rule as a score per digit: score_k = x·m_k − ½|m_k|². That is w_k·x + b_k, with weights w_k = m_k and bias b_k = −½|m_k|². So template matching is already a one-layer linear classifier, like the output layer of a neural network. Only its weights are fixed to the averages.

A trained network chooses its weights to tell the digits apart, not to look like them. A weight can be negative: "ink here means it is not a 3". Hidden layers can learn several partial patterns per digit, which covers several writing styles. That takes the 82 % to about 95 % with the small network on Draw a Digit: Neural Network Math. See also Neural Network: Recognizing Digits for what the hidden neurons of such a network really learn.

What the page leaves out

Better nearest-neighbour methods compare x with every training image instead of ten averages (k-nearest neighbours, about 97 % on MNIST), or with several templates per digit. Distances that allow small shifts and slants, such as tangent distance or deformable templates, do better still. Normalising the dot product by |m_k| (cosine similarity) also removes the fat-template bias and reaches 82.15 %. The page uses one template per digit and the plain squared distance. Your pen is also thicker or thinner than MNIST's, which changes every distance a little.