The idea: say no to some requests, early and cheaply

A server can handle only so many requests per second. A rate limiter sits in front of it and caps how fast each client may send. Requests over the limit are rejected at once, usually with 429 Too Many Requests and a Retry-After header, or delayed in a queue. Either way, the server never sees more than it can take.

Clients send requests to a rate limiter in front of the API server. The token bucket lets a request through if it can take a token and answers 429 Too Many Requests with Retry-After otherwise. The leaky bucket puts requests in a queue of 5 that drains one per tick to the server, and drops a request when the queue is full
Both limiters protect the same server; the token bucket rejects what it can't pay for, the leaky bucket makes requests wait in line.

The page feeds the same requests to two classic limiters, one tick at a time. Each guards its own copy of the API server. Pick how many requests arrive and press Next Tick, or run a demo. The timeline under the limiters records, per tick, what each one let through, rejected or kept waiting.

Token bucket

The bucket holds up to 5 tokens (its capacity, or burst size) and gains 1 token per tick (the refill rate). A token that arrives when the bucket is full is lost. Each request takes one token and goes straight to the server. With no token left, the request is rejected with 429 and Retry-After: 1: the next token comes in one tick.

A client that has been quiet has saved up tokens, so it may send a burst of up to 5 requests at once, with no delay (Demo 2: a burst of 8). After spending them it is held to the refill rate, then earns the burst back by being idle (Demo 3: burst after idle). In code it is two numbers per client: the token count and the time of the last refill. Tokens are added lazily when a request comes in: tokens = min(capacity, tokens + rate × elapsed).

Leaky bucket

The leaky bucket is a queue of 5 with a hole in the bottom: it lets out exactly 1 request per tick, whatever comes in. A request that arrives joins the tail. If 5 are already waiting, it is dropped. The output is perfectly smooth, so the server sees a constant rate. The price is delay: in a burst, the fifth request in line waits 4 ticks.

Requests sent to the API server per tick after a burst of 8 at tick 1. Token bucket: 5 at tick 1, then 0, 0, 0, 0, with 3 rejected with 429. Leaky bucket: 1 per tick for ticks 1 to 5, with 3 dropped; the last one waited 4 ticks
The same burst of 8: the token bucket passes 5 at once, the leaky bucket spreads 5 over 5 ticks; both turn away 3.

Side by side

token bucketleaky bucket (queue)
statea token counta queue of waiting requests
over the limitreject now: 429 + Retry-Afterwait in the queue; drop when it is full
burstup to the capacity passes at onceabsorbed as delay, sent 1 per tick
output to the serverbursty, at most capacity + rate × timesmooth, never above the rate
added latencynoneup to queue size ÷ rate
good forAPI quotas per client, where bursts are fineshaping traffic for a fragile backend or a link

At a steady rate within the limit, both let everything through and look the same (Demo 1). Under sustained overload they also agree in the long run: after the first ticks, both pass exactly 1 request per tick and turn away the rest (Demo 4: 3 per tick, 10 accepted and 8 rejected or dropped by each). The leaky bucket's accepted requests, though, sit in a full queue and wait 4 ticks each. Demo 5 runs mixed traffic and sums it up: the same 7 requests get through each limiter, at a peak of 4 per tick with no wait, or 1 per tick with up to 4 ticks of wait.

In nginx: limit_req

nginx's limit_req module is a leaky bucket per key (here the client address):

limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

location /api/ {
    limit_req zone=api burst=5;            # queue up to 5, delay them
    # limit_req zone=api burst=5 nodelay;  # serve the 5 at once
    limit_req_status 429;                  # default is 503
}

rate is the leak rate and burst is the queue size. Without nodelay, excess requests are delayed so they leave at the rate, as on this page. With nodelay, nginx forwards the burst at once but still frees the burst slots only at the rate, which behaves like a token bucket of capacity burst. delay=3 is the mix: the first 3 extra requests pass at once and the rest wait. Rejected requests get 503 unless limit_req_status says otherwise. See How nginx Handles an HTTP Request for where this sits in the request phases.

Windows, and many servers

A fixed window counter allows N requests per clock minute. It is simple but lets 2N through around the minute boundary. A sliding window (log or weighted counter) fixes that. With several API servers the counter must be shared, often in Redis with an atomic script, or each server takes a share of the limit. A rate limiter is often placed at the load balancer or API gateway, so that rejected requests cost the backend nothing.

See also Circuit Breaker and Bulkhead, which protects a caller from a failing service rather than a service from too many callers, and TCP Congestion Control, where the sender limits itself to what the network can carry.

What the page leaves out

Real time instead of ticks (fractional tokens, lazy refill); limits per client, per API key or per route; clients that honour Retry-After and come back; the response headers some APIs add (RateLimit-Limit, RateLimit-Remaining); distributed counters and their races; and the "leaky bucket as a meter" variant, which rejects instead of queueing and is then the same as a token bucket.