Rate Limiting at Scale.

The first time I really understood rate limiting, it was 2am and one customer's runaway script was quietly eating an entire service. Nothing was "down" — the dashboards were green — but every other user was crawling, because one client had decided retries were free. We shipped a rate limiter that week. I've had a soft spot for them ever since.

Every API that survives contact with the real world eventually grows one. It's the quietest, most load-bearing 30-odd lines of code in the whole system — the thing standing between a well-behaved product and a single misbehaving client taking everyone else down with it.

The idea sounds trivial: count requests, reject the ones over the line. And on your laptop, it is. But the moment there's more than one server, the obvious version quietly stops working — and that's where it gets fun. So let's walk the whole path together: from a counter in a variable, to the algorithms people actually reach for, to a distributed limiter that stays honest across a whole fleet. There's a live simulator about halfway down — go break it.

Why limit at all

It's easy to think of a rate limiter as a gatekeeper that says "no." Really, it's protecting three things at once:

• Availability — one client can't hog the CPU, connections, or database throughput that everyone else needs.
• Cost — you pay for compute and egress. Unbounded traffic is an unbounded bill, and that bill has your name on it.
• Fairness — your free tier and your enterprise tier should get measurably different slices of your capacity. That's a product decision as much as an engineering one.

The contract it exposes is refreshingly blunt: a request either passes, or it bounces back as 429 Too Many Requests with a Retry-After header telling the client when to come back. Everything from here on is really just about making that one yes/no decision correct, fast, and consistent — which turns out to be harder than it sounds.

The naive limiter

Here's the version everyone writes first — myself included. A map from client to a count, reset on a timer.

const counts = new Map();

function allow(clientId, limit = 100) {
  const n = (counts.get(clientId) ?? 0) + 1;
  counts.set(clientId, n);
  return n <= limit; // true = allowed, false = 429
}

// reset every minute
setInterval(() => counts.clear(), 60_000);

On one server, this is genuinely fine. It's O(1), it barely touches memory, and it ships today. Don't let anyone make you feel bad about it — plenty of production systems run on exactly this.

The problem isn't the logic. It's that little word: one. It holds up right up until the day you add a second server — and adding servers is the whole point of growing up.

But before we fix the distribution problem, we have to fix the algorithm — because "reset every minute" hides a nasty little edge of its own, and I'd rather you hear about it from me than from an angry customer.

Fixed windows, and the burst at the seam

The reset-on-a-timer approach is called a fixed window. It has one well-known flaw: the boundary.

Say the limit is 100 per minute. A client sends 100 requests at 12:00:59, then another 100 at 12:01:00. Both windows are individually, technically legal — and yet you just served 200 requests in one second. The limit you advertised is not the limit you enforced, and someone will eventually notice.

Sliding window fixes this by weighting the previous window's count by how much of it still overlaps the current one:

function slidingAllow(prev, curr, limit, elapsedFraction) {
  // elapsedFraction: 0..1 through the current window
  const estimate = prev * (1 - elapsedFraction) + curr;
  return estimate < limit;
}

It's an approximation, but a cheap and accurate one — and it kills the boundary burst. Most production limiters use this or the next one.

The token bucket

This is the one I reach for by default, and honestly the one I'd teach first if I could only teach one. The token bucket wins because it models the thing you actually care about: a steady allowed rate, plus a controlled amount of burst for the times real users legitimately spike.

The mental model is right there in the name — a bucket:

• It holds up to capacity tokens.
• Tokens are added at a fixed refill rate (say, 5 per second).
• Every request removes one token. No token, no service — that request gets a 429.

Capacity is your burst tolerance; refill rate is your sustained throughput. A client that's been quiet builds up tokens and gets to spike for a bit — which is exactly what a real user doing a real thing looks like. A client hammering you drains the bucket and gets throttled down to precisely the refill rate. Reward the well-behaved, gently starve the greedy. That's the behavior you want, and it falls out of the model for free.

Try it

Enough words — here's a real token bucket running right in your browser. Crank the arrival rate above the refill rate and watch the bucket drain and the 429s start piling up. Then give it a fat capacity and watch it soak up a burst before it settles into a rhythm. I genuinely find this thing hard to stop fiddling with; that's kind of the point.

Did you catch the steady state? Once the bucket empties, the pass rate settles at exactly refill ÷ arrival. Capacity only ever buys you the initial burst — after that, refill rate is destiny. Sit with that one observation for a second, because it's most of what you need to size a limiter in production without guessing.

Going distributed

Okay, here's the part that actually kept me up at night. Your limiter has to be correct across N servers, and the count simply can't live inside any one of them anymore. It has to move somewhere all of them can see.

The standard answer is a shared in-memory store, and Redis is the usual pick — it's fast, and it's single-threaded, which matters way more than it sounds like it should (hold that thought). The trap almost everyone falls into first is doing it in two round trips:

// ❌ race condition: two servers can both read 99, both write 100
let n = await redis.get(key);
if (n >= limit) return false;
await redis.set(key, n + 1);

In the gap between the get and the set, another server sneaks in with the same stale read. And here's the cruel irony: it leaks the most under heavy load — precisely the moment your limiter was supposed to save you. The fix is to make the whole read-modify-write atomic, and this is where single-threaded Redis pays off: it runs Lua scripts atomically, so the entire token-bucket decision collapses into one indivisible operation nothing can interleave with.

-- refill based on elapsed time, then try to spend one token
local tokens   = tonumber(redis.call('HGET', KEYS[1], 'tokens') or ARGV[1])
local last     = tonumber(redis.call('HGET', KEYS[1], 'ts') or ARGV[4])
local capacity = tonumber(ARGV[1])
local refill   = tonumber(ARGV[2])   -- tokens per second
local now      = tonumber(ARGV[4])

tokens = math.min(capacity, tokens + (now - last) * refill)

local allowed = tokens >= 1
if allowed then tokens = tokens - 1 end

redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], ARGV[3])
return allowed and 1 or 0

Every server runs the same script against the same key, Redis quietly serializes them one after another, and just like that you're back to a single source of truth and zero races. If you crack open express-rate-limit or peek behind a cloud API gateway, this is roughly what you'll find — no magic, just one atomic operation in the right place.

What it costs

Moving the counter to Redis buys you correctness — but nothing in this line of work is free, and it's worth being honest about the bill. You've traded a local memory read for a network round trip on every single request. Here's the rough shape of that trade-off:

The other cost is quieter and scarier: you've just made Redis a hard dependency in your hottest path. So ask yourself the uncomfortable question now, in the calm, rather than at 2am: if Redis is down, what does your limiter do?

The decisions that actually matter

Once the mechanics are in place, the real design work isn't the algorithm anymore — it's the numbers and the edges. These are the ones I've been burned by (or watched someone else get burned by), so learn them the cheap way:

Takeaways

Here's the thing I love about rate limiting: the arc is short, but it's the same arc you'll walk for half the hard problems in distributed systems.

1. The naive, single-node version is correct because there's only one of it.
2. Horizontal scale shatters that shared state, so you move it somewhere everyone can see.
3. Concurrency then forces you to make the read-modify-write atomic.
4. And a network dependency forces you to decide, on purpose, what happens when it fails.

That's it. A rate limiter is small enough to hold entirely in your head, which is exactly why it's such a good teacher — you can see the whole thing at once. Get the token bucket right, put the atomic update in the right place, choose your failure mode deliberately instead of by accident, and it'll quietly do its job at 2am while everything around it grows. And honestly? Go play with that simulator a little more. Intuition you build with your hands sticks around a lot longer than anything you read.