For most of the last decade, a rate limit was a round number someone picked in a planning meeting. Sixty requests a minute per key. A thousand an hour. It held up because the client behind the key was a browser tab, or a nightly sync that ran at 2 a.m. and touched endpoints in the same order every time.
The heavy clients on a lot of APIs now are agents. An LLM worker reads a tool list, calls twenty of those tools, retries whatever failed, spawns subagents, and keeps going for forty minutes with nobody watching. Agents do not get bored or close the tab. Put nine of them behind one API key and a limit that looked generous on paper turns into a wall of 429s by mid-morning.
Quota design for that traffic needs two things a request counter does not give you: an accurate idea of what each request costs, and a clear instruction for the client when it has to stop.
Rate limits do three jobs, and they need three different numbers
Most broken quota systems are broken because one number is doing all of this work at once.
The first job is protecting shared resources. Database connection pools, upstream vendor quotas, queue depth, worker memory. This limit exists so a traffic spike does not take the service down, and it should come from the measured capacity of the slowest thing in the chain, not from a marketing tier.
The second is fairness. One tenant running a backfill should not be able to starve another tenant's checkout calls. That is a per-tenant number and it has nothing to do with infrastructure capacity.
The third is commercial. Plan A gets 10,000 calls a month, plan B gets 100,000. That is a billing term, and it belongs in the same system as the invoice rather than in the gateway config.
When all three are expressed as a single rate_limit = 1000/hr column, someone eventually raises it to close a customer complaint and quietly removes the protection it was providing. Keep the numbers separate and name them separately in the logs. Then a capacity limit that trips pages someone, while a commercial limit that trips just sends an email.
None of this replaces authentication, authorization, or a WAF. A rate limit is not an access control. It slows an authenticated caller down; it does not decide what that caller may see.
Where the human assumption breaks
Bursts have a different shape
A person loads a dashboard: twelve XHRs in 400 milliseconds, then nothing for a while. A limiter built around that shape usually allows bursts, because a hard sixty per minute would break the page load itself.
An agent produces the opposite distribution. It is either idle or saturated, and saturation means a flat line at the ceiling for minutes. A bucket sized for a page load empties in the first second and stays empty.
Concurrency is the limit nobody configures
Requests per second is the wrong unit when each request holds a connection for four seconds. Two hundred requests a minute against an endpoint with a two second p99 is roughly seven requests in flight on average, and a good deal more at the peaks. If the pool has twenty connections, that is the number that actually matters.
GitHub publishes both kinds of limit and enforces them separately. Alongside its request quotas it caps concurrency at 100 in-flight requests, shared across the REST and GraphQL APIs, and returns a 403 or a 429 depending on which limit tripped. Concurrency limits catch the class of abuse that per-second counters miss: slow requests that are individually cheap and collectively ruinous.
Retries multiply everything
An agent that fails a call usually retries it. A supervisor with eight parallel subagents, each retrying three times, turns one user action into more than thirty upstream requests. If the client retries on a fixed five second timer because nobody told it otherwise, all eight come back at the same instant and collide again.
The server cannot fix a client's retry logic. It can make the correct behaviour obvious, which is what Retry-After is for, and it can refuse early instead of doing expensive work and then refusing.
Pick the algorithm by what you are protecting
There is no best limiter. There is a limiter that matches the failure you are trying to prevent.
| Algorithm | State per key | Bursts | Use it for |
|---|---|---|---|
| Fixed window | One counter | Allows double at the boundary | Monthly billing quotas |
| Sliding window counter | Two counters | Smooths the boundary | Public API quotas |
| Token bucket | Tokens plus timestamp | Deliberate burst up to capacity | Clients that batch on launch |
| GCRA / leaky bucket | One arrival time | None, output is smoothed | Edge nodes, high cardinality |
| Concurrency limiter | In-flight count | None | Slow or expensive endpoints |
A fixed window counter is the cheapest and the least precise. It resets on a boundary, so a caller can send the full allowance at 12:59:59 and the full allowance again at 13:00:00, which is double the intended rate for a few seconds. That is fine for a monthly quota. It is not fine in front of a database.
A sliding window counter interpolates between the current and the previous window, gets close to the true rate, and needs two counters per key. It is the sensible default for public API quotas, and it is what Redis recommends when a team wants accuracy without the memory cost of logging every request timestamp.
A token bucket lets you state a sustained rate and a burst allowance as separate parameters. Capacity is the burst, refill rate is the sustained rate. Use it where bursts are legitimate, and set capacity on purpose instead of leaving it at whatever the library defaults to.
A leaky bucket smooths output to a constant rate and rejects the excess. The GCRA variant of it stores one theoretical arrival time per key, which makes it the cheapest option at high cardinality and the easiest to run on an edge node where memory is scarce.
A concurrency limiter is a semaphore rather than a counter. It holds the number of in-flight requests per key or per endpoint below a threshold and queues or rejects the rest. This is the one to put in front of anything expensive and slow.
Most production systems end up with two layers: a coarse limiter at the edge keyed on identity, and a finer limiter inside the service keyed on the specific resource. The edge layer sheds obvious floods. The inner layer protects the thing that actually breaks first.
A distributed counter is a race unless you make it atomic
Read the count, compare it to the limit, write the new count. Three steps, one window, and every application instance in the fleet doing it at the same moment. Under load that sequence over-admits, sometimes badly, and the overshoot grows with replica count.
The fix is to make the decision one operation. In Redis that means a Lua script which reads, computes, and writes in a single atomic step, or one of the commands designed for the job. Do not approximate it with a GET followed by an INCRBY and hope the window is short enough to hide the gap.
Shard the state by key once a single Redis node becomes the bottleneck. The limiter is a hot path on every request, so it inherits the latency budget of the fastest endpoint you serve. A network round trip per request is often too expensive, which is why many teams keep a local token cache per instance and reconcile against the shared store on an interval. That design is approximate, and the documentation should say so, because a customer will eventually notice their limit enforced slightly differently on two nodes.
Then decide what happens when the counter store is unavailable, and write it down. Failing open means a Redis outage removes your abuse protection at the moment traffic is most likely to be strange. Failing closed means one dependency takes the whole API down. The middle path that usually works: fall back to a local limiter with a deliberately conservative budget, log the degradation loudly, and alert on the event. Pick per endpoint class rather than globally, and never let a fail-open pass without a metric.
Make the limit a contract
A 429 with no headers is a dead end. The client learns that it was refused and nothing else, so it guesses, and guessing at scale is how retry storms start.
The IETF has been working on standard fields for this since 2019. draft-ietf-httpapi-ratelimit-headers reached version 11 in May 2026 and is still an Internet-Draft; an HTTP Directorate early review of version 10 came back "not ready" in January 2026, so the syntax can still move. The older X-RateLimit-Limit, X-RateLimit-Remaining and X-RateLimit-Reset set is not deprecated, and the draft's RateLimit and RateLimit-Policy fields are different from it. Send both, document which one clients should trust, and keep the parsing tolerant.
Whatever the field names, the response has to answer four questions: what the limit is, what is left, when it resets, and how long to wait. Retry-After covers the last one and accepts either a delta in seconds or an HTTP date. Send the delta.
Keep 429 and 503 apart in your own API. A 429 tells the caller it is over its allowance and should back off. A 503 tells the caller the platform is overloaded and the problem is on your side. Collapsing them into one status means clients cannot tell whether to slow down or to fail over, and you cannot tell the two incidents apart in the logs.
Not every request should cost one unit. A GET /products/{id} and a faceted search that scans an index are not the same amount of work, and pretending otherwise means the cheap endpoint subsidises the expensive one until the expensive one falls over. Assign weights, publish the weight table in the docs, and report the cost in the response so a client can budget before it sends anything.
Return a body with the 429. application/problem+json with a type, a readable detail line, and a link to the limits page costs almost nothing and removes a share of the support tickets. Include the request id so the customer has something to hand you.
Agent clients break the retry contract
Everything above assumes the client reads the headers. Agents often do not, or read them and retry anyway because a framework default said so. Two server side measures help.
The first is a retry budget per key, enforced like any other quota. Retries may not exceed a fixed fraction of successful requests in a window; once the budget is spent, further retries get a 429 with a much longer Retry-After. This stops one broken client from converting a partial outage into a total one, and it makes the looping client visible in the metrics.
The second is deduplication. An idempotency key on unsafe methods, retained for longer than the longest plausible retry window, lets a retry return the original result instead of repeating the work. Payment APIs have done this for years for exactly this reason, and agent traffic makes it a general requirement rather than a payments speciality.
There is also an argument for serving 429s at the edge. A refusal decided in ten milliseconds costs a fraction of a request that authenticates, deserializes, queries, and then refuses. If the answer is going to be no, decide it before the request reaches the application server.
Fairness is a separate problem from capacity
Hierarchical limits handle most of it. Check the tenant bucket first, then the key bucket, then the global ceiling. Each check is cheap and each answers a different question: is this customer over their contract, is this particular integration runaway, and is the platform close to capacity.
For asynchronous work, hard rejection is the wrong tool. A batch import that hits its limit should be queued and deprioritised rather than deleted. Weighted queuing by tenant gives a large customer throughput without letting them consume the queue.
Watch the case where every tenant is inside its own allowance but the global ceiling is saturated. In that state the loudest tenant wins by default, which is the opposite of fairness. Global saturation needs a policy of its own, whether that is queueing, shedding low-priority work, or asking the largest callers to slow down before you have to force them.
What to instrument
Tuning a limiter requires seeing its decisions. The useful minimum:
- Decisions per endpoint, split by allowed, rejected, and fail-open.
- Reject rate as a share of traffic, per key and per tenant.
- Limiter latency at p99, since it sits in front of every endpoint you serve.
- Retry amplification, meaning 429 responses divided by total requests. Past roughly one in ten, either the limit is wrong or a client is looping.
- Time from a new integration's first request to its first 429, which tells you whether the documentation is doing its job.
Alert on fail-open events and on limiter p99 crossing the latency budget. Leave individual 429s alone. A steady trickle of them is the system doing what it was built to do.
A rollout order that will not page anyone at 3 a.m.
Inventory first. List every endpoint, note what it touches, and assign a rough cost. A handful of endpoints usually account for most of the load, and those are the only ones that need careful numbers.
Measure before you enforce. Two weeks of real traffic gives you a p99 per key per endpoint, which is a far better starting point than a number from a meeting. Include the traffic you do not like, since that is the traffic the limiter is for.
Run in shadow mode. Count violations, log them, send the headers, refuse nothing. This catches the integration that quietly sends four hundred requests in a burst at midnight, and it catches it before the customer does.
Then enforce generously and tighten on evidence. Nobody remembers a limit that was too loose. Everyone remembers the afternoon the API started rejecting production traffic.
If your API now serves agents as well as people, the quota layer is part of the product surface. Clients plan against it, agents loop on it, and customers escalate over it. Treating it as a config value rather than a designed interface is how teams end up writing incident reviews about number 429 every quarter.