Rate limiting is where reliability and customer experience collide. Set limits too loose and one noisy client can degrade the service for everyone. Set them too tight and you frustrate exactly the power users who pay you the most. This article explains how rate limiting really works, which algorithm to pick, how to communicate limits so integrators do not hate you, and the mistakes that turn a safety mechanism into a support nightmare. You will finish able to design a limit that protects your infrastructure and reads as fair.
Why rate limiting exists
An API is a shared resource. Without limits, a single misbehaving script, an accidental infinite loop, or an abusive actor can consume capacity meant for thousands of tenants. Rate limiting protects three things at once: infrastructure stability, fair allocation between customers, and cost control, since compute and database load map directly to money. It is also a security control, blunting credential-stuffing and scraping.
Choosing an algorithm
The algorithm you choose shapes how the limit feels to callers.
| Algorithm | How it behaves | Best for |
| Fixed window | Counts requests per calendar interval, resets at the boundary | Simple internal limits; tolerant of burst at edges |
| Sliding window | Smooths the count over a rolling period | Public APIs needing fairness without hard resets |
| Token bucket | Refills tokens steadily, allows controlled bursts | Most SaaS APIs; balances steady rate with occasional spikes |
| Leaky bucket | Processes at a fixed drain rate, queues or drops overflow | Protecting a downstream system with a strict throughput ceiling |
The fixed-window approach has a well-known flaw: a client can send a full window’s worth of requests just before the reset and again just after, doubling the intended rate at the boundary. Token bucket avoids that and is the pragmatic default for most SaaS products because it permits short, legitimate bursts while enforcing a long-run average.
Communicating limits clearly
A limit the caller cannot see is a trap. Return the state on every response using standard-style headers so clients can self-throttle rather than blindly retrying. Commonly used headers expose the limit, the remaining allowance, and the reset time. When you do reject a request, use HTTP status 429 Too Many Requests and include a Retry-After header telling the client exactly how long to wait. A well-behaved integrator will honor it; a naive one at least gets a clear signal instead of a mysterious failure.
Tie limits to plans, not just to IPs
Limit per API key or per tenant, not only per IP address, because many customers sit behind shared gateways or serverless functions with rotating IPs. Expressing limits as a plan benefit, higher tiers get higher throughput, also turns a constraint into an upgrade path rather than a wall.
A real scenario
A data-enrichment API I advised used a single global fixed-window limit for everyone. A customer running a nightly batch job would fire thousands of requests at midnight, hit the wall, and flood support with complaints, while the same customer’s daytime traffic was trivial. We switched to a token-bucket model scoped per API key, sized so the bucket could absorb a reasonable nightly burst but the refill rate held the daily average safe. We added the rate-limit headers and a Retry-After value. Support tickets about limits dropped sharply, and the batch users simply read the headers and paced themselves. The infrastructure ceiling never actually changed; the fairness and the communication did.
Common mistakes and how to fix them
Mistake: silent throttling. Dropping or slowing requests without a 429 or headers leaves integrators guessing. Fix: always signal the limit state explicitly.
Mistake: one global limit for all endpoints. A cheap read and an expensive report should not share a budget. Fix: weight costly endpoints more, or give them their own limits.
Mistake: no burst allowance. Real traffic is spiky; a rigid per-second cap punishes normal use. Fix: use token bucket so short bursts are fine as long as the average holds.
Mistake: limiting by IP only. This breaks customers behind proxies and NATs. Fix: key limits on the authenticated identity.
Mistake: no headroom for retries. If clients retry on 429 with no backoff, they amplify the problem. Fix: document exponential backoff and honor Retry-After.
Action checklist
- Identify what you are protecting: infrastructure, fairness, cost, or abuse.
- Default to token bucket unless a strict drain rate is required.
- Scope limits to the authenticated tenant or API key, not raw IPs.
- Weight expensive endpoints differently from cheap ones.
- Return limit, remaining, and reset headers on every response.
- Reject with 429 and a Retry-After value, never silently.
- Map limit tiers to pricing plans.
- Document the behavior and recommend exponential backoff to integrators.
- Monitor how often real customers hit limits and adjust before they churn.
Conclusion and next step
A good rate limit is invisible to well-behaved clients and unmistakable to misbehaving ones. Your next step: audit whether your API returns explicit rate-limit headers and a 429 with Retry-After today. If it does not, add them before you touch the limit values themselves. Communication fixes more complaints than raising the ceiling ever will.
FAQ
Should I rate limit per second, per minute, or per hour?
Use the granularity that matches your risk. Per-second protects against sudden floods; per-day protects cost and fairness. Many APIs enforce more than one window at once, a short burst limit plus a longer sustained limit.
What status code should a rate-limited request return?
HTTP 429 Too Many Requests, ideally with a Retry-After header. Avoid using generic 4xx or 5xx codes, which hide the real cause and encourage blind retries.
How do I set the initial numbers?
Start from your capacity and expected concurrency, then look at real usage percentiles. Set the limit above what legitimate heavy users need but below what would endanger the system, and adjust as you observe real traffic.
Does rate limiting replace autoscaling?
No. They are complementary. Autoscaling adds capacity for genuine demand; rate limiting prevents any single tenant from consuming a disproportionate share and controls cost even when you can scale.
References
The HTTP 429 status and Retry-After header are defined in the IETF RFCs for HTTP semantics (RFC 6585 and RFC 9110). The rate-limit header fields are the subject of ongoing IETF standardization work. Algorithm behavior described here reflects widely documented engineering practice rather than any single proprietary source.