System Design

Rate Limiter

Design a distributed rate limiter for a company's public APIs.

Functional requirements

• Enforce per-client rate limits (e.g., by API key or user ID) with configurable rules such as 100 requests/second or 10,000 requests/hour, supporting multiple rules per client simultaneously.
• Return a rejection response (HTTP 429) with informative headers (limit, remaining, retry-after) when a client exceeds their quota.
• Support dynamic rule updates (adding/changing a client's limits) that take effect within seconds, without redeploying the gateway fleet.
• Allow limits to be enforced globally across all gateway nodes: a client's traffic counts against one shared quota regardless of which node serves the request.
• Support burst tolerance: clients may briefly exceed steady-state rate up to a configurable burst size.

Non-functional requirements

• Low latency overhead: the rate-limit decision must add no more than ~1-2 ms p99 to the request path.
• High availability: the rate limiter must never take down the API itself; a defined fail-open or fail-closed policy is required when the limiter's backing store is unreachable.
• Accuracy stance: slight over-admission (a few percent) during failures or contention is acceptable; systematically blocking legitimate traffic is not.
• Horizontal scalability: adding gateway nodes or limiter capacity must not require re-sharding client rules manually.
• Counter state may be ephemeral (loss on limiter restart resets windows); rule configuration must be durable.

Scaling & constraints

• 10 million distinct API clients (API keys), of which ~1 million are active on any given day.
• Aggregate API traffic peaks at 500,000 requests/second across ~200 gateway nodes in 3 regions.
• Traffic is highly skewed: the top 100 clients account for roughly 40% of all requests, and a single large partner integration (the busiest client) can peak at roughly 75,000 requests/second on its own.
• Rule set: up to 3 limit rules per client; rules change infrequently (a few hundred updates/day).
• Per-client counter/bucket state is small (a few counters or a timestamp + token count, well under 100 bytes per client per rule).
• Traffic is growing ~2x per year.

Out of scope

• Authentication/authorization of the API requests themselves (assume client identity is already resolved).
• Billing, quota purchase flows, and self-serve rule management UI.
• DDoS mitigation at L3/L4 (IP-level flooding); assume an upstream layer handles it.
• Analytics dashboards on rate-limit metrics beyond basic counters/logging.