System Design

API Gateway

Design an API gateway like Kong or AWS API Gateway.

Functional requirements

• Route each request to the right backend by host, path, and method (e.g., /v2/orders/* goes to the orders service), with per-route path rewriting.
• Authenticate every request by validating an API key or JWT, pass the verified identity upstream as headers, and reject unauthenticated requests before they reach a backend.
• Enforce per-client rate limits (e.g., 1,000 requests per minute per API key), returning 429 with a retry hint when a limit is exceeded.
• Protect upstreams with per-route timeouts, bounded retries, and circuit breakers so a failing backend fails fast instead of taking the gateway down.
• Let operators add routes, rotate keys, change limits, and send a percentage of a route's traffic to a canary version through a control API, with no node restarts.

Non-functional requirements

• The gateway is on every request's critical path: p99 gateway overhead, excluding upstream time, under 10ms; p50 under 2ms.
• Availability over consistency for config: a node keeps serving its last known good config if the control plane is unreachable; a new route or limit may take a few seconds to reach every node, but no node may ever serve a partial config.
• Authentication is never wrong in the permissive direction: a revoked key must stop working on every node within 30 seconds, and no request reaches an upstream unauthenticated.
• Rate limits may be approximate under node loss (a client may briefly exceed its limit by ~10%) but must never be enforced against the wrong client.
• Config is durable and versioned: a control-plane crash never loses a published config and any version can be rolled back; API keys are stored hashed and never logged.

Scaling & constraints

• Peak of 500,000 requests per second across the fleet, ~150,000/sec average, with 3x spikes lasting minutes during product launches.
• 2,000 routes across ~300 upstream services and 50,000 active API keys; the full config snapshot is about 5 MB.
• Config changes ~200 times per day (mostly key rotations and limit tweaks) and each change must reach all ~400 gateway nodes across 3 regions.
• Requests average 2 KB in and 8 KB out; under 1% are uploads of up to 10 MB that must stream through without being buffered in memory.
• Upstream latency is p50 20ms, p99 400ms; about 1% of upstream instances are unhealthy or restarting at any time, and a whole upstream service goes down a few times a month.
• The top 20 API keys generate ~40% of all requests, and a single misbehaving client can send 50x its normal rate during a retry storm.

Out of scope

• Developer portal, API documentation, and self-service key signup.
• Request and response body transformation beyond header rewriting.
• Metering and billing for API monetization.
• TLS certificate issuance and renewal.