System Design

Distributed Cache

Design a distributed in-memory cache like Redis or Memcached.

Functional requirements

• The cache is a shared internal service used by multiple application teams across the company, each team caching data in front of its own backend database.
• Support GET, SET (with optional per-key TTL), and DELETE operations on string keys and binary values.
• Clients can address the cache as a single logical keyspace; the system routes each key to the correct node internally or via a client library.
• Support atomic operations on single keys (e.g., increment/decrement, compare-and-set) to enable counters and simple coordination.
• Allow online cluster resizing: nodes can be added or removed without a full restart or losing the entire cache.
• Evict entries automatically when a node approaches its memory limit, honoring TTL expirations.

Non-functional requirements

• Latency: p99 under 1 ms for GET/SET within a datacenter (excluding client network hops outside the DC).
• Availability over consistency: the cache must keep serving during node failures; brief staleness or cache misses are acceptable, but serving errors for the whole keyspace is not.
• Durability is NOT required (this is a cache, and the source of truth lives in the backing database), but losing an entire node's contents at once should degrade hit rate gracefully rather than cause a thundering herd on the database.
• A single node failure should impact only its fraction of the keyspace, with recovery (failover or rehash) completing within seconds.
• Multi-tenant isolation: one team's traffic spike or oversized values must not evict or starve another team's working set beyond its quota.

Scaling & constraints

• Aggregate traffic across tenants: ~500 million cache operations per day at launch, growing to ~5 billion/day within 2 years.
• Read:write ratio is roughly 20:1.
• Average value size 2 KB; p99 value size 64 KB; hard cap 1 MB per value. Keys average 64 bytes.
• Total working set to keep resident in memory: ~10 TB at launch, growing ~3x over 2 years.
• Traffic is highly skewed: the top 0.1% of keys receive ~30% of all reads (celebrity/hot-key pattern).
• Typical TTLs range from 60 seconds to 24 hours; ~20% of entries have no TTL and rely on eviction.

Out of scope

• Persistence to disk / snapshotting and point-in-time recovery.
• Cross-region / geo-replication of the cache tier.
• Rich data structures (lists, sorted sets, streams) and Lua-style server-side scripting.
• Billing, quota purchasing, and self-serve tenant onboarding UI.