System Design

Service Discovery

Design a service discovery system like Consul, Eureka, or Kubernetes DNS.

Functional requirements

• A service instance registers itself on startup with its service name, IP, port, and metadata (e.g., version, zone) and deregisters on graceful shutdown.
• Registered instances send periodic heartbeats; an instance that stops heartbeating is marked unhealthy and dropped from lookups after a grace period.
• A client resolves a service name (e.g., payments-api) to the current list of healthy instances and can filter by metadata such as zone or version.
• Clients can subscribe to a service name and be notified when instances are added or removed, instead of polling.
• Operators can see every service, its instances, and their health, and can drain an instance out of rotation without stopping it.

Non-functional requirements

• Resolution is on every caller's request path: p99 under 1ms from the client's local cache and under 50ms for a cache miss that reaches the registry.
• Availability over consistency for lookups: membership may be up to 30 seconds stale (calling an instance that just died is acceptable and handled by retries), but a client must never be left with an empty list because the registry is slow or down.
• Registrations are durable within the registry cluster: a registration survives the restart of any single registry node, and the registry has no single point of failure.
• Health marking must not flap: one missed heartbeat during a network blip must not evict an instance, and a truly dead instance must leave lookups within 90 seconds.
• Only the instance that registered a record, or an authenticated operator, can renew, drain, or remove it; a compromised client cannot remove other services.

Scaling & constraints

• 500 distinct services and ~20,000 instances across 3 availability zones; the largest service has 2,000 instances, the median has 10.
• Heartbeats every 10 seconds from every instance (~2,000/sec); a deploy wave replaces ~5,000 instances in 10 minutes (~10 registrations and deregistrations per second).
• ~50,000 resolve calls per second across all callers if every call reached the registry, with 20 services receiving 80% of that traffic.
• A registry record is ~500 bytes; the entire registry is ~10 MB and fits in memory on every registry node.
• ~15,000 subscribing clients, each watching 5-30 service names; a change to the largest service must reach ~3,000 subscribers.
• Instances crash without deregistering ~50 times per day; partitions between zones happen a few times a year and last 1-10 minutes.

Out of scope

• Key-value configuration storage beyond service records.
• Service mesh sidecars, mTLS, and traffic policy.
• External load balancer and public DNS provisioning.
• Multi-datacenter federation and WAN replication.