System Design

Message Queue

Design a distributed message queue like Kafka or SQS.

Functional requirements

• Producers publish messages to named topics; consumers subscribe and receive those messages.
• Support consumer groups: multiple consumer instances share a topic's traffic for parallel processing, and multiple independent groups can each receive the full stream.
• Guarantee ordering for messages that share a key (e.g., all events for a given user ID are delivered in publish order to their consumer).
• Consumers acknowledge processing; unacknowledged messages are redelivered. Consumers can also rewind/replay from a retained history window.
• Messages that repeatedly fail processing are routed to a dead-letter destination rather than blocking the stream.

Non-functional requirements

• Durability first: once a publish is acknowledged to the producer, the message must survive the permanent loss of any single node (and ideally a full availability-zone outage).
• At-least-once delivery is the baseline guarantee; the design should explain how consumers achieve effectively-exactly-once processing on top of it.
• End-to-end latency (publish to consumer receipt) p99 under 500ms under normal load; publish-ack p99 under 50ms.
• High availability for publishes: a single broker/node failure must not reject writes for more than a few seconds; prefer availability of the pipeline over strict global ordering across keys.
• Per-topic authentication/authorization and encryption in transit; tenants must not read each other's topics.

Scaling & constraints

• ~10,000 producing services and ~5,000 consumer groups across ~50,000 topics company-wide.
• Aggregate publish volume: ~2 million messages/second at peak, average message size 1 KB (max 1 MB).
• Fan-out: each message is read by 3 consumer groups on average (read:write roughly 3:1).
• Retention: 7 days of replayable history by default; some topics configure up to 30 days.
• Hot topics exist: the top 10 topics carry ~40% of total traffic, and traffic within a topic can be heavily skewed toward a few keys.
• Traffic grows ~2x year over year; bursts of 5x sustained for minutes must be absorbed without dropping publishes.

Out of scope

• Stream processing / transformation frameworks (windowing, joins, SQL-on-streams).
• Cross-region / multi-datacenter geo-replication.
• Schema registry and message format evolution.
• Billing, quotas UI, and multi-tenant cost accounting.