System Design

Trending Topics

Design a trending topics feature like Twitter Trends or Google Trends.

Functional requirements

• Ingest the platform's firehose of public posts and extract candidate terms from each post (hashtags and short phrases, e.g., #WorldCup or 'Taylor Swift').
• Return the top 10 trending terms for a region and a window (the last 5 minutes or the last hour), each with an approximate mention count.
• Serve a global list plus one list per region for ~200 countries and ~1,000 major cities, using the post's geotag or the author's profile location.
• Trends reflect what is spiking now: a term that stops being mentioned drops off the list within its window, and always-popular terms (e.g., 'lol', 'love') must not permanently occupy it.
• Operators can block specific terms (spam, slurs) so they never appear, with the block taking effect within one minute.

Non-functional requirements

• Reading a trending list is the hot path: p99 latency under 50ms, served from precomputed results, never computed per request.
• Freshness: a term that begins spiking must appear in the 5-minute list within 60 seconds of the spike starting.
• Availability over consistency: counts are approximate, two users may briefly see different lists, and if the pipeline stalls the service keeps serving the last computed list marked with its age. A blocked term must never be shown.
• Durability: the blocklist and each minute's published lists are durable (kept 7 days for auditing); in-flight counters and sketches are disposable and rebuildable from the stream's 24-hour retention.
• Abuse resistance: a few hundred coordinated accounts posting the same hashtag thousands of times must not push it into the list; a trend must reflect many distinct authors.

Scaling & constraints

• The firehose averages 6,000 public posts per second, peaking at 30,000/sec during global events; each post yields 1-5 candidate terms, up to ~100,000 mentions per second.
• About 250M distinct terms appear per day, most mentioned once or twice; the true top 1,000 terms account for ~40% of all mentions.
• Reads: 50,000 list requests per second at peak across ~1,200 region lists; the global list and the top 20 countries take ~90% of reads.
• A term averages 20 bytes (up to 100); post and author IDs are 8 bytes; counting every (term, region, minute) exactly means ~1.5B distinct keys per day.
• The firehose is a partitioned log (Kafka-style) with 24-hour retention; posts arrive up to 30 seconds out of order and ~1% are duplicates from producer retries.
• Skew: a breaking-news term can go from 0 to 20% of all mentions within 2 minutes, and read traffic spikes 5x when a major event starts.

Out of scope

• Post creation, storage, and the posting API itself.
• Natural-language processing beyond tokenization (entity linking, translation, sentiment).
• Personalized or interest-based trend selection for individual users.
• The trends UI, promoted trends, and advertiser placement.