System Design

Web Crawler

Design a large-scale distributed web crawler.

Functional requirements

• Given a set of seed URLs, crawl the web by fetching pages, extracting links, and enqueueing newly discovered URLs for future crawling.
• Store fetched page content (HTML) durably for downstream consumers (indexer, link-graph builder).
• Respect robots.txt directives and honor per-site crawl-delay rules.
• Support prioritized crawling: high-value pages (e.g., news, popular domains) are fetched sooner and re-crawled more frequently than low-value pages.
• Detect and skip URLs that have already been crawled (within a configurable re-crawl window).

Non-functional requirements

• Politeness is a hard requirement: never overwhelm any single host; bound concurrent connections and request rate per domain regardless of internal load.
• Prioritize availability and throughput over strict consistency: it is acceptable to occasionally re-fetch a page or briefly crawl a stale priority, but the crawl must never stall globally.
• Fault tolerant: crash of any crawler machine must not lose the frontier state or permanently orphan URLs assigned to it.
• Scalable and elastic: adding crawler machines should near-linearly increase aggregate fetch throughput.
• Robust against adversarial or broken web content: the crawler must not waste crawl resources on low-value or adversarial regions of the web, must bound what it fetches per site, and must tolerate slow or hanging servers and malformed HTML without degrading the fleet.

Scaling & constraints

• Target corpus: 10 billion pages crawled per month (steady state, including re-crawls).
• Average page size: ~500 KB raw HTML; average of ~100 outbound links per page.
• URL universe: assume ~50 billion distinct URLs seen over the crawler's lifetime; average URL length ~80 bytes.
• Domain distribution is highly skewed: a small fraction of domains (e.g., large sites) hold a large fraction of URLs, while the long tail is hundreds of millions of small hosts.
• Fetched content must be retained for 2 months for the indexing pipeline; the URL-seen set and link graph metadata are retained indefinitely.
• Re-crawl policy: high-priority pages re-fetched daily, typical pages every 1-4 weeks.

Out of scope

• Building the search index, ranking, or query serving; the crawler only produces stored content and link data.
• JavaScript rendering / headless-browser crawling of dynamic pages.
• Handling authentication-gated or paywalled content.
• Legal/copyright policy decisions beyond honoring robots.txt.