Design a Web Crawler

#Problem Statement

Design a large-scale web crawler that systematically downloads and indexes web content. Used by search engines (Googlebot), SEO tools, archiving services (Wayback Machine), and security scanners.

📖

Real-world: The original cautionary tale is the Morris Worm (1988) — arguably the first internet "crawler" gone wrong: a bug in how it tracked already-visited hosts made it re-infect machines repeatedly, crippling ~10% of the internet and producing the first CFAA conviction. It's the perfect illustration of why this design's hard parts are politeness (don't hammer one domain — respect robots.txt and rate-limit per host) and deduplication at scale (a Bloom filter for "have I seen this URL?", because tracking billions of URLs exactly is infeasible). Security angles worth raising: crawlers must avoid crawler traps (infinite calendars, dynamically generated link mazes), beware SSRF-style pitfalls (a malicious page linking to http://169.254.169.254/ or internal IPs — your crawler shouldn't fetch cloud metadata or internal services), and handle that most of the modern web is hostile or adversarial (cloaking, spam farms). Note the same design powers security scanners — a vuln scanner is a crawler with payloads.

#Requirements

#Functional

• Crawl seed URLs and recursively follow links
• Respect robots.txt (politeness)
• Avoid re-crawling unchanged pages (freshness)
• Store crawled content for downstream processing
• Handle redirects, different MIME types, duplicate content

#Non-Functional

• Crawl 1B pages/month → ~400 pages/s
• Crawl budget: don't overwhelm any single host (max 1 req/s per domain)
• Detect and avoid infinite crawl loops
• Scale horizontally across many crawler nodes

#Core Design

#URL Frontier (Priority Queue)

The URL frontier is the queue of URLs to crawl. It has two tiers:

Scheduler logic:
  for each crawler worker:
    pick next available host queue that hasn't been polled in last N seconds
    dequeue one URL from that host's queue
    dispatch to crawler worker

This enforces one request per host per N seconds without blocking all workers.

#Deduplication

Billions of URLs; many are equivalent or already crawled.

URL normalization

• Remove default ports (http://example.com:80/ → http://example.com/)
• Lowercase host
• Remove #fragment (fragment is client-side only)
• Sort query parameters consistently

Seen URL check (bloom filter):

bloom_filter.add(normalized_url)
if bloom_filter.contains(url): skip  # probably seen (false positives acceptable)

• Bloom filter for 10B URLs: ~10 GB memory (acceptable)
• False positive rate ~0.1% → occasionally re-crawl a URL (acceptable)

Content deduplication (SimHash):

• After downloading, compute SimHash of content
• Two pages with SimHash Hamming distance < 3 → near-duplicates (mirror content)
• Store SimHash of each page; skip near-duplicates

#robots.txt

Every crawler must respect robots.txt:

User-agent: *
Disallow: /admin/
Crawl-delay: 10

User-agent: Googlebot
Allow: /

• Fetch and cache robots.txt per domain on first visit
• Cache TTL: 24h (re-fetch regularly in case it changes)
• Respect Crawl-delay directive

#Architecture

Seed URLs → URL Frontier (Redis sorted set + per-domain queues)
                  ↓
         Crawler Workers (auto-scaling)
                  ↓
    ┌─────────────────────────────────────┐
    │  DNS Resolver (cached, rate-limited)│
    │  HTTP Client (timeout, redirect cap)│
    │  robots.txt Cache                   │
    └─────────────────────────────────────┘
                  ↓
         Raw Content Store (S3 / HDFS)
                  ↓
    ┌────────────────────────────────┐
    │  Link Extractor                │ → new URLs → URL Frontier
    │  Content Parser (HTML, PDF...) │ → Document Index (Elasticsearch)
    │  SimHash Dedup                 │
    └────────────────────────────────┘
         ↑
    URL Metadata DB (Postgres):
      - last_crawled_at
      - content_hash
      - crawl_status
      - next_crawl_at

#Recrawl Scheduling (Freshness)

Not all pages change at the same rate. Schedule recrawls based on change frequency:

• News sites: every 15 minutes
• Product pages: daily
• Static documentation: weekly

Track last_modified header and ETag. Send If-Modified-Since / If-None-Match on recrawl → server returns 304 Not Modified if unchanged → skip processing.

#Key Design Decisions

#Security Considerations

#Interview Tips