Week 8 · Tue Sep 15, 2026 · Sep 14-20, 2026

Day 2: Web Crawler CORE Deep Dives

A Staff-level crawler answer for frontier leases, retry/DLQ, robots, per-domain pacing, dedupe, and 10B-page throughput.

How do politeness and fault tolerance survive scale?Frontier · leases · retries · dedupe · recovery3 Heap/Priority Queue slots

Exact Time Blocks

Follow the Week 4 delivery pattern: blank-page first, source reading second, DDIA decision third, spoken close, contiguous algorithm practice, evening recall.

Time	Block	Required output
08:30	Wake + setup	Open this page, start the board, write the day question: How do politeness and fault tolerance survive scale?
08:50—09:00	Board set	List entities, queues, leases, retry states, dedupe keys, metrics before reading.
09:00—10:05	Blank-page attempt + Hello reading	Rebuild the deep dives from memory first: progress loss, robots/politeness, and 10B pages in five days. Then attach mechanisms: SQS visibility, frontier table, robots cache TTL, domain token bucket, DNS cache, canonical URL table, Bloom/content hash dedupe, crawler-trap caps.
10:05—10:45	Deep dive + DDIA decision	Write two recovery stories: worker dies after fetch before parse, and worker pauses while holding a domain lease. The correct answer names who can retry, what token prevents stale writes, and what metric proves the system is not abusing a domain.
10:45—10:55	Spoken close	State the crawler invariant: no URL is considered done until durable progress and dedupe-visible output exist; no host is fetched without a fresh robots decision and rate token.
14:30—16:15	NeetCode Heap/Priority Queue	Complete contiguous slots W8-H04—W8-H06; capture invariant and bug risk for each.
20:30—21:15	Recall / Q&A / mock	Use Staff Q&A, lecture assets, or mock script for the day's system.

Canonical Heading Anchors

Read only the listed Hello Interview headings for today's assignment. Every link is a direct day-page anchor.

Potential Deep DivesDay 2

CORE deep-dive index

1) How can we ensure we are fault tolerant and don't lose progress?Day 2

CORE restart-safe frontier and durable progress

2) How can we ensure politeness and adhere to robots.txt?Day 2

CORE per-domain politeness and robots enforcement

3) How to scale to 10B pages and efficiently crawl them in under 5 days?Day 2

CORE partitioning, batching, and throughput

Some additional deep dives you might considerDay 2

CORE optional pressure prompts

What is Expected at Each Level?Day 6

Mock rubric calibration

Mid-levelDay 6

Rubric contrast

SeniorDay 6

Senior+ bar

Staff+Day 6

Staff bar

Decision Notes

CORE means you must answer requirement, HLD, deep dive, failure, recovery, metric, and Staff-level tradeoff.
TRANSFER means blind attempt first, then delta against source, invalid assumption, new bottleneck, and follow-up pressure question.
Do not add open-ended AI radar unless it changes a concrete Week 8 design decision; this week it does not.

DDIA Decision Map

Use exact sections only from Ch5, Ch9, and Ch12.

Chapter	Exact DDIA decision section	Concrete Week 8 use
Ch9	故障与部分失效	Assume workers pause, networks lie, and queues redeliver.
Ch9	隔离僵尸进程和延迟请求	Per-domain crawl ownership needs fencing tokens so old workers cannot violate politeness after a pause.
Ch12	基于日志的消息代理	Replay and offset ownership make large frontier movement auditable.
Ch12	消费者偏移量	Commit offsets only after dedupe/write-side state is durable.

Heap/Priority Queue Slots

Three contiguous slots today. Keep one invariant and one failure mode per problem.

Slot	Problem	Pattern	Invariant / bug risk	Time · Space
W8-H04	K Closest Points to Origin	bounded max-heap	Compare squared distance and avoid floating point.	O(n log k) · O(k)
W8-H05	Top K Frequent Elements	count map + heap	Frequency is the priority; the value is just tie metadata.	O(n log k) · O(n)
W8-H06	Task Scheduler	cooldown priority queue	Time advances to the next available task, not one wasted tick at a time.	O(n log a) · O(a)

Evidence To Save

One compact artifact is enough if it proves mechanism depth.

Invariant

State safety

Name the state that survives restart and the write that makes retry safe.

Failure

Concrete break

Use a worker crash, zombie lease, redelivery, poison message, stale robots decision, or duplicate price event.

Metric

Operational proof

Attach lag, retry rate, duplicate suppression, stale lease reject count, host politeness violation, or alert freshness.

Local study materials
Detailed lecture notes, audio, recall scripts, PDFs, Staff Q&A, and mock packs are archived locally and are intentionally not published on this site.