Ingests, parses, classifies & serves billions of URLs/month. A Kafka-decoupled crawler pipeline — URL frontier in, an NLP agent answering natural-language SEO queries out. Python · FastAPI · Kafka · Redis · S3 · ClickHouse.
Assumed ~5–10% of URLs need Playwright. At 25% the compute bill multiplies instantly. Fix: 10K-URL sample from the customer seed list, measure render requirement.
The tiered cascade (90% zero-shot BART, 10% Gemini) is the linchpin of the cost model; real ratios depend on traffic mix. Fix: budget review at week 4 — smaller models / request batching on overrun.
curl_cffi TLS fingerprints work in isolation, but sustained >1K QPS from AWS IPs may trigger Cloudflare/Amazon aggregate detection. Fix: spike fetcher fleet at 1K QPS for a week, measure per-domain block rate → calibrate proxy need.
>1K QPS across millions of domains makes per-lookup overhead a hidden bottleneck; failures misread as site downtime. Fix: shared 24h DNS cache + multi-provider (Cloudflare/Google/Route53) round-robin.