Writing · Nº 041065 essays
The writing
Ideas from inside the work. Explore technology, entrepreneurship, and the practice of building things that last.
42 essays found · Page 1 of 4
Clear filtersEngineeringWhen Scraping Breaks at Scale, and How to Design Around ItSmall scrapers work until they do not. The failures at scale are predictable, and you can design for them up front instead of firefighting later.EngineeringLLM Parsing vs CSS Selectors for Scraping: When to Use EachShould you parse scraped pages with an LLM or CSS selectors? Selectors are cheap and exact; LLMs handle chaos. Here is how to choose without blowing your cost per page.EngineeringScraping Personal Data: What Compliance Actually RequiresPublic does not mean unregulated. Scraping personal data triggers GDPR and CCPA obligations even when the data is on a public page. Here is what compliance requires.EngineeringHow to Schedule and Orchestrate Scraping JobsHow to schedule and orchestrate scraping jobs at scale: decouple scheduling from execution, handle retries and dependencies, and keep long jobs resumable.EngineeringHow to Extract Clean Structured Data From Messy HTMLGetting the page is the easy part of scraping. Turning messy HTML into clean structured data is where most projects fail. Here is how to extract reliable fields.EngineeringHow to Monitor Scrapers Before They Break SilentlyHow to monitor web scrapers so they fail loud, not silent: track real success rate, watch for soft blocks and data drift, and alert before a job dies quietly.EngineeringBrowser Fingerprinting Is Why Your Scraper Gets BlockedRotating proxies will not save a scraper with a bot fingerprint. Browser fingerprinting blocks scrapers by the signals they send. Here is what leaks and how to fix it.EngineeringHow to Store Scraped Data at ScaleHow to store scraped data at scale: keep raw and parsed separate, pick storage by access pattern, deduplicate on identity, and plan for versioned change over time.EngineeringHow to Scrape Flight and Travel Prices at ScaleTravel prices change by the minute and hide behind heavy anti-bot walls. Here is how to scrape flight and hotel prices reliably without exploding your costs.EngineeringHow Often Should You Recrawl a Site?How often you recrawl a site should match how fast the data moves and how fast users need it, not a default. Here is how to set crawl frequency without wasting money.EngineeringIs Web Scraping Legal? What Actually MattersIs web scraping legal? The honest answer for operators: public data, terms of service, personal data, and rate limits matter more than the yes-or-no question.EngineeringHow to Scrape Product Reviews for Sentiment and ResearchProduct reviews are a goldmine and a minefield. Here is how to scrape reviews for research, filter fake ones, and turn raw text into signals you can trust.