Writing · Nº 041065 essays
The writing
Ideas from inside the work. Explore technology, entrepreneurship, and the practice of building things that last.
42 essays found · Page 3 of 4
Clear filtersEngineeringManaged Scraping API vs Self-Hosted Scraper: The Real TradeoffA managed scraping API trades control for convenience. Here is how to decide between a scraping API and a self-hosted scraper based on scale, cost, and lock-in.EngineeringHow to Validate Scraped Data Quality Before It Poisons YouScraped data quality fails silently. A selector drifts and you store nulls for a week. Here is how to validate scraped data at ingestion before it reaches users.EngineeringThe Myth That Public Data Is Free to ScrapePublic does not mean free to take. This breaks down the myth that any public web data is legal to scrape, and what actually determines your real risk.EngineeringWeb Scraping Mistakes That Get You BlockedThe common web scraping mistakes that get you blocked: bad rate limits, obvious fingerprints, ignoring soft blocks, and treating every target the same way.EngineeringIncremental Crawl vs Full Recrawl: How to DecideAn incremental crawl only fetches what changed since last time. A full recrawl refetches everything. Here is how to choose and cut most sites' cost by 90 percent.EngineeringAlternative Data From Web Scraping: What Investors MissAlt data from web scraping can front-run public filings, but only if it is timely, clean, and defensible. Here is what makes scraped alternative data usable.EngineeringBuy or Build Your Scraping Stack: How to DecideShould you buy a scraping API or build your own stack? A clear decision framework on cost, control, and scale for owning your scraping infrastructure.EngineeringHow to Deduplicate Scraped Data Without Losing RecordsDeduplicating scraped data is not a DISTINCT query. Here is how to pick a real identity key, handle near-duplicates, and merge records without deleting good data.EngineeringScraping for Market Research: What Data Is Worth CollectingNot all web data is worth scraping for market research. Here is how to decide what to collect, what to skip, and how to turn scraped data into real answers.EngineeringBuild Your Own SERP Scraper Instead of Renting Rank DataRank tracking tools resell SERP data at a markup. Here is when to build your own SERP scraper for SEO data, what breaks, and when renting is still the smart call.EngineeringResidential vs Datacenter Proxies: Which to Use WhenA practical breakdown of residential vs datacenter proxies for web scraping: cost, speed, block rates, and how to pick the right one per target.EngineeringHow to Calculate the True Cost Per Page of ScrapingYour cost per page of scraping is not the proxy bill. It is proxies plus compute plus retries plus parsing plus storage. Here is how to compute the real number.