Writing · Nº 041065 essays
The writing
Ideas from inside the work. Explore technology, entrepreneurship, and the practice of building things that last.
3 essays found · Page 1 of 1
Clear filtersEngineeringHow to Store Scraped Data at ScaleHow to store scraped data at scale: keep raw and parsed separate, pick storage by access pattern, deduplicate on identity, and plan for versioned change over time.EngineeringHow to Design a Data Pipeline for Scraped DataHow to design a data pipeline for scraped data: separate collection from processing, stage raw HTML, validate before storage, and make reprocessing cheap.EngineeringAlternative Data From Web Scraping: What Investors MissAlt data from web scraping can front-run public filings, but only if it is timely, clean, and defensible. Here is what makes scraped alternative data usable.