How to Validate Scraped Data Quality Before It Poisons You

The worst scraping failure is not a crash. A crash you notice. The worst failure is the scraper that keeps running, keeps returning HTTP 200, and quietly stores garbage because a selector drifted or the site changed a label. You find out a week later when a customer asks why every price is null. Validating scraped data quality at ingestion is how you catch this before it ships. The rule I hold to: a page that parsed into wrong or empty data is a failure, even though nothing errored, and your pipeline has to treat it that way.

Why does scraped data fail silently?

Because success and correctness are different things, and most pipelines only check success. The request succeeded, the page loaded, the parser ran without throwing. Everything is green. But the site moved the price into a new element, so your selector matched nothing and you stored a null. No exception fired. The crawl "succeeded" a hundred thousand times and collected a hundred thousand blanks.

This is the gap between "did the scraper run" and "did the scraper get correct data." Monitoring the first is not enough, which is why I treat validation as its own stage separate from uptime checks. I cover the operational side of catching broken crawls in how to monitor scrapers before they break silently, but monitoring tells you the machine is alive. Validation tells you the data is real.

What checks catch bad data at ingestion?

Validate every record against expectations before it enters your store. The checks that earn their keep:

• Required fields present. If price, title, or ID is null, reject or flag the record. A product with no price is not a product.
• Type and format. Price parses as a number. Date parses as a date. URL looks like a URL. A price of "Add to cart" means your selector grabbed the wrong element.
• Range and sanity. A price of zero or a million on a consumer product is almost always a parse error, not a real value. Bound your numeric fields.
• Enum membership. A status field should be one of a known set. A new value means the site changed or you are reading the wrong element.
• Cross field consistency. Sale price below list price. End date after start date. Cheap logic that catches expensive mistakes.

None of this is fancy. It is a schema plus a handful of assertions, run on every record at the door. The value is that it converts silent corruption into a loud, countable event.

How do I catch drift across the whole crawl, not one record?

Per record checks miss a class of failure: the crawl where every record is individually plausible but the batch as a whole is wrong. For that you watch aggregate statistics run over run.

Track the null rate per field, the record count, and the distribution of key fields. Then alert on movement. If your null rate for "price" jumps from two percent to sixty percent overnight, a selector broke, even though each null record looked fine on its own. If your record count halves, a listing page changed pagination. If the average price shifts by an order of magnitude, you are parsing the wrong number. These are the checks that catch site redesigns before your users do, and they pair directly with the layout change problem I dig into in how to handle website layout changes that break scrapers.

What do I do with data that fails validation?

Quarantine, do not delete and do not pass through. A record that fails validation goes to a holding area with the reason it failed and the raw HTML that produced it. That raw capture is what lets a human or a fixed parser reprocess it later without recrawling, which saves the fetch cost twice.

Quarantining also protects everything downstream. Bad records never reach dedup, never reach your customers, and never get treated as truth. This is why validation sits early in the flow, right after fetch and before dedup and storage, which is how I structure it in how to design a data pipeline for scraped data. Bad input that reaches dedup just creates confidently wrong merged records, and the dedup logic I described in how to deduplicate scraped data without losing records assumes the input already passed the door.

Data quality is not a phase you do at the end. It is a gate every record passes through on the way in. Write the schema, assert on every field, watch the aggregate stats, and quarantine the failures with their raw source. Do that and the silent week of nulls stops happening. If you would rather have field level validation and drift alerting built into the crawl itself, that is part of what PyroSync ships.