Offline Evals vs Production Monitoring for AI

Offline evals tell you if a change is safe to ship. Production monitoring tells you if the shipped thing is actually working. They are not the same job, and you cannot substitute one for the other. Teams that only run offline evals ship something that scores well on their test set and then quietly rots in the wild. Teams that only watch production find out about regressions from angry users instead of from a gate. An AI-native product needs both, wired into different points in the lifecycle.

The thesis: offline evals are your pre-ship gate against a frozen set, production monitoring is your live alarm against real traffic, and each catches exactly what the other misses.

What offline evals catch and miss

Offline evaluation runs your feature against a fixed golden eval set before you deploy. Its whole value is control: same inputs every time, so a score change means your system changed. That makes it the right tool for the question "is this new prompt or model better than the current one." It catches regressions before users see them.

What it misses is everything you did not put in the set. Offline evals cannot see the input distribution shifting, cannot see a new document format users started uploading, cannot see the model degrading on a category you never tested. It is a snapshot of the past used to gate the future. Necessary, not sufficient.

What production monitoring catches and misses

Production monitoring watches real traffic after ship. It catches what offline evals structurally cannot: distribution drift, novel failure modes, the slow slide where a feature that worked in March is worse in June because usage moved. This is the same failure pattern behind why AI agents fail in production, and you only see it if you are looking at live behavior.

The signals worth tracking are mostly proxies, because you rarely have ground-truth labels in real time. Watch regenerate rate, user edit distance from the model's output, thumbs-down rate, escalation-to-human rate, and hard errors. A spike in any of them is a smoke alarm. It will not tell you exactly what broke, but it tells you to look. Measuring these consistently is the same discipline as measuring agent reliability: pick the metric that correlates with a bad outcome and alert on it.

What monitoring misses is the counterfactual. It cannot tell you whether a different prompt would have done better on the same traffic. For that you need to feed real cases back into offline evals.

How the two work together

The loop closes when they feed each other. Production surfaces a failure you never imagined. You capture that case, label it, and add it to the offline eval set. Now the next change is gated against it. Over time the offline set hardens because production keeps teaching it, and monitoring quiets down because the gate keeps catching more before ship.

Concretely: offline evals run in CI on every prompt or model change and block the deploy if the score drops. Monitoring runs continuously and pages you when a live metric moves. The SLA you promise rests on both, because a gate with no live alarm is blind, and a live alarm with no gate means you learn about every regression the hard way.

The mistake that costs the most

The expensive mistake is treating a good offline score as proof the feature is fine. It is proof the feature is fine on the cases you tested, in the world as it looked when you froze the set. The world moves. So does the model, every time a provider updates it under you, which is how model upgrades erase capability edges without a changelog you control.

We run both across the model-backed features at Girard AI: a frozen eval gate in the pipeline and live metrics on real traffic, with production failures flowing back into the gate. Offline to ship safe. Online to stay safe. Skip either and you find out about your regressions from the people you least want to hear from.