MLOps Part 4: Monitoring and Surviving Silent Failure
Part 3 put the model in front of people. The dashboards are green, traffic is flowing, and the predictions are quietly rotting.
This is the pillar that gives the least warning and costs the most to get wrong.
The four layers
Monitoring works in four layers, and they differ in two ways that pull against each other: how fast they tell you something, and how much that something is worth.
The four layers of monitoring. The signals you get soonest are the ones that mean least.
Layer 1 is ordinary operations. Latency, error rates, saturation, memory, cost per thousand documents. Familiar territory, and worth remembering that a perfectly healthy server will serve wrong answers all day without complaint.
One metric here is easy to miss. If the online store is eventually consistent, serving can read a stale feature, so watch freshness: how old is the newest value, and what is the p99 of that age. That number is where your consistency trade-off actually surfaces.
Layer 2 is the data contract, checked continuously rather than only at ingestion. Null rates, value ranges, cardinality, row volumes, and freshness. Most of what teams later call “the model broke” turns out to have been a Layer 2 event nobody was watching for.
Layer 4 is the only absolute answer, and it is expensive. Worth being precise about why: the cost is not compute, it is labels. Every honest measurement of quality needs a human to say what the right answer was, and in our system that human is a reviewer working weeks behind the prediction.
Which leaves Layer 3 doing most of the real work, because it is the earliest signal that is worth anything.
The three drifts
“Drift” gets used for three different phenomena. They have different causes, different detection methods, and crucially different fixes, so it pays to keep them apart.
| What moves | In our system | What to do | |
|---|---|---|---|
| Covariate drift | P(X), the inputs | Documents start arriving in another language, posts get shorter, a new source system sends a different layout | Often retrain on newer data. First check it is not your own release. |
| Concept drift | P(Y|X), the labelling rule | Regulation changes and something that was not personal data now is | Relabel, then retrain. There is no code fix. |
| Label shift | P(Y), the class balance | A model trained where 3% of documents were positive now serves a customer where 20% are | Recalibrate. Often no retrain needed at all. |
Two traps in that table.
Covariate drift is not the same as changing your own pipeline. Adding a feature or altering a preprocessing step also moves the distribution the model sees, but that is something you did, not something that happened. If a drift alarm fires every time you deploy, it is measuring your release process. (A policy that adds a whole new category is not label shift either; that is a different problem needing a different model.)
Label shift is the cheapest to fix and the most over-treated. A classifier’s scores are calibrated to the base rate it trained on, so at 20% positives instead of 3% it is systematically under-confident and the threshold sits in the wrong place. Adjust the threshold for the new prior, or apply a prior correction, and you have probably fixed it without touching the model. Strictly this case is population shift rather than drift: nothing moved over time, you pointed the model at a population it was never trained on. Same arithmetic, different cause, and something you measure at onboarding rather than monitor for.
Detecting it
For tabular features, per-feature statistical tests do the job and tell you which feature moved: population stability index, Kolmogorov-Smirnov for continuous values, chi-squared for categorical ones. Cheap, interpretable, and the standard offering in tools like Evidently.
Text and embeddings are harder, because comparing high-dimensional distributions directly is genuinely difficult. The instinct to compare the embeddings themselves is right, and it has two workable versions.
The crude version is to summarise, then compare: reduce each batch of embeddings to something comparable, a mean vector or a set of distances to fixed reference points, and watch that series over time. It is blunt, and it is frequently enough.
The better version is the domain classifier, which deserves to be better known. Train a small classifier to distinguish your training data from this week’s production data. If it cannot tell them apart, you have not drifted. If it can, you have, and its feature importances point straight at where. It converts “has this distribution changed?” into an ordinary supervised problem you already know how to solve, and it works on embeddings without any dimensionality tricks.
It is also ambiguous, and that is the part to internalise. That jump could be covariate drift, concept drift, label shift, a broken upstream job dumping malformed text, or one large new customer with a genuinely different mix. Prediction drift tells you to go and look. It never tells you what you are going to find, and treating it as a retrain trigger is how teams end up retraining monthly against an upstream bug.
What actually runs this
Worth clearing up a confusion that costs teams a lot of time, because the tools in this space are usually presented as competitors when most of them are layers.
Grafana is not an alternative to Evidently. They do different jobs, and a working setup has all three of these:
They compose rather than compete. Evidently runs as a scheduled task inside a Prefect flow, reads the prediction log, computes a report, and emits the results as ordinary metrics. Prometheus stores those. Grafana dashboards and alerts on them next to latency and error rate, so there is one alerting stack rather than two. OpenTelemetry instruments the serving code that produced the predictions in the first place.
Picking the ML-specific layer
| Tool | What it is for | Reach for it when |
|---|---|---|
| Evidently | Drift, data quality, model quality reports | Default open-source choice. Python library, no account, handles text as well as tabular. |
| NannyML | Estimating performance without labels | Ground truth arrives late, which is our situation. Complements Evidently rather than replacing it. |
| whylogs / WhyLabs | Statistical profiles instead of raw records | The data must not leave your perimeter. You ship sketches, never documents. |
| Deepchecks | Validation suites with pass/fail | You want gates in CI rather than dashboards. |
| Arize, Fiddler, Aporia | Hosted platforms, embedding drift, explainability | Large scale, real budget, and no objection to sending data to a vendor. |
| Prometheus and Grafana alone | Numbers over time | You are willing to write the drift statistics yourself. Entirely viable, just more work. |
One practical wrinkle worth knowing before you wire it up. Prometheus is built for scraping live targets and is awkward with delayed writes, which is exactly what batch monitoring produces when a report runs nightly or weekly. The usual fix is to skip Prometheus for the batch path and write reports to Postgres, pointing Grafana at that instead. Evidently’s own reference blueprint for batch is Evidently, Prefect, PostgreSQL and Grafana, which happens to be the stack we already have.
Why your drift alerts stop being read
Here is where a monitoring section usually ends and where the real problem starts.
Suppose we monitor 400 features across 6 customers, and every one of those series gets a statistical drift test at the usual p < 0.05. That is 2,400 tests a day, so roughly 120 alerts fire daily purely by construction, before anything is actually wrong. Within a fortnight nobody reads them.
That is not a discipline failure. The system trained the team to ignore it, and an alert nobody acts on is worse than no alert at all, because it supplies false assurance. Four causes, in rough order of how much damage they do:
Multiple comparisons. Run enough tests and significance is guaranteed. With thousands of series you need to correct for that, or accept that your alarm is mostly measuring the number of things you decided to watch.
Statistical significance is not operational significance. At millions of documents, every test is significant. A distribution can move by an amount no human would notice and no prediction depends on. Threshold on effect size, a PSI band for instance, rather than on a p-value.
Monitoring inputs instead of outputs. This is the biggest simplification available and the one most often missed. Input drift only matters if it changes predictions. So collapse those 2,400 input monitors into a handful of output monitors, the prediction distribution, the confidence distribution, and the quality proxies, and drill into individual features only once an output signal fires. You go from thousands of alarms to a few.
No importance weighting. A feature the model barely uses drifting is not an incident. Weight what you watch by how much the model actually depends on it.
One cause, many symptoms. A single stale upstream table fans out across every downstream series at once. Two hundred pages for one root cause is a grouping problem, not two hundred problems.
The fixes are unglamorous. Severity tiers routed to different places: page for something customer-impacting right now, ticket for investigate-this-week, dashboard for context that never notifies anybody. Route by accountability rather than broadcasting to the team, so an infrastructure alert reaches the on-call and a data-quality alert reaches whoever owns the pipeline.
And one test that settles most arguments about whether something should alert:
If there is no documented action a person would take, it does not page. Every alert needs an owner and a runbook, or it is a dashboard with ambitions.
Deciding what to do about it
An alarm fires. The naive response is to retrain, and most automated pipelines are configured to do exactly that.
The production response is to investigate, and the reason is uncomfortable: most drift alarms are not the world changing. They are a broken pipe. An upstream export switched encoding. A parser started returning empty strings for a new file layout. Someone shipped a change to the chunker. In every one of those cases retraining does not fix anything, it laminates the bug into the weights.
So there is a triage order, and it is worth having written down before you need it at 2am.
- Is it real? Effect size, not p-value, for the reasons in the previous section.
- Is it us? Look at the deploy log before the data. A drift alarm that starts within an hour of a release is a release, not a drift.
- Localise it. Which customer, which source system, which document type, which time window. Drift that turns out to live entirely in one tenant’s scanned PDFs is a much smaller problem than it first appeared.
- Which kind is it? Covariate, label shift, or concept, because the three have different fixes and only one of them needs new labels.
- Does it matter? If no output signal moved, you have detected a fact rather than a problem.
- Only now, respond.
Two guards. Never let a drift alarm trigger training directly, only a ticket. And pin the reference distribution to a named, reviewed snapshot rather than "whatever we last trained on".
One more distinction belongs here, because the symptom is identical and the response is opposite. Skew is the training and serving paths disagreeing: broken since day one, our bug, fixed in code, and explained properly in Part 2. Drift is the two paths agreeing perfectly while the world moves: not a bug, fixed with new labels.
Accuracy sagging looks identical either way. Retraining to fix skew is like rebooting to fix a memory leak: it works until Thursday, teaches you nothing, and you will do it again next week.
Surviving the wait for ground truth
Layer 4 is the only answer that is actually true, and in our system it arrives weeks late. A reviewer confirms a verdict long after we produced it, and a regulator’s finding is slower still. So the practical question is what you can know in the meantime.
Four things, none of which need a label:
You can go further and estimate the metrics themselves without labels. Tools like NannyML use the confidence distribution together with observed input shift to estimate what precision and recall probably are, weeks before the labels land.
That is genuinely useful, and it rests on an assumption worth stating out loud: it assumes the labelling rule has not changed. It works by reasoning about how the model behaves on shifted inputs, which is exactly the wrong tool for the case where the inputs are identical and the definition moved. Useful, with its limits understood, and blind to the one thing this post ends on.
Which leaves the expensive answer, and there is no clever way around it.
Review a random sample of the documents the model cleared. It feels like wasted analyst time, and it is the only unbiased estimate of recall you will ever have.
Everything else you measure comes from the flagged pile. Corrections from that pile are all false positives being fixed, so precision looks like it is improving while recall has no denominator at all. It is not being measured badly, it is unmeasurable. Sampling the cleared pile is the only thing that supplies one.
Notice this is the second time we have arrived here from a different direction. The feedback-loop argument in Part 3 said you need an unbiased labelled sample because otherwise the model sharpens only on visible errors. The concept-drift argument says you need one because otherwise a changed rule is invisible. Two independent arguments landing on the same requirement usually means it is load-bearing. Budget it as a fixed percentage of review capacity and treat it as instrumentation, not overhead.
The drift none of this catches
Here is the case the whole series has been building towards, and it defeats every mechanism in this post.
The compliance policy changes. A category of identifier that was not personal data on Monday is personal data on Tuesday. Now look at what our monitoring sees:
| Layer | What it reports | Reality |
|---|---|---|
| 1 · System | Green. Latency and error rates unchanged. | Unchanged, correctly. |
| 2 · Data quality | Green. Same schema, same nulls, same volumes. | The documents genuinely are identical. |
| 3 · Drift | Green. P(X) has not moved by any measure. | The inputs did not move. The answer did. |
| 3b · Prediction drift | Green. The model flags the same share it always did. | Which is now the wrong share. |
| 4 · Model quality | Green, until relabelled data arrives. | Already wrong, for weeks. |
Every detector is working correctly and every one of them is useless, because they all watch inputs or outputs and neither moved. What moved is the meaning of the label, and no amount of instrumentation on our side of the system can observe a definition changing on somebody else’s.
Two consequences follow, and both are worse than they first sound.
It is retroactive. Every document you have ever labelled was labelled under the old rule. So “retrain on recent data” does not help, because recent data carries old labels too. The fix is to relabel history under the new definition, which is an annotation programme rather than a training job. Concept drift has no code fix and, in this form, no data-recency fix either.
Confidence does not degrade. The model is not uncertain about the documents it is now wrong about. It is exactly as confident as it was last week, because nothing in its input changed. There is no signal anywhere in the system.
And yet this is the most tractable drift in the series, for one reason that inverts everything above.
You can see it coming. Regulation is published before it is enforced.
Every other kind of drift is discovered after the fact and handled reactively. A policy change arrives with a publication date, a text, and usually months of lead time, so the correct response is not a better detector but a process: treat policy publication as a pipeline trigger, diff the definition to find which documents are affected, relabel a targeted sample starting at the old decision boundary, and train and shadow the replacement before the enforcement date. Record which policy version each decision was made under, because “under the rules as they stood in March” is only a defence if you can prove it.
Done that way, the hardest failure mode in the series stops being a failure at all. It becomes a scheduled release.
The takeaway
Monitoring is four layers arranged unhelpfully: the signals you get soonest mean the least, and the one that means everything arrives last. System and data quality are fast, cheap, and catch failures that are usually somebody else’s bug. Drift is the leading indicator, and it will drown you unless you watch outputs rather than every input. Model quality is the only truth and is bounded by how fast humans can label. And a drift alarm is always a reason to investigate, never a trigger to retrain.
That closes the series. Four posts, all circling one idea: a model is a file of numbers and some code that reads them, and everything difficult about running one follows from its behaviour living in data that will not hold still.
If a single line survives, make it this one.
A control is only a control if it is the only path.
A data contract nobody validates at a boundary is a schema file. A registry you can bypass is documentation. A feature definition that exists twice is not a definition. A drift alarm with no runbook is a notification. And an unbiased sample of what the model cleared is the only one of these you cannot fake, which is precisely why it is the first thing cut when the quarter gets busy.
None of that makes the model better. It makes the system honest about how good the model already is, which is the harder problem and the one worth building for.