MLOps Part 1: Why Models Fail to Ship
We talk about machine learning models like they’re minds. They’re files.
A model is two things, and neither of them is clever:
- A file of numbers. Weights, coefficients, split points. Whatever your algorithm accumulated on its way through the data.
- Code that knows what those numbers mean. That this particular block of floats is a twelve-layer transformer and not a lookup table.
On their own, the numbers are inert: a few hundred megabytes with no opinion about anything. Point the right code at them and you get a prediction about a document nobody has ever seen.
That is the whole object. Which makes the obvious question slightly awkward: if a model is just a file and some code, why is getting one into production so reliably miserable?
The hidden technical debt in ML
Because the file is not the thing you are actually shipping.
In normal software, behaviour lives in code, code lives in the repo, and the repo is the truth. Machine learning breaks that in one place: the behaviour lives in data, and data does not hold still.
Deploy a microservice and it does the same thing on Friday that it did on Monday. Deploy a model and it can be wrong by Friday having changed nothing at all, because the world moved and the model did not.
Sculley et al. called this the hidden technical debt of ML systems, and the name earns its keep: you ship fast by copying a pickle onto a server, and the bill arrives later.
- Reproducibility. Six months on, can you name the dataset, the hyperparameters, and the commit behind the model serving traffic right now?
- Unowned state. θ is not the only thing that got fitted. The tokenizer’s vocabulary, the scaler’s mean, the linker’s alias table: all learned from data, all part of the behaviour, all versioned by nobody.
- Silent decay. When the model stops being accurate, what exactly tells you?
Three symptoms, one shared cause. Finding it means asking a duller question than which architecture should we use?
Two workloads, one contract
The duller question is this: when do you use the numbers?
There are only two answers. You use them to build the file. That is training. You use them to answer a question. That is inference. That is the entire list, and the two jobs agree on almost nothing.
Two workloads, one θ. Two jobs that agree on nothing except the answer they owe you.
Most of that table is logistics. The fourth row is the one that makes bugs, because every convenience training enjoys is a privilege serving will never have, and borrowing one by accident gives you a model that scored beautifully offline and is worth nothing in production.
Keeping the promise at the bottom of that table is not something you achieve once and tick off. It is something you operate.
So what is MLOps, then?
MLOps did not arrive as a grand idea. It accumulated, one hard-won fix at a time.
Every practice in it exists because some team shipped a model that worked, watched it quietly stop working, and could not say why. The textbook definition (a set of engineering practices covering the end-to-end lifecycle of an ML system) is accurate and instantly forgettable. The version worth carrying around is narrower:
MLOps is the discipline of keeping the training workload and the serving workload telling the same story about the same world, and finding out fast when they stop.
That is the contract from the last section, restated as a job. Everything else is machinery in service of it.
The shape of the machinery
An ML system is not a pipeline that finishes. It is a loop, and each stage of the loop is where one practice lives.
The MLOps loop. Seven stages, and the one practice that keeps each of them honest.
Stages and pillars are not two vocabularies for the same list. They are joined by a third thing, and it is the interesting one: every stage has a characteristic way of going wrong, and each pillar exists to catch exactly one of them.
Read the loop as a sentence, left to right:
| Stage | What breaks if nobody is watching | The pillar that catches it |
|---|---|---|
| Data | Nobody can say which rows trained the model that is live | Data versioning |
| Features | Training and serving compute the same field two different ways | Feature management |
| Train | Last week’s best run cannot be found, explained, or repeated | Experiment tracking |
| Evaluate | The headline number improves while the cases you care about get worse | Model evaluation |
| Release | Nobody is quite certain which version is answering requests | Model registry |
| Serve | The answer is correct and arrives too slowly, or costs too much | Serving |
| Observe | The model is wrong and says nothing about it | Monitoring |
That middle column is the argument. Each pillar exists to catch one specific failure, and every one of those failures is a version of the debt from earlier in this post. It is the difference between a list of practices and a list of reasons.
Those seven names hold for the rest of the series. Everything else in MLOps, and there is a great deal of it, is detail living inside one of them.
The system we will build
Abstract MLOps is easy to agree with and impossible to act on, so everything from here to the end of the series is anchored to one system. It is deliberately not a toy.
A compliance agency has to decide, for every document that crosses its desk, whether that document contains personal data. Get it wrong in one direction and you have a leak. Get it wrong in the other and you bury your reviewers in false alarms.
The system. Three intakes, one classifier, and a label definition that does not belong to us.
Three intake paths, chosen because they stress completely different parts of the machinery:
One classifier, three arrival patterns. That single fact generates most of the interesting problems in this series, because the batch path and the streaming path are the two workloads from earlier in this post, wearing different clothes and still owing each other the same promise.
The model itself is deliberately boring to start with: a scikit-learn baseline over TF-IDF features. Boring is the point. A linear classifier you can retrain in ninety seconds keeps the operational machinery visible, and none of the seven pillars care what is inside θ. When the NER work arrives we will reach for something heavier, and the pipeline around it should not have to change.
What the pipeline has to contain
The pipeline, end to end. Enclosures mark the places where there has to be exactly one of something.
Every box belongs to one of the seven pillars. Here is what sits inside them, in one pass, before the later parts take them apart properly.
Data versioning. Immutable, content-hashed snapshots so a run can name the data that produced it, and a schema contract validated at the ingestion boundary so a changed column is caught there rather than inside the model.
Feature management. Where our problem actually lives, so it earns the most space:
- Clean and normalise. Encoding repair, whitespace, stripping headers and boilerplate, and language detection so the pipeline knows what it is holding.
- NER. Finding the people, organisations, locations and identifiers. In practice a hybrid, because the two halves of the problem are nothing alike: regex plus validators for structurally-shaped things (emails, IBANs, national ID numbers, many of which carry checksums you can actually verify), and a statistical model for names, which have no shape at all. spaCy for a fast baseline, a fine-tuned transformer token classifier when precision matters more than throughput.
- Entity resolution. Deciding that J. Smith, John Smith and Smith, J. are one person. Comparing every pair is quadratic, so it runs in stages: blocking to cut the candidate space, then scoring with string distance (Jaro-Winkler, Levenshtein) and embedding similarity, then clustering whatever survives. Splink and Dedupe are the usual libraries.
- Entity linking. Connecting a resolved mention to a record we already hold, with a deliberate path for when there is no match. Alias tables and approximate nearest-neighbour retrieval propose candidates; a ranker picks between them. This one carries legal weight, because an identifiable individual is a different proposition from an anonymous mention. There is a whole post on this.
- Vectorise. TF-IDF as the baseline, sentence embeddings when meaning matters, and the entity counts riding alongside as structured features.
All of that is one implementation writing to two stores: history for training, current values for serving. The moment the stream consumer acquires its own slightly different copy, the backlog and the stream begin disagreeing about the same document.
Experiment tracking. Parameters, metrics, artifacts and context, meaning the commit and the data hash, recorded automatically per run and broken out by slice rather than only in aggregate.
Model evaluation. A trivial baseline to beat first, then precision and recall at the operating point, the curve across all thresholds, and results per slice.
Model registry. Versions, aliases such as champion and challenger, transitions recorded as events with an actor, and a pointer from each version back to the run that produced it.
Serving. Three shapes, and our system needs all of them: batch for the backlog, streaming for the live feed, online for the interactive check.
Monitoring. Four layers, from infrastructure health up to model quality, plus drift tests on the inputs and on the model’s own outputs.
That is the map. Parts 2 to 4 take these one at a time and explain why each is harder than it looks.
That is a great deal of machinery for a binary classifier. But notice what is missing from all of it: nothing in that map carries a brand name. The architecture came out of the problem rather than a product page, which is the only order that produces one you can defend. Now it is sensible to ask what runs each box.
Now the stack
With the boxes settled, choosing tools becomes a much smaller question: what runs each one?
Pipeline object so the transformation and the classifier are one artifact rather than two things that can be versioned apart. spaCy or a transformer handles NER alongside it.
MLflow
Owns training and the registry. Parameters, metrics, artifacts and the fitted pipeline for every run, plus one auditable pointer to whichever version is currently answering.
FastAPI
The online API. It loads the model named by the registry, reads features from the online store, and answers per document. Batch scoring is the same model invoked from a Prefect flow instead of a request.
Evidently
Reads the prediction log and produces monitoring. Drift and data quality reports on a schedule, stored beside the run that produced them.
Two boxes are deliberately left without a product, and it is worth saying why rather than quietly leaving them off the list.
The feature store is a pattern before it is a purchase. Two ordinary stores will do: Parquet in object storage for the offline layer, Redis or Postgres for the online one. What makes it a feature store is not the software, it is the rule that both are written by the same transformation code. Feast is worth reaching for when the number of features outgrows your ability to remember them, and not a moment sooner.
Data versioning has no tool yet, and that is a choice rather than an omission. For a fixed 500,000-document backlog, an immutable content-hashed snapshot in object storage does the job, and logging that hash as a run parameter is enough to tie a model back to its data. Reach for DVC or lakeFS when the data starts changing faster than you can name the snapshots, not before.
Model evaluation is not a product. It is your test suite. No tool knows which slices of documents a regulator cares about, and buying one instead of writing those tests is precisely how a rising headline number ends up concealing the regression that matters.
📉 What to do when ground truth arrives months late
Our labels are slow. A reviewer confirms a document weeks after it was scored, and a regulator’s finding can be slower still. That is awkward, because most monitoring compares predictions against truth, and we will not have truth for a while.
Two things help.
Estimate performance without labels. Tools like NannyML estimate what your metrics probably are, using the model’s confidence distribution and the shift in inputs, well before the labels land. The estimate is not free of assumptions, and notably it assumes the labelling rule has not changed, which is exactly the case that breaks it. Useful, with its limits understood.
Sample deliberately. Send a small random slice of documents for review regardless of what the model said, including the ones it cleared confidently. It is the only way to measure the errors that never generate a complaint, and it costs real reviewer time. Budget for it rather than discovering the need for it.
The takeaway
A model is a file of numbers and the code that reads them. Everything difficult about shipping one comes from a single fact: its behaviour lives in data, and data does not hold still.
That gives us the spine for the rest of the series.
- The training and serving workloads have almost nothing in common, and owe each other exactly one thing: the same input, the same prediction, whichever side it arrives on.
- Seven pillars exist to keep that promise, one per stage of the loop, each catching one specific failure: Data versioning, Feature management, Experiment tracking, Model evaluation, Model registry, Serving, Monitoring.
- Our system is a personal-data classifier for a compliance agency, fed by a 500,000-document backlog, a JSON article feed, and a live stream.
- Our stack is Prefect, MLflow and Evidently.
In Part 2 we take the first four pillars one at a time and actually build them.