MLOps Part 3: Shipping and Running It

July 25, 2026 · 20 min read

Part 2 got us a model we can rebuild, explain and defend. It is still a file.

This post covers the two pillars that put it in front of people, Model registry and Serving, plus the two concerns that cut across the whole loop rather than sitting at one point on it:

Orchestration The thing that runs the stages, retries them, and decides when to retrain. Not a stage; the machinery that makes the stages happen at all. Governance Audit trail, access control, and who approved a version going live. In a compliance setting it is a functional requirement rather than paperwork bolted on at the end.

The seventh pillar, Monitoring, gets Part 4 to itself, because it gives the least warning and costs the most to get wrong.

Model registry

Prevents Nobody being certain which version is answering requests, and no quick way back when it is the wrong one. In our stack The MLflow registry with champion and challenger aliases, and a serving layer that can load a model only by asking it.

Experiment tracking records everything you have tried. It says nothing whatsoever about what is answering requests right now.

That is the registry’s job, and the distinction is worth keeping sharp:

Tracking is the lab notebook. The registry is the shipping manifest.

A registry holds the model name and version, an alias that says what each version is for (production, staging, champion, challenger, or a per-tier alias if different customers get different models), and a record of every transition as an event: version 3 was promoted by this person, at this time, with this approval. From that you get traceability, one-step rollback, and an answer to “what was live when this happened?”

🚪

The registry only counts if it is the only door If an engineer can copy a model file onto a server and skip the registry, then the registry is documentation rather than a control. The serving layer should be able to load a model only by asking the registry which version to load. Anything else is a strong suggestion.

Two details about what the registry is actually pointing at, both of which are usually treated as hygiene and are really security.

A pickled model is a supply-chain vulnerability. pickle.load() executes arbitrary code during deserialisation. That is not a quirk, it is how the format works. So a pickle pulled from shared storage is remote code execution waiting for someone to write to that bucket, and “we only load our own models” is a statement about intent rather than about permissions. Prefer safetensors or ONNX, which deserialise data rather than executing it, and sign the artifacts so serving can verify what it loaded.

Pin digests, not tags. A container tag is a mutable pointer, closer to a branch than to a commit. mlops-serving:v2.1 can quietly become a different image tomorrow, and then your “reproducible” rollback restores something that is not what was running. Pin by digest, which is a content hash and therefore an actual identity.

↩️

Rollback is not symmetric The container goes back in thirty seconds. The decisions do not. Six weeks of a bad model has already written verdicts into customer records, cleared documents that should have been flagged, and shaped a reviewer queue that in turn became training labels. Reverting the artifact stops the bleeding; it does not undo anything.
This is the part that makes ML rollback different from software rollback, and it is why the gates before a release matter more here than they do for a web service. You are not protecting uptime. You are protecting a record you cannot rewrite.

Why a better offline number is not a reason to deploy

Version two scores 0.3% higher than version one. Ship it?

No. Four reasons to be suspicious before anyone touches a deploy button.

Is it better on average, or where it matters? The mean is a poor summary of a distribution that has customers living in its tail. A model that is slightly better overall and clearly worse on long documents is a regression for whoever sends long documents, and they will not be comforted by the average.

Do you know your run-to-run variance? This is the one most teams skip. Change nothing but the random seed and rerun: if the result moves by ±0.8%, then a 0.3% delta is indistinguishable from having changed nothing at all. You have not measured an improvement, you have measured your own noise floor. Establish that floor before you believe any delta, and be aware that if you also picked this model as the best of fifty experiments, the winner’s margin is biased upward simply by having been selected.

What did it cost? A 0.3% gain that comes with 10% higher latency, a heavier architecture, and GPUs pinned at full load is not obviously a gain. Somebody pays for that, monthly, forever.

Offline is not online. Everything above was measured on historical data. The live distribution has moved, and the only honest position is that you do not yet know how version two behaves on real traffic.

Finding out for real: shadow, canary, A/B

Three techniques, and they are not alternatives. They answer different questions and are usually used in sequence.

	What it does	What it tells you	What it costs
Shadow	Both models see identical live traffic. Only the old one’s answers are used; the new one’s are logged.	Whether it works: latency, errors, resource use, and exactly which documents the two disagree about.	Double inference. No user risk at all.
Canary	A small slice of real traffic goes to the new model. 1%, then 5%, then 10%, often region by region.	Whether it survives contact with reality, with the damage bounded.	Real exposure, deliberately capped.
A/B test	Traffic split deliberately, with a hypothesis and a stopping rule agreed in advance.	Whether the difference is real, with a number attached.	Time, traffic, and statistical discipline.

Or compressed: shadow removes the risk, canary limits the blast radius, A/B produces the evidence.

Each has a way of being done badly. Shadow says almost nothing about quality, since without labels you cannot know who was right when the two disagreed; what it is superb at is catching skew, so read the disagreements rather than counting them. Canary is only a canary if the rollback trigger exists before you start, with named metrics and automatic reversion, otherwise it is a slow deploy with extra steps. A/B needs a power calculation up front, because with a 0.3% effect and modest traffic the honest answer to “how long until significance” is often “longer than anyone will wait”, and that is worth knowing on day one rather than week three.

⚖️

A/B testing is awkward in a compliance setting An A/B test deliberately routes some real documents to a model you believe may be worse. For a recommendation engine that is a fine trade. For a classifier whose failure mode is a privacy breach, knowingly assigning some people to the worse arm is a decision with an ethical dimension, not just a statistical one. Shadow, which risks nothing, does most of the work here. If you do run a live split, the safe direction is one where the challenger can only be *more* cautious than the champion.

The feedback you get about a release is biased

One caveat on all three techniques. Whatever you ship, you learn about it from reviewers, and reviewers only see what the model flagged. So every correction you collect is a false positive being fixed, while the cleared documents that should not have been are invisible by construction.

That skews your read on any release, and the fix costs real reviewer time. Part 4 deals with it properly.

Serving

Prevents Correct answers that arrive too late, or cost more than the decision is worth. In our stack FastAPI for the online path, Prefect-driven batch scoring for the backlog, and a stream consumer for live posts.

The new pillar. Three shapes, and the boundary between two of them is thinner than it looks.

	Batch	Streaming	Online
Trigger	A schedule	An event arrives	A caller asks
Is anyone blocked?	No	No	Yes
Latency budget	Hours	Seconds	Milliseconds
Under overload	Runs longer	Queue grows, you catch up later	Requests fail
Hardware	Rent it for the window, cheap, preemptible	Modest, always on	Always on, provisioned for peak
Delivery	Rerun the job	At-least-once, so be idempotent	At-most-once, caller retries

Streaming and online look alike, because both are fast. The latency numbers are the symptom; the actual difference is whether anything is blocked waiting.

An online request is synchronous. Someone is holding the line, and if you take too long they get an error rather than a slow answer. A stream event is asynchronous. Nothing is waiting on the result, so if you fall behind, the queue absorbs it and you catch up. That single property changes everything downstream: a stream can be buffered, replayed, and reprocessed, so it can survive a traffic spike that would take an online API down. In exchange, at-least-once delivery means you will see the same event twice and your processing had better be idempotent.

By that measure batch and streaming are siblings, both asynchronous, differing mainly in granularity. Online is the odd one out.

Our system runs all three: the backlog is batch, the live posts are streaming, and the interactive check when someone uploads a document is online. Same model, same features, three sets of operational constraints.

When “it feels slow” and the dashboard disagrees

Before the theory, the argument you will actually have. Average latency reads 90ms and customers say the product is slow. Both are true, and there are usually four reasons.

The mean describes a customer who does not exist. Ninety percent of documents are short and take 20ms; ten percent are long and take 700ms. The mean is 88ms and every person complaining is in that ten percent. Quote p95 and p99, never the mean.

Queueing delay, which is usually the big one. Your handler starts its clock when it picks the request up, and reports 90ms perfectly honestly. The request spent three seconds in a queue before that, and the handler has no idea. Worse, queue wait grows non-linearly as utilisation rises: going from 80% to 95% busy can multiply the wait tenfold. Systems feel fine, then fall off a cliff, which is why you never run inference infrastructure near full utilisation.

The measurement boundary. We measure server-side. They experience a proxy, the network, TLS, and their own rendering. “Where does the clock start and stop” is a real question, not pedantry.

Cold starts. Autoscaling brings up a new instance, the load balancer starts routing to it, and it spends the next ninety seconds loading a two-gigabyte model. Every request in that window is catastrophic and none of them are the model’s fault.

Notice that “the model got slower” is nowhere on that list. It is worth ruling out the other four first.

Latency and throughput

Latency is how long one request takes. Throughput is how many you get through per unit time. They pull against each other, and the mechanism is batching.

Processing 32 documents together takes longer in wall-clock terms than processing one. But the fixed costs, the kernel launches, the memory transfers, the framework overhead, are paid once instead of 32 times, so the per-document cost drops sharply. Throughput goes up. The individual document waited longer, partly to be computed and partly just to sit in the queue while the batch filled. Latency got worse.

📈

They are not inversely proportional It is tempting to write latency ∝ 1/throughput, but that is not the shape of it, and the real shape matters for capacity planning. The governing relationship is Little's Law: concurrency = throughput × latency. Rearranged, latency is concurrency divided by throughput, which means that as long as you have spare capacity you can raise throughput with almost no latency cost at all. The two only genuinely fight once you approach saturation, and then latency does not degrade gracefully. It goes vertical.

Latency against throughput. Capacity planning targets about two thirds of maximum, because the last third costs all of your latency.

One correction on hardware, because the intuition is a common one. Adding GPU memory buys you bigger batches and more concurrent model replicas, which is throughput. It does not make a single request faster. Single-request latency is bound by compute and memory bandwidth, not by capacity. If latency is the problem, the levers are a smaller or quantized model, better kernels, caching, or simply a shorter input, not a bigger card. Scaling hardware works, but it buys linear improvement for linear cost forever, where algorithmic changes are step functions. And no amount of hardware rescues a bad batching policy.

Orchestration

Every pillar so far describes a thing that must happen. Orchestration is what makes it happen, in the right order, on time, and without a person watching.

The instinct is to reach for cron, and cron holds until roughly week two, when three things become true at once. Steps depend on each other, and “ingestion finished” is not the same as “the clock says 3am”. Steps fail for boring reasons: the article API times out, a spot instance is reclaimed, object storage returns a 503. Not bugs, just Tuesday, and they need retries with backoff rather than a silent morning. And history needs reprocessing, because you changed the chunker and now a specific date range of the backlog has to run again. That last one is a backfill, and doing it by hand is how people lose weekends.

An orchestrator knows about all of that. Two of its concepts are worth naming because the code below will not show them:

Idempotency Running a task twice must leave the world as it was after running it once. Without it every retry risks double-counting, and at-least-once stream delivery guarantees you will retry. Backfill Re-running the pipeline over a historical window, which is a parameter rather than a code change if the flow was written for it.

🧱

The orchestrator schedules work. It should not contain the work. This is the mistake that costs the most later. If your feature engineering lives inside the body of a Prefect task, then FastAPI cannot call it, so somebody reimplements it for the serving path, and you have rebuilt training-serving skew with extra steps. Put the logic in an ordinary importable library. Let tasks be thin wrappers that call it. Then the same function serves the batch flow, the stream consumer and the API, which is the entire point of the feature management pillar.

Which looks like this. Note how little the flow actually knows:

from datetime import timedelta
from prefect import flow, task
from prefect.cache_policies import INPUTS

from features import build_features   # the shared library. The flow imports it like anyone else.


@task(retries=3, retry_delay_seconds=[10, 60, 300])
def pull_articles(since: datetime, until: datetime) -> pd.DataFrame:
    """The API times out sometimes. That is Tuesday, not an incident."""
    return article_api.fetch(since=since, until=until)


@task(cache_policy=INPUTS, cache_expiration=timedelta(days=7))
def featurise(docs: pd.DataFrame) -> pd.DataFrame:
    # a thin wrapper. Nothing in here that FastAPI could not also call.
    return pd.DataFrame(build_features(d, linker) for d in docs.itertuples())


@flow(name="ingest-and-featurise")
def ingest_and_featurise(since: datetime, until: datetime) -> None:
    raw = pull_articles(since, until)
    validated = ingest(raw)          # the Pandera contract from Part 2
    features = featurise(validated)
    write_offline(features)          # full history, for training
    write_online(features)           # current values, for serving

Prefect’s own concepts documentation covers the rest of the vocabulary; three things matter here. Retries are declared rather than coded. The featurise task caches on its inputs, so a rerun after a downstream failure skips the work it already did. And because the flow takes a date range as a parameter, reprocessing last March is ingest_and_featurise(since=march_1, until=march_31) rather than a branch.

Continuous training

The MLOps-specific job of an orchestrator is deciding when to build a new model, and the interesting half of that is deciding when not to.

The continuous training decision chain. Two gates, and both mostly say no. Without them, automation just ships worse models faster.

The admission gate is the underrated one. A drift alarm should not start a training run on its own, for the reason from the monitoring section: most drift alarms are an upstream bug rather than a changed world. And even a legitimate trigger should respect a cooldown and a minimum quantity of new labels, or you retrain nightly on forty new examples and mistake the resulting churn for progress.

The promotion gate is everything from the model registry section, made automatic: does the improvement clear your measured run-to-run variance, has cost or latency regressed, has any slice gone backwards. A pipeline that trains automatically and promotes automatically without these is not mature, it is unsupervised.

🪜

Three levels, and most teams are on the first A useful way to locate yourself. Level 0: a person runs a notebook and hands over a model file. Level 1: the training pipeline is automated, so retraining is a triggered flow rather than an afternoon. Level 2: the pipeline itself is under CI/CD, so a change to feature code is tested and deployed like any other software change.
The jump from 0 to 1 is where nearly all the value is, and it is mostly about deleting manual steps rather than adding tools. Level 2 matters once more than one person is changing the pipeline.

Governance

Governance is the pillar people assume is paperwork. In a system that decides whether documents contain personal data, it is closer to the product.

Strip away the compliance vocabulary and it asks two questions. Can you reconstruct, later, why the system did what it did? And can the things that should not happen actually be prevented, rather than merely discouraged?

One distinction does most of the work here, and it is the one people collapse:

Versioning answers "can I retrieve v3?". Lineage answers "what produced v3, and who signed off?" An auditor is never asking the first question.

A registry with version numbers and no pointers back to the run, the data snapshot and the approval is a folder with good filenames. Everything below is about keeping that chain unbroken.

Reconstructing a decision

Someone asks why a specific document was cleared for release on a specific Tuesday. Answering that means holding, for every prediction:

The model version Which registered version answered, not which one you believe was live at the time. The threshold Which cut-off was in force for that customer at that moment. If thresholds are per-customer policy, they are part of the decision and belong in the record. The inputs The feature values as they were, not as they are now. Recomputing them today gives you a different answer and an incorrect audit. The lineage behind it From that model version, back to the training run, the data snapshot, and the code commit. Each link exists because of an earlier pillar; governance is what makes the chain queryable end to end.

Notice that governance adds almost no new machinery. It is the payoff for the prediction log, the registry and the data snapshots already being in place. What it adds is the requirement that the chain is unbroken, because a lineage with one missing link answers no questions at all.

Preventing rather than discouraging

The second half is duller and more often skipped.

Separation of duties. The person who trains a model should not be the only person who can promote it. This is not about distrust; it is that a second pair of eyes catches the run where the evaluation set was accidentally included in training.

Promotion as a recorded event. Not “the model was updated” but “version 7 was promoted by this named person at this time, with this evaluation report attached”. The registry already stores transitions; governance is the rule that says every transition must have an actor and a reason.

Access control on the queue. The reviewer queue is full of documents flagged as containing personal data, which makes it one of the most sensitive datasets you own.

A real deletion story, not a tombstone. Worth checking early, because it is easy to discover late. Several vector indexes cannot truly delete: HNSW marks a vector as removed and leaves it in the graph, so tombstones accumulate, recall degrades, and the index has to be periodically rebuilt. That is an acceptable performance trade and an unacceptable compliance one. A right-to-erasure request is not satisfied by a record flagged as ignored, and in a system whose entire purpose is handling personal data, “we soft-deleted it” is the wrong sentence to be saying to a regulator. Know whether your storage layer can genuinely remove a record before someone asks you to.

🔁

The uncomfortable recursion A classifier that detects personal data is itself a system that processes personal data, at volume, and stores its findings. The prediction log is a searchable index of exactly which documents contain sensitive information, which is a genuinely attractive thing to steal.
So the same discipline applies inward. Set a retention limit on the log rather than keeping it forever. Store a document hash and the feature vector rather than the full text where you can. Restrict who can query it, and log the queries. It is easy to build a compliance tool that is itself the largest compliance liability in the building.

Documentation that is actually used

A model card travels with each registered version: what it is for, what it trained on, how it performs by slice, and where it is known to be weak. The ones that get read are honest about limitations, and the ones that stay accurate are generated from the run rather than written by hand. If the metrics, data snapshot and slice breakdown are already logged, the card is a rendering of things you have rather than a document somebody must remember to update. Google’s model cards paper is the original and still the clearest.

⚖️

The regulatory frame, briefly and without pretending to be legal advice Two things tend to come up for a system like ours. GDPR gives people rights around decisions made about them by automated means, which in practice pushes towards being able to explain and reproduce an individual decision, and towards keeping a human in the loop for consequential ones. The EU AI Act sorts systems into risk tiers, with documentation, logging and human-oversight obligations attached to the higher ones.
Whether either applies to a given deployment is a question for someone qualified to answer it. The engineering point stands regardless: the capabilities they ask for, reproducibility, per-decision lineage, slice-level evaluation, recorded human approval, are the same capabilities the previous seven pillars already produce. Governance is mostly the discipline of not letting any of those links break.

The takeaway

Two pillars and two cross-cutting concerns, and one idea running under all of them.

Pillar	The one sentence version
Model registry	One auditable answer to what is live, and a controlled path for changing it.
Serving	Batch, streaming and online are different problems; latency and throughput only fight near saturation.
Orchestration	Runs the stages, and gates the retrain so that automation does not just ship worse models faster.
Governance	Makes the chain from a prediction back to its data queryable, and makes approval a recorded event.

Every pillar here has a version that looks right in a diagram and does nothing in production, and the difference is always the same: whether the thing sits on the path or beside it. A registry you can bypass by copying a model onto a server is documentation, not a control.

In Part 4 we go back to the layer that gives the least warning and matters the most: what to do when a drift alarm actually fires, how to measure quality while ground truth is still weeks away, and the one kind of change that none of our monitoring will catch.