MLOps Part 2: Building the Model

July 25, 2026 · 18 min read

Part 1 ended with a pipeline and a promise: seven pillars, taken one at a time. This is that.

A quick reminder of the shape, because everything below hangs off it. Seven stages in the loop, and one practice per stage that keeps that stage honest:

The MLOps loop. One pillar per stage. This post works down it in order.

This post covers the first four, the ones that get you a model worth shipping. Part 3 takes the three that put it in front of users, along with two concerns that do not fit the loop at all:

Both get a section of their own at the end.

Data versioning

Two jobs live here, and they are usually confused. One is knowing which data trained a model. The other is knowing whether the data arriving today is the same shape as the data you agreed to receive.

The first is a snapshot problem and it is largely solved by discipline: an immutable, content-hashed copy of the training set, with the hash recorded on the run that consumed it. If the hash is in the run, the run is rebuildable. If it is not, the run is a story about a number.

The second is where the interesting failure lives, because upstream teams change things without malice and without telling you.

Data contracts

A data contract is an explicit, versioned agreement about what a producer will send: which fields exist, what types they are, what ranges are legal, what may be null, and what the producer promises not to change without a version bump. It is the schema, plus the part that says who is accountable when it breaks.

For our article feed, the contract would pin things like: document_id is a non-empty string and unique within a batch; published_at is UTC and never in the future; body_text is present and at least fifty characters; source_system is one of a fixed set.

Plenty of tools express this. Pydantic for record-level validation in Python, Pandera for dataframe-level checks, Great Expectations if you want the suite-and-report style, and for genuinely semantic constraints, SHACL over an ontology, which is a longer story of its own.

Written out, the contract is unremarkable, which is rather the point:

import pandas as pd
import pandera.pandas as pa
from pandera.typing import Series


class IncomingDocument(pa.DataFrameModel):
    """Anything entering the pipeline, whichever of the three paths it arrived on."""

    document_id: Series[str] = pa.Field(unique=True, str_length={"min_value": 1})
    source_system: Series[str] = pa.Field(isin=["backlog", "article_api", "stream"])
    published_at: Series[pd.DatetimeTZDtype] = pa.Field(dtype_kwargs={"unit": "ns", "tz": "UTC"})
    body_text: Series[str] = pa.Field(str_length={"min_value": 50})

    class Config:
        strict = True    # an unexpected column is a breach, not a bonus
        coerce = False   # never silently cast: a changed type is exactly what we are here to catch

    @pa.check("published_at")
    def not_in_the_future(cls, s: Series) -> Series[bool]:
        return s <= pd.Timestamp.now(tz="UTC")

Two lines in that Config are doing most of the work. coerce = False is the important one: coercion is a validator quietly repairing the exact signal you wanted to see.

And the contract only means anything at the point it is enforced:

def ingest(batch: pd.DataFrame) -> pd.DataFrame:
    try:
        return IncomingDocument.validate(batch, lazy=True)  # collect every failure, not the first
    except pa.errors.SchemaErrors as err:
        quarantine(batch, reason=err.failure_cases)         # decided in advance, not mid-incident
        raise

The genuinely hard part of data contracts is not technical. It is that the producer has to accept the constraint, and the producer usually reports to somebody else. A contract nobody upstream agreed to is a wish with a schema attached.

Feature management

Part 1 called this the structural fix for the contract between training and serving. Time to say plainly what that means, because “training-serving skew” is one of those phrases that gets repeated far more often than it gets explained.

Training-serving skew, in plain language

The whole idea in one sentence: you compute a number one way when you train, and a slightly different way when you serve, so the model is handed something it was never taught to read.

An example with no mathematics in it. Say one of our features is how many documents this sender has sent before.

When you train, you write a query over the warehouse. It counts every document that sender ever sent. Straightforward, because everything is sitting there in one table.

When you serve, you are handling one document, right now, and the warehouse is six hours behind. So the serving code counts from somewhere else: a cache, a live table, a different service. It returns 41 where the training query would have said 44.

Nobody made a mistake. Two reasonable engineers wrote two reasonable pieces of code. But the model learned what 44 means, and it is being shown 41, and it will be shown a number like that on every single request from now on.

What makes this so hard to catch is that nothing goes wrong. No exception. The type is right, the value is plausible, the model returns a confident answer. You find out when somebody eventually compares the offline numbers against what actually happened, which is usually months later and usually by accident.

A feature store fixes it structurally rather than by vigilance: one definition, written to both stores. There is then only one piece of code that can be wrong, and if it is wrong it is wrong identically on both sides. Wrong identically is a bug you can find. Wrong differently on each side is a discrepancy you cannot.

In practice that discipline is less exotic than the phrase “feature store” suggests. It starts as one module that nothing is allowed to bypass:

# features.py
# The only place a document becomes numbers. Imported by the batch flow,
# the stream consumer, and the API. No path is permitted its own copy.

@dataclass(frozen=True)
class DocumentFeatures:
    clean_text: str
    n_person_entities: int
    n_org_entities: int
    has_identifier: bool
    linked_to_known_person: bool


def build_features(doc: RawDocument, linker: EntityLinker) -> DocumentFeatures:
    clean = normalise(doc.body_text)
    entities = linker.resolve(ner(clean))
    return DocumentFeatures(
        clean_text=clean,
        n_person_entities=sum(e.label == "PERSON" for e in entities),
        n_org_entities=sum(e.label == "ORG" for e in entities),
        has_identifier=bool(IDENTIFIER_RE.search(clean)),
        # a named individual we already hold a record for is a different legal
        # proposition from an anonymous mention, so the model gets told which it is
        linked_to_known_person=any(e.kg_id and e.label == "PERSON" for e in entities),
    )

Then three call sites, and nothing else:

features = [build_features(d, linker) for d in backlog_partition]   # batch
features = build_features(event.to_document(), linker)              # stream
features = build_features(request.to_document(), linker)            # api

That is the entire trick. Not a product, a rule: three paths, one import, no second implementation to drift away from the first.

The two storage layers

Training and serving want opposite things from storage, so a feature store keeps two:

The failure that looks like success

Point-in-time correctness has a nastier sibling, and it announces itself in the most misleading way possible: your offline score suddenly gets much better.

Say we add a feature, documents from this customer flagged in the last 90 days, and PR AUC jumps from 0.72 to 0.94. That is not a result. That is a symptom, and the diagnosis is almost always leakage, in one of two flavours:

Rather than memorising a list of leaky features, carry the question that generates the list:

At the moment of prediction, in production, would this value be knowable? And is it made of the thing I am trying to predict?

Four checks settle it quickly. Ablate the feature and retrain: if 0.94 collapses back to 0.72, one aggregate is carrying nearly all the signal, which is not plausible. Recompute a few rows by hand as of their own timestamp and diff. Split chronologically rather than randomly, because leakage that survives a random split usually dies against a time-ordered one. And the decisive one: try to compute the feature at inference time, with only what exists at that instant. Frequently you simply cannot, and that ends the discussion.

When you do not need one

Feature stores are marketed hard and needed less often than that would suggest. The honest test is two questions: do two different code paths need the same values, and do those values depend on history? Take either away and the answer changes.

Three cases where the answer is no.

When features come from the document alone. Chunking, tokenising, embedding: each depends on nothing but the input in front of you. There is no state to keep in sync, so a shared library that both paths import does the whole job. Versioning the chunker or the embedding model is certainly necessary, but it is a code versioning problem, not a storage one. Much of our pipeline sits here.

When you only serve one way. If everything is batch, there is no online path to disagree with, and the two-store split is solving a problem you do not have.

When you can hold the feature list in your head. A feature store’s quieter job is discovery: letting someone find the feature a colleague already built. With twelve features and three engineers, that is not a problem worth buying software for.

Experiment tracking

Model development is a search problem. You are not building one thing; you are exploring a space of learning rates, feature sets, architectures and loss functions, and most of what you try will be worse than what you already have. That is fine. It is the shape of the work.

What is not fine is being unable to say which point in that space you are currently standing on.

Search needs a log, and the log has to record four different kinds of thing:

That fourth row deserves the emphasis. Parameters and metrics tell you what happened. Context is what lets you rebuild it. A run logged without its commit hash and data snapshot is an anecdote about a number.

mlflow.set_experiment("personal-data-classifier")

with mlflow.start_run(run_name="tfidf-logreg-baseline"):
    # context. MLflow records the git commit itself; the data hash is on us
    mlflow.log_param("data_snapshot", snapshot_hash)

    # the vectoriser and the classifier are logged as ONE artifact, so the
    # fitted vocabulary can never be versioned apart from the weights
    model = Pipeline([
        ("tfidf", TfidfVectorizer(min_df=5, ngram_range=(1, 2))),
        ("clf", LogisticRegression(class_weight="balanced", max_iter=1000)),
    ])
    model.fit(X_train, y_train)

    scores = model.predict_proba(X_val)[:, 1]
    mlflow.log_metric("pr_auc", average_precision_score(y_val, scores))

    # never only the aggregate: the average is where a slice regression hides
    for name, idx in slices.items():
        mlflow.log_metric(f"pr_auc__{name}", average_precision_score(y_val[idx], scores[idx]))

    mlflow.sklearn.log_model(sk_model=model, name="model")

Two details in there matter more than the API. The vectoriser and the classifier are logged as a single Pipeline, so the fitted vocabulary and the weights cannot drift apart: that is the unowned state problem from Part 1 being closed off. And the metrics loop logs every slice, because a run that only recorded its headline number cannot answer the question you will actually be asked later.

Model evaluation

Someone says accuracy went up five percent. Almost nothing useful follows from that sentence.

Accuracy hides class imbalance. If three percent of documents contain personal data, a model that says “no” to everything scores 97%. It is also completely worthless, and no amount of accuracy will reveal that.

That comparison is worth turning into a habit rather than an anecdote: before celebrating any number, work out what the stupidest possible model scores. Majority class for classification, “same as yesterday” for forecasting, keyword match for text. If the real model does not beat that by a wide margin, you do not have a model, you have a coincidence with a training loop attached.

Accuracy hides slices. The headline can improve while a specific group gets worse. One customer sends unusually long documents, the new model regresses badly on those, and the average absorbs it without complaint. That customer does not experience the average. They experience their own documents.

So the first move is to stop looking at one number and start looking at the two that carry the actual cost:

These are not remotely equal, which is why any metric weighing precision and recall equally is answering a question nobody asked.

The threshold is the actual decision

Underneath every classification is a score and a cut-off. The model emits a number; something decides whether that number counts as a yes. Move the cut-off and you move both metrics in opposite directions, every time:

Precision and recall against threshold. No setting improves both. Only a choice about which error you would rather make.

Plotting precision against recall across every threshold gives you the precision-recall curve, and the area under it, PR AUC, summarises how well the model orders documents independently of where you cut.

Two things PR AUC does not do, worth being clear about. It does not choose your operating point, because that is a business decision about relative cost. And it is still an offline number, computed on historical data that the world has since moved past. Which is Part 4’s problem.

When the threshold cannot help

Sooner or later both complaints arrive in the same week. Compliance says too much is slipping through. The analysts say they are drowning in false alarms.

Those are the same lever pulled in opposite directions, and it is worth saying so plainly rather than promising to look into it. Moving the threshold reallocates pain. It does not create value, and no setting satisfies both.

The distinction that matters:

You can slide along the curve, or you can move the curve. Only one of those makes both numbers better.

Thresholds, sampling and cost weighting slide along it. Better features, more labels on the hard cases near the boundary, a different architecture, and abstention move it.

Abstention is the one to reach for first, and it is unreasonably underused. Rather than forcing a binary answer, let the model return three: confident yes, confident no, and a middle band routed to a human. Analysts stop seeing cases the model was never sure about, compliance gets deliberate coverage on the ambiguous ones instead of a coin flip, and nothing about the model has changed. You have only stopped making it answer questions it cannot answer.

Two checks before any of that. Make both complaints numeric: how many missed documents a month is acceptable, and how many false alarms per analyst per day before they stop reading the queue? Two feelings cannot be traded against each other. Then check they concern the same population. If the misses concentrate in scanned PDFs and the false alarms in HR templates, those are two problems, and a single global threshold was never the right control for either.

Threshold as configurable policy

Here is the move that follows from all of the above, and it is a genuinely good one: the threshold does not belong in the model.

A bank sending legal disclosures and a marketing team sending campaign copy have different tolerances for the same mistake. The obvious response is to train two models. The better response is to train one model and give each customer their own cut-off, because the model’s job is to rank documents by risk and the threshold’s job is to encode how much risk this particular customer will accept.

One model, many operating points, no retraining. It is versioned configuration rather than a constant compiled into an artifact.

The takeaway

Four pillars, and one theme running under all of them: the useful version of each is the one that removes a choice.

A contract nobody validates at a boundary is a schema file. A feature definition that exists twice is not a definition. A tracked run missing its data hash is an anecdote about a number.

At this point we have a model we can rebuild, explain, and defend. It is still sitting on a laptop. Part 3 puts it in front of people.