The entity linking flywheel

Draft

July 19, 2026 · 5 min read

Most knowledge-graph writing is about querying, the elegant traversals you can run once the graph exists. The unglamorous truth is that in enterprise settings the graph mostly doesn’t exist yet. It has to be extracted, mention by mention, from millions of messy documents, and the extraction pipeline is itself a full ML system with its own failure modes, thresholds, and economics.

That pipeline decomposes into three problems, and keeping them separate is half the battle:

NER: find mention spans and coarse types: “Apple”, ORG, characters 12–17.
Entity linking (disambiguation): decide which real-world entity: Apple-the-company or apple-the-fruit → a KB identity.
Relation extraction: how linked entities connect: acquired, supplies-to, governed-by → the edges.

NER spots names in a crowd. Linking checks their ID against your registry. Relation extraction notes who shook hands with whom. Blur them together and you can’t tell which stage is losing you accuracy.

Choosing an NER approach: the ladder

Approach	Where it wins	Where it breaks
Rules & gazetteers	Closed vocabularies, product codes, ISO refs, internal IDs. Still the precision backbone of most enterprise stacks.	Paraphrase, unseen surface forms, anything open-ended.
Fine-tuned encoder (BIO tagging)	The production workhorse: accurate, fast, cheap at corpus scale.	Needs labelled data; degrades silently under domain shift.
LLM zero/few-shot	New entity types with zero training data; the long tail; bootstrapping labels.	Per-token cost at millions of documents; outputs need validation.

The pattern that works is not picking one rung but composing them: the LLM as teacher, the small model as workhorse, use few-shot LLM extraction to generate silver labels for new entity types, human-verify a sample, train the encoder on the result, and keep rules for the high-precision patterns you already trust.

One warning from the trenches: a “92% F1” NER model is 92% on its benchmark’s domain. Point it at legal or technical enterprise text and it can shed twenty points without a single error message. Build a gold set from your actual target documents before any model work, and report macro-F1 per entity type, the classes that matter to your users are usually the rare ones a micro average hides.

Disambiguation: two stages with opposite jobs

Entity linking has a canonical architecture, and its two stages optimize for opposite things, which is exactly why it works.

Stage one may not miss (recall); stage two may not guess (precision); the threshold θ decides when honesty beats a forced link.

Candidate generation answers “who could this possibly be?”, alias and synonym tables harvested from the KG, fuzzy string matching, and dense retrieval over embedded entity descriptions. Its only metric is recall@k: if the true entity isn’t in the candidate set, no downstream cleverness can recover it.

Ranking answers “which one is it actually?”, and its most interesting feature is graph coherence: entities mentioned together in a document tend to be connected in the graph. Take “Java performance tuning at Oracle”. The mention “Java” generates candidates: the programming language, the island, the coffee. Context embeddings help, but coherence settles it, the document also mentions Oracle, and Oracle and Java-the-language share a neighborhood in the graph; the island does not.

def rank(mention, candidates, doc_entities, kg):
    def score(c):
        s_ctx  = cosine(embed(mention.context), c.embedding)  # does the text fit?
        s_type = float(c.type in mention.ner_types)           # does NER agree?
        s_coh  = kg.coherence(c, doc_entities)                # are the doc's other
        return 0.55 * s_ctx + 0.15 * s_type + 0.30 * s_coh    # entities its neighbors?

    best = max(candidates, key=score, default=None)
    if best is None or score(best) < THETA:
        return NIL  # an honest "not in the KB" beats a confident wrong link
    return best

For higher precision on the shortlist, the same trick as retrieval reranking applies: bi-encoder to generate candidates cheaply, cross-encoder over mention-context × entity-description pairs to rerank the top few.

NIL is where production diverges from the benchmark

Academic entity linking mostly assumes a complete knowledge base. Production never has one, the KB is under construction; that’s the whole point. So the decision below the threshold θ is not an error path, it’s a first-class outcome: this mention refers to something we don’t know yet.

What you do with NILs is a policy, not a parameter:

Auto-create a provisional entity (provenance-stamped, flagged low-confidence) when stakes are low and volume is high.
Queue for curation when a wrong entity is expensive, compliance, master data, anything customer-facing.
Cluster before either. A genuinely new entity doesn’t appear once; it appears in a thousand documents before anyone curates it. Cross-document coreference, clustering NIL mentions by name and context embedding similarity, yields one provisional entity per cluster instead of a thousand duplicates. This is classic entity-resolution discipline applied at the graph boundary, and skipping it is how graphs silently rot.

Tuning θ is a product decision wearing a model parameter’s clothes: lower it and you force wrong links (silent corruption); raise it and you flood the curation queue (visible cost). The right setting depends on what a bad link costs your domain and how much curation capacity you actually have. That trade-off belongs in a conversation with the people doing the curating, not in a config file default.

The flywheel

Look back at the dashed arrows in the diagram, they’re the part that compounds:

Alias tables for candidate generation are harvested from the graph’s own labels, synonyms, and curated merge history.
Coherence features for ranking are literally graph queries, the denser and cleaner the graph, the sharper the signal.
Every curated decision: an approved link, a corrected NIL, a merged duplicate, flows back as training data, new aliases, and regression tests.

The system exhibits the property you want from any data asset: the graph improves its own construction. Early on, the linker leans on embeddings and string similarity because the graph is sparse. A million documents later, coherence and aliases dominate, and accuracy climbs without a single model retrain.

Measure it like a product, not a paper

Linking accuracy on a gold set is table stakes. The numbers that predict whether your graph is actually trustworthy:

NIL precision/recall, reported separately. A linker that links everything looks great on accuracy until NIL-heavy reality arrives.
Duplicate-entity rate: the KPI that catches identity-strategy failures months before users do.
Curation queue throughput and backlog age: if the queue grows without bound, your thresholds are writing checks your team can’t cash.

Model metrics tell you the pipeline is clever. These tell you the graph is sound, and it’s the graph, not the pipeline, that the business runs on. For how LLM-drafted assertions pass through validation gates before they’re allowed to become graph facts at all, see the governed write path, the two posts describe the same pipeline from opposite ends.