Ingestion is not import
Draft
Every document-intelligence project has the same optimistic sentence in its first design doc: “we’ll load the documents into the graph.” Load. One word, past tense in spirit, as if it happens once. It never happens once. You will reprocess that corpus a dozen times, a better OCR engine, a new entity type, a re-trained classifier, a schema change, and each pass has to be safe, targeted, and cheap. The projects that survive are the ones that understood this on day one.
So the mental model to hold is blunt: ingestion is not import. Import is a migration you run once and forget. Ingestion is a replayable pipeline that happens to run its first pass on day one and its hundredth pass whenever a model improves. Design for the hundredth pass and the first one is free; design for the first and the hundredth will cost you a weekend and a corrupted graph.
Here’s the reassuring part, if you come from the data-engineering or knowledge-graph world like I do: you already know this discipline. A document pipeline is ETL where the inputs are messy files at huge volume and the transforms went statistical (OCR, NER, embeddings). The engineering virtues are unchanged, idempotency, incrementality, observability, dead-letter handling. You’re not learning a new field; you’re scaling one you already have.
Dedup is free money, do it first
Enterprise corpora are 30–60% exact and near duplicates: version 3 of the same contract, the email and its four forwards, the shared drive that is a graveyard of final_FINAL_v2.docx. De-duplicating at the front door does two things at once, it cuts every downstream compute bill by that same 30–60%, and it stops retrieval from later returning five copies of one answer.
Do it at two levels. Exact dedup is a content hash; it’s nearly free and catches the byte-identical copies. Near-dup is MinHash or embedding similarity, and it catches the reformatted-but-same-meaning cases. Say “dedup” first in any pipeline design review, it’s the cheapest lever with the biggest number attached, which makes leading with it a small signal that you’ve done this before.
Extraction is the swamp
This is where optimism drowns. A real corpus has a hundred-plus formats, corrupted files, password-protected archives, scanned PDFs that are images pretending to be text, embedded spreadsheets, and thirty-year-old encodings nobody living remembers choosing. There is no single parser. There’s a fallback chain, native parser first, OCR when the text layer is empty or garbage, routed by detected file type.
The rule that separates a robust pipeline from a fragile one: always record an extraction-confidence score, and propagate it. Text pulled cleanly from a born-digital PDF and text guessed by OCR off a skewed fax are not equally trustworthy, and every downstream model, the classifier, the entity linker, the embedder, deserves to know which it’s looking at. Garbage text confidently embedded is garbage that retrieval will happily serve.
Enrich on a ladder, not a firehose
Once you have text, the temptation is to send every document to the biggest model you have. At ten million documents a day that’s not an architecture, it’s a monthly invoice. Enrichment climbs a cost-aware ladder, and each document takes the cheapest exit it can justify:
| Rung | Cost | Handles |
|---|---|---|
| Hash / metadata checks | free | dedupes, routing, format facts |
| Rules & gazetteers | ≈ free | high-precision known patterns, PII regexes |
| Small fine-tuned model | cheap | the bulk, classification, NER, embeddings |
| LLM analysis | expensive | the uncertain, the long tail, the high-stakes |
It’s the same cascade logic that governs serving cost in general, spend compute in proportion to how hard, uncertain, or valuable each input is. I’ve written up the routing math and the confidence-threshold mechanics separately in the cascade; here the point is just that ingestion is where that ladder gets built, per stage, per document.
Incremental everything (or: you will reprocess)
Back to the sentence at the top. Because you will reprocess, “re-run the corpus” must never be a thing you do. Instead, track per-document state, extracted@v2, embedded@model-v3, classified@2026-07, so that a new model version becomes a targeted backfill campaign over exactly the affected documents, with its own progress checkpoints, rate limits, and a cost estimate you can show someone before you start.
That only works if writes are idempotent: reprocessing a document lands the same state, never a duplicate. The key is content-derived, not filename-derived, a content hash plus source plus version, so the same file arriving twice from two shares is one entity with two provenance links, not two entities.
def ingest(doc_bytes: bytes, source_path: str) -> str:
content_hash = sha256(doc_bytes) # identity is the content, not the path
doc_id = derive_id(content_hash)
if store.seen(content_hash): # exact duplicate - the free win
store.add_provenance(doc_id, source_path) # same doc, another place it was found
return "skipped:duplicate"
text, confidence = extract(doc_bytes) # parser → OCR fallback
store.upsert(doc_id, { # re-running yields the SAME state
"content_hash": content_hash,
"text": text,
"extraction_confidence": confidence,
"extractor_version": EXTRACTOR_V, # stamp every stage's version
})
return "ingested"Which lets me say the sentence that reliably marks seniority in a system-design conversation: nobody does exactly-once end-to-end. You do at-least-once delivery plus idempotent writes, and the dedup key makes the inevitable duplicates harmless. Chasing true exactly-once is a tell that someone hasn’t run one of these at scale.
Not everything is a document, RML for the structured half
Alongside the file swamp there’s almost always a clean half: a supplier table, an asset registry, a CSV export. Those don’t need extraction; they need mapping. This is exactly what RML (RDF Mapping Language) is for, declarative rules that turn a relational or tabular row into typed graph triples, so structured and unstructured sources land in one semantic model instead of two disconnected ones.
<#SupplierMap> a rr:TriplesMap ;
rml:logicalSource [ rml:source "suppliers.csv" ; rml:referenceFormulation ql:CSV ] ;
rr:subjectMap [ rr:template "http://ex.org/supplier/{id}" ; rr:class ex:Supplier ] ;
rr:predicateObjectMap [ rr:predicate ex:name ; rr:objectMap [ rml:reference "name" ] ] .The upside of one model is that the same SHACL shapes validate everything at the door, the structured rows and the LLM-drafted assertions from the document side pass the identical gate. Extraction confidence and mapping rules are different machinery; the graph they feed, and the constraints that graph enforces, are the same.
Partial failure is the steady state
At ten million documents, “rare” means “several times an hour.” A pipeline that treats failure as exceptional will spend its life paged. Design failure in: retries with exponential backoff for transient errors, a dead-letter queue for the persistent weird ones (so the 0.1% never blocks the 99.9%), backpressure so a slow downstream makes the queue grow instead of dropping data, and autoscaling on queue depth.
And measure the pipeline, not just the boxes. The SLOs that matter are pipeline-level: documents per hour, p95 ingestion-to-searchable lag (how long from a file landing to it being answerable), DLQ rate, and a per-stage error budget. Those are the numbers a stakeholder actually feels.
The payoff: the pipeline is the audit trail
Here’s the part that turns all this plumbing into product value, especially anywhere governance or compliance is the point. Every stage stamped its work, which extractor, which model version, which mapping rule, when, with what confidence. That metadata isn’t logging exhaust; it’s lineage, and lineage is a first-class output. “Show me every fact in the graph that came from document X” and “find everything model v3 touched between Tuesday and Friday” stop being archaeology projects and become queries.
Build ingestion as a one-shot import and you get a graph you can’t trust and can’t cheaply rebuild. Build it as a replayable, idempotent, provenance-stamped pipeline and you get something better than clean data: you get defensible data. The extracted mentions then flow into linking and resolution, the entity-linking flywheel picks up exactly where this pipeline sets them down. Same system, one stage apart.