Giving RAG a sense of time

Draft

July 19, 2026 · 6 min read

Vector-based RAG demos brilliantly: embed the corpus, retrieve the nearest chunks, let the model answer. Then real users show up with real questions, “Which trial sites reported adverse events for compounds whose formulation later changed?”, and the demo architecture quietly returns the five most similar-sounding paragraphs instead of an answer. Nothing errored. The retrieval was just structurally incapable of the question.

The mental model I use to explain the gap:

Vectors find similar things. Graphs explain how things relate, and when.

These are two different index structures over the same corpus, good at two different query patterns. Mature document AI needs both, joined properly.

Where vector-only retrieval structurally fails

Four failure classes come up over and over, and none of them are fixable with a better embedding model:

1. Multi-hop questions. “Documents about suppliers of plants affected by recall X” requires traversing supplier → plant → recall. Chunk similarity has no concept of a hop; it can only hope one paragraph happens to mention the whole chain.
2. Aggregation and completeness. “How many contracts expose us to vendor Y?” Top-k retrieval is by construction a sample, not an enumeration. Counting requires a structure that knows the members of a set.
3. Exact identifiers. Dense embeddings blur PO-2024-00317 into everything else shaped like an ID. (Sparse keyword search has the opposite failure, it can’t see paraphrase. That asymmetry is why hybrid retrieval exists.)
4. Time. A chunk is a frozen snapshot of whenever its document was written. Embed “the formulation is X” from 2022 and “the formulation is Y” from 2024, and similarity search happily serves both, with no machinery to know one superseded the other.

That last one is the quiet killer in R&D and compliance corpora, where the interesting questions are almost always versioned: as of when, what changed, what superseded what.

The architecture: one ingestion, two indexes, one join key

The fix is not to replace vector search, it’s to make ingestion do double duty. Every document flows through two enrichment paths: chunk-and-embed into an ANN index, and NER-plus-entity-linking into a knowledge graph.

The Data Ingestion Pipeline & Databases

In a production GraphRAG solution, the ingestion pipeline handles the split:

• Path A (Vector Database): Documents are chunked and embedded into an Approximate Nearest Neighbor (ANN) index (like Milvus or Pinecone). This provides your fuzzy semantic search capability.
• Path B (Graph Database): The document passes through an NER (Named Entity Recognition) and Entity Linking pipeline. Mentions of entities are extracted as nodes and edges and stored in a Graph Database (like Neo4j or Amazon Neptune).

The entity link is the crucial move: each chunk in the vector database is annotated with the KG entities it mentions in the graph. This gives you a join key between embedding-space and graph-space.

At query time the two halves play their positions. The question gets entity-linked, hybrid retrieval pulls candidate chunks, and the graph expands from the linked entities, typed relationships, constraints, lineage, all filtered to the time of interest. The LLM receives both kinds of evidence, each carrying its source.

Hybrid retrieval, briefly

Dense and sparse retrieval fail in opposite directions, so production retrieval fuses them. Reciprocal Rank Fusion is embarrassingly simple and hard to beat as a baseline:

from collections import defaultdict

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    """Fuse dense + BM25 result lists by reciprocal rank."""
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for rank, chunk_id in enumerate(ranking):
            scores[chunk_id] += 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.__getitem__, reverse=True)

Then a cross-encoder reranks the fused top-50 into the top-5 that actually enter the context window. The division of labour is the same one SentenceTransformers users know well: bi-encoders for recall (cheap, indexable), cross-encoders for precision (expensive, so only on the shortlist). Retrieval quality beats context quantity, a tight, reranked context outperforms a stuffed one, and costs less.

The Role of the Ontology

Where does this deterministic structure come from? You cannot just let an LLM extract arbitrary nodes and edges; that results in a messy, unqueryable “fuzzball” graph.

Who defines it? Subject Matter Experts (SMEs) and Ontologists define the business reality. Where is it defined? In formal standards like OWL (Web Ontology Language) and validated by SHACL (Shapes Constraint Language). What value does it bring? The ontology acts as the strict schema. It dictates that a Trial Site must Report an Adverse Event, and a Compound must have a Formulation. It ensures the LLM’s extractions conform to the business rules before they are allowed into the Graph Database, providing a reliable foundation for enterprise interoperability.

Making the graph temporal

A knowledge graph becomes temporal when facts carry validity intervals instead of being timeless assertions. In a property graph that’s edge properties; in RDF it’s RDF-star annotations on the triple. The modelling rule: don’t overwrite, supersede. When the formulation changes, the old HAS_FORMULATION edge gets a valid_to, the new edge gets a valid_from, and a SUPERSEDED_BY edge records the succession explicitly.

That turns “as of” from a prayer into a WHERE clause:

// Formulation of a compound as the world stood on 2024-06-30
MATCH (c:Compound {id: $compound})-[r:HAS_FORMULATION]->(f:Formulation)
WHERE r.valid_from <= date('2024-06-30')
  AND (r.valid_to IS NULL OR r.valid_to > date('2024-06-30'))
RETURN f.name, r.valid_from, r.source_doc

Two refinements matter in practice. First, distinguish valid time (when the fact was true in the world) from transaction time (when your pipeline learned it), bitemporal modelling. It sounds academic until an auditor asks “what did the system believe on the day that decision was made?”, which is precisely the transaction-time axis. Second, stamp every edge with its source document and extractor version. That’s what lets query results arrive with citations, and what lets you re-extract selectively when your NER model improves, instead of rebuilding the graph.

Now the question from the opening, trial sites with adverse events for compounds whose formulation later changed, decomposes cleanly: a graph traversal (site → event → compound), a temporal filter (formulation edges where a successor exists), and vector search over the linked chunks for the narrative detail the graph doesn’t carry. Each index does the part it’s built for.

Caveats and Limitations

Building this architecture is powerful, but not a silver bullet. You must be aware of the trade-offs:

• Extraction Costs: Running high-quality NER and relation extraction models over millions of documents is significantly more computationally expensive than simple chunking and embedding.
• Bitemporal Complexity: Maintaining and querying bitemporal graphs can be conceptually challenging and requires careful database architecture.
• Ontology Maintenance: Business realities change, which means your OWL and SHACL models will require governance and versioning of their own.

Evaluate retrieval separately from generation

The most common GraphRAG mistake isn’t architectural, it’s evaluative: judging the system only by final answers. When an answer is wrong you need to know which stage failed, so measure them independently:

• Retrieval: a labelled query set with known relevant chunks/entities; track recall@k and MRR. Build this eval set first, it’s a few hundred judgements and it converts every future retrieval tweak from vibes into a number.
• Generation: faithfulness and citation-correctness on retrieved context, LLM-as-judge for coverage, a human sample for calibration.

In my experience most “the model hallucinated” bugs turn out to be “retrieval never surfaced the evidence” bugs. You only find that out if the two stages are measured apart.

The takeaway

Vector RAG isn’t wrong, it’s half. Similarity gets you into the corpus; structure answers the question. Entity links at ingestion join the two, validity intervals make the structure time-aware, and provenance turns every answer into something you can defend, most of all in domains like clinical trials, where “as of when” is not a nicety but the whole game.

The write-side counterpart of this architecture, how LLM-drafted facts get into a governed graph in the first place, is covered in Governing LLMs in the Enterprise.