A Year of RAG in Production

A language model has a cutoff. The world keeps moving. RAG treats that gap as an inference problem: fetch the current document at query time and the weights stop being the store of truth. Most teams file retrieval under freshness, but the question that matters is whether retrieval showed the model the right world. Flat cosine search often doesn't.

Production failures rarely look like dramatic hallucinations. They look like a plausible answer grounded in the wrong document, or a correct high-level answer that missed the exception in a version note, or a citation relevant enough to defend and wrong enough to hurt. The model did what it was asked. The retrieval layer handed it the wrong world.

The multi-hop case

Vector similarity search finds documents that sit close to the query in wording or topic. Ask about interest rates and housing affordability and you get chunks about interest rates and housing. That case works.

The hard cases look different. Either the framing shifts, or the answer exists only when two sources are combined. Ask which changes between API v2 and v3 would break backwards compatibility, and the chunks nearest to that sentence won't help you. The v2 docs and the v3 docs can both rank below a general overview, because the overview is semantically closest to a high-level question. The diffs are more technical, sit lower in the ranking, and barely register in cosine distance, so nobody retrieves them.

This is the multi-hop failure. The answer needs piece A and piece B, which share no chunk and may not sit next to each other in the source. Each looks less relevant than a page that discusses the topic from higher up. Flat-index retrieval can't walk from A to B. It returns the top-k nearest neighbors and stops, so an answer that needs the 8th and 14th neighbors together never turns up.

What most production systems actually are

The pipeline you ship is simpler than the diagram drawn around it. A chunker splits documents into overlapping 256–512 token segments. An embedding model encodes each chunk. An approximate nearest-neighbor index (FAISS, Pinecone, Weaviate, take your pick) returns the top-k most similar chunks at query time, and an LLM answers from that context.

That's fine for question-answering over one coherent document. It also works for a small, well-structured knowledge base. It degrades as the corpus grows, as queries get more specific, and as the reasoning turns multi-step.

Three failures show up in production. A chunk can be on topic and wrong, because it sounds right and predates an update. A chunk can be the correct topic at the wrong level of specificity, the overview rather than the implementation detail. And a query that combines evidence across documents can fail retrieval completely.

You can't fix this with bigger chunks, smaller chunks, better embedding models, or more overlap. Those are parameter adjustments on the same architecture. They move the failure rate. The failure mode stays.

Reranking helps only after the candidate set already holds the right evidence. A cross-encoder can reorder the top 50 retrieved chunks better than raw vector similarity, because it reads the query and the candidate together instead of comparing two isolated embeddings. It can't rescue a document that never entered the set. Hybrid search, sparse lexical retrieval plus dense embeddings, helps with exact names, error codes, and version identifiers. It still gives you no memory of relationships across documents.

A larger candidate set plus hybrid search can imitate multi-hop. You retrieve the top 50, a lexical match catches a version identifier, and the LLM stitches two chunks together. Sometimes that works. It's a wider neighborhood, though, not a traversal. When the two facts share no tokens and sit far apart in embedding space, a bigger set doesn't invent the path between them.

What the graph actually fixes

Retrieval doesn't have to mean nearest neighbors to the query. That's the claim in GraphRAG. Find the entities in the query. Follow the relationships from those entities to related ones. Pull the documents attached along the way.

Entities are nodes and relationships are edges. A query comes in, the system picks out the relevant entities, and retrieval proceeds by traversal. To find the document that explains how component A relates to component B, you follow the edge from A to B. You don't hope that a chunk containing both landed in the same 512-token window at the right similarity.

The multi-hop case becomes tractable. Extract entity A, traverse to entity B through the relationship the query implies, and retrieve the documents tied to those nodes. The path lives in the graph structure, not in the semantic neighborhood of the query string alone.

The paper is arXiv 2404.16130, and its numbers are internal evaluations on multi-hop question-answering. They claim GraphRAG consistently outperforms flat-index retrieval, because retrieval finds the right documents, not because the LLM is better. This essay has no named eval, seed, or n for that margin.

The cost

Four stages, four places to fail. Extraction. Co-reference resolution, because the company in paragraph 3 and the firm in paragraph 7 are the same entity. Relationship inference. Upkeep as documents change. Every step involves a model, and the failures compound.

Entity extraction accuracy on technical documentation is typically in the 85-92% range for a well-tuned system. So 8-15% of entities are missing, miscategorized, or duplicated. Relationship inference is harder, because "component A deprecates component B" depends on the meaning of the surrounding text and not on both names appearing. On specialized technical domains, relationship extraction can drop below 70%. Those bands are named magnitudes, not a measured extraction run on a named corpus.

GraphRAG pays for its construction when the current failure rate is high and the failure is multi-hop. If your queries are mostly single-document, single-hop lookups, flat-index retrieval is cheaper to build and easier to maintain, and close enough that the graph never earns its keep.

The heuristic that holds: if users regularly ask questions that combine information across more than two source documents, and the corpus is big enough that the relevant documents don't reliably land in the top-5 nearest neighbors, the graph earns the cost. Otherwise the overhead is high and the return is marginal.

Where both approaches miss

Neither flat-index RAG nor GraphRAG handles implicit knowledge. A correct answer there requires reasoning that sits in no single document or relationship. It comes from combining evidence with background the model already has.

This is where retrieval as an architecture runs out. Retrieval can surface documents. It can't synthesize new reasoning from the relationships between them. The LLM still does the synthesis, and what it retrieves shapes what it can synthesize, but the two steps are separate. Better retrieval doesn't help when the bottleneck is reasoning.

The practical test isn't whether retrieval improves benchmark accuracy. It's whether you can name a wrong answer as a retrieval failure, a synthesis failure, or stale source material. If you can't, the system isn't production-understandable. It's a demo with an observability problem.

The honest state of RAG in 2025 is a reliable fix for knowledge freshness on well-structured, single-hop queries over managed corpora. Reliability drops as query complexity and corpus size go up, and as the freshness and consistency of the indexed documents degrade. GraphRAG stretches the reliable range when the failure is multi-hop.

If you can't name whether a wrong answer came from retrieval, synthesis, or a stale source, instrument that before you change the index. If you can name it, and the miss is the 8th and 14th neighbors, price the graph against the 85-92% extraction tax.