A Year of RAG in Production
What retrieval-augmented generation actually does in production, where flat-index vector search fails, and what GraphRAG fixes — and what it doesn't.
The premise of RAG is sound: language models have fixed training data, the world keeps producing new information, and you can bridge that gap by retrieving relevant context at inference time. This solves a specific problem, especially in systems where the alternative is asking a model to answer from stale weights and hoping that confidence correlates with truth. The issue is what "retrieval" has come to mean in practice, which is almost always vector similarity search, cosine distance in embedding space, plus the assumption that semantic proximity is a reliable proxy for logical relevance — which it isn't, in most of the cases that matter.
This matters because most production failures in RAG systems do not look like dramatic hallucinations. They look like plausible answers grounded in the wrong document, or correct high-level answers that miss the one exception buried in a version note, or outputs that cite something relevant enough to be defensible and wrong enough to cause damage. The model is often doing what it was asked to do. The retrieval layer has simply shown it the wrong world.
The retrieval step is doing more work than it gets credit for
Vector similarity search has a specific property: it finds documents that are linguistically or topically similar to the query. A query about "interest rate impact on housing affordability" retrieves chunks that discuss interest rates and housing. That's the easy case, and it works fine.
The hard cases are where the question and the answer use different conceptual framing, or where answering the question correctly requires combining information from multiple sources that don't individually contain the answer. A user asking "what changes between our API v2 and v3 that would break backwards compatibility" is not going to get a useful result from retrieving the chunks most similar to that sentence. The v2 docs and v3 docs might both rank below some general overview text, because the overview is the piece that's most semantically similar to the user's high-level question. The specific diffs are lower-ranked, more technical, less visible to cosine distance, and therefore not retrieved.
This is the multi-hop failure. The answer requires combining piece A and piece B, which are not in the same chunk, may not even be adjacent in the source documents, and which individually look less relevant than something that discusses the topic at a higher level of abstraction. Flat-index retrieval has no mechanism to traverse from A to B. It retrieves the top-k nearest neighbors, and if the answer requires the 8th and 14th nearest neighbors together, it doesn't find it.
What most production RAG systems actually are
Most deployed RAG pipelines are simpler than the diagrams drawn around them. The honest description is a text chunker that splits documents into overlapping segments of 256–512 tokens, an embedding model that encodes each chunk into a high-dimensional vector, an approximate nearest-neighbor index (FAISS, Pinecone, Weaviate, take your pick), and a retrieval step that returns the top-k most similar chunks at query time. This is followed by an LLM call that's given the retrieved chunks as context and asked to answer the question.
This works well for question-answering over a single coherent document or a small, well-structured knowledge base. It works less well as the corpus grows, as the queries become more specific, and as the required reasoning becomes multi-step.
The failure modes I've seen most in production: retrieval of thematically relevant but factually wrong chunks (the model retrieves something that sounds right but predates an important update), retrieval of the correct general topic but the wrong level of specificity (the overview rather than the implementation detail), and complete retrieval failure on queries that require combining evidence across documents (the multi-hop case above).
What doesn't fix this: bigger chunks, smaller chunks, better embedding models, more overlap. These are parameter adjustments on the same architecture. They move the failure rate, they don't eliminate the failure mode.
Reranking helps, but only after the candidate set contains the right evidence. A cross-encoder reranker can reorder the top 50 retrieved chunks more intelligently than raw vector similarity, because it evaluates the query and candidate together rather than comparing two isolated embeddings. But it cannot rescue a document that never entered the candidate set. Hybrid search, which combines sparse lexical retrieval with dense embeddings, helps with exact names, error codes, and version identifiers. It still does not give the system a memory of relationships across documents. These are useful improvements, and in many systems they are enough, but they should not be mistaken for a different retrieval architecture.
What the graph actually fixes
The insight behind GraphRAG is that retrieval doesn't have to be "find the nearest neighbors to the query." It can be "find the entities in the query, traverse the relationships between those entities and related entities, and retrieve the set of documents that are connected to those entities."
A knowledge graph stores entities as nodes and relationships as edges. When a query comes in, the relevant entities are extracted and identified in the graph, and retrieval proceeds by traversal rather than similarity. The document that answers "what is the relationship between component A and component B" is found by following the edge from node A to node B, not by hoping that a chunk containing both A and B in the same 512-token window was indexed at the right level of similarity.
The multi-hop case becomes tractable: you extract entity A, traverse to entity B through the relationship the query implies, and retrieve the documents associated with those nodes. The answer exists in the graph structure, not just in the semantic neighborhood of the query string.
This changes the failure profile. In internal evaluations on multi-hop question-answering benchmarks, GraphRAG consistently outperforms flat-index retrieval by a meaningful margin, not because the LLM is better, but because the retrieval step is finding the right documents.
The cost
Building a knowledge graph is not free. You need a pipeline that extracts entities from your source documents, resolves co-references (the "company" in paragraph 3 and the "firm" in paragraph 7 are the same entity), infers relationships between entities, and maintains the graph as documents change. Each of these steps involves a model that can fail, and the failures compound.
Entity extraction accuracy on technical documentation is typically in the 85-92% range for a well-tuned system. That means 8-15% of entities are missing, miscategorized, or duplicated. Relationship inference is harder because the relationship "component A deprecates component B" requires understanding the semantic meaning of the surrounding text, not just the presence of both entities. On specialized technical domains, relationship extraction accuracy can drop below 70%.
What this means practically: GraphRAG is worth the investment when your retrieval failure rate on the current system is high and the failure mode is the multi-hop case. If your queries are mostly single-document, single-hop lookups, flat-index retrieval is cheaper to build, easier to maintain, and close enough in accuracy that the graph doesn't pay for itself.
The heuristic I've settled on: if users are regularly asking questions that require combining information across more than two source documents, and if your corpus is large enough that the relevant documents don't reliably land in the top-5 nearest neighbors, the graph earns the construction cost. Otherwise, the overhead is high and the return is marginal.
Where both approaches miss
Neither flat-index RAG nor GraphRAG handles the case where the required knowledge is implicit. In those cases, answering correctly requires reasoning that isn't contained in any single document or relationship, but emerges from combining multiple pieces of evidence with background knowledge the model has from training.
This is the limit of retrieval as an architecture. Retrieval can surface relevant documents. It cannot synthesize novel reasoning from the relationships between documents. The LLM is still responsible for the synthesis step, and what it retrieves shapes what it can synthesize, but the retrieval step and the reasoning step are genuinely separate, and improving retrieval doesn't help when the bottleneck is reasoning.
The practical test I trust is not "does retrieval improve benchmark accuracy?" It is "when this system answers incorrectly, can we explain whether the failure came from retrieval, synthesis, or stale source material?" If the answer is no, the system is not yet production-understandable; it is a demo with an observability problem.
The honest state of RAG in 2025 is that it's a reliable solution to the knowledge freshness problem for well-structured, single-hop queries over managed corpora. It becomes progressively less reliable as query complexity increases, corpus size increases, and the freshness and consistency of the indexed documents degrades. GraphRAG extends the reliable range somewhat. Neither approach is a universal answer to the knowledge integration problem, and treating them as one leads to the kind of production system that works great in the demo and fails in specific, hard-to-predict ways once it's deployed. The retrieval layer is not an accessory to the model. In many production systems, it is the system.