# When a Knowledge Graph Earns Its Keep > GraphRAG has accumulated enthusiasm and skepticism from the same people at different points in their experience. Both are justified. The question reduces to a specific structural property of your retrieval task. - Published: 2025-11-03 - Reading time: 8 min read - Tags: ML, Engineering - Author: Caio Theodoro (https://caio.theodoro.dev/about.md) - Canonical HTML: https://caio.theodoro.dev/blog/what-the-graph-actually-adds --- [GraphRAG](https://github.com/microsoft/graphrag) grounds retrieval-augmented generation in a knowledge graph rather than a flat vector index. It has accumulated a lot of enthusiasm and a lot of skepticism, often from the same people at different points in their experience with it. The enthusiasm comes from a retrieval problem it addresses well. The skepticism comes from the cost it imposes. Both are justified, and the question of whether to use it reduces to a specific structural property of the retrieval task you're trying to solve. --- ## The flat-index failure mode GraphRAG was built for Flat-index RAG retrieves documents by computing similarity between the query embedding and document chunk embeddings, then returning the top-k most similar chunks. This works well for a specific class of questions: questions whose answers are contained in a single document chunk that is semantically close to the query. The failure case is the multi-hop question: a question whose correct answer requires combining information from two or more source documents, where neither document individually contains the answer, and where the two documents are not necessarily in the top-k nearest neighbors of the query. A practical example from a technical knowledge base: "What happens if service A fails while process B is running?" The answer requires knowing how service A interacts with process B, what process B's failure handling behavior is, and what service A's recovery mechanism does. This information might be spread across the service A documentation, the process B documentation, and a system integration specification. Each document is relevant to one part of the question, but none is highly similar to the full query. A flat-index retrieval might return the overview documents for service A and process B, which are highly similar to the query as a whole, and miss the specific failure-mode documentation that actually contains the answer. --- ## What the graph encodes A knowledge graph represents entities as nodes and relationships as typed, directed edges. Entity nodes might be services, processes, API endpoints, data schemas, or any other domain concept. Relationship edges encode how entities relate: "service A calls service B," "process B reads from schema C," "endpoint D fails with error E when condition F." When a query arrives, the relevant entities are identified in the query, located in the graph, and retrieval proceeds by traversal rather than similarity. To answer "what happens if service A fails while process B is running," the system identifies service A and process B as entities, traverses the edges connecting them to find the interaction relationship, then retrieves the documents associated with that relationship path. The relevant documentation is found through the explicit semantic structure of the domain, not through the statistical proximity of text. This traversal mechanism handles multi-hop questions structurally. A three-hop question, "what are the downstream effects of changing schema C, given that process B reads from C and service A calls process B," is answered by traversing the path A to B to C and retrieving the documents associated with each edge and node along the path. The answer exists in the graph structure; it doesn't depend on any single document being sufficiently similar to the full query. ```mermaid flowchart TD subgraph Flat Index A([Query]) --> B[Embedding Search] B --> C([Similar Chunks]) end subgraph GraphRAG D([Query]) --> E[Entity Detection] E --> F[Graph Traversal] F --> G([Relevant Docs]) end ``` --- ## The construction cost The part of GraphRAG that looks elegant in a diagram is the traversal. The part that determines whether the system works is the construction process underneath it, because the graph can only retrieve through relationships it has actually captured. Building a knowledge graph from a document corpus requires extracting entities, resolving co-references, and inferring relationships. Each step is an error-prone process, and the errors compound. Entity extraction: a model reads each document and identifies named entities and domain concepts. Accuracy on technical documentation is typically in the 85–90% range for a well-tuned system. The 10–15% miss rate means a significant fraction of entities are absent from the graph or imprecisely represented. Co-reference resolution: the "firm" in document A and the "company" in document B need to be linked to the same entity node if they refer to the same organization. This is one of the harder problems in NLP, particularly across documents that use different terminology for the same concept. Errors here produce fragmented entity representations: multiple nodes for the same entity, none of which captures the complete information about it. Relationship extraction: identifying and typing the relationships between entities requires understanding the semantic content of the surrounding text. "Service A monitors process B" and "service A depends on process B" are different relationship types with different implications. Extracting these reliably requires a model that understands the domain well enough to make this distinction. On specialized technical documentation, relationship extraction accuracy can fall below 70%. The result: knowledge graphs built from large document corpora are imperfect representations of the domain knowledge. The graph is useful but noisy, and the retrieval benefits of graph traversal are modulated by the noise in the graph structure. There is also a schema-design problem that tends to be under-discussed. A graph whose relationship types are too generic, such as "related_to," "mentions," or "depends_on," becomes a more expensive version of search, because traversal returns broad neighborhoods without enough semantic precision. A graph whose relationship types are too granular becomes brittle, because extraction quality drops as the model has to distinguish between subtle edge labels that even human annotators would debate. The useful middle is domain-specific and operational: enough relationship structure to answer the questions users actually ask, not enough to satisfy an ontology designer. --- ## The maintenance problem A knowledge graph built from a static corpus can be constructed once and then queried. A knowledge graph built from a dynamic corpus, where documentation changes with product releases, policies update, and integration specs evolve, requires continuous maintenance. Entities change, relationships change, documents are deprecated. Maintaining graph consistency as the underlying documents change is an operational challenge that flat-index RAG doesn't have to the same degree. A flat index is re-embedded as documents change; the embedding process is independent for each document, so a change in one document doesn't cascade through the index structure. A knowledge graph has structural dependencies: if the specification for service A changes in a way that affects its relationship to process B, that relationship edge needs to be updated, and any downstream relationships that depend on it need to be re-evaluated. This is the practical reason that GraphRAG is deployed less frequently than its performance characteristics would suggest. The operational overhead of graph maintenance is non-trivial, and teams that have tried to use it at scale often find that the graph degrades faster than they can maintain it, particularly in fast-moving engineering documentation contexts. --- ## When GraphRAG pays off The conditions under which GraphRAG is worth the construction and maintenance cost are more specific than "we have complex queries." The query distribution needs to contain a significant fraction of multi-hop questions, and flat-index retrieval needs to be measurably failing on those questions. That measurement matters because it tells you the retrieval architecture is the bottleneck, not the model, the prompt, the chunking strategy, or the absence of a reranker. The corpus also needs to be stable enough that graph maintenance is tractable. High-churn documentation is a poor fit because relationships decay as quickly as they are extracted; reference documentation with defined versioning cycles is a better fit because the graph has time to remain useful between updates. The domain needs well-structured entity relationships that can be reliably extracted. Generic text, such as news articles, customer support tickets, and unstructured notes, is usually a poor substrate for knowledge graph construction. Technical documentation with named components, defined APIs, and explicit integration relationships is much better. Before committing to the architecture, the useful experiment is small and concrete: run extraction on a representative slice of the corpus, manually inspect the entities and relationships, then replay actual failed queries against the graph. If extraction quality is below about 80%, the graph will have enough noise that its retrieval benefits are offset by retrieval failures caused by missing or incorrect structure. If the graph answers the failed queries cleanly, the architecture has earned its cost. --- ## The honest position GraphRAG is not a universal upgrade to RAG. It is a specialized architecture that addresses a specific retrieval failure mode, multi-hop questions requiring cross-document reasoning, at a cost in construction complexity, operational overhead, and sensitivity to extraction quality. For organizations whose retrieval failures are concentrated in the multi-hop case and whose corpus is stable and well-structured enough to support a maintained knowledge graph, the investment can make sense. For organizations whose retrieval failures are more diffuse, such as documents not indexed correctly, embedding model quality issues, or chunk size mismatches, the graph doesn't address the actual problem and adds cost without proportionate benefit. The most common mistake is treating GraphRAG as a general quality improvement rather than a targeted fix for a specific failure mode. Diagnosing the failure mode before choosing the architecture is the step that most frequently gets skipped. That diagnosis is the whole point. A graph adds value when the missing thing is structure; it adds very little when the missing thing is better indexing, fresher documents, or a model that can reason over the evidence it already has. --- More posts: https://caio.theodoro.dev/blog.md · About the author: https://caio.theodoro.dev/about.md · Machine-readable index: https://caio.theodoro.dev/llms.txt