Five questions before you build a knowledge graph for RAG

A knowledge graph can make a RAG system better at connecting facts. It can also make it slower, more expensive and harder to secure, without moving answer quality at all. Which one you get depends less on the framework you pick than on your questions, your data and your organisation.

In GraphRAG vs RAG: what the evidence actually says I went through three studies. This post turns them into five questions to answer before you commit budget. Each comes with the evidence behind it and the answer that points each way.

The scorecard

1. How many of your questions need several hops?

This is the question that caps the upside. In the most careful comparison so far (Han et al. 2025), plain RAG beat Microsoft's GraphRAG on single-hop questions, and GraphRAG's local search won on multi-hop ones. With Llama 3.1 70B, the multi-hop gain was 0.7 F1 points on HotpotQA and 5.4 accuracy points on MultiHop-RAG. On time-based questions the gap was large: 49.1% against 25.7%.

GraphRAG-Bench (Xiang et al. 2025) found the same split: on simple fact retrieval, vector RAG with a reranker matched or beat the graph methods on its novel corpus.

How to answer it: pull a few hundred real questions from search logs, chat logs or support tickets. Label each as a lookup, a multi-hop question or a whole-corpus question. Have two people label the same 50 and check that they agree; if they do not, your categories are not clear yet.

Whatever share comes out as multi-hop is the ceiling for what a graph can improve. If it is small, the money is better spent on retrieval quality.

2. Do people ask about the whole corpus?

Questions like "what are customers complaining about this quarter?" have no single source chunk. This is where GraphRAG's global search, built on community summaries, did best: in the original paper (Edge et al. 2024) global approaches beat vector RAG on comprehensiveness in 72 to 83% of LLM-judged comparisons.

Global search also has a known failure mode. In Han et al., it answered only 13.95% of questions that had no answer in the corpus correctly, against 91.36% for vector RAG: it produced an answer anyway. It also lost detail on specific questions.

How to answer it: look for summary questions in your logs. If they exist, consider serving them as a separate feature (a periodic report, a dashboard of themes) rather than routing every query through global search.

3. Can you extract the entities reliably?

A graph is only as good as its extraction. In Han et al., retrieving knowledge-graph triplets alone was the worst method tested: 28.5 F1 on Natural Questions against 68.2 for plain RAG, with the 70B model. One reason they give: for NQ and HotpotQA, only about 65% of the answer entities made it into the graph at all.

How to answer it: run extraction on 50 representative documents and have a domain expert check the entities and relations it found and missed. Watch for synonyms, part numbers written three ways and people who appear under two names.

Enterprises have one advantage here that benchmarks do not: structured sources. An ERP, a product catalogue or an asset register already knows which part goes into which product. Taking entities and relations from there, and using the LLM only for what lives in free text, is often more reliable than extracting everything from documents.

4. How fast does the corpus change, and who pays for re-indexing?

Building a graph means running an LLM over the corpus to extract entities and, for Microsoft's GraphRAG, to write community summaries. GraphRAG-Bench measured what that costs on one book of about 56,000 tokens:

Time matters too: in the same table LightRAG needed over 710 seconds for that one book. Note that LazyGraphRAG, which Microsoft describes as indexing at 0.1% of GraphRAG's cost, used about 90% of GraphRAG's tokens in this benchmark. Implementations and settings may differ, so measure on your own data.

How to answer it: multiply your corpus size in tokens by the ratio for the method you are considering and by your model's price, then ask how often you would need to rebuild. Check whether the framework supports incremental updates, and what happens to community summaries when one document changes.

5. Do permissions differ between documents?

This one is not from a benchmark; it is an architecture constraint, and in enterprises it is often the one that decides.

Vector RAG can respect permissions at query time: each chunk carries the access rules of its source document, and the search only returns chunks the user may see. A graph merges facts from many documents into shared nodes, and GraphRAG's community summaries mix many documents into one text. If a user may see only some of those documents, a summary can tell them what they should not know.

Your options, from simplest to most work:

• Separate graphs per audience (for example per business unit), each built only from documents that audience may see.
• Access rules on every node and edge, inherited from the source documents, with filtering at query time and no shared summaries across permission boundaries.
• A graph only for content everyone can see, with vector RAG and security trimming for the rest.

Putting it together

1. Mostly lookups? Improve hybrid retrieval and add a reranker. Revisit the graph when the question mix changes.
2. Many multi-hop questions and reliable entities? Pilot a graph method on a sample of your questions. Compare at least two variants and score against ground truth.
3. Whole-corpus questions? Offer them as a summary or report feature rather than for every query.
4. In every case, keep the vector index. In Han et al., routing between the two or combining both retrievals beat either one alone.

Once the answers point to a pilot, the next step is to put numbers on it. That is the subject of A GraphRAG business case, done the evidence-based way, and the token and indexing costs are in The real cost of GraphRAG: indexing, queries and keeping the graph fresh.

Sources

• Han et al. (2025). RAG vs. GraphRAG: A Systematic Evaluation and Key Insights.
• Xiang et al. (2025). When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation (GraphRAG-Bench).
• Edge et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization. Microsoft Research.
• Microsoft Research (2024). LazyGraphRAG: Setting a new standard for quality and cost.