# How to evaluate GraphRAG honestly: golden sets, metrics and judge bias

> Most GraphRAG wins come from an LLM picking the better of two answers, and independent re-evaluations shrink or reverse them. How judge bias works, which metrics to use instead, how many questions you need, and how to use a judge if you must.

- URL: https://bardak.ai/posts/graphrag-evaluation
- Author: Ivan Bardak (https://bardak.ai/pages/about)
- Published: 2026-11-17

Most claims that GraphRAG beats plain RAG rest on one kind of test: show a language model two answers to the same question and ask which is better. It is quick, it needs no reference answers, and it produces impressive win rates. It is also the weakest way to evaluate a retrieval system, and the published research now shows by how much.

This post covers what goes wrong with the usual GraphRAG evaluations, which metrics to use instead, how big a question set you need, and how to use an LLM judge if you must.

**The short answer:** build a golden set from your own questions, with reference answers and the question type labelled, and score against it. Use an LLM judge only with controls: both answer orders, similar lengths, repeated runs and ties allowed. Report a margin of error. When independent studies applied these controls, large published GraphRAG wins shrank to a few points or reversed.

## Why published wins shrink

The original GraphRAG paper generated 125 questions per dataset with a language model, had no reference answers, and let an LLM judge compare answers pairwise; global GraphRAG won 72% to 83% of comparisons on comprehensiveness ([Edge et al.](https://arxiv.org/abs/2404.16130)). Later work tested the method itself. Han et al. found the judge gave substantially different, and sometimes opposite, verdicts depending on which answer it saw first, and that against reference summaries global GraphRAG generally lost to plain RAG ([Han et al. 2025](https://arxiv.org/abs/2502.11371)).

Zeng et al. went further. Under the usual protocol, two identical copies of LightRAG judged against each other came out 90% to 10%, which should be 50:50. With answer lengths aligned, both orders, repeated runs and ties allowed, LightRAG's reported wins disappeared ([Zeng et al. 2025](https://arxiv.org/abs/2506.06331)):

**LightRAG win rate: as reported and under a controlled protocol**

| comparison | As reported | Controlled protocol |
| --- | --- | --- |
| LightRAG vs NaiveRAG | 66.7 | 39.1 |
| LightRAG vs MGRAG | 56.4 | 32.3 |

Agriculture dataset. Under the controlled protocol plain NaiveRAG beat LightRAG, and more than a fifth of comparisons were ties in most settings. Source: Zeng et al., How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG (arXiv 2506.06331, 2025). Data: https://bardak.ai/data/graphrag-evaluation/lightrag-win-rate-as-reported-and-under-a-controlled-protocol.csv

## LLM judges have measurable biases

The biases are well documented outside GraphRAG. In the study that introduced MT-Bench, GPT-4 gave the same verdict after the two answers were swapped in only 65% of cases, GPT-3.5 in 46% and Claude v1 in 24%; padding an answer with a rephrased repeat of its own list fooled GPT-3.5 and Claude v1 in 91% of cases ([Zheng et al. 2023](https://arxiv.org/abs/2306.05685)). In another study, simply reordering answers let Vicuna-13B beat ChatGPT on 66 of 80 questions with ChatGPT as the judge ([Wang et al. 2023](https://arxiv.org/abs/2305.17926)).

**Position consistency of LLM judges**

| Judge | MT-Bench | DevBench |
| --- | --- | --- |
| GPT-4 | 0.82 | 0.83 |
| Llama-3.3-70B | 0.80 | 0.89 |
| GPT-4o | 0.76 | 0.80 |
| Claude-3.5-Sonnet | 0.82 | 0.76 |
| Gemini-1.5-pro | 0.62 | 0.84 |
| Llama-3.1-8B | 0.69 | 0.47 |
| Claude-3-Haiku | 0.57 | 0.23 |

Share of comparisons with the same verdict after the answers are swapped (1.0 means fully consistent). Consistency depends on the judge and the task, and falls when the two answers are close in quality, which is exactly the GraphRAG versus RAG situation. Source: Shi et al., Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge (AACL 2025), Table 2. Data: https://bardak.ai/data/graphrag-evaluation/position-consistency-of-llm-judges.csv

Judges are not useless. On clear differences, GPT-4 agreed with human experts 85% of the time, against 81% between two humans ([Zheng et al.](https://arxiv.org/abs/2306.05685)). The trouble starts when the systems are close, and two retrieval methods on the same documents usually are.

## Score against ground truth where you can

With reference answers, the differences between methods look very different. In Han et al.'s question-answering tests, the gaps between plain RAG and the best graph methods were a few points either way ([Han et al. 2025](https://arxiv.org/abs/2502.11371)):

**RAG and GraphRAG scored against reference answers**

| Method | Natural Questions F1 | HotpotQA F1 | MultiHop-RAG accuracy |
| --- | --- | --- | --- |
| Plain RAG | 64.78 | 60.04 | 67.02 |
| Community-GraphRAG (local) | 63.01 | 61.66 | 69.01 |
| HippoRAG 2 | 61.03 | 63.01 | 70.27 |
| Knowledge-graph triplets only | 34.28 |  |  |

Plain RAG leads on single-fact questions, graph methods on multi-hop ones, by narrow margins. Only about two thirds of the answer entities existed in the graphs built from the documents. Source: Han et al., RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (arXiv 2502.11371, v3). Data: https://bardak.ai/data/graphrag-evaluation/rag-and-graphrag-scored-against-reference-answers.csv

Two more findings are worth carrying into your own evaluation. On MultiHop-RAG, 13.6% of questions were answered correctly only by GraphRAG and 11.6% only by RAG, so routing questions between the two can beat either. And on a benchmark of 1,018 computer science questions built from textbooks, the model alone scored 70.7% average accuracy, plain BM25 retrieval 71.7%, the best method 73.6%, and two graph methods fell below using no retrieval at all ([Xiao et al. 2025](https://arxiv.org/abs/2506.02404)). A separate GraphRAG-Bench found plain RAG comparable or better on simple fact retrieval, with graphs ahead on complex reasoning and summarisation ([Xiang et al. 2025](https://arxiv.org/abs/2506.05690)). The pattern from [GraphRAG vs RAG: what the evidence actually says](https://bardak.ai/posts/graphrag-vs-rag-evidence) holds: graphs help on some question types, not across the board.

## Metrics, in plain language

**Which metric measures what**

| Metric | What it measures | Needs reference answers? |
| --- | --- | --- |
| Exact match and F1 | Whether the answer matches a reference answer exactly, or how many words overlap | Yes |
| Claim-level recall | Share of the facts in the reference answer that the answer contains | Yes |
| Recall@k | Share of the relevant passages that appear in the top k retrieved | Labelled passages |
| nDCG@10 | Retrieval quality that rewards putting the best passages first | Labelled passages |
| Faithfulness (RAGAS) | Share of the answer's statements supported by the retrieved context | No |
| Answer relevance (RAGAS) | How closely the answer addresses the question; ignores whether it is true | No |
| Pairwise LLM judgement | Which of two answers a model prefers | No, which is why it is weakest |

Reference-free metrics are useful for monitoring, but they cannot tell you whether an answer is correct. Source: Rajpurkar et al. 2016 (SQuAD); Thakur et al. 2021 (BEIR); Es et al. 2023 (RAGAS). Data: https://bardak.ai/data/graphrag-evaluation/which-metric-measures-what.csv

Reference answers have limits too. Even people do not score perfectly against them: on SQuAD, human answers reached 77% exact match and 87% F1 ([Rajpurkar et al. 2016](https://arxiv.org/abs/1606.05250)), so collect more than one acceptable answer where you can. RAGAS's answer relevance explicitly ignores factuality ([Es et al. 2023](https://arxiv.org/abs/2309.15217)). And in retrieval, passages nobody labelled count as wrong: in BEIR, almost a third of one dense retriever's top-ten results on TREC-COVID had never been labelled at all ([Thakur et al. 2021](https://arxiv.org/abs/2104.08663)).

## How to build a golden set

1. **Sample real questions** from search logs, tickets or chat transcripts, not questions a model writes about your documents.
2. **Label the type of each question:** single fact, multi-hop, whole-corpus summary, and questions with no answer in the corpus. Report results per type; the averages hide everything that matters.
3. **Write a reference answer and the supporting passages** for each question, with a second acceptable answer where wording varies.
4. **Write the grading rules before you grade.** People change their criteria while looking at outputs, a pattern called criteria drift ([Shankar et al. 2024](https://arxiv.org/abs/2404.12272)).
5. **Label the pooled results.** When you compare systems, label the passages any of them retrieved, or the newcomer is penalised for finding things nobody checked.

Size matters more than most teams expect. Anthropic's statistical guidance works through an example in which detecting a three-point difference between two systems with reasonable confidence takes about a thousand questions ([Miller 2024](https://arxiv.org/abs/2411.00640)). With around 200 questions, only differences of roughly 13 points are reliably detectable, or about 7.5 points if each question is run ten times. With fewer than a few hundred questions, use Wilson or Bayesian intervals rather than the usual normal approximation, which gives error bars that are too narrow ([Bowyer et al. 2025](https://arxiv.org/abs/2503.01747)). Always report the margin of error next to the score.

## If you must use an LLM judge

- **Judge both orders** and count disagreement as a tie, or average the two.
- **Align answer lengths,** or the longer answer wins regardless of content.
- **Repeat the run** several times and report the spread.
- **Allow ties** and a scale, rather than forcing a winner.
- **Validate against people** on at least a hundred or so labelled items before trusting the judge; ARES gets confidence intervals from about 150 human labels ([Saad-Falcon et al. 2024](https://arxiv.org/abs/2311.09476)).
- **Run the sanity check:** judge a system against itself. Anything far from 50:50 means the protocol, not the systems, is deciding the result.

Then put the result where it belongs: in the pilot threshold of [A GraphRAG business case, done the evidence-based way](https://bardak.ai/posts/graphrag-business-case), next to the cost from [The real cost of GraphRAG](https://bardak.ai/posts/graphrag-cost). Whether to build a graph at all is the subject of [Five questions before you build a knowledge graph for RAG](https://bardak.ai/posts/knowledge-graph-rag-five-questions), and the same discipline of measuring rather than demoing applies to agents ([Agentic AI from demo to production](https://bardak.ai/posts/agentic-ai-from-demo-to-production)).

## How much weight this evidence deserves

**Appraising the evidence**

| Evidence | Strength | Main limitation |
| --- | --- | --- |
| Studies of judge bias (Zheng, Shi, Wang) | Large, repeated, consistent across judges | General chat tasks, not retrieval; judges improve over time |
| GraphRAG evaluation studies (Han, Zeng) | Independent re-evaluations with controls | Specific datasets and methods; Zeng et al. is a single preprint |
| Benchmarks (Xiao, Xiang) | Reference answers, many methods | Their corpora, not yours |
| Statistical guidance (Miller, Bowyer) | Sound methods with worked examples | Assumptions about variance that you should check on your data |

Data: https://bardak.ai/data/graphrag-evaluation/appraising-the-evidence.csv

## Sources

- [Edge et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization. arXiv:2404.16130.](https://arxiv.org/abs/2404.16130)
- [Han et al. (2025). RAG vs. GraphRAG: A Systematic Evaluation and Key Insights. arXiv:2502.11371.](https://arxiv.org/abs/2502.11371)
- [Zeng et al. (2025). How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG. arXiv:2506.06331.](https://arxiv.org/abs/2506.06331)
- [Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.](https://arxiv.org/abs/2306.05685)
- [Shi et al. (2024, revised 2025). Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge. arXiv:2406.07791.](https://arxiv.org/abs/2406.07791)
- [Wang et al. (2023). Large Language Models are not Fair Evaluators. arXiv:2305.17926.](https://arxiv.org/abs/2305.17926)
- [Xiao et al. (2025). GraphRAG-Bench: Challenging Domain-Specific Reasoning for Evaluating Graph Retrieval-Augmented Generation. arXiv:2506.02404.](https://arxiv.org/abs/2506.02404)
- [Xiang et al. (2025). When to use Graphs in RAG (GraphRAG-Bench). arXiv:2506.05690.](https://arxiv.org/abs/2506.05690)
- [Es et al. (2023). Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217.](https://arxiv.org/abs/2309.15217)
- [Saad-Falcon et al. (2024). ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. arXiv:2311.09476.](https://arxiv.org/abs/2311.09476)
- [Rajpurkar et al. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250.](https://arxiv.org/abs/1606.05250)
- [Thakur et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. arXiv:2104.08663.](https://arxiv.org/abs/2104.08663)
- [Miller (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640.](https://arxiv.org/abs/2411.00640)
- [Bowyer, Aitchison and Ivanova (2025). Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints. arXiv:2503.01747.](https://arxiv.org/abs/2503.01747)
- [Shankar et al. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. arXiv:2404.12272.](https://arxiv.org/abs/2404.12272)
