A magnifying glass on a white table
Field note7 min read
All posts

How to evaluate GraphRAG honestly: golden sets, metrics and judge bias

Most GraphRAG wins come from an LLM picking the better of two answers, and independent re-evaluations shrink or reverse them. How judge bias works, which metrics to use instead, how many questions you need, and how to use a judge if you must.

Photo: Markus Winkler / Unsplash

On this page

    Most claims that GraphRAG beats plain RAG rest on one kind of test: show a language model two answers to the same question and ask which is better. It is quick, it needs no reference answers, and it produces impressive win rates. It is also the weakest way to evaluate a retrieval system, and the published research now shows by how much.

    This post covers what goes wrong with the usual GraphRAG evaluations, which metrics to use instead, how big a question set you need, and how to use an LLM judge if you must.

    The short answer: build a golden set from your own questions, with reference answers and the question type labelled, and score against it. Use an LLM judge only with controls: both answer orders, similar lengths, repeated runs and ties allowed. Report a margin of error. When independent studies applied these controls, large published GraphRAG wins shrank to a few points or reversed.

    Why published wins shrink

    The original GraphRAG paper generated 125 questions per dataset with a language model, had no reference answers, and let an LLM judge compare answers pairwise; global GraphRAG won 72% to 83% of comparisons on comprehensiveness (Edge et al.). Later work tested the method itself. Han et al. found the judge gave substantially different, and sometimes opposite, verdicts depending on which answer it saw first, and that against reference summaries global GraphRAG generally lost to plain RAG (Han et al. 2025).

    Zeng et al. went further. Under the usual protocol, two identical copies of LightRAG judged against each other came out 90% to 10%, which should be 50:50. With answer lengths aligned, both orders, repeated runs and ties allowed, LightRAG's reported wins disappeared (Zeng et al. 2025):

    LightRAG win rate: as reported and under a controlled protocol

    LightRAG vs NaiveRAG
    LightRAG vs MGRAG

    LightRAG win rate

    Agriculture dataset. Under the controlled protocol plain NaiveRAG beat LightRAG, and more than a fifth of comparisons were ties in most settings.

    Source: Zeng et al., How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG (arXiv 2506.06331, 2025)

    Download the data (CSV)

    Show the data
    comparisonAs reportedControlled protocol
    LightRAG vs NaiveRAG66.739.1
    LightRAG vs MGRAG56.432.3

    LLM judges have measurable biases

    The biases are well documented outside GraphRAG. In the study that introduced MT-Bench, GPT-4 gave the same verdict after the two answers were swapped in only 65% of cases, GPT-3.5 in 46% and Claude v1 in 24%; padding an answer with a rephrased repeat of its own list fooled GPT-3.5 and Claude v1 in 91% of cases (Zheng et al. 2023). In another study, simply reordering answers let Vicuna-13B beat ChatGPT on 66 of 80 questions with ChatGPT as the judge (Wang et al. 2023).

    Position consistency of LLM judges

    GPT-40.82 (best)0.83
    Llama-3.3-70B0.800.89 (best)
    GPT-4o0.760.80
    Claude-3.5-Sonnet0.82 (best)0.76
    Gemini-1.5-pro0.620.84
    Llama-3.1-8B0.690.47
    Claude-3-Haiku0.570.23

    Share of comparisons with the same verdict after the answers are swapped (1.0 means fully consistent). Consistency depends on the judge and the task, and falls when the two answers are close in quality, which is exactly the GraphRAG versus RAG situation.

    Source: Shi et al., Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge (AACL 2025), Table 2

    Download the data (CSV)

    Judges are not useless. On clear differences, GPT-4 agreed with human experts 85% of the time, against 81% between two humans (Zheng et al.). The trouble starts when the systems are close, and two retrieval methods on the same documents usually are.

    Score against ground truth where you can

    With reference answers, the differences between methods look very different. In Han et al.'s question-answering tests, the gaps between plain RAG and the best graph methods were a few points either way (Han et al. 2025):

    RAG and GraphRAG scored against reference answers

    Plain RAG64.78 (best)60.0467.02
    Community-GraphRAG (local)63.0161.6669.01
    HippoRAG 261.0363.01 (best)70.27 (best)
    Knowledge-graph triplets only34.28

    Plain RAG leads on single-fact questions, graph methods on multi-hop ones, by narrow margins. Only about two thirds of the answer entities existed in the graphs built from the documents.

    Source: Han et al., RAG vs. GraphRAG: A Systematic Evaluation and Key Insights (arXiv 2502.11371, v3)

    Download the data (CSV)

    Two more findings are worth carrying into your own evaluation. On MultiHop-RAG, 13.6% of questions were answered correctly only by GraphRAG and 11.6% only by RAG, so routing questions between the two can beat either. And on a benchmark of 1,018 computer science questions built from textbooks, the model alone scored 70.7% average accuracy, plain BM25 retrieval 71.7%, the best method 73.6%, and two graph methods fell below using no retrieval at all (Xiao et al. 2025). A separate GraphRAG-Bench found plain RAG comparable or better on simple fact retrieval, with graphs ahead on complex reasoning and summarisation (Xiang et al. 2025). The pattern from GraphRAG vs RAG: what the evidence actually says holds: graphs help on some question types, not across the board.

    Metrics, in plain language

    Which metric measures what

    Exact match and F1Whether the answer matches a reference answer exactly, or how many words overlapYes
    Claim-level recallShare of the facts in the reference answer that the answer containsYes
    Recall@kShare of the relevant passages that appear in the top k retrievedLabelled passages
    nDCG@10Retrieval quality that rewards putting the best passages firstLabelled passages
    Faithfulness (RAGAS)Share of the answer's statements supported by the retrieved contextNo
    Answer relevance (RAGAS)How closely the answer addresses the question; ignores whether it is trueNo
    Pairwise LLM judgementWhich of two answers a model prefersNo, which is why it is weakest

    Reference-free metrics are useful for monitoring, but they cannot tell you whether an answer is correct.

    Source: Rajpurkar et al. 2016 (SQuAD); Thakur et al. 2021 (BEIR); Es et al. 2023 (RAGAS)

    Download the data (CSV)

    Reference answers have limits too. Even people do not score perfectly against them: on SQuAD, human answers reached 77% exact match and 87% F1 (Rajpurkar et al. 2016), so collect more than one acceptable answer where you can. RAGAS's answer relevance explicitly ignores factuality (Es et al. 2023). And in retrieval, passages nobody labelled count as wrong: in BEIR, almost a third of one dense retriever's top-ten results on TREC-COVID had never been labelled at all (Thakur et al. 2021).

    How to build a golden set

    1. Sample real questions from search logs, tickets or chat transcripts, not questions a model writes about your documents.
    2. Label the type of each question: single fact, multi-hop, whole-corpus summary, and questions with no answer in the corpus. Report results per type; the averages hide everything that matters.
    3. Write a reference answer and the supporting passages for each question, with a second acceptable answer where wording varies.
    4. Write the grading rules before you grade. People change their criteria while looking at outputs, a pattern called criteria drift (Shankar et al. 2024).
    5. Label the pooled results. When you compare systems, label the passages any of them retrieved, or the newcomer is penalised for finding things nobody checked.

    Size matters more than most teams expect. Anthropic's statistical guidance works through an example in which detecting a three-point difference between two systems with reasonable confidence takes about a thousand questions (Miller 2024). With around 200 questions, only differences of roughly 13 points are reliably detectable, or about 7.5 points if each question is run ten times. With fewer than a few hundred questions, use Wilson or Bayesian intervals rather than the usual normal approximation, which gives error bars that are too narrow (Bowyer et al. 2025). Always report the margin of error next to the score.

    If you must use an LLM judge

    • Judge both orders and count disagreement as a tie, or average the two.
    • Align answer lengths, or the longer answer wins regardless of content.
    • Repeat the run several times and report the spread.
    • Allow ties and a scale, rather than forcing a winner.
    • Validate against people on at least a hundred or so labelled items before trusting the judge; ARES gets confidence intervals from about 150 human labels (Saad-Falcon et al. 2024).
    • Run the sanity check: judge a system against itself. Anything far from 50:50 means the protocol, not the systems, is deciding the result.

    Then put the result where it belongs: in the pilot threshold of A GraphRAG business case, done the evidence-based way, next to the cost from The real cost of GraphRAG. Whether to build a graph at all is the subject of Five questions before you build a knowledge graph for RAG, and the same discipline of measuring rather than demoing applies to agents (Agentic AI from demo to production).

    How much weight this evidence deserves

    Appraising the evidence

    Studies of judge bias (Zheng, Shi, Wang)Large, repeated, consistent across judgesGeneral chat tasks, not retrieval; judges improve over time
    GraphRAG evaluation studies (Han, Zeng)Independent re-evaluations with controlsSpecific datasets and methods; Zeng et al. is a single preprint
    Benchmarks (Xiao, Xiang)Reference answers, many methodsTheir corpora, not yours
    Statistical guidance (Miller, Bowyer)Sound methods with worked examplesAssumptions about variance that you should check on your data

    Download the data (CSV)

    Sources

    How to cite

    Ivan Bardak. "How to evaluate GraphRAG honestly: golden sets, metrics and judge bias". bardak.ai, November 17, 2026. https://bardak.ai/posts/graphrag-evaluation

    Data behind the charts (JSON)

    Discussion