We introduce Legal RAG Bench, a benchmark and evaluation methodology for assessing the end-to-end performance of legal RAG systems. As a benchmark, Legal RAG Bench consists of 4,876 passages from the Victorian Criminal Charge Book alongside 100 complex, hand-crafted questions demanding expert knowledge of criminal law and procedure. Both long-form answers and supporting passages are provided. As an evaluation methodology, Legal RAG Bench leverages a full factorial design and novel hierarchical error decomposition framework, enabling apples-to-apples comparisons of the contributions of retrieval and reasoning models in RAG. We evaluate three state-of-the-art embedding models (Isaacus' Kanon 2 Embedder, Google's Gemini Embedding 001, and OpenAI's Text Embedding 3 Large) and two frontier LLMs (Gemini 3.1 Pro and GPT-5.2), finding that information retrieval is the primary driver of legal RAG performance, with LLMs exerting a more moderate effect on correctness and groundedness. Kanon 2 Embedder, in particular, had the largest positive impact on performance, improving average correctness by 17.5 points, groundedness by 4.5 points, and retrieval accuracy by 34 points. We observe that many errors attributed to hallucinations in legal RAG systems are in fact triggered by retrieval failures, concluding that retrieval sets the ceiling for the performance of many modern legal RAG systems. We document why and how we built Legal RAG Bench alongside the results of our evaluations. We also openly release our code and data to assist with reproduction of our findings.
Table 1: Dimension scores by LLM by embedding model.
Model
Correctness (%)
Groundedness (%)
Retrieval accuracy (%)
Embedders
Kanon 2 Embedder
94.0
96.0
86.0
Text Embedding 3 Large
76.5
91.5
52.0
Gemini Embedding 001
74.0
87.0
53.0
LLMs
Gemini 3.1 Pro
82.3
94.3
63.7
Table 2: Average embedding model and LLM dimension scores.
Figure 2: Decomposed error rates for each combination of embedding model and LLM.
Figure 3: Model impact on RAG accuracy relative to sample average.
Correct
Grounded
Both
Interaction
χ2(2)=0.08(p=0.962)
χ2(2)=8.12(p=0.017∗)
χ2(2)=1.47(p=0.479)
LLM main effect
χ2(1)=0.46(p=0.499)
χ2(1)=5.36(p=0.021∗)
χ2(1)=2.43(p=0.119)
Embedder main effect
χ2(2)=28.63(p<0.001∗∗∗)
χ2(2)=8.95(p=0.011∗)
χ2(2)=32.00(p<0.001∗∗∗)
Table 3: ANOVA-style Wald χ2 tests from the linear probability model yeli=αi+βe+γl+δel+εeli with question fixed effects and cluster-robust SEs clustered by question, reported separately for celi (correct), geli (grounded), and beli (both; beli=1 iff celi=geli=1 ).
Effect
Contrast
Est. (pp) (SE)
p
Δ LLM
with Text Emb. 3 L.
-9.0 (4.2) ∗
0.031
Δ LLM
with Gem. Emb. 001
-10.0 (4.8) ∗
0.038
Δ LLM
with Kanon 2 Emb.
2.0 (2.7)
0.459
ΔΔ
Kanon 2 Emb. − Text Emb. 3 L.
11.0 (5.2) ∗
0.033
ΔΔ
Kanon 2 Emb. − Gem. Emb. 001
12.0 (5.0) ∗
0.017
ΔΔ
Gem. Emb. 001 − Text Emb. 3 L.
-1.0 (6.2)
0.871
Table 4: Simple LLM effects and embedder–LLM interaction contrasts for groundedness ( geli ).