How Much Do Legal RAG Systems Still Hallucinate?

Souvick Das; Sallam Abualhaija; Domenico Bianculli

cs.CLAug 14, 2026

How Much Do Legal RAG Systems Still Hallucinate?

Authors: Souvick Das, Sallam Abualhaija, Domenico Bianculli

Organizations: SnT, University of Luxembourg Luxembourg

Abstract

Hallucination is a major challenge for retrieval-augmented generation (RAG) systems in the legal domain, where ungrounded answers can lead to serious consequences. To better understand this problem, we conduct a fine-grained analysis of hallucination behavior in eight legal RAG systems across two legal corpora, the GDPR (in English) and a national civil law (in French). Using claim-level and answer-level evaluation, we report on hallucination density and severity, analyze performance across question categories and user personas, and validate our findings on an independent set of 142 legal-expert-authored questions. Our results show that hallucinations remain pervasive, ranging from less than 10% of responses for the best-performing systems to nearly half in the worst case. We further find that false-premise questions, containing incorrect assumptions that must be rejected, produce high hallucination rates on the manually-drafted questions.

Figures & tables

	GDPR					CIVIL
RAG System	$M$	CL $\downarrow$	CL ${}^{s}\downarrow$	AL $\downarrow$	AL ${}^{s}\downarrow$	$M$	CL $\downarrow$	CL ${}^{s}\downarrow$	AL $\downarrow$	AL ${}^{s}\downarrow$
BM25+Llama3-8B	$10.6{\scriptstyle\pm 5.6}$	$4.6{\scriptstyle\pm 8.6}$	$3.5{\scriptstyle\pm 8.5}$	$29.5$	$20.8$	$6.1{\scriptstyle\pm 2.6}$	$17.9{\scriptstyle\pm 25.8}$	$5.0{\scriptstyle\pm 10.1}$	$45.8$	$24.4$
BM25+Mixtral-8x7B	$10.1{\scriptstyle\pm 5.8}$	$3.4{\scriptstyle\pm 9.3}$	$3.3{\scriptstyle\pm 7.5}$	$\mathbf{20.1}$	$20.8$	$6.4{\scriptstyle\pm 3.2}$	$16.4{\scriptstyle\pm 24.4}$	$6.6{\scriptstyle\pm 14.1}$	$41.1$	$27.9$
BM25+GPT-4	$13.5{\scriptstyle\pm 6.7}$	$2.9{\scriptstyle\pm 5.0}$	$2.3{\scriptstyle\pm 7.3}$	$30.9$	$14.7$	$6.9{\scriptstyle\pm 3.6}$	$4.1{\scriptstyle\pm 14.6}$	$\mathbf{4.4}{\scriptstyle\pm 11.0}$	$15.5$	$20.8$
BM25+GPT-5	$17.9{\scriptstyle\pm 12.2}$	$\mathbf{1.5{\scriptstyle\pm 3.8}}$	$\mathbf{0.9{\scriptstyle\pm 4.6}}$	$\mathbf{20.1}$	$10.7$	$7.5{\scriptstyle\pm 5.6}$	$\mathbf{1.2{\scriptstyle\pm 4.7}}$	$6.1{\scriptstyle\pm 18.2}$	$\mathbf{8.3}$	$16.6$
E5+Llama3-8B	$9.2{\scriptstyle\pm 4.4}$	$9.5{\scriptstyle\pm 12.4}$	$3.4{\scriptstyle\pm 6.8}$	$48.3$	$24.1$	$5.7{\scriptstyle\pm 3.0}$	$11.2{\scriptstyle\pm 21.1}$	$8.8{\scriptstyle\pm 18.7}$	$32.1$	$27.9$

Table 1: Hallucination rates (%) of RAG systems in legal application context ( RQ1 ).

M

denotes the average number of claims per response (mean

\pm

standard deviation).

\downarrow

indicates that lower values are better.

	CIVIL										GDPR
RAG System	H0	H1	H2	H3	H4	H5	H6	H7	H8	H9	H0	H1	H2	H3	H4	H5	H6	H7	H8	H9
BM25+Llama3-8B	91	33	23	13	4	4	0	0	0	0	105	27	13	3	0	1	0	0	0	0
BM25+Mixtral-8x7B	99	30	12	19	3	2	2	0	1	0	119	21	7	1	0	1	0	0	0	0
BM25+GPT-4	142	23	0	3	0	0	0	0	0	0	103	32	11	3	0	0	0	0	0	0
BM25+GPT-5	154	11	1	1	1	0	0	0	0	0	119	15	11	3	0	1	0	0	0	0
E5+Llama3-8B	114	29	19	4	1	0	1	0	0	0	77	38	20	9	3	2	0	0	0	0

Table 2: Distribution of hallucinated claims per answer ( RQ1 ).

	GDPR						CIVIL
	GR ( $Q=131$ )		FR ( $Q=9$ )		FP ( $Q=8$ )		GR ( $Q=137$ )		FR ( $Q=24$ )		FP ( $Q=5$ )
RAG System	CL $\downarrow$	AL $\downarrow$	CL $\downarrow$	AL $\downarrow$	CL $\downarrow$	AL $\downarrow$	CL $\downarrow$	AL $\downarrow$	CL $\downarrow$	AL $\downarrow$	CL $\downarrow$	AL $\downarrow$
BM25+Llama3-8B	$4.5{\scriptstyle\pm 8.3}$	$29.8$	$10.9{\scriptstyle\pm 13.7}$	$44.4$	$0.9{\scriptstyle\pm 2.5}$	$12.5$	$13.4{\scriptstyle\pm 22.0}$	$39.4$	$35.1{\scriptstyle\pm 29.9}$	$75.0$	$56.0{\scriptstyle\pm 37.8}$	$80.0$
BM25+Mixtral-8x7B	$3.6{\scriptstyle\pm 9.3}$	$22.1$	$\mathbf{0.0{\scriptstyle\pm 0.0}}$	$\mathbf{0.0}$	$5.0{\scriptstyle\pm 14.1}$	$12.5$	$11.6{\scriptstyle\pm 21.1}$	$32.1$	$39.1{\scriptstyle\pm 29.4}$	$75.0$	$24.2{\scriptstyle\pm 12.4}$	$100.0$
BM25+GPT-4	$2.9{\scriptstyle\pm 5.0}$	$31.3$	$1.8{\scriptstyle\pm 3.6}$	$22.2$	$3.0{\scriptstyle\pm 6.1}$	$25.0$	$4.4{\scriptstyle\pm 15.7}$	$16.1$	$2.4{\scriptstyle\pm 8.3}$	$8.3$	$6.5{\scriptstyle\pm 9.3}$	$40.0$
BM25+GPT-5	$\mathbf{1.6{\scriptstyle\pm 3.6}}$	$22.1$	$2.5{\scriptstyle\pm 7.4}$	$11.1$	$\mathbf{0.0{\scriptstyle\pm 0.0}}$	$\mathbf{0.0}$	$\mathbf{1.4{\scriptstyle\pm 5.2}}$	$\mathbf{10.2}$	$\mathbf{0.0{\scriptstyle\pm 0.0}}$	$\mathbf{0.0}$	$\mathbf{0.0{\scriptstyle\pm 0.0}}$	$\mathbf{0.0}$

Table 3: Hallucination rates (%) across question categories ( RQ2 ). GR = General Research, FR = Factual Recall, and FP = False Premise. CL and AL denote claim-level and answer-level hallucination rates, where

\pm

is the standard deviation.

	GDPR						CIVIL
	LE ( $Q=52$ )		CO ( $Q=77$ )		CI ( $Q=20$ )		LE ( $Q=104$ )		CO ( $Q=29$ )		CI ( $Q=35$ )
RAG System	CL $\downarrow$	AL $\downarrow$	CL $\downarrow$	AL $\downarrow$	CL $\downarrow$	AL $\downarrow$	CL $\downarrow$	AL $\downarrow$	CL $\downarrow$	AL $\downarrow$	CL $\downarrow$	AL $\downarrow$
BM25+Llama3-8B	$4.3{\scriptstyle\pm 7.6}$	$30.8$	$4.8{\scriptstyle\pm 8.8}$	$31.2$	$4.9{\scriptstyle\pm 10.5}$	$20.0$	$6.0{\scriptstyle\pm 11.9}$	$26.2$	$36.3{\scriptstyle\pm 31.2}$	$72.4$	$37.9{\scriptstyle\pm 30.3}$	$80.0$
BM25+Mixtral-8x7B	$4.7{\scriptstyle\pm 9.3}$	$28.9$	$3.5{\scriptstyle\pm 10.3}$	$\mathbf{19.5}$	$\mathbf{0.0{\scriptstyle\pm 0.0}}$	$\mathbf{0.0}$	$5.7{\scriptstyle\pm 14.4}$	$20.4$	$28.3{\scriptstyle\pm 23.8}$	$75.9$	$37.9{\scriptstyle\pm 29.7}$	$71.4$
BM25+GPT-4	$2.5{\scriptstyle\pm 3.8}$	$34.6$	$3.3{\scriptstyle\pm 5.7}$	$31.2$	$2.2{\scriptstyle\pm 4.7}$	$20.0$	$\mathbf{1.7{\scriptstyle\pm 6.1}}$	$\mathbf{10.7}$	$13.8{\scriptstyle\pm 30.7}$	$31.0$	$3.3{\scriptstyle\pm 8.1}$	$17.1$
BM25+GPT-5	$\mathbf{1.6{\scriptstyle\pm 3.7}}$	$23.1$	$\mathbf{1.9{\scriptstyle\pm 4.3}}$	$23.4$	$\mathbf{0.0{\scriptstyle\pm 0.0}}$	$\mathbf{0.0}$	$\mathbf{1.7{\scriptstyle\pm 5.8}}$	$11.7$	$\mathbf{0.3{\scriptstyle\pm 1.7}}$	$\mathbf{3.5}$	$0.4{\scriptstyle\pm 2.1}$	$2.9$

Table 4: Hallucination rates across personas (%) ( RQ2 ). LE = Legal Expert, CO = Civil Officer, and CI = Citizen.

	GDPR					CIVIL
RAG System	$M$	CL $\downarrow$	CL ${}^{s}\downarrow$	AL $\downarrow$	AL ${}^{s}\downarrow$	$M$	CL $\downarrow$	CL ${}^{s}\downarrow$	AL $\downarrow$	AL ${}^{s}\downarrow$
BM25+Llama3-8B	$6.0{\scriptstyle\pm 4.8}$	$15.8{\scriptstyle\pm 26.2}$	$7.7{\scriptstyle\pm 18.9}$	$42.3$	$25.4$	$4.1{\scriptstyle\pm 2.6}$	$15.9{\scriptstyle\pm 24.8}$	$18.0{\scriptstyle\pm 27.4}$	$40.8$	$43.7$
BM25+GPT-4	$3.8{\scriptstyle\pm 3.4}$	$5.8{\scriptstyle\pm 15.7}$	$9.6{\scriptstyle\pm 25.1}$	$18.3$	$19.7$	$2.7{\scriptstyle\pm 2.2}$	$\mathbf{3.1}{\scriptstyle\pm 10.8}$	$10.7{\scriptstyle\pm 23.7}$	$9.9$	$23.9$
BM25+GPT-5	$5.2{\scriptstyle\pm 4.6}$	$7.2{\scriptstyle\pm 16.5}$	$\mathbf{4.5}{\scriptstyle\pm 15.3}$	$22.5$	$\mathbf{12.7}$	$4.9{\scriptstyle\pm 2.7}$	$11.8{\scriptstyle\pm 22.2}$	$\mathbf{9.0}{\scriptstyle\pm 23.7}$	$33.8$	$\mathbf{19.7}$
E5+Llama3-8B	$5.8{\scriptstyle\pm 4.9}$	$12.6{\scriptstyle\pm 24.9}$	$7.1{\scriptstyle\pm 19.8}$	$25.4$	$18.3$	$4.0{\scriptstyle\pm 2.8}$	$14.1{\scriptstyle\pm 25.1}$	$17.5{\scriptstyle\pm 32.8}$	$33.8$	$33.8$
E5+GPT-4	$4.4{\scriptstyle\pm 4.3}$	$\mathbf{1.9}{\scriptstyle\pm 10.1}$	$9.6{\scriptstyle\pm 25.1}$	$\mathbf{4.2}$	$18.3$	$3.0{\scriptstyle\pm 1.7}$	$\mathbf{3.1}{\scriptstyle\pm 11.6}$	$15.6{\scriptstyle\pm 31.5}$	$\mathbf{8.5}$	$23.9$

Table 5: Hallucination rates (%) of RAG systems evaluated on manually curated legal expert questions within a legal application context.

M

represents the average number of claims per response, reported as mean

\pm

standard deviation.

	GDPR			CIVIL
RAG System	GR	FR	FP	GR	FR	FP
$Q=$	$32$	$12$	$26$	$26$	$7$	$31$
BM25+Llama3-8B	34.4	33.3	53.9	30.8	71.4	41.9
BM25+GPT-4	21.9	0.0	23.1	11.5	14.3	9.7
BM25+GPT-5	15.6	16.7	30.8	30.8	57.1	32.3
E5+Llama3-8B	28.1	8.3	30.8	19.2	14.3	54.8

Table 6: Answer-level hallucination rates (%) by question category on expert-authored legal questions.

Explore similar work

Legal RAG Bench: an end-to-end benchmark for legal RAG
Mar 2, 2026Abdur-Rahman Butler, Umar ButlerRetrieval-Augmented Generation RAG Evaluation
Isaacus
Fine-grained Claim-level RAG Benchmark for Law
May 20, 2026Souvick Das, Sallam Abualhaija, Domenico BianculliLegal IR Legal QA
University of Luxembourg
CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law
May 28, 2026Ethan Zhao, Maksym Taranukhin, Wei Cui +2Retrieval-Augmented Generation Legal QA
Department of Computer Science, University of British Columbia · Vector Institute · Peter A. Allard School of Law, University of British Columbia +1

Mar 2, 2026J/K move · Enter open · S save

Legal RAG Bench: an end-to-end benchmark for legal RAG

Abdur-Rahman Butler, Umar Butler

Isaacus

Retrieval-Augmented Generation RAG Evaluation Legal QA IR Evaluation

Open paper AskPDF arXiv

We introduce Legal RAG Bench, a benchmark and evaluation methodology for assessing the end-to-end performance of legal RAG systems. As a benchmark, Legal RAG Bench consists of 4,876 passages from the Victorian Criminal Charge Book alongside 100 complex, hand-crafted questions demanding expert knowledge of criminal law and procedure. Both long-form answers and supporting passages are provided. As an evaluation methodology, Legal RAG Bench leverages a full factorial design and novel hierarchical error decomposition framework, enabling apples-to-apples comparisons of the contributions of retrieval and reasoning models in RAG. We evaluate three state-of-the-art embedding models (Isaacus' Kanon 2 Embedder, Google's Gemini Embedding 001, and OpenAI's Text Embedding 3 Large) and two frontier LLMs (Gemini 3.1 Pro and GPT-5.2), finding that information retrieval is the primary driver of legal RAG performance, with LLMs exerting a more moderate effect on correctness and groundedness. Kanon 2 Embedder, in particular, had the largest positive impact on performance, improving average correctness by 17.5 points, groundedness by 4.5 points, and retrieval accuracy by 34 points. We observe that many errors attributed to hallucinations in legal RAG systems are in fact triggered by retrieval failures, concluding that retrieval sets the ceiling for the performance of many modern legal RAG systems. We document why and how we built Legal RAG Bench alongside the results of our evaluations. We also openly release our code and data to assist with reproduction of our findings.

Abstract

Explore similar work

Legal RAG Bench: an end-to-end benchmark for legal RAG

Fine-grained Claim-level RAG Benchmark for Law

CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law