Evaluating retrieval-augmented generation (RAG) systems in Persian law remains challenging because benchmarks lack realism, grounding, and coverage of legal reasoning. Existing resources are either synthetically generated from statutory texts, lacking natural information needs, or derived from informal legal question--answer sources without explicit linkage to authoritative legal documents, making them unsuitable for rigorous evaluation of grounded legal reasoning. To address this gap, we introduce TUSI-Bench, a structured Persian legal benchmark designed for the evaluation of RAG-based systems. TUSI-Bench is built upon a multi-layered legal knowledge framework containing 7,261 legal articles and integrating codified statutes, legal commentaries, doctrinal writings, and Supreme Court unification rulings. We also construct a legal knowledge graph linking articles and concepts for structure-aware retrieval. Based on this foundation, TUSI-Bench provides two complementary evaluation tasks: (i) TUSI-QA, an open-ended question answering benchmark containing 1,999 questions that requires grounded responses with explicit legal citations, and (ii) TUSI-Bar, a multiple-choice benchmark containing approximately 500 four-option questions derived from Iranian Bar examination questions. Furthermore, we propose a graph-based retrieval baseline that leverages relational structure within the legal knowledge graph. Experimental results demonstrate that incorporating graph-based relations improves retrieval recall by up to 8 percentage points compared to standard baselines, highlighting the importance of structured legal representations for RAG systems. The proposed benchmark provides a standardized resource for evaluating grounded legal reasoning in Persian and enables future research on structured legal intelligence.
Figures & tables
Figure 1: Overview of the structured legal knowledge layer and the proposed TUSI-Bench. TUSI-Bench consists of two complementary subsets: TUSI-QA, a citation-grounded question answering set derived from real lawyer–client consultations; and TUSI-Bar, a multiple-choice question set derived from Iranian Bar Exam questions.
Figure 2: Length distributions of statutory articles (left) and their corresponding legal interpretations (right).
Figure 3: Temporal distribution of legal articles in the dataset across decades. The total number of articles is (n = 5,323).
Figure 4: Structural properties of the legal reference graph. (a) Out-degree distribution on a log scale, showing a heavy-tailed pattern consistent with a power-law ( α=2.50 , R2=0.961 ), indicating highly uneven citation behavior. (b) Coverage of explanation annotations across nodes, where 27.7% of articles have associated interpretations. (c) Coverage of reference annotations, showing that 33.6% of nodes participate in at least one reference link, highlighting moderate but sparse connectivity in the graph structure.
Figure 5: Top 10 laws by article count in the dataset.
Figure 6: Overview of the TUSI-QA dataset construction pipeline.
Figure 7: Distribution of citation frequencies across ranked legal provisions in the training and test sets of TUSI-QA. Legal provisions are ranked according to their citation frequency in the training set. The figure shows that the long-tail citation distribution is preserved after the stratified train-test split.
Figure 8: Distribution of the number of relevant legal articles required to answer each question in TUSI-QA (training and test sets). Each question may depend on one, two, or three legal provisions, reflecting different levels of retrieval and reasoning complexity.
Figure 9: KDE plots of cosine similarity scores before and after citation filtering in TUSI-QA. Left: question–legal provision similarity. Right: answer–legal provision similarity. The rightward shift after filtering indicates stronger semantic alignment.
Dataset
Split
Total Number
Average Length
Q
P
A
Q
P
A
TUSI-QA (Ours)
Train
1490
7261
1490
448.04
555.22
397.50
Test
509
7261
509
426.85
548.42
362.95
Comparison
PersianQuAD [ 31 ]
Train
18567
3149
18567
59.07
1032.47
16.14
Test
1000
35
2339
57.79
1525.66
12.05
Table 1: Summary of datasets. We report number of queries (Q), number of passages (P), number of answers (A), and the average length of queries, passages and answers (calculated with the GPT-2 tokenizer [ 38 ] ).
Dataset
LS retrieval
LS QA
TUSI-QA (Ours)
0.03±0.00
0.19±0.02
PersianQuAD [ 31 ]
0.12±0.00
0.09±0.00
PQuAD [ 30 ]
0.16±0.00
0.12±0.00
ParSQuAD [ 29 ]
0.10±0.00
0.12±0.00
Bar Exam QA [ 3 ]
0.07±0.00
0.14±0.01
Housing Statute QA [ 3 ]
0.09±0.00
0.00±0.00
Table 2: Lexical similarity scores for the retrieval task and downstream QA task across datasets. Values are reported as mean ± 95% confidence interval.
Figure 10: Distribution of cosine similarity scores across datasets for Query–Gold Passage and Answer–Gold Passage settings. The figure illustrates lexical similarity patterns and highlights the distinct behavior of the proposed dataset compared with existing benchmarks.
Figure 11: Retrieval evaluation results across different embedding models and the proposed Structure-Aware Legal RAG framework. The figure illustrates consistent improvements over the baseline in most configurations.
Method
Correctness
Effectiveness
Objectiveness
Clarity
Completeness
Gemma3 27B 8bit [ 39 ]
1.947
2.092
3.314
3.827
1.782
Gemma3 27B 8bit [ 39 ] + Structure-Aware
Legal RAG
2.420
2.540
3.487
3.644
2.206
GPT-5-nano [ 49 ]
2.110
2.450
4.150
4.000
2.010
Table 3: Downstream evaluation results on the Cited Question-Answer Set.
Table 4: Downstream evaluation results on Bar Exam multiple-choice QA.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : An example instance from TUSI-QA, consisting of a statutory article and its corresponding expert interpretation.
Figure A.2 : Examples of judicial precedents included in TUSI-QA: Unification Judgments No. 847 and No. 794 of the General Assembly of the Supreme Court of Iran. Each example presents a concise summary of the corresponding judicial decision.
Figure A.3 : Examples of citation-grounded legal question answering instances in TUSI-QA. Each instance contains a legal question, the corresponding answer, and the referenced legal article providing the supporting citation.
Figure A.4 : Illustrative examples from TUSI-Bar. Each example consists of a multiple-choice legal question, four candidate answers, the ground-truth answer, and the associated legal source for citation-based legal reasoning.
Retrieval-augmented generation (RAG) evaluation measures how well systems retrieve, use, and cite evidence while balancing reliability, cost, and speed.
We introduce Legal RAG Bench, a benchmark and evaluation methodology for assessing the end-to-end performance of legal RAG systems. As a benchmark, Legal RAG Bench consists of 4,876 passages from the Victorian Criminal Charge Book alongside 100 complex, hand-crafted questions demanding expert knowledge of criminal law and procedure. Both long-form answers and supporting passages are provided. As an evaluation methodology, Legal RAG Bench leverages a full factorial design and novel hierarchical error decomposition framework, enabling apples-to-apples comparisons of the contributions of retrieval and reasoning models in RAG. We evaluate three state-of-the-art embedding models (Isaacus' Kanon 2 Embedder, Google's Gemini Embedding 001, and OpenAI's Text Embedding 3 Large) and two frontier LLMs (Gemini 3.1 Pro and GPT-5.2), finding that information retrieval is the primary driver of legal RAG performance, with LLMs exerting a more moderate effect on correctness and groundedness. Kanon 2 Embedder, in particular, had the largest positive impact on performance, improving average correctness by 17.5 points, groundedness by 4.5 points, and retrieval accuracy by 34 points. We observe that many errors attributed to hallucinations in legal RAG systems are in fact triggered by retrieval failures, concluding that retrieval sets the ceiling for the performance of many modern legal RAG systems. We document why and how we built Legal RAG Bench alongside the results of our evaluations. We also openly release our code and data to assist with reproduction of our findings.
RAG-based legal assistants have been growing in popularity, but LLM hallucinations remain a key issue and potentially undermines justice. While benchmarks have been developed to evaluate progress, many rely on synthetic queries rather than realistic legal scenarios. Moreover, Canadian law remains underrepresented in existing evaluations. To address this gap, we introduce CanLegalRAGBench, a Canadian legal QA benchmark based on realistic queries and expert-annotated answers grounded in case law. Our evaluation shows that retrieval performance is sensitive to design choices and that open-source embedding models are competitive with closed source models. However, it also reveals the limitation of automatic evaluations that penalize systems for retrieving alternative relevant documents. We also find that generated answers often diverge from gold responses, either with hallucinations or by producing overly detailed or irrelevant content, with 8-29% of claims not being supported by the retrieved documents. We hope this benchmark will help drive continued progress in addressing limitations of legal RAG systems.
Ethan Zhao, Maksym Taranukhin, Wei Cui +2
Department of Computer Science, University of British Columbia · Vector Institute · Peter A. Allard School of Law, University of British Columbia +1
The rapid progress of large language models (LLMs) is shifting semantic search toward a question-answering paradigm, where users ask questions and LLMs generate responses. In high-stake domains such as law, retrieval-augmented generation (RAG) is commonly used to mitigate hallucinations in generated responses. Nonetheless, prior work shows that RAG systems, whether general-purpose or legal-specific, still hallucinate at varying rates, making fine-grained evaluation essential. Despite the need, existing evaluation frameworks for legal RAG systems lack the granularity required to provide detailed analysis of retrieval and generation performance separately. Moreover, current benchmarks are largely English-only and centered on legal expert queries, overlooking non-expert needs. We introduce ClaimRAG-LAW, a comprehensive dataset for legal RAG that supports French and English, targets both experts and non-experts, and includes diverse question types reflecting realistic scenarios. We further apply a fine-grained evaluation framework of state-of-the-art legal RAG systems, revealing limitations in retrieval, generation, and claim-level analysis in the legal domain.