Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents

Yaocong Li; Qiang Lan; Leihan Zhang; Le Zhang

cs.CLMar 12, 2026

Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents

Authors: Yaocong Li, Qiang Lan, Leihan Zhang, Le Zhang

Organizations: School of Economics and Management, Beijing University of Posts and Telecommunications, No. 10 Xitucheng Road, Haidian District, Beijing, 100876, China · College of Computing, Beijing Information Science and Technology University, No. 12 Xiaoying East Road, Haidian District, Beijing, 100085, China

Abstract

Retrieval-Augmented Generation (RAG) has emerged as a promising technology for legal document consultation, yet its application in Chinese legal scenarios faces two key limitations: existing benchmarks lack specialized support for joint retriever-generator evaluation, and mainstream RAG systems often fail to accommodate the structured nature of legal provisions. To address these gaps, this study advances two core contributions: First, we constructed the Legal-DC benchmark dataset, comprising 480 legal documents (covering areas such as market regulation and contract management) and 2,475 refined question-answer pairs, each annotated with clause-level references, filling the gap for specialized evaluation resources in Chinese legal RAG. Second, we propose the LegRAG framework, which integrates legal adaptive indexing (clause-boundary segmentation) with a dual-path self-reflection mechanism to ensure clause integrity while enhancing answer accuracy. Third, we introduce automated evaluation methods for large language models to meet the high-reliability demands of legal retrieval scenarios. LegRAG outperforms existing state-of-the-art methods by 1.3% to 5.6% across key evaluation metrics. This research provides a specialized benchmark, practical framework, and empirical insights to advance the development of Chinese legal RAG systems. Our code and data are available at https://github.com/legal-dc/Legal-DC.

Figures & tables

	Min	Max	Average
Query	6	79	21.69
Answer	3	672	100.05
Document	12	2006	132.26

Table 1: Token lengths of questions, answers and reference documents in the Legal-DC benchmark.

Evaluation Dimension	Weight	Rationale
Accuracy	40%	Legal answers directly affect rights and obligations. Any deviation from statutory provisions or factual inaccuracies may cause serious legal risks, making correctness the fundamental requirement.
Relevance	30%	Answers must align with query intent and retrieved evidence. Irrelevant content has no legal consultation value.
Completeness	20%	Legal queries often involve multiple clauses. Completeness improves usefulness, but correctness remains more critical.
Fluency	10%	Fluency enhances readability, but content validity outweighs linguistic quality in legal consultation.

Table 2: Weight Allocation and Rationale for Legal Answer Evaluation Dimensions

Panel of Figure 4 — Fig. 2: Core prompt snippet for automatic legal RAG evaluation.

Panel of Figure 5 — Fig. 3: The prompt for answer generation.

Fig. 4: Architecture of the LegRAG framework.

Method	Recall	MRR@5
Weknora	61.35	35.44
QAnything	63.69	36.67
LightRAG	75.86	47.56
LegRAG	78.02	50.23

Table 3: Retrieval Results (%) of Different Methods.

Model	Precision(%)
Qwen3-max	45.32
DeepSeek-V3.2	40.23

Table 4: Precision Comparison of Different Models

Framework	Model	Recall (%)	MRR@5 (%)	Precision (%)	BLEU (%)	ROUGE (%)	MAP (%)	F1 (%)	LLM-eval (%)
Weknora	DeepSeek-V3.2	57.47	32.74	50.98	38.43	41.42	30.59	55.86	60.82
	Qwen3-8B	59.36	33.44	51.86	37.49	38.89	31.84	57.08	62.71
	Qwen3-max	61.35	35.44	52.28	38.61	42.59	33.49	57.48	64.68
QAnything	DeepSeek-V3.2	58.61	32.33	50.19	36.18	42.78	30.46	55.43	62.03
	Qwen3-8B	59.78	34.01	51.21	36.67	44.15	31.69	55.05	63.15
	Qwen3-max	63.69	36.67	52.58	40.93	43.34	34.70	58.99	67.02

Table 5: Performance Comparison of Different RAG Frameworks with Various LLMs

Framework	Model	Conceptual			Generalization			Logic
		Acc (%)	BLEU (%)	ROUGE (%)	Acc (%)	BLEU (%)	ROUGE (%)	Acc (%)	BLEU (%)	ROUGE (%)
Weknora	DeepSeek-V3.2	57.86	37.96	39.96	59.04	37.48	41.37	57.57	38.51	40.87
	Qwen3-8B	59.68	38.23	42.70	58.93	39.46	40.04	59.49	38.21	43.19
	Qwen3-max	61.62	41.12	42.79	62.71	38.58	43.29	62.72	39.52	41.29
QAnything	DeepSeek-V3.2	60.02	40.41	42.86	61.39	38.89	40.20	60.32	38.64	42.08
	Qwen3-8B	61.36	40.29	43.13	59.93	38.61	43.98	60.94	38.19	40.78

Table 6: Performance Comparison on Conceptual, Generalization, and Logic Task Dimensions

Processing Strategy	Model	Recall (%)	MRR@5 (%)	Precision (%)	BLEU (%)	ROUGE (%)	MAP (%)	F1 (%)	LLM-eval (%)
Chunk-based	Qwen3-8B	74.51	44.43	57.91	48.96	48.12	38.23	64.34	67.25
	DeepSeek-V3.2	77.01	45.77	58.80	43.84	46.43	38.97	65.62	63.19
	Qwen3-max	78.02	48.23	59.56	46.68	51.27	42.21	65.76	62.54
Article-based	Qwen3-8B	76.21	45.21	58.21	49.12	49.66	39.25	66.89	75.02
	DeepSeek-V3.2	80.43	47.33	60.27	45.23	48.65	40.87	66.45	64.68
	Qwen3-max	82.12	50.32	62.45	52.12	56.23	46.23	67.34	66.15

Table 7: Performance Comparison of Different Processing Strategies under Various LLMs

Entity Augmentation	Model	Recall (%)	MRR@5 (%)	Precision (%)	BLEU (%)	ROUGE (%)	MAP (%)	F1 (%)
NO	DeepSeek-V3.2	73.52	43.23	57.63	47.86	47.13	37.25	63.22
	Qwen3-8B	76.70	44.56	57.43	46.47	45.34	38.09	64.52
	Qwen3-max	77.12	46.63	58.54	46.34	50.12	43.52	64.35
Yes	DeepSeek-V3.2	75.32	46.25	57.34	50.56	49.27	38.62	65.48
	Qwen3-8B	81.44	47.25	61.76	46.33	48.34	41.89	65.44
	Qwen3-max	82.73 (+5.61)	51.34 (+4.71)	62.90 (+4.36)	53.15 (+5.29)	54.26 (+4.14)	45.25 (+1.73)	66.24 (+1.72)

Table 8: Effect of Entity Augmentation on Retrieval and Generation Performance

Reflection	Model	Recall (%)	MRR@5 (%)	Precision (%)	BLEU (%)	ROUGE (%)	MAP (%)	F1 (%)
No	DeepSeek-V3.2	74.51	44.43	57.91	48.96	48.12	38.23	64.34
	Qwen3-8B	77.01	45.77	58.80	43.84	46.43	38.97	65.62
	Qwen3-max	78.02	48.23	59.56	46.68	51.27	42.21	65.76
Yes	DeepSeek-V3.2	79.65	48.20	61.24	52.49	52.47	42.65	68.26
	Qwen3-8B	84.21	49.78	62.44	48.12	51.24	44.19	69.44
	Qwen3-max	86.34 (+2.13)	53.54 (+5.31)	65.60 (+6.04)	57.23 (+8.27)	59.11 (+7.84)	50.32 (+8.11)	69.28 (+3.52)

Table 9: Performance Comparison of Legal-DC Systems With and Without Self-Reflection

Model	Conceptual (%)	Generalization (%)	Logic (%)	Avg (%)
DeepSeek-V3.2	69.90	66.10	62.80	66.27
Qwen3-8B	73.00	69.50	66.30	69.60
Qwen3-max	76.00	72.50	69.60	72.70

Table 10: Performance Comparison Across Different Question Types

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Question	Answer	Reference content	Type
外国投资者或者外商投资企业发现其存在未报、错报、漏报有关投资信息的应该怎么办？	外国投资者或者外商投资企业发现其存在未报、错报、漏报有关投资信息的，应当及时进行补报或更正。商务主管部门发现相关情形的，应当通知其于20个工作日内补报或更正；涉及公示事项的，更正前后的信息应当同时公示。	第十九条外国投资者或者外商投资企业发现其存在未报、错报、漏报有关投资信息的，应当及时进行补报或更正。商务主管部门发现相关情形的，应当通知其于 20 个工作日内补报或更正；涉及公示事项的，更正前后的信息应当同时公示。	Concept explanation
(What should foreign investors or foreign-invested enterprises do if they discover that they have failed to report, misreported, or omitted relevant investment information?)	(Foreign investors or foreign-invested enterprises that discover any unreported, misreported, or omitted investment information shall promptly submit supplementary reports or corrections \CJK@punctchar \CJK@uniPunct 0”80”A6 \CJK@punctchar \CJK@uniPunct 0”80”A6)	(Article 19 Where a foreign investor or a foreign-invested enterprise discovers that it has failed to report, misreported, or omitted relevant investment information, it shall promptly make supplementary reports or corrections \CJK@punctchar \CJK@uniPunct 0”80”A6 \CJK@punctchar \CJK@uniPunct 0”80”A6 \CJK@punctchar \CJK@uniPunct 0”80”A6 \CJK@punctchar \CJK@uniPunct 0”80”A6)
某外国投资者拟设立外资企业，计划认缴注册资本为 800 万美元，且以工业产权作价出资。同时，该企业的投资总额拟定为 2000 万美元。该企业的注册资本是否符合 \CJK@punctchar \CJK@uniPunct 0”80”9C注册资本与投资总额比例 \CJK@punctchar \CJK@uniPunct 0”80”9D 的法定要求?	符合规定。该企业投资总额为 2000 万美元，属于 \CJK@punctchar \CJK@uniPunct 0”80”9C1000 万美元以上至 3000 万美元 \CJK@punctchar \CJK@uniPunct 0”80”9D 的区间，法定注册资本不得低于投资总额的 2/5。计算：2000万美元2/5=800万美元，而该企业拟定的注册资本为 800 万美元，恰好满足法定最低要求，因此符合规定。	第三条投资总额在 1000 万美元以上至 3000 万美元（含 3000 万美元）的，其注册资本至少应占投资总额的 2/5；其中投资总额在 1250 万美元以下的，注册资本不得低于 500 万美元。	Logical reasoning
(A foreign investor intends to establish a foreign-invested enterprise with a planned subscribed registered capital of US$8 million, to be contributed in the form of industrial property rights. Concurrently \CJK@punctchar \CJK@uniPunct 0”80”A6 \CJK@punctchar \CJK@uniPunct 0”80”A6)	(Compliant with regulations. The enterprise’s total investment amount is US $20 million, falling within the range of ”US$ 10 million to $30 million” range, where the statutory registered capital must not be less than 2/5 of the total investment. Calculation:$ 20 million 2/5 = $8 million. The enterprise’s proposed registered capital of$ 8 million precisely meets the statutory minimum requirement and is therefore compliant.)	(For investments totaling between US $10 million and US$ 30 million (inclusive), the registered capital shall constitute at least two-fifths of the total investment amount. Where the total investment amount is below US $12.5 million, the registered capital shall not be less than US$ 5 million.)
违反《安全技术防范产品管理办法》的规定，相关工作人员滥用职权、玩忽职守的，将面临怎样的处罚？	由有关部门按照干部管理权限给予行政处分；构成犯罪的，依法追究刑事责任。	第十七条滥用职权、玩忽职守的，依法给予处分；构成犯罪的，依法追究刑事责任。	Generalization and Synthesis
(What penalties will staff members face for abuse of authority or dereliction of duty?)	(They shall be subject to administrative sanctions; criminal liability shall be pursued where a crime is constituted.)	(Article 17 Staff members who abuse authority or neglect duties shall be punished according to law.)

Appendix

Table A1: Data Samples

Query	谁负责食品安全抽样检验工作？(Who is responsible for food safety sampling and inspection?)
Answer	食品安全抽样检验工作由国家市场监督管理总局和县级以上地方市场监督管理部门协同负责。 (Food safety sampling and inspection are jointly managed by the State Administration for Market Regulation and local market supervision departments at or above the county level.)
Gold Content	第六条国家市场监督管理总局建立国家食品安全抽样检验信息系统，定期分析食品安全抽样检验数据，加强食品安全风险预警，完善并督促落实相关监督管理制度。县级以上地方市场监督管理部门应当按照规定通过国家食品安全抽样检验信息系统，及时报送并汇总分析食品安全抽样检验数据。 (Article 6: The State Administration for Market Regulation establishes a national food safety sampling and inspection information system, periodically analyzes food safety sampling and inspection data, strengthens food safety risk warnings, and improves and supervises the implementation of relevant supervision and management systems. Local market supervision departments at or above the county level shall, in accordance with regulations, promptly submit and summarize food safety sampling and inspection data through the national food safety sampling and inspection information system.)
Retrieved Content	第二十四条食品安全抽样检验实行承检机构与检验人负责制。承检机构出具的食品安全检验报告应当加盖机构公章，并有检验人的签名或者盖章。承检机构和检验人对出具的食品安全检验报告负责。承检机构应当自收到样品之日起 20个工作日内出具检验报告。市场监督管理部门与承检机构另有约定的，从其约定。未经组织实施抽样检验任务的市场监督管理部门同意，承检机构不得分包或者转包检验任务。 (Article 24: Food safety sampling and inspection follow a responsibility system for inspection agencies and inspectors. Inspection agencies and inspectors are responsible for issued inspection reports.)
	第三、四、七、八条略。(Articles 3, 4, 7, 8: omitted.)
LLM Answer	食品安全抽样检验工作由国家市场监督管理总局和县级以上地方市场监督管理部门协同负责。 (Food safety sampling and inspection are jointly managed by the State Administration for Market Regulation and local market supervision departments at or above the county level.)

Appendix

Table A2: Information about Responsibility for Food Safety Sampling and Inspection

Query	购买微型无人机必须实名登记吗？(Is real-name registration required for purchasing micro drones?)
Answer	是的，民用无人驾驶航空器所有者应当依法进行实名登记。 (Yes, owners of civil unmanned aerial vehicles shall legally perform real-name registration.)
Gold Content	第十条民用无人驾驶航空器所有者应当依法进行实名登记，具体办法由国务院民用航空主管部门会同有关部门制定。 (Article 10: Owners of civil unmanned aerial vehicles shall legally perform real-name registration, with specific measures formulated by the civil aviation authority under the State Council in conjunction with relevant departments.)
Retrieved Content	第十七条购买第二类、第三类易制毒化学品的，应当在购买前将所需购买的品种、数量，向所在地的县级人民政府公安机关备案。个人自用购买少量高锰酸钾的，无须备案。 (Article 17: For the purchase of second or third category precursor chemicals, the variety and quantity to be purchased shall be filed with the local public security organ at the county level or above before purchase.)
	第十二、七十八、四十三、八十七条略。(Articles 12, 78, 43, 87: omitted.)
LLM Answer	民用无人驾驶航空器所有者应当依法进行实名登记。 (Owners of civil unmanned aerial vehicles shall legally perform real-name registration.)

Appendix

Table A3: Example of Logical Reasoning Retrieval Failure

Query	合同示范文本由谁负责制定？(Who is responsible for formulating contract template texts?)
Answer	省级以上市场监督管理部门可以根据有关法律法规规定，针对特定行业或者领域，联合有关部门制定合同示范文本。(Market supervision and administration departments at or above the provincial level may, in accordance with relevant laws and regulations, jointly formulate contract template texts for specific industries or fields with relevant departments.)
Gold Content	第十三条省级以上市场监督管理部门可以根据有关法律法规规定，针对特定行业或者领域，联合有关部门制定合同示范文本。(Article 13: Market supervision and administration departments at or above the provincial level may, in accordance with relevant laws and regulations, jointly formulate contract template texts for specific industries or fields with relevant departments.)
Retrieved Content	第十六条进行二手车交易应当签订合同。合同示范文本由国务院工商行政管理部门制定。(Article 16: A contract shall be signed for second-hand car transactions. The contract template text is formulated by the State Council’s administrative department for industry and commerce.)
	第十三条省级以上市场监督管理部门可以根据有关法律法规规定，针对特定行业或者领域，联合有关部门制定合同示范文本。根据前款规定制定的合同示范文本，应当主动公开，供社会公众免费阅览、下载、使用。(Article 13: Market supervision and administration departments at or above the provincial level may, in accordance with relevant laws and regulations, jointly formulate contract template texts for specific industries or fields with relevant departments. Contract template texts formulated under the preceding paragraph shall be proactively disclosed for free public access, download, and use.)
	第十四、十五、四十二条略。(Articles 14, 15, 42: omitted.)
LLM Answer	合同示范文本由国务院工商行政管理部门制定。(The contract template texts are formulated by the State Council’s administrative department for industry and commerce.)

Appendix

Table A4: Example of Response Failure Due to Incorrect Document Selection

Explore similar work

Legal RAG Bench: an end-to-end benchmark for legal RAG
Mar 2, 2026Abdur-Rahman Butler, Umar ButlerRetrieval-Augmented Generation RAG Evaluation
Isaacus
Fine-grained Claim-level RAG Benchmark for Law
May 20, 2026Souvick Das, Sallam Abualhaija, Domenico BianculliLegal IR Legal QA
University of Luxembourg
CoAL-RAG: A Complexity-Aware Legal Retrieval-Augmented Generation Method
Aug 18, 2026Jin Su, Zhuofeng Zhao, Huanhuan Wang +1Retrieval-Augmented Generation Legal QA
North China University of Technology, Beijing, 100144, China · Beijing Key Laboratory of Key Technologies for AI+ Domain Applications, Beijing, 100144, China · Beijing Key Laboratory on Integration and Analysis of Large-scale Stream Data, Beijing 100144, China

Mar 2, 2026J/K move · Enter open · S save

Legal RAG Bench: an end-to-end benchmark for legal RAG

Abdur-Rahman Butler, Umar Butler

Isaacus

Retrieval-Augmented Generation RAG Evaluation Legal QA IR Evaluation

Open paper AskPDF arXiv

We introduce Legal RAG Bench, a benchmark and evaluation methodology for assessing the end-to-end performance of legal RAG systems. As a benchmark, Legal RAG Bench consists of 4,876 passages from the Victorian Criminal Charge Book alongside 100 complex, hand-crafted questions demanding expert knowledge of criminal law and procedure. Both long-form answers and supporting passages are provided. As an evaluation methodology, Legal RAG Bench leverages a full factorial design and novel hierarchical error decomposition framework, enabling apples-to-apples comparisons of the contributions of retrieval and reasoning models in RAG. We evaluate three state-of-the-art embedding models (Isaacus' Kanon 2 Embedder, Google's Gemini Embedding 001, and OpenAI's Text Embedding 3 Large) and two frontier LLMs (Gemini 3.1 Pro and GPT-5.2), finding that information retrieval is the primary driver of legal RAG performance, with LLMs exerting a more moderate effect on correctness and groundedness. Kanon 2 Embedder, in particular, had the largest positive impact on performance, improving average correctness by 17.5 points, groundedness by 4.5 points, and retrieval accuracy by 34 points. We observe that many errors attributed to hallucinations in legal RAG systems are in fact triggered by retrieval failures, concluding that retrieval sets the ceiling for the performance of many modern legal RAG systems. We document why and how we built Legal RAG Bench alongside the results of our evaluations. We also openly release our code and data to assist with reproduction of our findings.

Abstract

Explore similar work

Legal RAG Bench: an end-to-end benchmark for legal RAG

Fine-grained Claim-level RAG Benchmark for Law

CoAL-RAG: A Complexity-Aware Legal Retrieval-Augmented Generation Method