Organizations: School of Artificial Intelligence, Beihang University, Beijing, China · KoGuan School of Law, Shanghai Jiao Tong University, Shanghai, China · School of Criminal Law, China University of Political Science and Law, Beijing, China
Large language models (LLMs) have demonstrated strong reasoning abilities across specialized domains, motivating research into their application to legal reasoning. However, existing legal benchmarks often conflate factual recall with genuine inference, fragment the reasoning process, and overlook the quality of reasoning. To address these limitations, we introduce MSLR, the first Chinese multi-step legal reasoning dataset grounded in real-world judicial decision making. MSLR adopts the IRAC framework (Issue, Rule, Application, Conclusion) to model structured expert reasoning from official legal documents. In addition, we design a scalable Human-LLM collaborative annotation pipeline that efficiently produces fine-grained step-level reasoning annotations and provides a reusable methodological framework for multi-step reasoning datasets. Evaluation of multiple LLMs on MSLR shows only moderate performance, highlighting the challenges of adapting to complex legal reasoning. Further experiments demonstrate that Self-Initiated Chain-of-Thought prompts generated by models autonomously improve reasoning coherence and quality, outperforming human-designed prompts. MSLR contributes to advancing LLM reasoning and Chain-of-Thought strategies and offers open resources for future research. The dataset and code are available at https://github.com/yuwenhan07/MSLR-Bench and https://law.sjtu.edu.cn/flszyjzx/index.html.
Figures & tables
Figure 1: A schematic of the MSLR construction and evaluation framework.
Characteristic
MSLR
Document time span
2005–2024
Total number of documents
1,389
Avg. characters per document (chars)
2,516
Avg. structured fields per document
43.03
Total number of structured entries
59,771
Avg. fill rate of key fields (%)
90.58
Table 1: Descriptive statistics of the MSLR dataset.
Model
FCR
Overall Acc
Llama3-8B-Chinese
76.37
57.43
GLM4-9B
68.85
67.01
Qwen2.5-7B
58.12
57.69
Qwen2.5-72B
57.26
58.69
Llama3.3-70B-Instruct
57.19
64.72
DeepSeek-V3
69.65
77.43
Table 2: Comparative performance of LLMs on the automatic annotation task. The first block of rows corresponds to General LLMs, the second block corresponds to Legal LLMs, and the third block corresponds to Reasoning LLMs.
Figure 2: Pearson correlation analysis and linear fit between IRAC Recall and LLM Score.
Figure 3: Zero-Shot vs. One-Shot performance on the legal reasoning task.
Model
Standard Input
Human-Designed CoT
Delta
IRAC Recall
LLM Score
IRAC Recall
LLM Score
IRAC Recall
LLM Score
Llama3-8B-Chinese
46.76
29.63
53.93
60.37
7.17
30.74
GLM4-9B
50.79
52.38
57.00
70.88
6.21
18.50
Qwen2.5-7B
56.78
65.73
57.70
65.44
0.92
-0.29
Qwen2.5-72B
61.25
68.50
58.09
57.70
-3.16
-10.80
Llama3.3-70B-Instruct
53.47
58.06
58.87
79.73
5.40
21.67
Table 3: Comparing Standard Input and Human-Designed CoT on the legal reasoning task.
Figure 4: H-CoT vs. S-CoT on the legal reasoning task.
Figure 5: Typology and coverage of Self-Initiated CoT reasoning traces in three mainstream LLMs.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
File from Lawyer LLaMA
ROUGE-1 F1
ROUGE-2 F1
ROUGE-L F1
alpaca_gpt4_data_zh.json
0.0303
0.0033
0.0247
fakao_gpt4.json
0.0171
0.0008
0.0155
judical_examination_v2.json
0.0083
0.0007
0.0071
legal_counsel_multi_turn_with_article_v2.json
0.0032
0.0002
0.0029
legal_counsel_v2.json
0.0168
0.0026
0.0144
zixun_gpt4.json
0.0017
0.0001
0.0014
Appendix
Table 4: Document similarity between MSLR and Lawyer LLaMA.
Field Type
Section
Field Name
Exa-Fields
Name of the Traded Stock
Insider Information Determination
Date of Insider Information Formation
Date of Public Disclosure
Name
Basic Information of the Party
Gender
Year of Birth
Appendix
Table 5: Field categorization for evaluation.
Category
Model
Zero-shot
One-shot
FCR
Overall Acc
FCR
Overall Acc
General LLMs
Llama3-8B
76.37
57.43
76.40
58.50
GLM4-9B
68.85
67.01
61.37
57.17
DeepSeek-V3
69.65
77.43
73.07
75.11
Legal LLMs
Cleverlaw
78.43
61.79
64.76
55.73
Lawyer-LLM
70.13
57.49
83.77
58.91
Appendix
Table 6: Zero-shot vs. One-shot performance on the automatic annotation task.
Analysis Dimension
Standard Output
Human-Designed Output
Evaluation Conclusion
Case Fact Recognition
Accurate and complete, covering the full process of insider information formation and trading
Omits the early stage of insider information formation, which does not affect substantive determination
Roughly equivalent
Insider Information Identification
Correctly cites Article 75 of the 2005 Securities Law; applies the historical law accurately
Incorrectly cites the 2019 version and Article 80 (misapplies information disclosure clauses)
Std clearly superior; CoT lacks legal clarity
Insider Awareness Analysis
Reasonably judged based on Article 74, considering official position and timing of awareness
Introduces a “time of awareness” path, but the logical chain is unclear and partially fabricated
Small language models (SLMs) are promising for real-world deployment due to their efficiency and low operational cost. However, their limited capacity struggles with high-stakes legal reasoning tasks that require coherent statute interpretation and logically consistent deduction. Furthermore, training SLMs for such tasks demands high-quality, concise reasoning trajectories, which are prohibitively expensive to manually collect and difficult to curate via standard rejection sampling, lacking granularity beyond final verdicts. To address these challenges, we propose {LegalDrill}, a diagnosis-driven synthesis framework that extracts and iteratively refines reasoning trajectories from a capable teacher via fine-grained prompting, then a self-reflective verification is employed to adaptively select the most effective data for the SLM student. The resulting data empower SLM training through supervised fine-tuning and direct preference optimization. Extensive experiments on several legal benchmarks demonstrate that {LegalDrill} significantly bolsters the legal reasoning capabilities of representative SLMs while bypassing the need for scarce expert annotations, paving a scalable path toward practical legal reasoning systems.
Tianchun Li, Haochen Liu, Vishwa Pardeshi +5
Purdue University, West Lafayette, IN, USA · Fidelity Investments, Boston, MA, USA
Large language models are increasingly applied to high-risk domains such as law, yet complex legal reasoning remains limited by two structural challenges. First, existing RAG and GraphRAG methods emphasize lexical or semantic similarity while overlooking normative relations among legal provisions. Second, vanilla Chain-of-Thought prompting may generate plausible rationales without enforcing the normative structure of legal reasoning. To deal with the bottleneck of pipelines in the legal reasoning domain, we propose LEGO, a dual-module framework that synergizes Legal Expert GraphRAG and expert Chain-of-thought for complex legal reasoning. ExpertGraphRAG uses an expert-annotated civil code graph encoding these normative relations with a greedy normative-coverage retrieval algorithm to dynamically extract instance-specific provision subgraphs, while ExpertCoT organizes the retrieved provisions and case facts into structured Provision-Fact-Conclusion reasoning. With a Qwen3-8B backbone, LEGO achieves 40.53% exact-match accuracy on LawExamQA_Civil, outperforming the evaluated RAG and CoT baselines and performing comparably to the evaluated larger models, while remaining robust on multi-hop questions. It also achieves the best results among the evaluated baselines on the open-ended benchmarks. Ablation studies confirm the individual and complementary contributions of both modules, demonstrating LEGO's effectiveness in improving LLMs' complex legal reasoning ability. Code and dataset can be found in the link: https://github.com/BLK-WHT/LEGO
Qingjing Chen, Junkai Zhang, Shaochun Wang +7
Alma AI, University of Bologna, Italy · Tsinghua University, China · Modelbest Inc. +2
In the legal domain, LLMs can improve reliability and interpretability by generating explicit reasoning traces. However, training LLMs to produce high-quality reasoning traces remains challenging. Existing supervised fine-tuning methods struggle when reasoning traces are significantly longer than final answers, as the learning signal for the answer becomes diluted. Meanwhile, reinforcement learning methods such as Group Relative Policy Optimization also have several limitations, including costly reward design, performance plateaus, and reward hacking. We therefore propose a two-stage training framework. In Stage I, JurisCoT-SFT employs a length-normalized dual objective to balance learning signals between reasoning traces and final answers. In Stage II, Lifecycle-Aware Backtrackable Policy Optimization dynamically evaluates and retires auxiliary reward signals based on their impact on final evaluation objectives, enabling efficient reward utilization. Our framework substantially improves legal reasoning performance on relevant tasks selected from multiple legal benchmarks, including ChJGBench, LawBench, and LegalBench. Experimental results demonstrate consistent improvements in both reasoning trace quality and final answer accuracy, while our models remain competitive with larger models.
Yinghao Hu, Qian Xiao, Yuting Huang +7
Zhejiang University, Hangzhou, China · Alibaba Group, Hangzhou, China · Northeastern University, Shenyang, China +1