Organizations: School of Artificial Intelligence, Beihang University, Beijing, China · KoGuan School of Law, Shanghai Jiao Tong University, Shanghai, China · School of Criminal Law, China University of Political Science and Law, Beijing, China
Large language models (LLMs) have demonstrated strong reasoning abilities across specialized domains, motivating research into their application to legal reasoning. However, existing legal benchmarks often conflate factual recall with genuine inference, fragment the reasoning process, and overlook the quality of reasoning. To address these limitations, we introduce MSLR, the first Chinese multi-step legal reasoning dataset grounded in real-world judicial decision making. MSLR adopts the IRAC framework (Issue, Rule, Application, Conclusion) to model structured expert reasoning from official legal documents. In addition, we design a scalable Human-LLM collaborative annotation pipeline that efficiently produces fine-grained step-level reasoning annotations and provides a reusable methodological framework for multi-step reasoning datasets. Evaluation of multiple LLMs on MSLR shows only moderate performance, highlighting the challenges of adapting to complex legal reasoning. Further experiments demonstrate that Self-Initiated Chain-of-Thought prompts generated by models autonomously improve reasoning coherence and quality, outperforming human-designed prompts. MSLR contributes to advancing LLM reasoning and Chain-of-Thought strategies and offers open resources for future research. The dataset and code are available at https://github.com/yuwenhan07/MSLR-Bench and https://law.sjtu.edu.cn/flszyjzx/index.html.
Figures & tables
Figure 1: A schematic of the MSLR construction and evaluation framework.
Characteristic
MSLR
Document time span
2005–2024
Total number of documents
1,389
Avg. characters per document (chars)
2,516
Avg. structured fields per document
43.03
Total number of structured entries
59,771
Avg. fill rate of key fields (%)
90.58
Table 1: Descriptive statistics of the MSLR dataset.
Model
FCR
Overall Acc
Llama3-8B-Chinese
76.37
57.43
GLM4-9B
68.85
67.01
Qwen2.5-7B
58.12
57.69
Qwen2.5-72B
57.26
58.69
Llama3.3-70B-Instruct
57.19
64.72
DeepSeek-V3
69.65
77.43
Table 2: Comparative performance of LLMs on the automatic annotation task. The first block of rows corresponds to General LLMs, the second block corresponds to Legal LLMs, and the third block corresponds to Reasoning LLMs.
Figure 2: Pearson correlation analysis and linear fit between IRAC Recall and LLM Score.
Figure 3: Zero-Shot vs. One-Shot performance on the legal reasoning task.
Model
Standard Input
Human-Designed CoT
Delta
IRAC Recall
LLM Score
IRAC Recall
LLM Score
IRAC Recall
LLM Score
Llama3-8B-Chinese
46.76
29.63
53.93
60.37
7.17
30.74
GLM4-9B
50.79
52.38
57.00
70.88
6.21
18.50
Qwen2.5-7B
56.78
65.73
57.70
65.44
0.92
-0.29
Qwen2.5-72B
61.25
68.50
58.09
57.70
-3.16
-10.80
Llama3.3-70B-Instruct
53.47
58.06
58.87
79.73
5.40
21.67
Table 3: Comparing Standard Input and Human-Designed CoT on the legal reasoning task.
Figure 4: H-CoT vs. S-CoT on the legal reasoning task.
Figure 5: Typology and coverage of Self-Initiated CoT reasoning traces in three mainstream LLMs.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
File from Lawyer LLaMA
ROUGE-1 F1
ROUGE-2 F1
ROUGE-L F1
alpaca_gpt4_data_zh.json
0.0303
0.0033
0.0247
fakao_gpt4.json
0.0171
0.0008
0.0155
judical_examination_v2.json
0.0083
0.0007
0.0071
legal_counsel_multi_turn_with_article_v2.json
0.0032
0.0002
0.0029
legal_counsel_v2.json
0.0168
0.0026
0.0144
zixun_gpt4.json
0.0017
0.0001
0.0014
Appendix
Table 4: Document similarity between MSLR and Lawyer LLaMA.
Field Type
Section
Field Name
Exa-Fields
Name of the Traded Stock
Insider Information Determination
Date of Insider Information Formation
Date of Public Disclosure
Name
Basic Information of the Party
Gender
Year of Birth
Appendix
Table 5: Field categorization for evaluation.
Category
Model
Zero-shot
One-shot
FCR
Overall Acc
FCR
Overall Acc
General LLMs
Llama3-8B
76.37
57.43
76.40
58.50
GLM4-9B
68.85
67.01
61.37
57.17
DeepSeek-V3
69.65
77.43
73.07
75.11
Legal LLMs
Cleverlaw
78.43
61.79
64.76
55.73
Lawyer-LLM
70.13
57.49
83.77
58.91
Appendix
Table 6: Zero-shot vs. One-shot performance on the automatic annotation task.
Analysis Dimension
Standard Output
Human-Designed Output
Evaluation Conclusion
Case Fact Recognition
Accurate and complete, covering the full process of insider information formation and trading
Omits the early stage of insider information formation, which does not affect substantive determination
Roughly equivalent
Insider Information Identification
Correctly cites Article 75 of the 2005 Securities Law; applies the historical law accurately
Incorrectly cites the 2019 version and Article 80 (misapplies information disclosure clauses)
Std clearly superior; CoT lacks legal clarity
Insider Awareness Analysis
Reasonably judged based on Article 74, considering official position and timing of awareness
Introduces a “time of awareness” path, but the logical chain is unclear and partially fabricated