cs.AINov 11, 2025

Benchmarking Multi-Step Legal Reasoning and Analyzing Chain-of-Thought Effects in Large Language Models

Authors: Wenhan Yu, Xinbo Lin, Lanxin Ni, Jinhua Cheng, Lei Sha

Organizations: School of Artificial Intelligence, Beihang University, Beijing, China · KoGuan School of Law, Shanghai Jiao Tong University, Shanghai, China · School of Criminal Law, China University of Political Science and Law, Beijing, China

Abstract

Large language models (LLMs) have demonstrated strong reasoning abilities across specialized domains, motivating research into their application to legal reasoning. However, existing legal benchmarks often conflate factual recall with genuine inference, fragment the reasoning process, and overlook the quality of reasoning. To address these limitations, we introduce MSLR, the first Chinese multi-step legal reasoning dataset grounded in real-world judicial decision making. MSLR adopts the IRAC framework (Issue, Rule, Application, Conclusion) to model structured expert reasoning from official legal documents. In addition, we design a scalable Human-LLM collaborative annotation pipeline that efficiently produces fine-grained step-level reasoning annotations and provides a reusable methodological framework for multi-step reasoning datasets. Evaluation of multiple LLMs on MSLR shows only moderate performance, highlighting the challenges of adapting to complex legal reasoning. Further experiments demonstrate that Self-Initiated Chain-of-Thought prompts generated by models autonomously improve reasoning coherence and quality, outperforming human-designed prompts. MSLR contributes to advancing LLM reasoning and Chain-of-Thought strategies and offers open resources for future research. The dataset and code are available at https://github.com/yuwenhan07/MSLR-Bench and https://law.sjtu.edu.cn/flszyjzx/index.html.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models

    Apr 26, 2026Tianchun Li, Haochen Liu, Vishwa Pardeshi +5Synthetic Data GenerationSmall Language Models

  2. LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning

    Sep 22, 2026Qingjing Chen, Junkai Zhang, Shaochun Wang +7LLM Reasoning with GraphsGraph Retrieval-Augmented Generation

  3. Towards Better Legal Reasoning LLMs: Signal Balancing and Reward Scheduling

    Nov 11, 2026Yinghao Hu, Qian Xiao, Yuting Huang +7Legal ReasoningStructured Reasoning in Language Models