LexIssue: Benchmarking Legal Issue Identification in Chinese Civil Litigation

Huiyuan Xie; Yuqin Huang; Zhicheng Hao; Yida Cai; Shaochun Wang; Zhenghao Liu; Yuxiao Ye

cs.CLSep 2, 2026

LexIssue: Benchmarking Legal Issue Identification in Chinese Civil Litigation

Authors: Huiyuan Xie, Yuqin Huang, Zhicheng Hao, Yida Cai, Shaochun Wang, Zhenghao Liu, Yuxiao Ye

Organizations: Tsinghua University · Peking University · Modelbest Inc. · Northeastern University

Abstract

Identifying the issues disputed between litigating parties is a crucial component of real-world litigation. However, legal issues remain comparatively underexplored in legal AI research. In this work, we study the computational modelling of legal issue identification in litigation. We introduce a legally grounded hierarchical schema that represents legal issues through both free-form issue descriptions and structured legal categories, and formulate legal issue identification as two complementary tasks: legal issue generation and legal issue classification. Based on this formulation, we construct LexIssue, a benchmark containing 430 real-world Chinese civil litigation cases and 1,303 expert-annotated disputed legal issues. We further develop an issue-centric legal knowledge base spanning 27 causes of action and 441 candidate legal issue entries to support retrieval-augmented reasoning. Experimental results across a diverse set of models show that retrieval-augmented generation using the constructed legal issue knowledge base consistently improves performance in identifying disputed legal issues and their corresponding legal attributes.

Figures & tables

Figure 1: Overview of the legal issue identification process . Left : An illustrative product liability dispute in which the claimant alleges that a freezer purchased from the defendant is defective and seeks compensation for losses, while the defendant denies the existence of defects, disputes causation and fault, and refuses compensation. The issue analysis content is extracted from the judge’s reasoning section in the original court judgment. Right : An illustration of the legal issue identification task, where models identify disputed legal issues from the litigated case and generate both descriptive issue summaries and structured legal category labels. In the illustrated three-tier legal category hierarchy, the first-tier labels law & fact and law correspond to mixed issues of law and fact and substantive issues of law , respectively. The second-tier labels elements and effect correspond to elements of a claim and legal effect . The third-tier labels act , causation , fault , and liability presence correspond to the schema categories tortious act , causation , fault , and presence or absence of liability , respectively.

Model	LIG	LIC
Model	LIG	Tier-1	Tier-2	Tier-3
GPT-5.2	0.542	0.346	0.255	0.185
w/ LP	0.475	0.279	0.191	0.133
w/ RAG	0.634	0.460	0.335	0.243
Gemini-3-Flash	0.701	0.581	0.459	0.357
w/ LP	0.699	0.544	0.394	0.292

Table 1: Evaluation results on the LexIssue benchmark. LIG denotes the legal issue generation task, whilst LIC denotes the legal issue classification task. Entries labelled with the model name alone (e.g., GPT-5.2) correspond to the zero-shot inference setting. The variant “w/ LP” denotes models evaluated under the legal prompting setting, while “w/ RAG” denotes models augmented with RAG using the external legal issue knowledge base.

Rater	Sim.	Align.	Agr.
Qwen-3.5-27B	0.965	0.852	0.668
ROUGE_1	0.870	0.635	0.178
gte-large-zh	0.870	0.665	0.254

Table 2: Comparison of representative instances for three categories of rating approaches: LLM-as-a-Judge (Qwen-3.5-27B), NLG-based metrics (ROUGE_1), and embedding-based methods (gte-large-zh). Sim. denotes the accuracy achieved in the similarity mapping step; Align. denotes the accuracy achieved in the alignment verification step; Agr. denotes the overall agreement with human judgments.

Model	Tier-1	Tier-2	Tier-3
GPT-5.2	0.971	0.707	0.513
Gemini-3-Flash	0.900	0.709	0.550
DeepSeek-R1	0.877	0.653	0.421
DeepSeek-V4-Flash	0.851	0.657	0.467
Qwen-3-8B	0.908	0.583	0.310
Llama-3.1-8B	0.838	0.658	0.326

Table 3: Model performance on legal issue classification under Tier-1, Tier-2, and Tier-3 settings, calculated only on the subset of successfully aligned issues.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: The proposed dual-layer hierarchical schema for representing legal issues.

Figure 3: Prompt specification (Part 1/2): Task overview and execution steps (Sections I & II). Covers the role/task definition, input variables, and the three execution steps (Substantive/Procedural $\to$ Legal/Factual $\to$ Legal Logic Subdivision) with the Fact-Mapping Procedure for mixed Legal–Factual pairs. The Chain-of-Thought analysis is written exclusively in JSON fields.

Figure 4: Prompt specification (Part 2/2): Scoring framework, semantic adjustments, and evaluation example (Sections III, IV & V). Covers the final assessment step (Step 4), the quantitative scoring framework with the 6-level benchmark table, semantic distance adjustment rules (terminology, redundancy, bonus), and a fully worked example with complete classification path and score breakdown.

Figure 5: Illustration of the alignment workflow used in the issue-level alignment verification stage. The reference and predicted issues are compared by traversing the hierarchical legal issue taxonomy, with the alignment level determined by the deepest hierarchical tier at which their category paths match. The resulting alignment level forms part of the rubric used by the LLM judge to determine whether the prediction and reference represent the same underlying disputed legal issue.

Explore similar work

LePREC: Reasoning as Classification over Structured Factors for Assessing Relevance of Legal Issues
Apr 21, 2026Fanyu Wang, Xiaoxi Kang, Paul Burgess +6Neuro-Symbolic Reasoning Legal Reasoning
Faculty of Information Technology, Monash University, Australia · School of Information Technology, Monash University Malaysia · Faculty of Law, Monash University, Australia +1
InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries
Aug 20, 2026Samuel J. Vincent, Daniel Calloway, Fangyi Yu +2Legal QA LLM Evaluation
Thomson Reuters Foundational Research, 5 Canada Square, London · Imperial College London
ViLegalExpert: A Large-Scale Benchmark for Vietnamese Legal Retrieval and Question Answering from Real-World Consultations
Sep 30, 2026Dat Tien Nguyen, Nghia Hieu Nguyen, Anh Thi-Hoang Nguyen +3Legal QA Legal NLP
University of Information Technology · Vietnam National University, Ho Chi Minh city, Viet Nam

Apr 21, 2026J/K move · Enter open · S save

LePREC: Reasoning as Classification over Structured Factors for Assessing Relevance of Legal Issues

Fanyu Wang, Xiaoxi Kang, Paul Burgess, Aashish Srivastava+5

Faculty of Information Technology, Monash University, Australia · School of Information Technology, Monash University Malaysia · Faculty of Law, Monash University, Australia+1

Neuro-Symbolic Reasoning Legal Reasoning

Open paper AskPDF arXiv

More than half of the global population struggles to meet their civil justice needs due to limited legal resources. While Large Language Models (LLMs) have demonstrated impressive reasoning capabilities, significant challenges remain even at the foundational step of legal issue identification. To investigate LLMs' capabilities in this task, we constructed a dataset from 769 real-world Malaysian Contract Act court cases, using GPT-4o to extract facts and generate candidate legal issues, annotated by senior legal experts, which reveals a critical limitation: while LLMs generate diverse issue candidates, their precision remains inadequate (GPT-4o achieves only 62%). To address this gap, we propose LePREC (Legal Professional-inspired Reasoning Elicitation and Classification), a neuro-symbolic framework combining neural generation with structured statistical reasoning. LePREC consists of: (1) a neuro component leverages LLMs to transform legal descriptions into question-answer pairs representing diverse analytical factors, and (2) a symbolic component applies sparse linear models over these discrete features, learning explicit algebraic weights that identify the most informative reasoning factors. Unlike end-to-end neural approaches, LePREC achieves interpretability through transparent feature weighting while maintaining data efficiency through correlation-based statistical classification. Experiments show a 30-40% improvement over advanced LLM baselines, including GPT-4o and Claude, confirming that correlation-based factor-issue analysis offers a more data-efficient solution for relevance decisions.

Abstract

Explore similar work

LePREC: Reasoning as Classification over Structured Factors for Assessing Relevance of Legal Issues

InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries

ViLegalExpert: A Large-Scale Benchmark for Vietnamese Legal Retrieval and Question Answering from Real-World Consultations