Organizations: Zhejiang University, Hangzhou, China · National University of Defense Technology, Changsha, China · Alibaba Group, Hangzhou, China · Northeastern University, Shenyang, China
With the rapid development of LLMs, increasing attention has been paid to whether they can reliably generate complete and legally valid judicial documents. This calls for a benchmark capable of evaluating whether models can accurately apply statutes and resolve outcome-critical issues, including claims, charges, penalties, and monetary obligations. However, existing benchmarks typically focus on isolated legal tasks or coarse-grained evaluation. We introduce ChJGBench, a benchmark for generating the reasoning and dispositive sections of Chinese judgments across civil, criminal, and administrative cases. Built from 3,462 real-world judgments, ChJGBench provides a case-type-aware diagnostic evaluation framework comprising 13 metric types and adopts a hybrid paradigm. Experiments reveal no uniform advantage for legal-domain models over general-purpose models, with model strengths varying across diagnostic dimensions. Performance is generally stronger on criminal cases, while civil and administrative cases continue to present substantial challenges.
Figures & tables
Benchmark
M-Type
Samp.
Dims.
Eval. Method
JuDGE
×
501
4
Det.
CaseGen
×
500
15
LLM
J1-EVAL
×
162
6
Det. + LLM
PLawBench
N/A
40
6
Rubric + LLM
ChJGBench
✓
3,462
13
Diag. + LLM
Table 1: Comparison of ChJGBench with prior legal judgment and document generation benchmarks. M-Type : all three litigation types; Samp. : distinct source cases; Dims. : dimensions for document drafting and judgment generation; Det. : deterministic evaluation, e.g., exact, pattern, or regex matching; Diag. : diagnostic evaluation with litigation-type-specific criteria.
Abbr.
Dimension
Section
Law
Statutory Provision Application
4.3
ISS
Disputed Issues Identification
4.3
JUS
Judicial Justification
4.3
Charge
Criminal Charge Prediction
4.4
Fine
Monetary Fine
4.4
REC
Recidivism Assessment
4.4
Table 2: Case-type-aware evaluation dimensions in ChJGBench, grouped by background color into shared, criminal, civil, and administrative dimensions. Each entry reports the metric abbreviation and points to the section where the metric is defined.
Figure 1: Structure of a legal judgment document.
Case / Metric
Model
WisdomI.
LegalOne
Claude Sonnet 4.5
DS-R1
Gemini 3 Pro
GPT-5.4
Grok-4
LLaMA 405B
Qwen3 Max
Criminal
Law
0.393 ± 0.006
0.718 ± 0.007
0.677 ± 0.003
0.687 ± 0.004
0.746 ± 0.002
0.642 ± 0.003
0.610 ± 0.002
0.504 ± 0.006
0.643 ± 0.003
JUS
0.369 ± 0.025
0.714 ± 0.002
0.734 ± 0.009
0.758 ± 0.002
0.738 ± 0.001
0.702 ± 0.001
0.739 ± 0.002
0.639 ± 0.003
0.734 ± 0.005
ISS
0.493 ± 0.020
0.802 ± 0.003
0.812 ± 0.005
0.814 ± 0.006
0.823 ± 0.000
0.786 ± 0.002
0.824 ± 0.002
0.702 ± 0.010
0.797 ± 0.003
Charge
0.468 ± 0.028
0.973 ± 0.000
0.974 ± 0.003
0.970 ± 0.004
0.974 ± 0.003
0.973 ± 0.002
0.977 ± 0.001
0.960 ± 0.004
0.972 ± 0.004
Table 3: Model performance across evaluation dimensions and case types. Values are reported as mean ± sample standard deviation over three runs. Macro Avg is computed per run as the unweighted mean of the unrounded component scores within each case type. Values are rounded to three decimals. Bold and underlined values indicate the best and second-best means, respectively; means that round to the same three-decimal value share the same rank. Abbreviations: WisdomI. = WisdomInterrogatory; DS-R1 = DeepSeek-R1.
Figure 2: Task difficulty and cross-model variability across evaluation dimensions.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Setting
Function
Power-temperature coefficient τ
0.5
Controls the degree of smoothing applied to the empirical category distribution.
Interpolation coefficient λ
0.3
Controls the balance between the empirical distribution and the temperature-smoothed distribution.
Appendix
Table 4: Hyperparameter settings used in the coverage-oriented sampling strategy. Setting τ=0.5 applies a square-root transformation to the empirical category distribution, reducing the relative dominance of high-frequency categories and increasing the relative sampling probabilities of low-frequency categories. With λ=0.3 , the final sampling distribution combines 30% of the empirical distribution with 70% of the temperature-smoothed distribution, thereby emphasizing long-tail coverage while retaining the overall structure of the empirical distribution.
Dimension
LLM Extraction
LLM Semantic Matching
Final Scoring Method
Main-text Section
Law
✓
×
Case-level set F 1
Sec. 4.3
ISS
✓
✓
Soft semantic F 1
Sec. 4.3
JUS
✓
✓
Soft semantic F 1
Sec. 4.3
Charge
✓
×
Defendant-level set F 1
Sec. 4.4
Fine
✓
×
NVS
Sec. 4.4
REC
✓
×
Binary macro-F 1
Sec. 4.4
Appendix
Table 5: Evaluation components and scoring methods for each ChJGBench evaluation dimension. LLM extraction denotes the structured extraction or normalization of predefined legal elements. LLM semantic matching denotes constrained element-level comparison between the generated and reference judgments.
Dimension
Cohen’s κ
Krippendorff’s α
ISS
0.8203
0.8216
JUS
0.7036
0.7047
DIS
0.8909
0.8919
Appendix
Table 6: Dimension-specific agreement between Qwen3-Plus and the human majority annotations.
Family
Metrics
Legal Basis
Law
Issue & Qualification
Charge, ISS SUP Adm , SUP Civ
Judicial Reasoning
JUS, REC MIT
Disposition & Quantification
Fine, Pen_M, DIS PAY Adm , PAY Civ
Appendix
Table 7: Legal ability family taxonomy used for task-family analysis.
Statistic (abbr.)
Value
# Samples
3,462
Lang.
Chinese
Prompt len.
2,562.95 characters
Reason len.
595.54 characters
Judgment len.
119.94 characters
Cited provisions
4.68
Appendix
Table 8: Basic statistics of the benchmark dataset. Abbreviations: average prompt length (Prompt len.), average length of the Reasoning section (Reason len.), average length of the Judgment section (Judgment len.), and average number of cited legal provisions (Cited provisions).
Figure 3: Case study comparing different evaluation metrics on a legally inconsistent judgment pair.
Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored. We present TW-LegalBench that utilizes Taiwanese legal system's rich official corpus open to the public to fill the gap in evaluating LLMs on Taiwanese law, among common-law benchmarks that focus on English sources and civil-law benchmarks focusing on sources of Simplified Chinese. TW-LegalBench comprises three task types: (1) over 16,000 multiple-choice questions (MCQs) across five years of official examinations in 18 professional domains; (2) 117 open-ended essay questions (OEQs) from examinations for legal professionals with official scoring rubrics; and (3) more than 14,000 legal judgment prediction (LJP) instances covering hundreds of crime categories. We evaluate 13 LLMs using accuracy for MCQs, a decomposed LLM-as-Judge framework based on the scoring rubric points for OEQs, and metrics for sentencing accuracy and statute citation for LJP. Our results reveal that top-performing models exceed the passing threshold for qualified lawyers (passing rate: 11%) but fall short of that for judges and prosecutors (passing rate: 1~2%). For LJP, while models demonstrate reasonable verdict type accuracy and sentence prediction capability, they struggle to cite exact legal articles. These findings highlight that reliable legal text generation remains challenging for LLMs, even though their performance on qualification examinations approaches human level.
Fei-Yueh Chen, Chun Huang Lin, Chan Wei Hsu +4
University of Rochester Rochester, NY, USA · National Taiwan University Taipei, Taiwan · NVIDIA Santa Clara, CA, USA
As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses has become essential. These tasks require context-sensitive answers and allow little room for error, motivating fine-grained and diagnostic evaluation that can identify specific sources of response quality failures. We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks. LexRubric contains 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning and cover 14 legal scenarios. It further includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework, enabling accurate evaluation and diagnostic analysis across tasks and evaluation dimensions. To validate the reliability of the evaluation, we test multiple judge models and compare model-based judgments with human judgments. We further evaluate 18 recent general and legal-domain LLMs on LexRubric. Results show that different models exhibit distinct capability profiles, and that open-ended legal tasks remain challenging for current LLMs. Data is available at: https://github.com/foggpoy/LexRubric.
Yifan Chen, Haitao Li, Yiran Hu +6
1Beijing University of Posts and Telecommunications · 2Tsinghua University · University of Waterloo +1
Existing benchmarks for legal AI focus primarily on tasks where LLMs must produce legal arguments or documents, yet the capacity to judge such arguments -- weighing competing claims, applying doctrine to facts, and rendering reasoned decisions -- is arguably as fundamental to a well-functioning legal system as advocacy itself. We introduce Magis-Bench, a benchmark for evaluating LLMs on magistrate-level writing tasks derived from recent Brazilian competitive examinations for judicial positions. Magis-Bench comprises 74 questions from eight examinations conducted between 2023 and 2025, including discursive legal analysis questions with multi-turn structure and practical exercises requiring the composition of complete civil and criminal judicial sentences. We evaluate 23 state-of-the-art LLMs using an LLM-as-a-judge methodology with four independent frontier models as evaluators. Our results show strong inter-judge agreement (Kendall's W=0.984; pairwise Kendall's τ≥0.897), with Google's Gemini-3-Pro-Preview achieving the highest average score (6.97/10), followed by Gemini-3-Flash-Preview (6.67) and Claude-4.5-Opus (6.46). Even the best-performing models score below 70% of the maximum, indicating that judicial-level legal reasoning and writing remain challenging for current LLMs. We release the complete benchmark, model outputs, and evaluation code to support further research on legal AI capabilities.
Ramon Pires, Thales Sales Almeida, Celio Larcher Junior +6
Maritaca AI Campinas, São Paulo, Brazil · Jusbrasil Salvador, Bahia, Brazil