Organizations: Zhejiang University, Hangzhou, China · National University of Defense Technology, Changsha, China · Alibaba Group, Hangzhou, China · Northeastern University, Shenyang, China
With the rapid development of LLMs, increasing attention has been paid to whether they can reliably generate complete and legally valid judicial documents. This calls for a benchmark capable of evaluating whether models can accurately apply statutes and resolve outcome-critical issues, including claims, charges, penalties, and monetary obligations. However, existing benchmarks typically focus on isolated legal tasks or coarse-grained evaluation. We introduce ChJGBench, a benchmark for generating the reasoning and dispositive sections of Chinese judgments across civil, criminal, and administrative cases. Built from 3,462 real-world judgments, ChJGBench provides a case-type-aware diagnostic evaluation framework comprising 13 metric types and adopts a hybrid paradigm. Experiments reveal no uniform advantage for legal-domain models over general-purpose models, with model strengths varying across diagnostic dimensions. Performance is generally stronger on criminal cases, while civil and administrative cases continue to present substantial challenges.
Figures & tables
Benchmark
M-Type
Samp.
Dims.
Eval. Method
JuDGE
×
501
4
Det.
CaseGen
×
500
15
LLM
J1-EVAL
×
162
6
Det. + LLM
PLawBench
N/A
40
6
Rubric + LLM
ChJGBench
✓
3,462
13
Diag. + LLM
Table 1: Comparison of ChJGBench with prior legal judgment and document generation benchmarks. M-Type : all three litigation types; Samp. : distinct source cases; Dims. : dimensions for document drafting and judgment generation; Det. : deterministic evaluation, e.g., exact, pattern, or regex matching; Diag. : diagnostic evaluation with litigation-type-specific criteria.
Abbr.
Dimension
Section
Law
Statutory Provision Application
4.3
ISS
Disputed Issues Identification
4.3
JUS
Judicial Justification
4.3
Charge
Criminal Charge Prediction
4.4
Fine
Monetary Fine
4.4
REC
Recidivism Assessment
4.4
Table 2: Case-type-aware evaluation dimensions in ChJGBench, grouped by background color into shared, criminal, civil, and administrative dimensions. Each entry reports the metric abbreviation and points to the section where the metric is defined.
Figure 1: Structure of a legal judgment document.
Case / Metric
Model
WisdomI.
LegalOne
Claude Sonnet 4.5
DS-R1
Gemini 3 Pro
GPT-5.4
Grok-4
LLaMA 405B
Qwen3 Max
Criminal
Law
0.393 ± 0.006
0.718 ± 0.007
0.677 ± 0.003
0.687 ± 0.004
0.746 ± 0.002
0.642 ± 0.003
0.610 ± 0.002
0.504 ± 0.006
0.643 ± 0.003
JUS
0.369 ± 0.025
0.714 ± 0.002
0.734 ± 0.009
0.758 ± 0.002
0.738 ± 0.001
0.702 ± 0.001
0.739 ± 0.002
0.639 ± 0.003
0.734 ± 0.005
ISS
0.493 ± 0.020
0.802 ± 0.003
0.812 ± 0.005
0.814 ± 0.006
0.823 ± 0.000
0.786 ± 0.002
0.824 ± 0.002
0.702 ± 0.010
0.797 ± 0.003
Charge
0.468 ± 0.028
0.973 ± 0.000
0.974 ± 0.003
0.970 ± 0.004
0.974 ± 0.003
0.973 ± 0.002
0.977 ± 0.001
0.960 ± 0.004
0.972 ± 0.004
Table 3: Model performance across evaluation dimensions and case types. Values are reported as mean ± sample standard deviation over three runs. Macro Avg is computed per run as the unweighted mean of the unrounded component scores within each case type. Values are rounded to three decimals. Bold and underlined values indicate the best and second-best means, respectively; means that round to the same three-decimal value share the same rank. Abbreviations: WisdomI. = WisdomInterrogatory; DS-R1 = DeepSeek-R1.
Figure 2: Task difficulty and cross-model variability across evaluation dimensions.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Setting
Function
Power-temperature coefficient τ
0.5
Controls the degree of smoothing applied to the empirical category distribution.
Interpolation coefficient λ
0.3
Controls the balance between the empirical distribution and the temperature-smoothed distribution.
Appendix
Table 4: Hyperparameter settings used in the coverage-oriented sampling strategy. Setting τ=0.5 applies a square-root transformation to the empirical category distribution, reducing the relative dominance of high-frequency categories and increasing the relative sampling probabilities of low-frequency categories. With λ=0.3 , the final sampling distribution combines 30% of the empirical distribution with 70% of the temperature-smoothed distribution, thereby emphasizing long-tail coverage while retaining the overall structure of the empirical distribution.
Dimension
LLM Extraction
LLM Semantic Matching
Final Scoring Method
Main-text Section
Law
✓
×
Case-level set F 1
Sec. 4.3
ISS
✓
✓
Soft semantic F 1
Sec. 4.3
JUS
✓
✓
Soft semantic F 1
Sec. 4.3
Charge
✓
×
Defendant-level set F 1
Sec. 4.4
Fine
✓
×
NVS
Sec. 4.4
REC
✓
×
Binary macro-F 1
Sec. 4.4
Appendix
Table 5: Evaluation components and scoring methods for each ChJGBench evaluation dimension. LLM extraction denotes the structured extraction or normalization of predefined legal elements. LLM semantic matching denotes constrained element-level comparison between the generated and reference judgments.
Dimension
Cohen’s κ
Krippendorff’s α
ISS
0.8203
0.8216
JUS
0.7036
0.7047
DIS
0.8909
0.8919
Appendix
Table 6: Dimension-specific agreement between Qwen3-Plus and the human majority annotations.
Family
Metrics
Legal Basis
Law
Issue & Qualification
Charge, ISS SUP Adm , SUP Civ
Judicial Reasoning
JUS, REC MIT
Disposition & Quantification
Fine, Pen_M, DIS PAY Adm , PAY Civ
Appendix
Table 7: Legal ability family taxonomy used for task-family analysis.
Statistic (abbr.)
Value
# Samples
3,462
Lang.
Chinese
Prompt len.
2,562.95 characters
Reason len.
595.54 characters
Judgment len.
119.94 characters
Cited provisions
4.68
Appendix
Table 8: Basic statistics of the benchmark dataset. Abbreviations: average prompt length (Prompt len.), average length of the Reasoning section (Reason len.), average length of the Judgment section (Judgment len.), and average number of cited legal provisions (Cited provisions).
Figure 3: Case study comparing different evaluation metrics on a legally inconsistent judgment pair.