A sound judgment applies the law to established facts and weighs the circumstances in which they arose. However, existing methods swing between rigid statute matching and ungrounded discretion, benchmarks score a label or a rubric, and the experience that would supply the balance stays unverified. We formalize legal judgment as a reference-anchored task, whose object is a single decision that stays tied to the statute and to the circumstances at once. We introduce JusticeAxis, 256 real-world criminal cases from 18 countries with audio, image, and text evidence, and three lawyer-written judgments for every case: the recorded one and one for each failure. We further propose JusticeAgent, a harness whose element agents establish the facts and whose judge agent applies the law under skills carrying experience of the circumstances. Skills are distilled from execution trajectories and admitted only under Bayesian credible bounds. Experiments show that failure turns direction with scale: open-weight backbones drift to unsupported grounds, frontier models to the statutory default. We further verify that JusticeAgent, as a simple yet effective plugin, carries a frozen open-weight backbone to commercial level. Project resources are available at https://github.com/beita6969/JusticeAxis.
Figures & tables
Figure 1: Overview: the axis a judgment sits on (a), the two failure modes of existing systems, rigid statute matching (b) and ungrounded discretion (c), and the JusticeAgent framework (d).
Figure 2: The JusticeAgent harness. C1 : an orchestrator edits a shared Element Graph while one agent per offence element calls tools and returns findings until no node is open. C2 : the judge applies the law to the closed graph under contextual skills and grounding constraints. C3 : skills mined from trajectories are verified independently and retained, deferred, refined or pruned on credible bounds, not on judgment scores.
Given per instance
Benchmark
Jur.
Input
Law
Prec.
Bg.
Refs
Fail.
CAIL2018 [ 27 ]
1
T
1
LegalBench [ 10 ]
1
T
1
CrossLex [ 29 ]
3
T
✓
1
Multi-Legal-Bench [ 22 ]
6
T
1
CLAUSE [ 3 ]
1
T
✓
1
LLM
Table 1: Comparison with legal and multimodal benchmarks. A/I/T: audio, image, text; Law, Prec., Bg.: statutes, precedents, background; Refs: references per case; Fail.: who wrote the failure ones.
Table 2: Main results (128 evaluation cases). Acc, Fam, Disp and Sent. are the shares of cases with charge, charge family, disposition and sentence exactly correct, Avg their mean; Acc/Fam is exact charge among family-correct cases, Sent/Disp exact sentence among disposition-correct cases. Nearest anchor is the share landing closest to each reference under ρ , Pol. the signed share of misses toward the rigid pole, (Jrig−Jung)/(Jrig+Jung) . Miss is 100−J∗ . ± are Wilson 95% half-widths; Δ is the gain over the Qwen3.8-27B backbone, RMR the relative reduction of its misses. Higher is better except Jrig , Jung , Miss. † 304B mixture-of-experts.
Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative judgment elicits pairwise preferences between outputs. Although both methodologies are widely used, the choice between them is rarely justified. We release JudgmentBench, a benchmark of 30 real-world legal tasks, paired with 1,539 rubric scores and 1,530 pairwise preference judgments collected from practicing attorneys--including at major U.S. law firms--with substantial experience. The annotations constitute the first publicly available dataset in a high-expertise domain in which both supervision signals are elicited from the same experts on the same items. Using LLM-generated outputs at three constructed quality levels, we provide an initial empirical comparison: comparative judgments recover the intended quality ordering substantially better than rubrics under both a per-task rank-correlation metric (mean Spearman's rank correlation of 0.908 vs. 0.150, estimated difference = 0.758 [0.494, 1.021]) and a per-judgment pairwise win-rate metric (0.669 vs. 0.542, estimated difference = 0.127 [0.067, 0.186]), while requiring less than half the annotation time. The patterns hold for human annotators and LLM autograders. Beyond this initial comparison, the paired structure of the dataset supports a broader research agenda on how expert judgment should be elicited, aggregated, and used as supervision in domains without verifiable ground truth.
Legal outcome prediction must disentangle objective case facts from adjudicative context. Merit-based rulings rely on factual evidence while technical disposals may hinge on judicial discretion. We propose a Judge-Aware Gated Multi-Task Learning architecture that explicitly models this distinction. We introduce a fine-grained outcome taxonomy to supervise the encoder, enforcing a structural regularization that disentangles distinct semantic pathways. This granular legal curriculum enables our Gated Fusion mechanism to dynamically modulate reliance on judge identity. We evaluate our approach on 13,937 UK Employment Tribunal decisions. We benchmark our design against supervised fine-tuning (SFT) of a Gemma-4 26B-A4B backbone, in which judge identity and the taxonomy are injected as prompt tokens or autoregressive output targets. The two contextual signals compose only weakly when forced through a single autoregressive channel. In contrast, coupling a LoRA-adapted Gemma-4 encoder with our gated architecture defines a new state of the art on this benchmark while requiring an order of magnitude fewer trainable parameters than the generative SFT baselines, with gains concentrated on the most ambiguous and rarest outcome classes. Beyond accuracy, the architecture is interpretable; learned judge embeddings and calibration profiles localize the cases where adjudicative context drives the prediction. These results indicate that, for identity-conditioned classification of legal outcomes, the choice of conditioning interface dominates scale: differentiable structured composition yields more accurate, more parameter-efficient models than prompt-based composition over a substantially larger backbone.
Stanisław Sójka, Felix Steffek, Matthias Grabmair
School of Computation, Information, and Technology, Technical University of Munich, Germany · Faculty of Law, University of Cambridge
Reward models are central to preference alignment for legal tasks, yet existing reward model benchmarks are still primarily centered on general-domain preferences. In this work, we introduce LegalRewardBench, the first human-verified benchmark for evaluating reward models in the legal domain. The benchmark organizes legal preference evaluation into five capability levels: knowledge, understanding, analysis, decision, and writing, and preference examples are constructed through rubric-guided automatic judging combined with human verification. We evaluate three families of judges, covering both general and fine-tuned models, and generative and discriminative architectures. Experimental results show that strong general-purpose generative models are currently the most robust legal reward models, while existing fine-tuned reward models transfer less reliably to legal preference evaluation. We also conduct detailed comparisons of input organizations and training strategies.
Hui Huang, Jie Zhou, Hongli Zhou +7
Harbin Institute of Technology · ModelBest Inc. · The University of Osaka