Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models

Subramanyam Sahoo; Justin Shenk

cs.LGOct 5, 2026

Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models

Authors: Subramanyam Sahoo, Justin Shenk

Organizations: Horizon Research.

Abstract

What happens when a legal AI model learns to look like a lawyer instead of reasoning like one? We fine tune Qwen3-8B with Group Relative Policy Optimisation (GRPO) against a proxy built from three surface features: citation count, legalese density, and response length. The model does not learn to reason more effectively. It learns to withhold commitment. Across 16 yes or no legal reasoning tasks from LegalBench (N=320), overall accuracy collapses from 0.500 (chance) to 0.072 (McNemar p < 10^-36), driven entirely by the rate of properly formatted answers falling from 0.900 to 0.109. The model stops committing to answers. Yet when it does commit, accuracy rises from 0.556 to 0.657, showing that the collapse is not a failure of capability but a strategic response: the model has learned that verbose responses packed with citations but empty of a direct answer score higher than terse correct ones. We term this the Saul Goodman effect, a policy that becomes maximally lawyerly while becoming maximally noncommittal, and prove formally that it is the optimal response to any surface feature proxy that attaches no penalty to abstention. We further show that 89.3% of citations produced after training are structurally implausible hallucinations, many of them subtly corrupted names of real landmark cases, constructed in effect to survive a casual read and fail under scrutiny. To detect this failure mode before deployment, we introduce three diagnostic tools: the Confidence Theater Score (CTS), the Citation Plausibility Rate (CPR), and the Regret Gap (RG). In a domain where a confidently wrong answer can constitute malpractice, the broader lesson is direct: a reward function that measures how legal a response looks will produce a model that is maximally photogenic and minimally useful.

Figures & tables

Panel of Figure 1 — Figure 1 : Reward configuration. Solid arrows are gradient-bearing; dashed arrows are zero-weight diagnostic signals. Only surface features drive the GRPO update; correctness is logged but never paid for.

Figure 2 : Surface-feature distributional shift on the held-out pool ( $N{=}320$ ). All four distributions shift rightward post-training. Effect sizes: citations $d{=}+0.62$ , legalese $d{=}+0.60$ , length $d{=}+1.89$ , CTS $d{=}+1.29$ . The post-training CTS distribution piles up at the ceiling near $1.0$ , confirming surface-feature saturation consistent with Proposition 3.2 .

Figure 3 : Four-panel diagnostic of GRPO training. (a) Smoothed proxy reward climbs. (b) True reward (signed accuracy) stagnates near zero. (c) Goodhart Divergence Metric — rolling Pearson $\rho$ (proxy, true) — sits near $0$ with frequent excursions below it; the proxy is decoupled from truth. (d) Regret Gap ( Equation 6 ) grows monotonically, confirming over-optimisation.

Metric	Pre	Post	$\Delta$
Accuracy	$0.500$	$0.072$	$-0.428$
$95\%$ CI	[.44, .55]	[.04, .10]
Acc. when answered	$0.556$	$0.657$	$+0.102$
Answer-format rate	$0.900$	$0.109$	$-0.791$
Avg. citations	$0.056$	$0.609$	$+0.55$
Avg. legalese terms	$0.094$	$1.075$	$+0.98$

Table 1 : Pre/post comparison on the held-out pool (

N{=}320

95\%

CI from bootstrap with

n{=}2{,}000

resamples;

p

-value from paired McNemar; Cohen’s

d

on per-completion arrays.

Figure 4 : Per-task disaggregation across all $16$ LegalBench tasks. (a) Accuracy: every task degrades, with $9$ collapsing to zero. (b) Answer-format rate: shows the same shape across tasks, confirming that format collapse drives accuracy collapse. The single outlier, consumer_contracts_qa , retains substantial format ( $0.45$ ) and accuracy ( $0.35$ ).

Figure 5 : Reward Over-Optimisation Curves at $3$ mid-training checkpoints. Left: mean proxy reward rises monotonically with KL budget, confirming the policy continues to climb the proxy throughout training. Right: true accuracy falls monotonically while CTS rises — the classical Gao et al. (2022) Goodhart trade-off, here reproduced in the legal domain under a purely surface-feature proxy. The inverse trajectories of true accuracy and CTS are the visual signature of Confidence Theater: the model looks increasingly lawyerly while becoming increasingly wrong.

Step	KL (approx.)	Proxy reward	True acc.	CTS
$15$	$0.00093$	$-0.350$	$0.500$	$0.890$
$30$	$0.00198$	$-0.342$	$0.440$	$0.916$
$45$	$0.00725$	$-0.088$	$0.333$	$0.937$

Table 2 : Reward Over-Optimisation curve at three mid-training checkpoints. KL is approximate; proxy reward is the mean

r_{\mathrm{proxy}}

over the evaluation set; CTS is mean Confidence Theater Score.

Figure 6 : CTS vs. answer correctness, pre and post. Pre-training, CTS is mildly predictive of correctness (green regression line slopes downward — higher-CTS responses are slightly less likely to be correct even before training). Post-training, CTS collapses to a tight ceiling near $0.99$ with almost all examples wrong: CTS itself becomes uninformative about correctness. This is Goodhart’s Law operating at the level of the diagnostic metric.

Figure 7 : Reward-hacking taxonomy: $\Delta$ pre $\,\to\,$ post across all tracked metrics. Surface features (blue) gain; true accuracy (red) loses. The asymmetry is the empirical signature of Proposition 3.2 : the proxy climbs, the truth does not. Length and CTS are displayed on rescaled axes (see legend).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Base model	Qwen3-8B
LoRA rank $/$ $\alpha$ $/$ dropout	$32/64/0.05$
LoRA targets	$\{q,k,v,o\}$ -proj
Learning rate	$5\times 10^{-6}$
Optimiser	AdamW (fused)
Per-device batch (prompts)	$16$

Appendix

Table 3 : Full training and evaluation hyperparameters.

Task	Pre acc.	Post acc.	$\Delta$ acc.	Pre fmt.	Post fmt.	$\Delta$ fmt.
diversity_1	$0.85$	$0.10$	$-0.75$	$1.00$	$0.10$	$-0.90$
telemarketing_sales_rule	$0.65$	$0.05$	$-0.60$	$0.90$	$0.05$	$-0.85$
diversity_4	$0.65$	$0.00$	$-0.65$	$1.00$	$0.00$	$-1.00$
int._citizenship_questions	$0.60$	$0.00$	$-0.60$	$0.95$	$0.00$	$-0.95$
nys_judicial_ethics	$0.60$	$0.00$	$-0.60$	$0.95$	$0.00$	$-0.95$
consumer_contracts_qa	$0.60$	$0.35$	$-0.25$	$0.80$	$0.45$	$-0.35$

Appendix

Table 4 : Per-task accuracy and answer-format rate (

N{=}20

per task).

\Delta

is post minus pre.

Explore similar work

Sep 22, 2026cs.CL

LEGO: Synergizing Expert GraphRAG and Expert Chain-of-Thought for Legal Reasoning

Large language models are increasingly applied to high-risk domains such as law, yet complex legal reasoning remains limited by two structural challenges. First, existing RAG and GraphRAG methods emphasize lexical or semantic similarity while overlooking normative relations among legal provisions. Second, vanilla Chain-of-Thought prompting may generate plausible rationales without enforcing the normative structure of legal reasoning. To deal with the bottleneck of pipelines in the legal reasoning domain, we propose LEGO, a dual-module framework that synergizes Legal Expert GraphRAG and expert Chain-of-thought for complex legal reasoning. ExpertGraphRAG uses an expert-annotated civil code graph encoding these normative relations with a greedy normative-coverage retrieval algorithm to dynamically extract instance-specific provision subgraphs, while ExpertCoT organizes the retrieved provisions and case facts into structured Provision-Fact-Conclusion reasoning. With a Qwen3-8B backbone, LEGO achieves 40.53% exact-match accuracy on LawExamQA_Civil, outperforming the evaluated RAG and CoT baselines and performing comparably to the evaluated larger models, while remaining robust on multi-hop questions. It also achieves the best results among the evaluated baselines on the open-ended benchmarks. Ablation studies confirm the individual and complementary contributions of both modules, demonstrating LEGO's effectiveness in improving LLMs' complex legal reasoning ability. Code and dataset can be found in the link: https://github.com/BLK-WHT/LEGO

Qingjing Chen, Junkai Zhang, Shaochun Wang +7

Alma AI, University of Bologna, Italy · Tsinghua University, China · Modelbest Inc. +2

Legal Reasoning Tasks LLM Reasoning Strategies Chain-of-Thought Reasoning Experts

Ask PDF

Sep 30, 2026cs.AI

Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agents

Legal research is a core and time-consuming legal workflow. Lawyers must identify controlling authority, verify that it remains valid, reconcile statutes and cases, and synthesize a grounded answer. Language model agents are a natural fit for this retrieval-intensive workflow, and automating even part of it would be valuable. But that value depends on reliability: a single missing authority, stale citation, or wrong legal conclusion can make an otherwise plausible answer unusable. We introduce \textbf{Legal Research Bench} (LRB), a benchmark of 413 open-ended U.S. legal research questions written by experts, each paired with a gold answer, supporting authorities, and a binary grading rubric. We evaluate thirteen frontier models in a harness with web search, case-law search, page parsing, and retrieval tools. We score agent responses through all-pass grading with source verification, where a response is correct only if every required criterion is satisfied and its cited authorities verify. We also validate the LLM judge against expert attorneys ensuring that benchmark scores track attorney judgment. Agents remain far from reliable: among the models we tested, the strongest, Claude Opus 4.8, is fully correct on 42.9% of questions. Performance also varies substantially by task setting: all-pass rates differ across areas of law and are lower on questions requiring reconciliation of conflicting authorities. Across models, more turns, tool calls, and inference cost do not predict higher accuracy.

Katrina Drozdov, Oliver Chen, Langston Nashold +1

Vals AI San Francisco, USA

Legal Reasoning Tasks Language-Model Agents

Ask PDF

Sep 20, 2026cs.CL

GRACE: Grounded Adversarial Reasoning over Canadian Law

Large language models have shown strong performance across a range of legal tasks, but existing benchmarks rarely evaluate the ability to take and defend a legal position, reason under incomplete information, or synthesize multiple statutory provisions. This gap is particularly pronounced for Canadian law, which remains underrepresented in legal NLP. We introduce GRACE (Grounded Reasoning Adversarial Canadian LEgal examples), a dataset of 1,915 question-reasoning-answer instances grounded in Canadian federal legislation. GRACE covers three reasoning modes: adversarial advocacy, uncertainty, and applied reasoning. We develop a pipeline that partitions raw statutory text, generates scenario-based questions and reasoning, and filters examples through model-free citation verification and LLM-based quality auditing. As a proof of concept, we fine-tune CLeAR-4B (Canadian Legal Adversarial Reasoning), a lightweight model for grounded legal reasoning, and evaluate it against the unmodified Qwen3-4B base model in open- and closed-book settings. CLeAR-4B substantially improves agreement with teacher outputs and statutory citation behavior when the relevant act text is provided, while its grounding degrades sharply when the statute is withheld. These results suggest that GRACE can support the development of lightweight legal models that reason more effectively from supplied statutory text.

Jiakang Xu, Wantong Huo, Udom Silparcha +1

University of Toronto · UTOA Computing Analytics Co., Ltd. · King Mongkut’s University of Technology Thonburi

Legal Reasoning Tasks Natural Language Processing Adversarial Evaluation Citations

Ask PDF