Building Legal Reward Models for Grounding and Abstention

Rilton Franzone; Valentin Noël; Puyu Wang; Philip Torr; Fabio J. Fehr

cs.CLSep 13, 2026

Building Legal Reward Models for Grounding and Abstention

Authors: Rilton Franzone, Valentin Noël, Puyu Wang, Philip Torr, Fabio J. Fehr

Organizations: Devoteam

Abstract

Large language models are increasingly used in high-stakes domains such as law, where systems must ground their reasoning in retrieved evidence and abstain when that evidence is insufficient. However, existing reward models are largely optimised for general preferences rather than contextual grounding, limiting their ability to evaluate these behaviours in retrieval-augmented generation (RAG) settings. We introduce a framework for transforming existing legal QA datasets into contextual preference data and use it to construct LegalRewardBench (LRB), a benchmark for evaluating grounded legal generation under noisy and insufficient retrieval conditions. Across general and legal contextual evaluation, we find that contextual DPO improves grounded evaluation, but performance is sensitive to preference-data construction. Length-balanced augmentation substantially improves grounded legal evaluation, with the strongest configuration combining length-balanced legal and general contextual preference data and improving performance by up to $\mathbf{+25.6}$ pp over baseline. We further find evidence of cross-jurisdiction transfer: models contextually refined primarily on Victorian criminal-law data improve grounded evaluation on external US legal benchmarks, including a $\mathbf{+16.2}$ pp improvement on Housing Statute QA. Together, these results provide a reproducible foundation for constructing and evaluating grounded legal reward models in retrieval-augmented settings.

Figures & tables

Figure 1: Legal preference-pair construction framework. Starting from Legal RAG Bench triples $(c_{\mathrm{gold}},q,a)$ (Section 2.1 ), we construct six retrieved-context variants by combining the gold passage with distractor passages to control answerability and retrieval noise (Section 2.2 ). Four LLMs generate candidate responses across all $(c,q)$ instances (Section 2.3 ). A judge-model assigns pointwise labels for answer behaviour, faithfulness, correctness, and completeness (Section 2.4 ). A hierarchical preference rule then converts labelled responses into contextual preference pairs $(c,q,r^{+},r^{-})$ for reward-model training and evaluation (Section 2.5 ).

Failure Type	Question ${q}$	Preferred Response ${r^{+}}$	Rejected Response ${r^{-}}$
Refusal	What sentence was imposed on the accused?	“The provided context is insufficient to answer this question.”	“The accused was sentenced to five years imprisonment.”
Faithfulness	Which Act introduced cyberstalking offences?	“The Crimes (Stalking) Act 2003 introduced cyberstalking offences.”	“The Crimes Amendment Act 2001 introduced cyberstalking offences.”
Correctness	Does Victoria protect against double jeopardy?	“Yes. Victoria protects against double jeopardy.”	“No. Victoria does not recognise double jeopardy protections.”
Completeness	Must jurors be excused after seeing pre-trial publicity?	“No. Judges can usually address prejudice through directions to the jury.”	“No.”

Table 1: Legal preference pairs. Representative pairwise preference examples. In the dataset,

r^{+}

and

r^{-}

are selected from generated candidate responses after pointwise annotation; they are not manually written reference or corrupted responses. Refusal examples test whether a model abstains when the retrieved evidence is insufficient. Faithfulness, correctness, and completeness examples reflect distinct failures of grounded reasoning. Full examples including retrieved context are provided in Appendix A.4 .

Figure 2: DPO refinement improves grounded evaluation on ContextualJudgeBench . Compared to the Ministral-8B baseline (grey), contextual DPO refinement on CJB (gold) improves performance across the three contextual dimensions we focus on: refusal, faithfulness, and completeness.

	Gemma 2-2B		Ministral-8B
Training	Overall	$\Delta$	Overall	$\Delta$
Random	50.0	–	50.0	–
Baseline	61.8	–	59.3	–
CJB	66.4	+4.6	67.6	+8.3
LRB	63.6	+1.8	61.9	+2.6
CJB+LRB	65.7	+3.9	66.0	+6.7

Table 2: Legal contextual reward modelling is sensitive to dataset construction. Evaluation on LegalRewardBench (LRB). Length-balanced augmentation and general grounded pairs improves grounded legal evaluation, particularly for baseline Ministral-8B. Higher is better. Wilson 95% confidence intervals are provided in Appendix - Table 12 . Best results are shown in bold ; second-best results are underlined .

Figure 3: Length-balanced augmentation improves legal contextual reward modelling. Compared to the Ministral-8B baseline (grey), DPO refinement on CJB alone transfers strongly to LegalRewardBench (blue). Joint training on CJB and LRB (pink) is limited by structural response-length artefacts in the legal preference data. After applying length-balanced augmentation (gold), performance improves substantially across evaluation dimensions.

Model	Bar Exam	$\Delta$	Housing	$\Delta$
Random	50.0	–	50.0	–
Ministral-8B	56.4 [47,65]	–	54.2 [50,58]	–
+DPO	60.7 [52,69]	+4.3	70.4 [66,74]	+16.2
Qwen3.5-9B	54.7 [46,63]	–	52.8 [48,57]	–
+DPO	54.7 [46,63]	+0.0	55.4 [51,60]	+2.6
Qwen3.5-27B	60.7 [52,69]	–	50.8 [46,55]	–

Table 3: Contextual reward models partially generalise across legal jurisdictions. Models refined on CJB and LRB improve grounded evaluation on external US legal reasoning QA benchmarks despite training primarily on Victorian criminal law data. Results report pairwise reward accuracy with Wilson 95% confidence intervals. Higher is better. Best results are shown in bold ; second-best results are underlined .

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dimension	Train	Val	Test	Total
refusal_answerable	451	68	91	610
refusal_unanswerable	264	36	47	347
faithfulness	145	12	12	169
correctness	49	3	4	56
completeness	31	5	2	38
Total	940	124	156	1,220

Appendix

Table 4: Main LegalRewardBench preference pair counts by split and first distinguishing hierarchy dimension. The 80/10/10 split is performed at question level.

First distinguishing dimension	Main six	All variants
refusal_answerable	610	1,443
refusal_unanswerable	347	564
faithfulness	169	330
correctness	56	147
completeness	38	113
Strict pairs	1,220	2,597

Appendix

Table 5: Strict pair yield by corpus. “Main six” is the preference corpus used for training; “all variants” is the context-variation ablation.

Dimension	$N$	Chance	Raw	Fleiss’ $\kappa$
Answer behaviour	2,400	50.1%	99.9%	0.998
Faithfulness	1,139	74.7%	84.3%	0.379
Correctness	1,139	74.7%	94.2%	0.771
Completeness	1,139	69.4%	91.5%	0.722

Appendix

Table 9: Agreement across LLM judges. Chance and raw agreement are reported as percentages. Chance agreement reflects expected agreement from the marginal label distributions, while raw agreement is the average observed agreement across all six judge pairs.

Preference reconstruction	Result
Overall pair directions preserved	1,119 / 1,220 (91.7%)
Test-set pair directions preserved	148 / 156 (94.9%)
Pair reversals	15 / 1,220 (1.2%)
Answer-behaviour pair directions preserved	99.8%

Appendix

Table 10: Robustness of preference construction across LLM judges. Agreement is measured after propagating independent annotations through the same hierarchical preference-construction procedure used to construct LegalRewardBench .

Baselines
Model	Overall	C-QA	Cc-QA	F-QA	C-S	Cc-S	F-S	R-A	R-U
Random	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0	50.0
Qwen2.5-0.5B	55.4	24.0	88.0	66.0	36.2	71.8	45.5	84.1	32.0
SmolLM2-1.7B	57.1	30.0	86.0	68.0	34.0	74.4	47.7	88.4	34.0
Gemma3-1B	58.1	30.0	90.0	72.0	38.3	65.9	50.0	86.0	36.0
Llama3.2-1B	55.9	24.0	88.0	68.0	29.8	79.5	47.7	84.1	32.0

Appendix

Table 11: Open-weight and specialist reward models perform poorly, while DPO fine-tuning improves grounded evaluation. Per-dimension pairwise reward accuracy on the ContextualJudgeBench (CJB) test set (

\sim

400 preference pairs). Results are grouped into open-weight models, specialist reward models with scalar reward heads, and DPO fine-tuned contextual judge models. Dimensions evaluate completeness (C), conciseness (Cc), faithfulness (F), and refusal under answerable (R-A) and unanswerable (R-U) retrieval settings. QA denotes question answering subsets and S denotes summarisation subsets. Random performance corresponds to

50\%

pairwise accuracy. Higher is better. Best scores in each column are shown in bold ; second-best scores are underlined .

	Gemma 2-2B		Ministral-8B
Training	Overall	95% CI	Overall	95% CI
Baseline	61.8	[56.0, 67.3]	59.3	[53.4, 64.9]
CJB	66.4	[60.7, 71.7]	66.8	[61.1, 72.0]
LRB	63.6	[57.8, 69.0]	60.7	[54.9, 66.3]
CJB+LRB	65.7	[60.0, 71.0]	65.4	[59.6, 70.7]
LRB-Length-Aug	81.1	[76.1, 85.2]	81.8	[76.8, 85.9]

Appendix

Table 12: Wilson 95% confidence intervals for Table 2 . Pairwise reward accuracy on LegalRewardBench-v2 val+test (

n{=}280

). Gemma 2-2B values are from a single training seed; Ministral-8B CIs correspond to the seed=42 run (main table reports 3-seed means).

Eval set	Raw	SFT	DPO	$\Delta$ (DPO $-$ SFT)
CJB ( $n=373$ )	$56.8$	$60.3$	$\mathbf{72.7}$	$+12.4$ pp
LRB-v2 test ( $n=156$ )	$56.4$	$58.3$	$\mathbf{91.7}$	$+33.4$ pp

Appendix

Table 13: SFT vs. DPO on Ministral-8B, identical combined corpus and training budget. Evaluated on the CJB test split (

n=373

) and the LRB-v2 test split (

n=156

Seed	CJB reward accuracy (95% CI)
42	72.7 [67.9, 76.9]
1	71.6 [66.8, 75.9]
2	71.9 [67.1, 76.2]
Mean $\pm$ $\sigma$	$\mathbf{72.0\pm 0.6}$

Appendix

Table 14: Ministral-8B DPO-Combined-v2 (CJB+LRB-v2) across three random seeds on the CJB test split (

n=373

). Wilson 95% CI in brackets. Unnormalised sum log-probability scoring per the convention used in the main results table.

Hyperparameter	Value	CJB Acc.
DPO $\beta$	0.01	70.1
	0.05	71.8
	0.10	73.7
	0.20	72.9
LoRA rank $r$	8	71.0
	16	72.2

Appendix

Table 15: Hyperparameter sensitivity, Ministral-8B DPO-Combined on the CJB test split. Shaded row = default configuration carried forward to the DPO-Combined-v2 model. At these defaults, DPO-Combined-v2 reaches

72.7

at seed 42 and

72.0\pm 0.6

across three seeds (cf. Table 14 ).

Raw model benchmark (14 models $\times$ CJB)
Phase	Wall-clock (h)	GPU-hours
0.5B–3B models ( $\times$ 8)	8.0	8.0
8B models ( $\times$ 6)	9.0	9.0
Specialist model evaluation ( $\times$ 3)
Skywork / Selene / SFR-Judge	1.5	1.5
DPO and SFT training (all corpora and models)

Appendix

Table 16: Estimated GPU-hours by experimental phase. Wall-clock times are rounded to the nearest 5 minutes.

Package	Version	Role
Python	3.11.15	Runtime
PyTorch	2.4.1+cu124	Tensor operations, autograd
Transformers	4.45.2	Model loading, tokenisation
PEFT	0.12.0	LoRA and QLoRA adapters
TRL	0.11.4	DPO / SFT training loop
BitsAndBytes	0.49.2	4-bit NF4 quantisation (QLoRA)

Appendix

Table 17: Software dependencies. Versions correspond to the locked environment used for all reported experiments.

Model name	Hugging Face repository
Instruction-tuned baselines
SmolLM2-1.7B	HuggingFaceTB/SmolLM2-1.7B-Instruct
Qwen2.5-0.5B	Qwen/Qwen2.5-0.5B-Instruct
Qwen3.5-2B	Qwen/Qwen3.5-2B
Llama-3.2-1B	meta-llama/Llama-3.2-1B-Instruct
Llama-3.2-3B	meta-llama/Llama-3.2-3B-Instruct

Appendix

Table 18: Hugging Face repository identifiers for all evaluated models.

Explore similar work

LegalRewardBench: Benchmarking Reward Models for Legal Preference Evaluation
Oct 13, 2026Hui Huang, Jie Zhou, Hongli Zhou +7Reward Model Evaluation Pairwise Preference Evaluation
Harbin Institute of Technology · ModelBest Inc. · The University of Osaka
Legal RAG Bench: an end-to-end benchmark for legal RAG
Mar 2, 2026Abdur-Rahman Butler, Umar ButlerRetrieval-Augmented Generation RAG Evaluation
Isaacus
Legal-R1: Agentic Retrieval for Evidence-Based Legal Reasoning
Oct 13, 2026Huaiwen Pang, Jiale Han, Junying Chen +2Legal QA Legal Reasoning
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · Shenzhen Shenten Digital Technology Co., Ltd.

Oct 13, 2026J/K move · Enter open · S save

LegalRewardBench: Benchmarking Reward Models for Legal Preference Evaluation

Hui Huang, Jie Zhou, Hongli Zhou, Qundong Shi+6

Harbin Institute of Technology · ModelBest Inc. · The University of Osaka

Reward Model Evaluation Pairwise Preference Evaluation Legal NLP

Open paper AskPDF arXiv

Reward models are central to preference alignment for legal tasks, yet existing reward model benchmarks are still primarily centered on general-domain preferences. In this work, we introduce LegalRewardBench, the first human-verified benchmark for evaluating reward models in the legal domain. The benchmark organizes legal preference evaluation into five capability levels: knowledge, understanding, analysis, decision, and writing, and preference examples are constructed through rubric-guided automatic judging combined with human verification. We evaluate three families of judges, covering both general and fine-tuned models, and generative and discriminative architectures. Experimental results show that strong general-purpose generative models are currently the most robust legal reward models, while existing fine-tuned reward models transfer less reliably to legal preference evaluation. We also conduct detailed comparisons of input organizations and training strategies.

Abstract

Explore similar work

LegalRewardBench: Benchmarking Reward Models for Legal Preference Evaluation

Legal RAG Bench: an end-to-end benchmark for legal RAG

Legal-R1: Agentic Retrieval for Evidence-Based Legal Reasoning