Large language models are increasingly used in high-stakes domains such as law, where systems must ground their reasoning in retrieved evidence and abstain when that evidence is insufficient. However, existing reward models are largely optimised for general preferences rather than contextual grounding, limiting their ability to evaluate these behaviours in retrieval-augmented generation (RAG) settings. We introduce a framework for transforming existing legal QA datasets into contextual preference data and use it to construct LegalRewardBench (LRB), a benchmark for evaluating grounded legal generation under noisy and insufficient retrieval conditions. Across general and legal contextual evaluation, we find that contextual DPO improves grounded evaluation, but performance is sensitive to preference-data construction. Length-balanced augmentation substantially improves grounded legal evaluation, with the strongest configuration combining length-balanced legal and general contextual preference data and improving performance by up to +25.6pp over baseline. We further find evidence of cross-jurisdiction transfer: models contextually refined primarily on Victorian criminal-law data improve grounded evaluation on external US legal benchmarks, including a +16.2pp improvement on Housing Statute QA. Together, these results provide a reproducible foundation for constructing and evaluating grounded legal reward models in retrieval-augmented settings.
Figures & tables
Figure 1: Legal preference-pair construction framework. Starting from Legal RAG Bench triples (cgold,q,a) (Section 2.1 ), we construct six retrieved-context variants by combining the gold passage with distractor passages to control answerability and retrieval noise (Section 2.2 ). Four LLMs generate candidate responses across all (c,q) instances (Section 2.3 ). A judge-model assigns pointwise labels for answer behaviour, faithfulness, correctness, and completeness (Section 2.4 ). A hierarchical preference rule then converts labelled responses into contextual preference pairs (c,q,r+,r−) for reward-model training and evaluation (Section 2.5 ).
Failure Type
Question q
Preferred Response r+
Rejected Response r−
Refusal
What sentence was imposed on the accused?
“The provided context is insufficient to answer this question.”
“The accused was sentenced to five years imprisonment.”
Faithfulness
Which Act introduced cyberstalking offences?
“The Crimes (Stalking) Act 2003 introduced cyberstalking offences.”
“The Crimes Amendment Act 2001 introduced cyberstalking offences.”
Correctness
Does Victoria protect against double jeopardy?
“Yes. Victoria protects against double jeopardy.”
“No. Victoria does not recognise double jeopardy protections.”
Completeness
Must jurors be excused after seeing pre-trial publicity?
“No. Judges can usually address prejudice through directions to the jury.”
“No.”
Table 1: Legal preference pairs. Representative pairwise preference examples. In the dataset, r+ and r− are selected from generated candidate responses after pointwise annotation; they are not manually written reference or corrupted responses. Refusal examples test whether a model abstains when the retrieved evidence is insufficient. Faithfulness, correctness, and completeness examples reflect distinct failures of grounded reasoning. Full examples including retrieved context are provided in Appendix A.4 .
Figure 2: DPO refinement improves grounded evaluation on ContextualJudgeBench . Compared to the Ministral-8B baseline (grey), contextual DPO refinement on CJB (gold) improves performance across the three contextual dimensions we focus on: refusal, faithfulness, and completeness.
Gemma 2-2B
Ministral-8B
Training
Overall
Δ
Overall
Δ
Random
50.0
–
50.0
–
Baseline
61.8
–
59.3
–
CJB
66.4
+4.6
67.6
+8.3
LRB
63.6
+1.8
61.9
+2.6
CJB+LRB
65.7
+3.9
66.0
+6.7
Table 2: Legal contextual reward modelling is sensitive to dataset construction. Evaluation on LegalRewardBench (LRB). Length-balanced augmentation and general grounded pairs improves grounded legal evaluation, particularly for baseline Ministral-8B. Higher is better. Wilson 95% confidence intervals are provided in Appendix - Table 12 . Best results are shown in bold ; second-best results are underlined .
Figure 3: Length-balanced augmentation improves legal contextual reward modelling. Compared to the Ministral-8B baseline (grey), DPO refinement on CJB alone transfers strongly to LegalRewardBench (blue). Joint training on CJB and LRB (pink) is limited by structural response-length artefacts in the legal preference data. After applying length-balanced augmentation (gold), performance improves substantially across evaluation dimensions.
Model
Bar Exam
Δ
Housing
Δ
Random
50.0
–
50.0
–
Ministral-8B
56.4 [47,65]
–
54.2 [50,58]
–
+DPO
60.7 [52,69]
+4.3
70.4 [66,74]
+16.2
Qwen3.5-9B
54.7 [46,63]
–
52.8 [48,57]
–
+DPO
54.7 [46,63]
+0.0
55.4 [51,60]
+2.6
Qwen3.5-27B
60.7 [52,69]
–
50.8 [46,55]
–
Table 3: Contextual reward models partially generalise across legal jurisdictions. Models refined on CJB and LRB improve grounded evaluation on external US legal reasoning QA benchmarks despite training primarily on Victorian criminal law data. Results report pairwise reward accuracy with Wilson 95% confidence intervals. Higher is better. Best results are shown in bold ; second-best results are underlined .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Dimension
Train
Val
Test
Total
refusal_answerable
451
68
91
610
refusal_unanswerable
264
36
47
347
faithfulness
145
12
12
169
correctness
49
3
4
56
completeness
31
5
2
38
Total
940
124
156
1,220
Appendix
Table 4: Main LegalRewardBench preference pair counts by split and first distinguishing hierarchy dimension. The 80/10/10 split is performed at question level.
First distinguishing dimension
Main six
All variants
refusal_answerable
610
1,443
refusal_unanswerable
347
564
faithfulness
169
330
correctness
56
147
completeness
38
113
Strict pairs
1,220
2,597
Appendix
Table 5: Strict pair yield by corpus. “Main six” is the preference corpus used for training; “all variants” is the context-variation ablation.
Dimension
N
Chance
Raw
Fleiss’ κ
Answer behaviour
2,400
50.1%
99.9%
0.998
Faithfulness
1,139
74.7%
84.3%
0.379
Correctness
1,139
74.7%
94.2%
0.771
Completeness
1,139
69.4%
91.5%
0.722
Appendix
Table 9: Agreement across LLM judges. Chance and raw agreement are reported as percentages. Chance agreement reflects expected agreement from the marginal label distributions, while raw agreement is the average observed agreement across all six judge pairs.
Preference reconstruction
Result
Overall pair directions preserved
1,119 / 1,220 (91.7%)
Test-set pair directions preserved
148 / 156 (94.9%)
Pair reversals
15 / 1,220 (1.2%)
Answer-behaviour pair directions preserved
99.8%
Appendix
Table 10: Robustness of preference construction across LLM judges. Agreement is measured after propagating independent annotations through the same hierarchical preference-construction procedure used to construct LegalRewardBench .
Model
Overall
C-QA
Cc-QA
F-QA
C-S
Cc-S
F-S
R-A
R-U
Random
50.0
50.0
50.0
50.0
50.0
50.0
50.0
50.0
50.0
Baselines
Qwen2.5-0.5B
55.4
24.0
88.0
66.0
36.2
71.8
45.5
84.1
32.0
SmolLM2-1.7B
57.1
30.0
86.0
68.0
34.0
74.4
47.7
88.4
34.0
Gemma3-1B
58.1
30.0
90.0
72.0
38.3
65.9
50.0
86.0
36.0
Llama3.2-1B
55.9
24.0
88.0
68.0
29.8
79.5
47.7
84.1
32.0
Appendix
Table 11: Open-weight and specialist reward models perform poorly, while DPO fine-tuning improves grounded evaluation. Per-dimension pairwise reward accuracy on the ContextualJudgeBench (CJB) test set ( ∼ 400 preference pairs). Results are grouped into open-weight models, specialist reward models with scalar reward heads, and DPO fine-tuned contextual judge models. Dimensions evaluate completeness (C), conciseness (Cc), faithfulness (F), and refusal under answerable (R-A) and unanswerable (R-U) retrieval settings. QA denotes question answering subsets and S denotes summarisation subsets. Random performance corresponds to 50% pairwise accuracy. Higher is better. Best scores in each column are shown in bold ; second-best scores are underlined .
Gemma 2-2B
Ministral-8B
Training
Overall
95% CI
Overall
95% CI
Baseline
61.8
[56.0, 67.3]
59.3
[53.4, 64.9]
CJB
66.4
[60.7, 71.7]
66.8
[61.1, 72.0]
LRB
63.6
[57.8, 69.0]
60.7
[54.9, 66.3]
CJB+LRB
65.7
[60.0, 71.0]
65.4
[59.6, 70.7]
LRB-Length-Aug
81.1
[76.1, 85.2]
81.8
[76.8, 85.9]
Appendix
Table 12: Wilson 95% confidence intervals for Table 2 . Pairwise reward accuracy on LegalRewardBench-v2 val+test ( n=280 ). Gemma 2-2B values are from a single training seed; Ministral-8B CIs correspond to the seed=42 run (main table reports 3-seed means).
Eval set
Raw
SFT
DPO
Δ (DPO − SFT)
CJB ( n=373 )
56.8
60.3
72.7
+12.4 pp
LRB-v2 test ( n=156 )
56.4
58.3
91.7
+33.4 pp
Appendix
Table 13: SFT vs. DPO on Ministral-8B, identical combined corpus and training budget. Evaluated on the CJB test split ( n=373 ) and the LRB-v2 test split ( n=156 ).
Seed
CJB reward accuracy (95% CI)
42
72.7 [67.9, 76.9]
1
71.6 [66.8, 75.9]
2
71.9 [67.1, 76.2]
Mean ±σ
72.0±0.6
Appendix
Table 14: Ministral-8B DPO-Combined-v2 (CJB+LRB-v2) across three random seeds on the CJB test split ( n=373 ). Wilson 95% CI in brackets. Unnormalised sum log-probability scoring per the convention used in the main results table.
Hyperparameter
Value
CJB Acc.
DPO β
0.01
70.1
0.05
71.8
0.10
73.7
0.20
72.9
LoRA rank r
8
71.0
16
72.2
Appendix
Table 15: Hyperparameter sensitivity, Ministral-8B DPO-Combined on the CJB test split. Shaded row = default configuration carried forward to the DPO-Combined-v2 model. At these defaults, DPO-Combined-v2 reaches 72.7 at seed 42 and 72.0±0.6 across three seeds (cf. Table 14 ).
Phase
Wall-clock (h)
GPU-hours
Raw model benchmark (14 models × CJB)
0.5B–3B models ( × 8)
8.0
8.0
8B models ( × 6)
9.0
9.0
Specialist model evaluation ( × 3)
Skywork / Selene / SFR-Judge
1.5
1.5
DPO and SFT training (all corpora and models)
Appendix
Table 16: Estimated GPU-hours by experimental phase. Wall-clock times are rounded to the nearest 5 minutes.
Package
Version
Role
Python
3.11.15
Runtime
PyTorch
2.4.1+cu124
Tensor operations, autograd
Transformers
4.45.2
Model loading, tokenisation
PEFT
0.12.0
LoRA and QLoRA adapters
TRL
0.11.4
DPO / SFT training loop
BitsAndBytes
0.49.2
4-bit NF4 quantisation (QLoRA)
Appendix
Table 17: Software dependencies. Versions correspond to the locked environment used for all reported experiments.
Model name
Hugging Face repository
Instruction-tuned baselines
SmolLM2-1.7B
HuggingFaceTB/SmolLM2-1.7B-Instruct
Qwen2.5-0.5B
Qwen/Qwen2.5-0.5B-Instruct
Qwen3.5-2B
Qwen/Qwen3.5-2B
Llama-3.2-1B
meta-llama/Llama-3.2-1B-Instruct
Llama-3.2-3B
meta-llama/Llama-3.2-3B-Instruct
Appendix
Table 18: Hugging Face repository identifiers for all evaluated models.
Reward models are central to preference alignment for legal tasks, yet existing reward model benchmarks are still primarily centered on general-domain preferences. In this work, we introduce LegalRewardBench, the first human-verified benchmark for evaluating reward models in the legal domain. The benchmark organizes legal preference evaluation into five capability levels: knowledge, understanding, analysis, decision, and writing, and preference examples are constructed through rubric-guided automatic judging combined with human verification. We evaluate three families of judges, covering both general and fine-tuned models, and generative and discriminative architectures. Experimental results show that strong general-purpose generative models are currently the most robust legal reward models, while existing fine-tuned reward models transfer less reliably to legal preference evaluation. We also conduct detailed comparisons of input organizations and training strategies.
Hui Huang, Jie Zhou, Hongli Zhou +7
Harbin Institute of Technology · ModelBest Inc. · The University of Osaka
We introduce Legal RAG Bench, a benchmark and evaluation methodology for assessing the end-to-end performance of legal RAG systems. As a benchmark, Legal RAG Bench consists of 4,876 passages from the Victorian Criminal Charge Book alongside 100 complex, hand-crafted questions demanding expert knowledge of criminal law and procedure. Both long-form answers and supporting passages are provided. As an evaluation methodology, Legal RAG Bench leverages a full factorial design and novel hierarchical error decomposition framework, enabling apples-to-apples comparisons of the contributions of retrieval and reasoning models in RAG. We evaluate three state-of-the-art embedding models (Isaacus' Kanon 2 Embedder, Google's Gemini Embedding 001, and OpenAI's Text Embedding 3 Large) and two frontier LLMs (Gemini 3.1 Pro and GPT-5.2), finding that information retrieval is the primary driver of legal RAG performance, with LLMs exerting a more moderate effect on correctness and groundedness. Kanon 2 Embedder, in particular, had the largest positive impact on performance, improving average correctness by 17.5 points, groundedness by 4.5 points, and retrieval accuracy by 34 points. We observe that many errors attributed to hallucinations in legal RAG systems are in fact triggered by retrieval failures, concluding that retrieval sets the ceiling for the performance of many modern legal RAG systems. We document why and how we built Legal RAG Bench alongside the results of our evaluations. We also openly release our code and data to assist with reproduction of our findings.
Legal question answering requires answers that are not only plausible, but also grounded in legally valid and citable evidence. Standard retrieval-augmented generation often relies on one-shot semantic matching, which is insufficient when legal evidence must be selected according to authority, applicability, and diversity across heterogeneous sources. We introduce Legal-R1, an agentic retrieval framework for evidence-based legal reasoning. Legal-R1 decomposes legal reasoning into a multi-agent workflow, where a Lawyer agent learns when to retrieve, which legal source to query, what evidence to seek, and when the collected evidence is sufficient. A Legal Assistant agent retrieves from normative legal sources and precedent cases with authority-aware and diversity-aware selection, while a Judge agent generates the final answer from the complete evidence trajectory. We post-train only the Lawyer agent with masked supervised fine-tuning and trajectory-aware reinforcement learning. Experiments on multiple Chinese legal benchmarks show that Legal-R1 improves over general-purpose, legal-specific, and retrieval-augmented baselines.
Huaiwen Pang, Jiale Han, Junying Chen +2
The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute · Shenzhen Shenten Digital Technology Co., Ltd.