Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following

Sangyeon Yoon; Wooseok Seo; Hamish Ivison; Hyesoo Hong; Donghoon Kim; Wonjun Kang; Albert No; Jaehyung Kim

cs.LGOct 14, 2026

Diagnosing and Correcting Reward Hacking in Verifiable Instruction Following

Authors: Sangyeon Yoon, Wooseok Seo, Hamish Ivison, Hyesoo Hong, Donghoon Kim, Wonjun Kang, Albert No, Jaehyung Kim

Organizations: Yonsei University · FuriosaAI · University of Washington

Abstract

Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underlying request, creating an evaluation blind spot. We show that frontier models can receive full credit for responses with little substantive answer content. Using the same verifier-based criterion as a training reward can induce reward hacking, as publicly released checkpoints exhibit higher constraint satisfaction but lower response quality after reinforcement learning with verifiable rewards. To address this reward hacking, we introduce CALIF, a lightweight method that augments verifier rewards with a calibrated likelihood-based bonus to preserve response quality. CALIF evaluates response likelihood under the request alone using the existing frozen reference policy, requiring no additional judge or learned reward model. Across models ranging from 1.7B to 8B in reasoning and non-reasoning modes, CALIF improves constraint satisfaction on IFBench while maintaining response quality near or above pre-RL levels. It also outperforms corresponding base policies in 40 of 45 model-metric comparisons across three rubric-based instruction-following benchmarks, demonstrating gains beyond training verifiers.

Figures & tables

Panel of Figure 1 — Figure 1: An evaluation blind spot in VIF: evidence from frontier models on IFBench. Constraint Satisfaction evaluates whether responses satisfy the constraints using programmatic verifiers; Response Quality evaluates how well they answer the request using three LLM judges: GPT-5.6 Sol, Claude Opus 5, and Gemini 3.1 Pro, whose scores are averaged. Pink arrows show the drop in Response Quality from answering the request alone to answering the request with added constraints.

Figure 2: DeepSeek V4 Pro responses with and without an appended constraint. The model achieves a high Response Quality score of 9.33 when given the request alone, but its constrained response includes exactly five numbers as required while scoring only 2.00 in Response Quality .

Figure 3: Comparison of DPO and RLVR checkpoints in the OLMo 3 family on IFBench. Log-likelihood is computed under the corresponding DPO checkpoint, conditioned on the request alone.

Figure 4: Illustration of reference-calibrated likelihood region in CaLIF .

Figure 5: CaLIF mitigates reward hacking in IF-RLVR on IFBench across five model settings.

Figure 6: Log-likelihood under the request alone on IFBench across five model settings.

	AdvancedIF (CIF)		FollowBench			ComplexBench
Model	Overall pass rate $\uparrow$	Micro rubric pass $\uparrow$	HSR $\uparrow$	SSR $\uparrow$	CSL $\uparrow$	Lexical $\uparrow$	Semantic $\uparrow$	Utility $\uparrow$	Format † $\uparrow$
Qwen3-1.7B	$\underline{18.08}\,{\scriptscriptstyle\pm 1.12}$	$\underline{56.82}\,{\scriptscriptstyle\pm 0.53}$	$\underline{53.81}\,{\scriptscriptstyle\pm 1.76}$	$\underline{68.33}\,{\scriptscriptstyle\pm 1.10}$	$\underline{2.54}\,{\scriptscriptstyle\pm 0.06}$	$\underline{57.46}\,{\scriptscriptstyle\pm 1.29}$	$\underline{53.62}\,{\scriptscriptstyle\pm 0.49}$	$\underline{51.09}\,{\scriptscriptstyle\pm 0.05}$	$\underline{50.70}\,{\scriptscriptstyle\pm 0.42}$
+ IF-RLVR	$12.44\,{\scriptscriptstyle\pm 0.00}$	$50.13\,{\scriptscriptstyle\pm 0.29}$	$48.14\,{\scriptscriptstyle\pm 1.00}$	$64.02\,{\scriptscriptstyle\pm 0.51}$	$2.29\,{\scriptscriptstyle\pm 0.04}$	$49.68\,{\scriptscriptstyle\pm 0.37}$	$47.43\,{\scriptscriptstyle\pm 0.28}$	$40.93\,{\scriptscriptstyle\pm 0.11}$	$50.45\,{\scriptscriptstyle\pm 0.70}$
+ CaLIF	$\mathbf{19.32}\,{\scriptscriptstyle\pm 0.29}$	$\mathbf{60.08}\,{\scriptscriptstyle\pm 0.28}$	$\mathbf{60.29}\,{\scriptscriptstyle\pm 0.60}$	$\mathbf{72.51}\,{\scriptscriptstyle\pm 0.60}$	$\mathbf{2.81}\,{\scriptscriptstyle\pm 0.05}$	$\mathbf{58.60}\,{\scriptscriptstyle\pm 0.12}$	$\mathbf{56.34}\,{\scriptscriptstyle\pm 0.24}$	$\mathbf{51.69}\,{\scriptscriptstyle\pm 0.53}$	$\mathbf{50.94}\,{\scriptscriptstyle\pm 0.91}$
Qwen3-1.7B (Thinking)	$\underline{23.88}\,{\scriptscriptstyle\pm 0.25}$	$62.53\,{\scriptscriptstyle\pm 1.05}$	$\underline{58.66}\,{\scriptscriptstyle\pm 0.65}$	$71.54\,{\scriptscriptstyle\pm 0.23}$	$\underline{2.79}\,{\scriptscriptstyle\pm 0.05}$	$\underline{59.46}\,{\scriptscriptstyle\pm 1.61}$	$\underline{60.07}\,{\scriptscriptstyle\pm 1.01}$	$\underline{54.86}\,{\scriptscriptstyle\pm 0.22}$	$51.47\,{\scriptscriptstyle\pm 0.58}$
+ IF-RLVR	$23.63\,{\scriptscriptstyle\pm 0.90}$	$\underline{63.43}\,{\scriptscriptstyle\pm 0.15}$	$58.42\,{\scriptscriptstyle\pm 0.46}$	$\underline{72.73}\,{\scriptscriptstyle\pm 0.59}$	$2.78\,{\scriptscriptstyle\pm 0.05}$	$56.67\,{\scriptscriptstyle\pm 0.62}$	$57.03\,{\scriptscriptstyle\pm 1.25}$	$50.91\,{\scriptscriptstyle\pm 0.81}$	$\mathbf{55.10}\,{\scriptscriptstyle\pm 0.80}$

Table 1: Performance on three rubric-based instruction-following benchmarks. Format † is most closely aligned with the training constraints. See Appendix C for details on the benchmarks and evaluation metrics. Bold and underline indicate the best and second-best within each setting.

Figure 7: Performance across constraint types on IFBench. Bars and error bars indicate the mean and standard deviation across the five model settings. Response length is measured in tokens.

	$\Delta$ Response Quality $\uparrow$
Method	Pass $\rightarrow$ Pass	Fail $\rightarrow$ Pass
IF-RLVR	$-1.26$ $\pm 0.37$	$-2.05$ $\pm 0.80$
CaLIF	$\mathbf{+0.01}$ $\pm 0.06$	$\mathbf{-0.28}$ $\pm 0.47$

Table 2: Response Quality changes for verifier-passing outcomes. Transitions are from base to trained policies, aggregated over five model configurations.

	IFBench		AdvancedIF (CIF)		FollowBench			ComplexBench
Reward	Constraint Satisfaction $\uparrow$	Response Quality $\uparrow$	Overall Pass Rate $\uparrow$	Micro Rubric Pass $\uparrow$	HSR $\uparrow$	SSR $\uparrow$	CSL $\uparrow$	Lexical $\uparrow$	Semantic $\uparrow$	Utility $\uparrow$	Format $\uparrow$
Tülu 3 8B DPO
CaLIF	29.78 $\pm 0.19$	5.23 $\pm 0.02$	32.92 $\pm 0.38$	73.32 $\pm 0.53$	67.88 $\pm 0.64$	77.50 $\pm 0.21$	3.23 $\pm 0.01$	63.81 $\pm 1.30$	64.76 $\pm 0.91$	58.60 $\pm 0.19$	55.10 $\pm 0.40$
Self judge	28.11 $\pm 0.69$	4.64 $\pm 0.01$	30.18 $\pm 0.94$	71.40 $\pm 0.50$	60.28 $\pm 1.32$	69.04 $\pm 1.10$	2.86 $\pm 0.04$	60.53 $\pm 0.49$	62.12 $\pm 0.08$	57.69 $\pm 0.57$	51.05 $\pm 0.36$
Stronger judge	26.89 $\pm 0.38$	5.43 $\pm 0.01$	28.94 $\pm 0.57$	70.53 $\pm 0.33$	63.26 $\pm 1.58$	74.29 $\pm 1.05$	3.01 $\pm 0.11$	60.60 $\pm 0.37$	60.48 $\pm 0.08$	55.73 $\pm 1.06$	51.12 $\pm 0.84$
Qwen3-4B (Thinking)

Table 3: Comparison with LLM-judge rewards on IFBench and three rubric-based instruction-following benchmarks. Self-judge uses the base policy as the judge; Stronger judge uses Qwen3-30B-A3B. Bold and underline indicate the best and second-best within each setting.

Method	Constraint Satisfaction $\uparrow$	Response Quality $\uparrow$	Reference Log-Likelihood	Response Length
Qwen3-1.7B	22.22	3.98	$-1.50$	485
+ IF-RLVR	41.44 (+19.22)	2.16 (-1.82)	$-5.91$ $(-4.41)$	152 $(-333)$
+ CaLIF	28.78 (+6.56)	4.01 (+0.03)	$-1.42$ $(+0.09)$	550 $(+65)$
Direct likelihood reward (Section 6.1 )
Direct bonus	36.78 (+14.56)	1.91 (-2.07)	$-0.68$ $(+0.82)$	1677 $(+1192)$
Single-bound variants (Section 6.2 )

Table 4: Effect of likelihood reward design and region construction on IFBench. We compare CaLIF with a direct likelihood reward, single-bound variants, and alternative region constructions.

Figure 8: Reward and KL coefficient sweeps on IFBench. Squares vary

\alpha

in CaLIF ; circles vary

\beta

in IF-RLVR. Larger coefficients shift points leftward.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model configuration	Wins	Ties	Losses	Adjusted WR (%)
Qwen3-1.7B	229	34	37	82.00
Qwen3-1.7B (Thinking)	193	45	62	71.83
Qwen3-4B	219	36	45	79.00
Qwen3-4B (Thinking)	195	56	49	74.33
Tülu 3 8B DPO	229	42	29	83.33
Overall	1,065	213	222	78.10

Appendix

Table 5: Pairwise comparison of CaLIF and IF-RLVR on IFBench. Counts reflect CaLIF ’s perspective. Ties include order-inconsistent judgments and receive half credit in adjusted win rates.

Figure 9: CaLIF without KL regularization on IFBench. We compare CaLIF with and without the KL penalty (

\beta=0.001

and

0

, respectively), keeping

\alpha=0.1

fixed. IF-RLVR uses

\beta=0.001

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 182 papers

LIFT: Scoped Constraints for Evaluating and Training Instruction Following
Oct 20, 2026Eran Goldstein, Yuval Belfer, Yonatan Belinkov +1Instruction Following Generative AI Evaluation
AI21 Labs
IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation
Mar 5, 2026Bosi Wen, Yilin Niu, Cunxiang Wang +5LLM-as-a-Judge Instruction Following
The Conversational Artificial Intelligence (CoAI) Group, Tsinghua University · Zhipu AI · University of Electronic Science and Technology of China
Detecting and Suppressing Reward Hacking with Gradient Fingerprints
Apr 17, 2026Songtao Wang, Quang Hieu Pham, Fangcong Yin +4Reward Hacking Reinforcement Learning with Verifiable Rewards
AUniversity of Alberta · N New York University · LLMU Munich +1

Oct 20, 2026J/K move · Enter open · S save

LIFT: Scoped Constraints for Evaluating and Training Instruction Following

Eran Goldstein, Yuval Belfer, Yonatan Belinkov, Daniel Gissin

AI21 Labs

Instruction Following Generative AI Evaluation RL for Language Models

Open paper AskPDF arXiv

Common instruction following (IF) benchmarks, such as IFEval, evaluate the ability of language models (LMs) to follow simple constraints that can easily be verified programmatically. These same constraints, in turn, are used as reward signals for reinforcement learning. This design may lead to reward hacking of constraints, pushing models away from the main task. Recent attempts to address this resulted in benchmarks with richer constraints, but required expert labor, remained closed-access, and cannot be easily extended. We present Lift-Bench, an IF benchmark that asks a model to draft a document given a topic and a structured outline. Constraints are scoped to certain document sections, providing finer control over constraint density. Lift-Bench includes both verifiable and judged constraints, as well as a training counterpart, Lift-Train. Importantly, Lift-Bench provides both main task and instruction-following constraints, mitigating shallow reward hacking. On experiments with multiple models, we find that frontier models struggle with the harder subsets of our benchmark and analyze model performance along multiple axes. We also find that training on Lift-Train is much less susceptible to reward hacking than training on the common IFTrain. Lift-Bench and Lift-Train are publicly available, as well as the data generating pipeline.

Abstract

Reward Hacking

Explore similar work

LIFT: Scoped Constraints for Evaluating and Training Instruction Following

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation

Detecting and Suppressing Reward Hacking with Gradient Fingerprints