Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization

Suyash Maniyar; Armaan Sandhu; Abhishek Mishra

cs.AIAug 15, 2026

Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization

Authors: Suyash Maniyar, Armaan Sandhu, Abhishek Mishra

Organizations: University of Massachusetts Amherst

Abstract

When a reward is correct on every training example yet consistent with more than one goal, a model can acquire an unintended one, a failure known as goal misgeneralization. Endpoint accuracy on the training distribution cannot tell the two apart, because solving the task and exploiting a surface feature can satisfy the reward equally well. We treat this as a measurement problem: what does a benchmark score measure once a model has been optimized against a correct but confounded signal? We train language models with GRPO on multiple-choice math problems where the correct answer is always option A, then evaluate on an unseen test set with unbiased answer positions. Across Qwen2.5, Llama 3.x and Gemma-3 models, biased training often drives option-A rates above 0.90 in smaller models and collapses unbiased accuracy toward chance, so accuracy stops measuring math ability and instead measures an answer-position policy. We further find reasoning-answer decoupling: capable models generate reasoning that reaches the correct numeric answer while still selecting A. We track this with numeric extraction and an LLM judge (GPT-4.1-mini; Qwen2.5-3B decoupling rate is about 0.66). The broken construct generalizes beyond the training domain: biased models inflate A-rates on out-of-domain MMLU and value-laden prompts. Continued training on unbiased data reverses the in-domain shift unevenly and only partially reverses the out-of-domain one, so a model can appear restored on its training distribution while remaining biased on unseen inputs. Reasoning-answer decoupling rate, together with answer distributions and out-of-domain behavior, separates capability loss from a learned, transferable shortcut.

Figures & tables

Figure 1: Final in-domain performance on the position-randomized math test set. Left: endpoint accuracy. Right: option- A rate. Biased-curriculum models are evaluated on the same unbiased test distribution as unbiased-curriculum controls. Under biased training, many models show reduced unbiased accuracy together with elevated option- A rates, indicating transfer of the training-time answer-position regularity to the unbiased evaluation distribution.

Figure 2: Shortcut susceptibility under biased training. Panel A shows validation option- A rate during the biased curriculum; white circles mark the first checkpoint where the model’s mean validation option- A rate reaches $0.75$ . Panel B shows final option- A rate on the position-randomized test set, with gray dots for individual seeds and red points for seed means. Collapse timing and final shortcut strength vary substantially across models, indicating that position-bias susceptibility is an empirical property of the learner–evaluator interaction.

Figure 3: Out-of-domain MMLU-50 probe. Panel A shows MMLU accuracy and Panel B shows option- A rate. Several biased-curriculum models show elevated option- A rates relative to unbiased-curriculum controls on questions never seen during training, indicating that the learned position policy can transfer out of domain. (Qwen2.5-0.5B is omitted because its MMLU generations are mostly unparseable under the tagged-output evaluator.)

Figure 4: Recovery after switching from the biased curriculum to unbiased training. Panel A shows validation option- A rate over recovery steps. Some models rapidly reduce their option- A rate, while others retain shortcut residue throughout recovery. Panel B shows final post-recovery in-domain outcomes, plotting final accuracy against final option- A rate; faint points are individual seeds and larger markers are model means. Recovery improves some models but does not uniformly restore low-shortcut behavior.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Dense unbiased-validation accuracy over training steps. Complements the stage-checkpoint view in Figure 2 .

Figure 6: Final option-letter distribution on unbiased test ( $\%A$ , $\%B$ , $\%C$ , $\%D$ ) by model and training condition.

Condition	Harmful in A	Harmful in B
Base	0.04 [0.01, 0.14]	0.00 [0.00, 0.07]
Unbiased	0.04 [0.01, 0.14]	0.00 [0.00, 0.07]
Biased	0.84 [0.71, 0.92]	0.00 [0.00, 0.07]
Recovered	0.70 [0.56, 0.81]	0.00 [0.00, 0.07]

Appendix

Table 2: Position-swap control on the value-laden prompt for Qwen2.5-1.5B. Each cell is the harmful-option selection rate over 50 samples, with 95% Wilson intervals. Harmful in A is the original placement; harmful in B is the swapped placement. Holding content fixed, moving the harmful option from A to B collapses the biased model’s harmful selection from

0.84

0.00

, while base and unbiased checkpoints stay near zero in both placements. The placement

\times

condition interaction isolates a positional policy from a content preference.

Figure 7: Exploratory value-laden binary-choice transfer probe. The harmful option is placed in A and the honest-work option in B . Panel A shows harmful- A selection rate over 50 sampled completions per checkpoint; Panel B shows refusal rate, with honest-work B selections making up the remainder. Qwen2.5-1.5B shows the clearest shift, rising from 4% harmful- A selections for both base and unbiased checkpoints to 84% after biased training, and remaining at 70% after recovery. Qwen2.5-3B shows a smaller shift, from 4% to 26%, while most larger Qwen and Llama checkpoints stay near 0% harmful- A on this prompt.

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 182 papers

Sep 29, 2026

Beyond Compression: Diagnosing How Post-Training Changes Mathematical Reasoning

Post-training is central to mathematical reasoning in modern large language models (LLMs), but endpoint pass@1 alone underidentifies what has changed. Gains may reflect newly reachable solutions, cheaper sampling of latent solutions, surface robustness, or memorisation. We compare three post-training paths under a common diagnostic readout: our sufficiently trained off-policy distillation trajectories, released Qwen3 off-policy-plus-on-policy distillation endpoints, and a released DeepSeek-Math endpoint trained with Group Relative Policy Optimisation (GRPO). Our probe uses cross-surface pass@K over verbatim prompts, paraphrases, numerical isomorphisms, and translations, plus consistency, distribution-shape, and verified supervised-fine-tuning (SFT) membership analyses. We find two regimes. On easier AMC problems, large-K ceilings are near saturation, so post-training mainly compresses sample cost. On harder AIME problems, post-training expands the large-K ceiling over the base model: sufficient off-policy distillation already raises this ceiling, Qwen3 released endpoints raise it further, and DeepSeek-Math GRPO does not dominate sufficient off-policy distillation at large K. English-dominant distillation improves non-English reasoning but preserves language-tier gaps. A controlled-overfit audit finds limited sensitivity in current SFT-membership probes. Compression is one regime of post-training, not a universal explanation.

Hongyang Li, Yiming Zhu, Xiao Li +3

University of Luxembourg · Seafill Open-Source Community · Université Paris-Saclay

Language Model Distillation RL Post-Training LLM Post-Training LLM Mathematical Reasoning

Ask PDF

Oct 5, 2026

COMPASS: Finding Where Reasoning Lives in Language Models

Explicitly eliciting reasoning substantially improves LLM performance. Existing approaches require a predefined characterization of reasoning, whether through CoT prompt design, contrastive CoT directions, or via SAE derived reasoning features. For mathematical reasoning with verifiable answers, we show that a much simpler signal suffices, which is the correctness of the model's own direct answer attempts. This signal yields a latent direction that elicits reasoning. This direction is decodable within the activations of most attention heads, but only a small subset of them can be effectively intervened. We introduce COMPASS, an inference-time steering method that identifies these heads using a logit-space attribution score and steers their activations along the correctness direction, requiring only per-head activation statistics. Across three model families and multiple math benchmarks, COMPASS outperforms the activation-steering baselines we compare against, improves GSM8K accuracy by 16 percentage points on average, and approaches CoT accuracy with 20-70% fewer generated tokens. Interventions transfer without re-fitting to unseen benchmarks, and ablations show that both the correctness direction and the small set of heads carrying it are necessary, with the effect concentrated in remarkably few heads.

Pratyay Dutta, Kowshik Thopalli, Vivek Narayanaswamy

University of California, Riverside, CA 92521. Work performed during a summer internship at Lawrence Livermore National Laboratory · Lawrence Livermore National Laboratory, Livermore, CA 94550.

Efficient Language Model Reasoning Activation Steering LLM Interpretability Numerical Reasoning in Language Models

Ask PDF

May 27, 2026

GeneralThinker: Domain-General Reasoning through Likelihood-Guided Answer-Conditioned Optimization

Reinforcement learning with verifiable rewards improves language model reasoning, but its reliance on domain-specific verifiers, sparse outcome rewards, and coarse-grained credit assignment limits its applicability. We introduce GeneralThinker, an on-policy framework that reformulates reasoning supervision as dense answer-conditioned optimization, enabling response-level evaluation and token-level credit assignment without domain-specific verifiers. GeneralThinker evaluates generated reasoning trajectories using the likelihood of the ground-truth answer and derives token-wise compatibility signals for fine-grained credit assignment. To stabilize optimization, it constrains token-level updates through clipping and direction-preserving modulation. Across 11 benchmarks spanning mathematics, STEM, and general reasoning, GeneralThinker achieves the best average performance. Further analyses show that uncontrolled token-level modulation can destabilize training, whereas controlled modulation makes fine-grained credit assignment consistently effective.

Shengmin Piao, Sanghyun Park

Yonsei University Seoul, South Korea

Token-Level Credit Assignment RL for Language Model Reasoning RL Fine-Tuning LLM Reasoning

Ask PDF

Abstract

Reward Hacking

Explore similar work

Beyond Compression: Diagnosing How Post-Training Changes Mathematical Reasoning

COMPASS: Finding Where Reasoning Lives in Language Models

GeneralThinker: Domain-General Reasoning through Likelihood-Guided Answer-Conditioned Optimization