Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense

Yushu Li; Jiin Woo; Xiaoxiao Li; Youngsuk Park

cs.LGOct 15, 2026

Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense

Authors: Yushu Li, Jiin Woo, Xiaoxiao Li, Youngsuk Park

Organizations: University of British Columbia · Amazon Annapurna Labs

Abstract

To improve task performance while encouraging desirable response properties, LLM post-training often combines a primary reward (e.g., correctness) with auxiliary rewards (e.g., brevity or efficiency). However, how these rewards are aggregated can critically affect training outcomes. We show that symmetric aggregation, which treats reward channels equally, can induce a failure where aggregate reward improves while primary-task performance degrades, even when individual rewards are meaningful. We investigate this aggregation-induced failure through a one-step multi-reward decision formulation, characterizing when it constitutes multi-reward hacking. We prove that, under symmetric aggregation, deterministic conditional-mean GRPO converges to auxiliary-favoring behavior when auxiliary rewards are more reliably obtained, regardless of initialization. We propose Trust-Anchored Gating (TAG), which first elects an anchor channel based on the policy drift induced by optimizing each channel alone, then gates auxiliary credit on anchor satisfaction. Unlike calibration-sensitive static reweighting, TAG structurally eliminates credit for auxiliary-only outcomes while preserving additional credit for joint success. In the same analytical setting, TAG provably reverses this selection, converging to the primary-favoring action from any initialization when the primary channel is anchored. Experiments across Qwen and Llama models on mathematics and program synthesis reveal substantial degradation under symmetric, constrained, and normalized aggregation, whereas TAG mitigates this failure. On Qwen3-4B with a brevity reward, TAG achieves 0.581 pass@4 versus 0.220 for additive aggregation, while shortening responses by 44% relative to correctness-only training.

Figures & tables

Panel of Figure 1 — Figure 1: Multi-reward hacking (Qwen3-1.7B; $R_{\mathrm{P}}$ : correctness, $R_{\mathrm{A}}=\mathbf{1}[\text{len}<200]$ : brevity). Summed, the two rewards yield the bare template, short and wrong; TAG credits $R_{\mathrm{A}}$ only when $R_{\mathrm{P}}=1$ ; its response is short and right.

Panel of Figure 2 — Figure 2: The two-channel MDP. Outcome distributions of the auxiliary-favoring action $a_{1}$ and the primary-favoring action $a_{2}$ .

Figure 3: Symmetric aggregation selects the auxiliary-favoring action $a_{1}$ over the primary-favoring action $a_{2}$ . Exact conditional-mean recursions ( $n=8$ , $\alpha=0.1$ , $\rho=0.9$ , $\eta=0.1$ ) from $\pi_{\theta_{0}}=(0.5,0.05,0.45)$ . GRPO, Dr.GRPO and exact policy gradient first raise $\pi_{\theta_{t}}(a_{2}\mid x)$ and then hand the policy to $a_{1}$ , as does MO-GRPO (Appendix A.1 ); TAG keeps $a_{2}$ . Right: for an assumed true reward that favors the primary channel, $R^{\star}=0.9R_{\mathrm{P}}+0.1R_{\mathrm{A}}$ , the symmetric rules first gain and then lose true reward as $a_{1}$ takes over, while TAG retains it.

Figure 4: The reverse KL sees how far an invented mode lies. Top: base $\pi_{0}$ and a probe $\pi_{T}$ that adds a mode at distance $\Delta$ . Bottom: $D_{\mathrm{KL}}(\pi_{T}\|\pi_{0})$ grows with $\Delta^{2}$ ; $D_{\mathrm{KL}}(\pi_{0}\|\pi_{T})$ stays at the added mass (Lemma B.1 ).

		Qwen3-1.7B					Qwen3-4B
		Correct & brief			Correct & negative		Correct & brief			Correct & negative
Advantage	Aggregation $f$	pass@4 $\uparrow$	len $\downarrow$	both $\uparrow$	pass@4 $\uparrow$	neg. rate $\downarrow$	pass@4 $\uparrow$	len $\downarrow$	both $\uparrow$	pass@4 $\uparrow$	neg. rate $\downarrow$
Base model	/	0.140	479	0.006	0.140	0.007	0.076	497	0.003	0.076	0.005
GRPO	$R_{\mathrm{P}}$ -only	0.481	328	0.043	0.481	0.036	0.634	310	0.118	0.634	0.039
	$(R_{\mathrm{P}}{+}R_{\mathrm{A}})/2$	0.138	9	0.119	0.026	0.999	0.220	9	0.207	0.041	1.000
	$0.9R_{\mathrm{P}}{+}0.1R_{\mathrm{A}}$	0.202	40	0.154	0.040	0.999	0.213	9	0.200	0.616	0.041

Table 1: Main grid (Qwen3-1.7B and Qwen3-4B). Held-out pass@4 and the rewarded behavior at the end of training: under Correct & brief the mean response length (len, tokens) and the rate of correct-and-short responses (both); under Correct & negative the rate of negative final answers. Shaded: pass@4 below the base model. Best bold, second underlined. Auxiliary rates: Table 7 .

Figure 5: The auxiliary rate rises first and correctness follows it down. Held-out $R_{\mathrm{P}}$ and $R_{\mathrm{A}}$ during training on Qwen3-1.7B under Correct & brief and Correct & negative; arms as in Table 1 , dotted line the base model. Most tested methods exhibit auxiliary takeover, with exceptions depending on the reward rule and setting; under TAG the auxiliary rate rises only as far as correct responses carry it. Qwen3-4B: Figure 9 .

Figure 6: Reweighting only delays the takeover. Held-out $R_{\mathrm{P}}$ for $w_{A}\in\{0.1,\dots,0.5\}$ ; TAG and $R_{\mathrm{P}}$ -only for reference.

		Qwen3-1.7B					Qwen3-4B
		Correct & fast			Correct & no-op		Correct & fast			Correct & no-op
Advantage	Aggregation $f$	pass@4 $\uparrow$	lat. $\downarrow$	both $\uparrow$	pass@4 $\uparrow$	rate $\downarrow$	pass@4 $\uparrow$	lat. $\downarrow$	both $\uparrow$	pass@4 $\uparrow$	rate $\downarrow$
Base model	/	0.176	137	0.059	0.176	0.641	0.459	383	0.113	0.459	0.297
GRPO	$R_{\mathrm{P}}$ -only	0.325	359	0.103	0.325	0.318	0.593	486	0.164	0.593	0.255
GRPO	$(R_{\mathrm{P}}{+}R_{\mathrm{A}})/2$	0.096	12	0.043	0.000	0.984	0.350	63	0.201	0.108	0.914
Dr.GRPO	$(R_{\mathrm{P}}{+}R_{\mathrm{A}})/2$	0.132	11	0.066	0.008	0.976	0.391	91	0.163	0.075	0.930

Table 2: Program synthesis on DeepCoder-RT (Qwen3-1.7B and Qwen3-4B). Held-out pass@4 at the end of training and, under Correct & fast, the mean running time of the programs that ran (largest CPU time over the executed tests, timeouts excluded; lat., ms) and the rate of programs that are correct and faster than the reference solution (both); under Correct & no-op the rate of programs that exit within

3

ms. Shaded: pass@4 below the base model. Best bold, second underlined (pass@4 and both only). Protocol in Appendix C.4 .

Figure 7: Anchor-election probes. Left: Correct & brief and Correct & negative on mathematics; right: Correct & fast and Correct & no-op on program synthesis. Each auxiliary probe is paired with the correctness probe of its setting, and the correctness probe drifts less in every setting; values and the forward KL in Table 4 .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

	exact PG / Dr.GRPO	GRPO, $\eta=0$ : largest safe $w_{2}/w_{1}$		GRPO, $\eta=0.1$
$\rho$	crossing $w_{2}/w_{1}$	from $(0.1,0.05,0.85)$	from $(0.3,0.6,0.1)$	$w_{1}/w_{2}$ reaching $\mathbb{E}_{a_{2}}[R^{\star}]$
0.9	0.9	1	1	1.1
0.7	0.7	0.88	0.88	1.3
0.5	0.5	0.52	0.29	2.0
0.3	0.3	0.21	0	3.9

Appendix

Table 3: Weight thresholds of the conditional-mean recursions (

n=8

\alpha=0.1

). Exact policy gradient and Dr.GRPO select

a_{2}

for

w_{2}/w_{1}<\rho

and

a_{1}

for

w_{2}/w_{1}>\rho

(expected-reward crossing). Middle columns: at

\eta=0

, the largest ratio

w_{2}^{\star}/w_{1}

below which the GRPO recursion selects

a_{2}

(bisection over

[10^{-3},1]

) from a primary-dominated and an auxiliary-favored initialization;

1

means that every

w_{2}<w_{1}

selects

a_{2}

0

that

a_{1}

is selected for every

w_{2}/w_{1}\geq 10^{-3}

. Last column: in the shared joint-success model at

\eta=0.1

from

\pi_{\theta_{0}}=(0.5,0.05,0.45)

, the smallest grid ratio

w_{1}/w_{2}

at which the converged true reward

J^{\star}

(

R^{\star}=0.9R_{\mathrm{P}}+0.1R_{\mathrm{A}}

) lies within

0.01

\mathbb{E}_{a_{2}}[R^{\star}]

(grid values

1.14

1.30

1.96

3.88

). TAG selects

a_{2}

for every

\rho>0

(Theorem 2 ).

Model	Setting	auxiliary	correctness	ratio	auxiliary	correctness	ratio
		$\mathrm{KL}^{\mathrm{rev}}$ / token			$\mathrm{KL}^{\mathrm{fwd}}$ / token
Qwen3-1.7B	Correct & brief	3.47	0.29	$12.0\times$	0.16	0.16	$1.0\times$
	Correct & negative	4.25	0.18	$23.4\times$	0.36	0.13	$2.8\times$
	Correct & fast	0.82	0.069	$11.9\times$	0.50	0.097	$5.2\times$
	Correct & no-op	1.11	0.069	$16.2\times$	0.50	0.097	$5.2\times$
Qwen3-4B	Correct & brief	4.18	1.01	$4.2\times$	0.15	0.68	$0.2\times$

Appendix

Table 4: Anchor-election probes in both KL directions (per token;

T=300

, 256 held-out prompts). Each auxiliary probe is paired with the correctness probe of the same setting; ratio = auxiliary/correctness. The reverse KL separates every pair (Figure 7 ); the forward KL fails on mathematics (inverted on Qwen3-4B) and is less sharp on program synthesis.

Figure 8: Forward vs reverse KL under an invented mode (Monte Carlo, $N=2048$ samples). Left: the two distributions. Middle: $\varepsilon$ sweep at $\Delta=6$ — both KLs are proportional to $\varepsilon$ , $\mathrm{KL}^{\mathrm{rev}}$ higher by $\ell-1$ . Right: $\Delta$ sweep at $\varepsilon=5\%$ — $\mathrm{KL}^{\mathrm{fwd}}$ is flat at $\approx\varepsilon$ whatever the separation, $\mathrm{KL}^{\mathrm{rev}}$ grows with $\Delta^{2}$ .

$\delta$	$\mathrm{KL}^{\mathrm{fwd}}$ on/off	excess	$z$	$\mathrm{KL}^{\mathrm{rev}}$ on/off	excess	$z$	$\mathrm{KL}^{\mathrm{rev}}-\mathrm{KL}^{\mathrm{fwd}}$
0.3	0.094 / 0.043	$+117\%$	7.8	0.745 / 0.043	$+1614\%$	108	0.65
0.5	0.171 / 0.120	$+43\%$	4.7	0.819 / 0.120	$+583\%$	65	0.65
1.0	0.528 / 0.477	$+11\%$	2.4	1.161 / 0.477	$+143\%$	32	0.63
2.0	2.051 / 2.000	$+3\%$	1.2	2.624 / 2.000	$+31\%$	14	0.57

Appendix

Table 5: Invented mode on vs off at

\Delta=6

\varepsilon=5\%

under a benign drift

\delta

;

z=(\text{on}-\text{off})/(\sigma_{\rm off}/\sqrt{N})

N=2048

	Qwen3-1.7B / 4B	Llama-3.2-3B / 3.1-8B	Qwen3-30B-A3B	Program synthesis
Training data	Big-Math hard split	Llama-calibrated Big-Math subset	Big-Math hard split	DeepCoder-RT
Response budget (tok)	512	1024	4096	4096
Rollout batch	256	256	256	256
Training steps	500	500	500	500
Probe horizon $T$ (steps, rollout batch)	300, 32	300, 32	300, 32	300, 32
Held-out problems	1,498	1,500	1,498	185

Appendix

Table 6: Training and evaluation configuration by testbed. All arms use GRPO with learning rate

10^{-6}

, no KL penalty (except the additive

+

KL arms), clipping

0.2/0.28

; Llama arms sample at temperature

0.6

, top-

p

0.9

. The compatible auxiliary channel rewards responses shorter than the listed threshold (math) or fast execution (program synthesis); the incompatible one rewards a negative final answer (math) or execution faster than the listed latency on problems whose reference solution needs at least

10

ms (program synthesis).

Figure 9: Training trajectories (Qwen3-4B): Figure 5 repeated for the larger model, with the same arms, panels and evaluation cadence.

	Qwen3-1.7B				Qwen3-4B
	Correct & brief		Correct & negative		Correct & brief		Correct & negative
Aggregation	$R_{\mathrm{A}}$	$R_{\mathrm{P}}$	$R_{\mathrm{A}}$	$R_{\mathrm{P}}$	$R_{\mathrm{A}}$	$R_{\mathrm{P}}$	$R_{\mathrm{A}}$	$R_{\mathrm{P}}$
Additive	1.000	0.119	0.999	0.021	1.000	0.207	1.000	0.034
Weighted sum $0.9R_{\mathrm{P}}{+}0.1R_{\mathrm{A}}$	0.996	0.155	0.999	0.027	1.000	0.200	0.041	0.492
MO-GRPO	0.999	0.137	0.999	0.024	1.000	0.202	1.000	0.020
AND ( $\min$ )	1.000	0.125	0.039	0.215	1.000	0.205	0.761	0.037

Appendix

Table 7: Main grid (Qwen), auxiliary rate and pass@1. Held-out auxiliary-behavior rate

R_{\mathrm{A}}

and accuracy

R_{\mathrm{P}}

(pass@1) at the end of training for the arms of Table 1 and the additional ports of Appendix C.1 , same evaluation.

	Qwen3-1.7B (ref. $R_{\mathrm{P}}$ $0.310$ )			Qwen3-4B (ref. $R_{\mathrm{P}}$ $0.503$ )
Aggregation	neg. answer	short	$R_{\mathrm{P}}$	neg. answer	short	$R_{\mathrm{P}}$
Base model (step $0$ )	0.007	0.024	0.068	0.005	0.011	0.049
Additive (three channels)	1.000	1.000	0.016	1.000	1.000	0.018
Rubric Dropout (three channels) ( Yang et al., 2026 )	1.000	1.000	0.016	1.000	1.000	0.018
TAG (three channels)	0.032	0.594	0.255	0.042	0.661	0.452

Appendix

Table 8: Three channels trained simultaneously (negative answer, brevity, correctness; cap 512, end of training). Held-out rates of the two auxiliary behaviors and held-out accuracy. Additive and Rubric Dropout cascade through both auxiliary behaviors; TAG keeps the negative-answer rate at the

R_{\mathrm{P}}

-only level, collects the compatible brevity signal, and retains

0.82\times

and

0.90\times

of the

R_{\mathrm{P}}

-only accuracy.

	Held-out		Outcome cells
$\lambda$	$R_{\mathrm{A}}$	$R_{\mathrm{P}}$	$00$	$01$	$10$	$11$
0.25	0.526	0.262	0.355	0.383	0.120	0.142
0.5	0.432	0.270	0.430	0.300	0.138	0.132
1 (Table 1 )	0.550	0.282	0.332	0.386	0.118	0.164
2	0.620	0.256	0.294	0.451	0.087	0.169
4	0.612	0.254	0.298	0.447	0.089	0.165

Appendix

Table 9:

\lambda

sensitivity of TAG (Qwen3-1.7B, Correct & brief, cap 512, end of training). Held-out short-response rate

R_{\mathrm{A}}

and accuracy

R_{\mathrm{P}}

, the four outcome cells (first digit

R_{\mathrm{P}}

, second

R_{\mathrm{A}}

Figure 10: $\lambda$ sensitivity of TAG (Qwen3-1.7B, Correct & brief, cap 512, end of training). Held-out $R_{\mathrm{P}}$ , $R_{\mathrm{A}}$ and the auxiliary-only and joint-success cells against $\lambda$ on a log scale; dashed and dotted lines mark the $R_{\mathrm{P}}$ -only reference and the base model.

Arm (Qwen3-30B-A3B, cap 4096)	Correct & brief	Correct & negative
$R_{\mathrm{P}}$ -only	0.708	0.708
Additive	0.579	0.044
TAG	0.663	0.711

Appendix

Table 10: Qwen3-30B-A3B-Instruct-2507 (Big-Math hard split, cap 4096). Held-out

R_{\mathrm{P}}

at the end of training. Under Correct & brief the additive arm oscillates around the short template and ends at

0.82\times

the

R_{\mathrm{P}}

-only reference, TAG at

0.94\times

; under Correct & negative the additive arm collapses to

0.06\times

while TAG stays at

1.00\times

. The

R_{\mathrm{P}}

-only reference is shared by both settings.

	Llama-3.2-3B			Llama-3.1-8B
Aggregation	pass@4 $\uparrow$	pass@1 $\uparrow$	neg. rate $\downarrow$	pass@4 $\uparrow$	pass@1 $\uparrow$	neg. rate $\downarrow$
Base model (step $0$ )	0.811	0.499	0.069	0.873	0.551	0.065
$R_{\mathrm{P}}$ -only	0.939	0.767	0.072	0.926	0.766	0.075
Additive	0.081	0.065	0.990	0.075	0.061	0.999
AND ( $\min$ )	0.083	0.067	0.948	0.235	0.121	0.824
Dr.GRPO (additive) ( Liu et al., 2025 )	0.077	0.065	1.000	0.096	0.077	1.000

Appendix

Table 11: Second model family (Llama), Correct & negative (Llama-calibrated Big-Math subset, cap

1024

). Held-out pass@4, pass@1 (the accuracy

R_{\mathrm{P}}

) and the rate of negative final answers; arrows and bold as in Table 1 . Every symmetric and constrained aggregator collapses on both models while TAG stays at the

R_{\mathrm{P}}

-only reference. Evaluation sets differ across families, so absolute values are not comparable with Table 1 ; baselines as in Table 1 . The Correct & brief setting is summarized in the text of this section.

Panel of Figure 11 — Figure 11: Real outputs on one held-out problem per setting, I: mathematics (Correct & brief and Correct & negative) (Qwen3-1.7B; final evaluation of each arm). Each panel shows the prompt, the response of the symmetric-aggregation arm ( $(R_{\mathrm{P}}{+}R_{\mathrm{A}})/2$ ), the single-reward reference ( $R_{\mathrm{P}}$ only) and TAG ; the header of each response states whether it is correct, whether it meets the auxiliary condition, and its length; [ … ] marks omitted lines, and shaded lines carry the behavior named in the panel note. Under Correct & brief the symmetric-aggregation arm emits the bare template with a guessed number; under Correct & negative it appends a negative constant to a sentence of pseudo-reasoning; TAG solves both problems.

Panel of Figure 12 — Figure 12: Real outputs on one held-out problem per setting, II: program synthesis (Correct & no-op and Correct & fast) (Qwen3-1.7B; layout as in Figure 11 ; code panels show the core lines re-wrapped to the column; prompts abridged where marked). In both settings the symmetric-aggregation arm prints a constant without reading the input; under Correct & fast TAG replaces the reference’s $O(N^{2})$ loop by an $O(N)$ prefix-sum pass.

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 182 papers

Aug 31, 2026

Uncovering and Mitigating Aggregation-Induced Reward Hacking in Multi-Reward Reinforcement Learning

Reinforcement learning fine-tuning of large language models increasingly adopts multiple reward dimensions, including verifiable rules, task-specific evaluators, and learned reward models, to provide richer supervision across diverse capabilities. These dimensions are commonly scalarized with fixed aggregation weights. We identify a failure mode in which aggregation itself induces reward hacking: static projection aliases qualitatively different reward profiles into a single scalar, steering optimization toward whichever dimensions are easiest, densest, or systematically favored by the reward signal. Over training, this traps the policy in suboptimal profiles and prevents convergence to better-balanced ones that would yield higher task performance. To address this, we propose Adaptive Multi-Reward Projection (AMRP), a lightweight online method that reallocates aggregation weights using three signals, relative shortfall, reward volatility, and recent progress, increasing pressure on lagging, unstable, or stagnant dimensions while relieving saturated ones. Across structured reasoning, citation-grounded generation, and open-ended alignment under GRPO, AMRP consistently improves reward-profile balance and downstream performance over fixed and dynamic weighting baselines; it also remains effective with GDPO and PPO, supporting compatibility across RL algorithms. Our code is available at https://github.com/yyhappier/AMRP.git.

Yu Yuan, Yaoyou Fan, Lili Zhao +5

University of Science and Technology of China · 2State Key Laboratory of Cognitive Intelligence · 3Peking University +1

Multi-Objective Reinforcement Learning Reward Hacking RL Fine-Tuning

Ask PDF

Oct 27, 2026

CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning

When a Large Language Model (LLM) is trained with Reinforcement Learning (RL) on multiple tasks, a task's primary reward can gate auxiliary rewards such as a length reward: on a failing response, an auxiliary cannot exceed the primary. However, group-relative advantage estimators treat a gated auxiliary like any other component, so it can reverse the primary's order, and at the primary's ceiling only auxiliaries order responses. Keeping gated auxiliaries hurts some tasks, while dropping them hurts others. We propose CONCUR, an advantage estimator that replaces the standardized scores of each gated auxiliary with the closest scores that satisfy two conditions: responses with higher primary rewards cannot receive lower scores, and responses at the primary's ceiling receive equal scores. CONCUR-T applies both conditions throughout training, and CONCUR-A lifts the second for an auxiliary that agrees with the primary's order more often than it reverses it among passing responses. On three models from two families and across math, code, instruction following, and writing, both variants score higher on average than all baselines and than when all the gated auxiliaries are dropped.

Jixuan Leng, Si Si, Fei Wang +4

CMU · Google DeepMind · Google +1

Reinforcement Learning RL for Language Models Multi-Objective Reinforcement Learning Multi-Task RL

Ask PDF

Nov 11, 2025

Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents

Large Language Models (LLMs) in multi-agent systems (MAS) have shown promise for complex tasks, yet current training methods lack principled ways to connect system-level evaluation with agent- and message-level learning. We propose a theoretical framework that unifies cooperative game-theoretic attribution with process reward modeling to transform system evaluation to agent credit to response-level signals. Unlike prior approaches that rely only on attribution (Shapley) or step-level labels (PRM), our method produces local, signed, and credit-conserving signals. In success cases, Shapley-based credit assignment fairly allocates outcomes across agents and is refined into per-message rewards that promote cooperation while discouraging redundancy or sabotage; in failure cases, first-error localization yields repair-aware preferences that penalize harmful steps while rewarding corrective attempts. The resulting signals are bounded, cooperative, and directly compatible with reinforcement- or preference-based post-training, providing a unified and auditable pathway from global evaluation to local supervision in LLM multi-agent training. Our contribution is conceptual: we present a theoretical foundation and training signals, leaving empirical validation for future work.

Chih-Hsuan, Yang, Tanwi Mallick +5

Bella · Argonne National Laboratory · & University of Chicago

Multi-Agent LLM Systems Cooperative Game Theory Process Reward Models Credit Assignment in RL LLM Agent Training

Ask PDF

Abstract

Reward Hacking

Explore similar work

Uncovering and Mitigating Aggregation-Induced Reward Hacking in Multi-Reward Reinforcement Learning

CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning

Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents