CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning

Jixuan Leng; Si Si; Fei Wang; Cho-Jui Hsieh; Jiong Zhang; Hsiang-Fu Yu; Inderjit S. Dhillon

cs.CLOct 27, 2026

CONCUR: Concordant Auxiliary Rewards for Multi-Task, Multi-Reward Reinforcement Learning

Authors: Jixuan Leng, Si Si, Fei Wang, Cho-Jui Hsieh, Jiong Zhang, Hsiang-Fu Yu, Inderjit S. Dhillon

Organizations: CMU · Google DeepMind · Google · UCLA

Abstract

When a Large Language Model (LLM) is trained with Reinforcement Learning (RL) on multiple tasks, a task's primary reward can gate auxiliary rewards such as a length reward: on a failing response, an auxiliary cannot exceed the primary. However, group-relative advantage estimators treat a gated auxiliary like any other component, so it can reverse the primary's order, and at the primary's ceiling only auxiliaries order responses. Keeping gated auxiliaries hurts some tasks, while dropping them hurts others. We propose CONCUR, an advantage estimator that replaces the standardized scores of each gated auxiliary with the closest scores that satisfy two conditions: responses with higher primary rewards cannot receive lower scores, and responses at the primary's ceiling receive equal scores. CONCUR-T applies both conditions throughout training, and CONCUR-A lifts the second for an auxiliary that agrees with the primary's order more often than it reverses it among passing responses. On three models from two families and across math, code, instruction following, and writing, both variants score higher on average than all baselines and than when all the gated auxiliaries are dropped.

Figures & tables

Figure 1: Left: GDPO standardizes reward components in a group and sums them at the declared weights, and the insets show where an auxiliary departs from its primary. Right: CONCUR projects each gated auxiliary’s scores onto the closest ones that concur with the primary, and in CONCUR-A earned trust decides whether a ceiling is tied.

Figure 2: GDPO at the declared weights on Qwen3-4B-Base , one line per gated pair. The panels show the auxiliary’s cosine with its primary, its shares of energy against the order and at the ceiling from Equation 3 , as 10-step running means, and the cumulative mean of the agreement of Equation 4 , which is not defined for Math’s binary primary.

	Math							Code				IF				Writing
	AIME24	AIME25	MATH	OBench	Minerva	AMC23	Mean	HEval+	MBPP+	LCB-v6	Mean	IFEval	IFBench	GBench	Mean	WBench	CW-v3	LBWrite	Mean	Avg
Qwen3-4B-Base
Base	0 7.81	0 7.08	67.26	35.62	31.61	43.91	32.21	65.06	55.58	13.20	44.61	47.36	17.00	24.86	29.74	32.42	24.76	44.44	33.87	34.53
GRPO	11.98	12.60	80.25	44.63	45.28	54.69	41.57	82.74	71.16	24.00	59.30	86.45	37.67	77.93	67.35	74.25	47.32	69.21	63.59	54.68
GDPO	0 9.06	0 9.90	79.29	44.66	45.48	59.61	41.33	81.83	70.87	24.40	59.03	85.73	36.33	80.61	67.56	71.94	45.97	69.01	62.31	54.31
GD 2 PO	11.88	0 6.77	77.67	43.55	47.19	57.73	40.80	83.41	71.46	23.49	59.45	87.65	37.67	79.08	68.13	70.16	44.68	68.94	61.26	54.09

Table 1: Evaluation scores of the seven estimators and the two variants of CONCUR. Each model’s best score is in bold, its second best is underlined, and the base policy is in gray italics. Eight benchmark names are shortened: MATH is MATH-500, OBench is OlympiadBench, HEval+ is HumanEval+, LCB-v6 is LiveCodeBench v6, GBench is GuideBench, WBench is WritingBench, CW-v3 is Creative Writing v3, and LBWrite denotes LongBench-Write.

Panel of Figure 3 — Figure 3: Panels (a) to (c): each task group’s score and Avg minus those of GDPO. Panel (d): each gated auxiliary’s mean score over the passing responses, as the mean over the last 20 training steps minus the base policy’s level.

Figure 4: Cosine between each task’s advantages and the standardized scores of its primary (top) and the primary’s mean score over all responses (bottom) during training on Qwen3-4B-Base , shown as 10-step running means.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Primary	Gated auxiliaries	Not gated
Math	correctness (binary)	length reward	–
Code	unit-test pass rate	length reward	–
IF	rule-checked criteria met	length reward	judged criterion
Writing	judged criteria met	length reward, RM	–

Appendix

Table 2: Reward components of the mixture: a task’s primary gates each auxiliary in its row, all at weight

1

Task group	Benchmark	Problems	Responses	Response Len	Temperature	Top- $p$	Other	Metric	Judge
Math	AIME24	30	32	32,768	0.6	0.95	–	accuracy	–
	AIME25	30	32	32,768	0.6	0.95	–	accuracy	–
	MATH-500	500	32	32,768	0.6	0.95	–	accuracy	–
	OlympiadBench	674	32	32,768	0.6	0.95	–	accuracy	–
	Minerva	272	32	32,768	0.6	0.95	–	accuracy	–
	AMC23	40	32	32,768	0.6	0.95	–	accuracy	–

Appendix

Table 3: Settings of the evaluation for each benchmark. Every metric is reported on a scale from 0 to 100.

Hyperparameter	Value
Backend	megatron
Rollout	sglang
Optimizer	Adam
Learning Rate	1e-6
LR Scheduler	constant
Weight Decay	0.0

Appendix

Table 4: Hyperparameters of the runs.

	Cosine		Against order		At ceiling		Passing at ceiling		Agreement
Pair	Early	Late	Early	Late	Early	Late	Early	Late
Math	0.89	0.89	0.00	0.00	0.19	0.19	1.00	1.00	–
Code	0.85	0.79	0.03	0.03	0.09	0.04	0.70	0.74	+0.003
IF	0.75	0.48	0.04	0.05	0.05	0.18	0.29	0.57	+0.015
Writing	0.51	0.04	0.29	0.60	0.00	0.00	0.00	0.00	$-$ 0.036
Writing (RM)	0.67	0.15	0.22	0.51	0.00	0.00	0.00	0.00	+0.058

Appendix

Table 5: The gated auxiliaries under GDPO at the declared weights on Qwen3-4B-Base , early (steps 1–20) and late (the last 20). Passing at ceiling is the share of passing responses at the ceiling, and Agreement is pooled over the run.

Qwen3-4B-Base
	Passing	Pooled	In standard errors			Not
Pair	at ceiling	agreement	-A	-T	-O	tied
Math	1.00	–	–	–	–	0.00
Code	0.74	+0.012	+4.2	+1.9	+1.4	0.85
IF	0.55	+0.018	+7.4	+6.5	+7.0	0.95
Writing	0.02	$-$ 0.054	$-$ 36.8	$-$ 32.7	$-$ 35.1	0.00

Appendix

Table 6: What earned trust reads and decides for each gated pair: the share of passing responses at the ceiling over the last 20 training steps, the pooled agreement of CONCUR-A at its last training step, each run’s pooled agreement in standard errors, which CONCUR-T and CONCUR-O only log, and the share of all training steps at which CONCUR-A leaves the pair’s ceiling not tied.

Panel of Figure 5 — Figure 5: Each gated auxiliary’s mean score over the passing responses on Qwen3-8B-Base (left) and OLMo-3-7B-Instruct-SFT (right): the mean over the last 20 training steps minus the base policy’s level of the same model.

Figure 6: The primary’s mean score over all responses during training on Qwen3-8B-Base (top) and OLMo-3-7B-Instruct-SFT (bottom) under GDPO, the reference, and CONCUR, shown as 10-step running means.

	Math							Code				IF				Writing
	AIME24	AIME25	MATH	OBench	Minerva	AMC23	Mean	HEval+	MBPP+	LCB-v6	Mean	IFEval	IFBench	GBench	Mean	WBench	CW-v3	LBWrite	Mean	Avg
Qwen3-4B-Base
GDPO	0 9.06	0 9.90	79.29	44.66	45.48	59.61	41.33	81.83	70.87	24.40	59.03	85.73	36.33	80.61	67.56	71.94	45.97	69.01	62.31	54.31
GDPO ( $w=0.5$ )	14.58	10.31	81.79	48.46	46.12	58.36	43.27	82.01	71.19	24.17	59.12	86.81	39.33	79.56	68.57	70.74	38.11	67.41	58.75	54.60
GDPO (no aux.)	23.44	18.75	84.76	53.16	49.52	65.70	49.22	85.49	72.33	24.74	60.85	84.77	40.33	75.91	67.01	67.35	39.33	64.36	57.01	56.66
GRPO (no std)	12.40	0 8.33	79.49	44.81	45.80	54.53	40.89	80.49	71.19	24.46	58.71	87.29	33.67	78.79	66.58	65.63	42.53	66.00	58.06	53.03

Appendix

Table 7: Evaluation scores of GDPO, GDPO (

w=0.5

), the reference GDPO (no aux.), GRPO (no std), CONCUR-O, and the two variants of CONCUR on every benchmark, with the same benchmark names and formatting as Table 1 .

Reinforcement Learning

Reinforcement learning studies how agents learn sequential decisions from rewards and interaction, balancing exploration, performance, safety, and adaptation.

See all 1,727 papers

Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense
Oct 15, 2026Yushu Li, Jiin Woo, Xiaoxiao Li +1Reward Hacking RL for Language Models
University of British Columbia · Amazon Annapurna Labs
PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs
Aug 31, 2026Yuanqiang Yu, Yanzhao Zheng, Zhentao Zhang +8RL for Language Model Reasoning Curriculum RL
Alibaba Group, Hangzhou, China
Lexicographic Multi-Objective On-Policy Distillation
Oct 1, 2026Doseok Jang, Jon Ander Campos, Youran QiMulti-Objective Reinforcement Learning Multi-Objective Optimization
Cohere · Mila, Universit´e de Montr´eal

Oct 15, 2026J/K move · Enter open · S save

Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense

Yushu Li, Jiin Woo, Xiaoxiao Li, Youngsuk Park

University of British Columbia · Amazon Annapurna Labs

Reward Hacking RL for Language Models RL Post-Training Reward Design for RL Multi-Objective Reinforcement Learning

Open paper AskPDF arXiv

To improve task performance while encouraging desirable response properties, LLM post-training often combines a primary reward (e.g., correctness) with auxiliary rewards (e.g., brevity or efficiency). However, how these rewards are aggregated can critically affect training outcomes. We show that symmetric aggregation, which treats reward channels equally, can induce a failure where aggregate reward improves while primary-task performance degrades, even when individual rewards are meaningful. We investigate this aggregation-induced failure through a one-step multi-reward decision formulation, characterizing when it constitutes multi-reward hacking. We prove that, under symmetric aggregation, deterministic conditional-mean GRPO converges to auxiliary-favoring behavior when auxiliary rewards are more reliably obtained, regardless of initialization. We propose Trust-Anchored Gating (TAG), which first elects an anchor channel based on the policy drift induced by optimizing each channel alone, then gates auxiliary credit on anchor satisfaction. Unlike calibration-sensitive static reweighting, TAG structurally eliminates credit for auxiliary-only outcomes while preserving additional credit for joint success. In the same analytical setting, TAG provably reverses this selection, converging to the primary-favoring action from any initialization when the primary channel is anchored. Experiments across Qwen and Llama models on mathematics and program synthesis reveal substantial degradation under symmetric, constrained, and normalized aggregation, whereas TAG mitigates this failure. On Qwen3-4B with a brevity reward, TAG achieves 0.581 pass@4 versus 0.220 for additive aggregation, while shortening responses by 44% relative to correctness-only training.

Abstract

Reinforcement Learning

Explore similar work

Multi-Reward Hacking: From Aggregation-Induced Drift to Trust-Anchored Defense

PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs

Lexicographic Multi-Objective On-Policy Distillation