Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL

Tong Zheng; Skylar Zhai; Zhan Cheng; TianMing Sha; Youling Huang; Shuo Zhou; Shaotong Qi; Jingcheng Liang

cs.LGSep 30, 2026

Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL

Authors: Tong Zheng, Skylar Zhai, Zhan Cheng, TianMing Sha, Youling Huang, Shuo Zhou, Shaotong Qi, Jingcheng Liang, +2 more

Organizations: University of Chinese Academy of Sciences · University of Minnesota Twin Cities · University of Wisconsin–Madison · Stony Brook University · Dalian University of Technology · Beijing Foreign Studies University · Southeast University · Kuaishou Technology

Abstract

Multi-reward reinforcement learning trains large language models to satisfy multiple behavioral objectives simultaneously. Reward-wise normalization, as used in GDPO, preserves reward-specific relative information within rollout groups, but different objectives can still exhibit uneven learning progress. We study this behavior through advantage energy, the sum of a reward's squared advantages over a batch. Under idealized GDPO normalization, we show that this energy is proportional to active-group density: the fraction of rollout groups in which the reward provides nonzero relative advantages. This reveals a residual batch-level signal imbalance and provides a basis for calibrating reward contributions. Based on this relation, we propose Density-Aware Reward Aggregation (DARA). We derive an inverse-square-root density correction that gives greater weight to signals from less frequently active rewards. DARA computes its weights from each rollout batch, adapting to changes in reward activity throughout training without modifying the underlying policy optimization objective. Experiments on tool calling and mathematical reasoning show that DARA learns the targeted behaviors faster than GDPO, reaching high format compliance in up to 26% fewer training steps on tool calling and near-saturated length compliance in up to 65% fewer steps on mathematical reasoning, while remaining competitive in final performance. Our code is available at https://github.com/zhaihaotian/DARA.

Figures & tables

Figure 1: (a) DARA performs reward-wise group normalization, estimates active-group density, and applies density-based calibration before reward aggregation and batch-wise normalization. (b) Median format reward and active-group density, with shaded regions indicating the interquartile range. DARA reaches high format compliance in fewer training steps.

		Live		Non-Live		Multi-Turn		Average
Size	Method	Acc.	Format	Acc.	Format	Acc.	Format	Acc.	Format
1.5B	Base	9.33	0.37	12.06	0.12	0.38	0.21	7.25	0.23
	GRPO	64.80	99.88	78.04	99.91	2.55	43.24	48.46	81.01
	GDPO	67.05	99.94	79.30	99.97	4.75	91.40	50.37	97.11
	DVAO	64.99	99.75	76.84	99.62	4.08	81.19	48.64	93.52
	GD 2 PO-Hard	67.06	99.97	78.96	100.00	4.38	86.59	50.13	95.52

Table 1: BFCL-v4 performance at step 100. Acc. denotes tool-calling accuracy and Format denotes output-format compliance. Average is the mean over Live, Non-Live, and Multi-Turn.

Figure 2: Checkpoint-wise BFCL-v4 evaluation of Qwen2.5-1.5B-Instruct. Left: Average Accuracy; right: Average Format, each averaged over Live, Non-Live, and Multi-Turn.

Figure 3: Format active-group density across rollout-group sizes. Curves show the median and shaded regions the interquartile range. Markers on the dashed line show when format reward first reaches 0.8; open markers indicate that it does not reach 0.8 within 100 steps.

		Live			Non-Live			Multi-Turn			Average
Size	Method	Acc.	Format	Len.	Acc.	Format	Len.	Acc.	Format	Len.	Acc.	Format	Len.
1.5B	Base	9.33	0.37	1.33	12.06	0.12	0.73	0.38	0.21	3.58	7.25	0.23	1.88
	GRPO	67.80	100.00	86.34	78.26	99.96	78.62	4.06	64.14	72.59	50.04	88.03	79.19
	GDPO	66.62	100.00	100.00	80.94	100.00	99.93	3.81	83.17	93.97	50.46	94.39	97.96
	DARA-Asym	68.02	100.00	99.41	80.00	100.00	100.00	3.88	90.80	94.74	50.63	96.93	98.05
	DARA-Sym	67.95	100.00	100.00	80.21	100.00	99.99	3.31	92.30	96.16	50.49	97.43	98.72

Table 2: BFCL-v4 performance at step 100 of Qwen2.5-1.5B-Instruct trained with correctness, format, and length rewards. Len. denotes compliance; the other columns follow Table 1 .

Figure 4: Training reward dynamics for mathematical reasoning on DeepSeek-R1-1.5B. Left: correctness reward; right: length reward. Static denotes the Static-Sym control in Table 4 (a).

Figure 5: First stable crossing step for increasingly strict training-side length-compliance thresholds on Qwen3-4B-Instruct (left) and DeepSeek-R1-7B (right). Lower values indicate faster acquisition.

Figure 6: Checkpoint-wise held-out mathematical reasoning evaluation on DeepSeek-R1-7B. Left: macro pass@1 accuracy; right: percentage of responses exceeding 4,000 tokens.

	DeepSeek-R1-1.5B			Qwen3-4B-Instruct			DeepSeek-R1-7B
Method	Acc $\uparrow$	Exceed $\downarrow$	Joint $\uparrow$	Acc $\uparrow$	Exceed $\downarrow$	Joint $\uparrow$	Acc $\uparrow$	Exceed $\downarrow$	Joint $\uparrow$
Base	48.93	65.23	27.75	68.15	33.60	51.69	64.66	56.16	36.32
GRPO	45.52	11.03	44.69	62.14	8.07	60.48	58.26	9.73	56.74
GDPO	44.12	8.66	43.47	60.82	7.64	59.19	57.45	6.82	56.48
DVAO	45.58	11.35	44.57	61.37	4.77	60.34	58.15	5.75	57.53
GD 2 PO-Hard	44.24	11.41	43.45	60.58	7.28	59.16	57.78	7.96	56.35

Table 3: Aggregate mathematical reasoning results at step 50. Acc is the macro-average pass@1 accuracy over the five benchmarks, Exceed is the macro-average percentage of responses longer than 4,000 tokens, and Joint is the macro-average percentage of responses that are both correct and within the length limit. Base denotes the pretrained checkpoint. All values are percentages.

Method	Acc $\uparrow$	Exceed $\downarrow$	Joint $\uparrow$
DARA-Asym	48.03	4.21	47.76
Static-Asym ( $w=5$ )	46.84	3.14	46.72
DARA-Sym	46.71	0.89	46.67
Static-Sym ( $w=5$ )	45.97	0.87	45.96

Table 4: Left: dynamic versus fixed reward weighting on DeepSeek-R1-1.5B. Right: comparison between Qwen3-4B Instruct and Thinking; underlined values mark DARA-Asym.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Reward-wise advantage energy on batches of mathematical reasoning training at step 40, before (GDPO) and after (DARA) density calibration. Values are from the recorded active-group densities and weights as

\pi_{k}

and

w_{k}^{2}\pi_{k}

, which equal

E_{k}/[B(G-1)]

by Equation 5 .

Figure 8: Ratio of the length to the correctness energy estimate on the same logged batches throughout training, without (GDPO normalization) and with (DARA weighting) density calibration. The dashed line marks equal energy.

Method	Reward weighting	Advantage
GRPO	Rewards summed before normalization	$\displaystyle\frac{r_{\mathrm{sum}}^{(i,j)}-\mu_{i,\mathrm{sum}}}{\sigma_{i,\mathrm{sum}}+\epsilon},\quad r_{\mathrm{sum}}^{(i,j)}=\sum_{k}r_{k}^{(i,j)}$
GDPO	Equal weights on normalized rewards	$\mathcal{N}_{\mathcal{B}}\bigl(\sum_{k}A_{k}^{(i,j)}\bigr)$
DVAO	Within-group reward standard deviation	$\displaystyle\sum_{k}\tilde{w}_{i,k}A_{k}^{(i,j)},\quad\tilde{w}_{i,k}=\frac{w_{k}\sigma_{i,k}}{\sum_{l}w_{l}\sigma_{i,l}}$
SAW	Batch-level coefficient of variation $\mathrm{CV}_{k}$ of the offset reward $r_{k}-r_{k}^{\min}+\delta$	$\displaystyle\mathcal{N}_{\mathcal{B}}\bigl(\sum_{k}\omega_{k}A_{k}^{(i,j)}\bigr),\quad\omega_{k}=K\,\frac{\mathrm{CV}_{k}}{\sum_{l}\mathrm{CV}_{l}}$
SA-MRPO	Batch-level saturation $s_{k}=\frac{\bar{r}_{k}-r_{k}^{\min}}{r_{k}^{\max}-r_{k}^{\min}}$	$\mathcal{N}_{\mathcal{B}}\bigl(\sum_{k}\tilde{w}_{k}A_{k}^{(i,j)}\bigr),\quad\tilde{w}_{k}=w_{k}(1-s_{k})^{\gamma}$
GD 2 PO-Hard	Sign consistency of $\{A_{k}^{(i,j)}\}_{k}$	$\mathcal{N}_{\mathcal{B}}\bigl(\sum_{k}A_{k}^{(i,j)}\bigr)$ over retained responses with $\delta^{(i,j)}=\mathbf{1}[\text{no sign conflict among }A_{k}^{(i,j)}]$ ; the loss of query $i$ is scaled by $\hat{\kappa}_{i}=\frac{1}{G}\sum_{j}\delta^{(i,j)}$

Appendix

Table 5: Advantage computation of multi-reward methods. The second column gives the statistic that determines each reward’s contribution, and the third column gives the sequence-level advantage.

Setting	Value
Backbones	Qwen2.5-1.5B-Instruct / Qwen2.5-3B-Instruct
Training / rollout	FSDP / vLLM, tensor parallelism 1, bfloat16
Training / validation / checkpoint	100 steps / every 10 steps / step 100
Prompts / responses per prompt / total	512 / 4 / 2,048
PPO minibatch / microbatch / epochs	512 responses / 256 responses / 1
Learning rate / gradient clip / PPO clip	$10^{-6}$ / 1.0 / 0.2

Appendix

Table 6: Training configuration for the tool-calling experiments.

Setting	Value
Evaluator	BFCL-v4, gorilla commit 6ea57973
Subsets	Live, Non-Live, and Multi-Turn
Trajectories	One per evaluation case
Temperature / top- $p$	0.6 / 0.95
Maximum new tokens / context length	8,192 / 32,768
Inference seed / top- $k$	0 / $-1$

Appendix

Table 7: Inference and evaluation configuration for BFCL-v4.

$G$	Prompts	PPO mini	PPO micro	Responses
4	512	128	64	2,048
8	256	64	32	2,048
16	128	32	16	2,048
32	64	16	8	2,048

Appendix

Table 8: Batch configuration for the rollout-group analysis.

Setting	Value
Backbones	DeepSeek-R1-Distill-Qwen-1.5B / 7B
	Qwen3-4B-Instruct-2507 / Qwen3-4B-Thinking-2507
Training data	DeepScaleR-Preview, full training collection
Training / rollout	FSDP / vLLM, rollout tensor parallelism 1
Prompt / response length	1,024 / 8,000 tokens
Accepted prompt groups / candidate groups	512 / 768 per generation batch

Appendix

Table 9: Training configuration for the mathematical-reasoning experiments.

Setting	Value
MATH	MATH-500, test split, 500 questions
AIME	AIME 2024, 30 questions
AMC	AMC 2022/2023, 83 questions
Minerva	Minerva Math, test split, 272 questions
OlympiadBench	OE_TO_maths_en_COMP , 674 questions
Samples per question	16

Appendix

Table 10: Inference and evaluation configuration for mathematical reasoning.

Explore similar work

Dependency-Aware Reward Shaping for Agentic Reinforcement Learning
Oct 1, 2026Ziyi Chen, Yan Zhang, Jianhui Wei +2Agentic Reinforcement Learning Potential-Based Reward Shaping
University of Illinois Urbana-Champaign · National University of Singapore · Zhejiang University +1
Uncovering and Mitigating Aggregation-Induced Reward Hacking in Multi-Reward Reinforcement Learning
Aug 31, 2026Yu Yuan, Yaoyou Fan, Lili Zhao +5Reward Functions
University of Science and Technology of China · 2State Key Laboratory of Cognitive Intelligence · 3Peking University +1
SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation
Aug 4, 2026Wen Wang, Jiahua Bao, Tu Yongsiqi +8Reward-Only Policy Optimization Variants Online-Policy Distillation
Qwen Large Model Application Team, Alibaba · University of Chinese Academy of Sciences

Oct 1, 2026, cs.AIJ/K move · Enter open · S save

Dependency-Aware Reward Shaping for Agentic Reinforcement Learning

Ziyi Chen, Yan Zhang, Jianhui Wei, Daoan Zhang+1

University of Illinois Urbana-Champaign · National University of Singapore · Zhejiang University+1

Agentic Reinforcement Learning Potential-Based Reward Shaping Dependency Graph

Open paper AskPDF arXiv

When training large language models with reinforcement learning, terminal rewards provide little guidance about which steps matter. Common methods for assigning step credit overlook that work built on uncorrected mistakes is wasted while independent work remains valid. With only a final success/failure reward, every step in a failed episode has zero total future reward, even when it made progress. We propose Dependency-Aware Reward Shaping (DARS), which represents task progress as predicates linked by prerequisite relations and assigns step-level credit over the dependency graph. An annotator marks which predicates each step verifies, invalidates, or repairs. Verified predicates are discounted according to graph distance from the nearest broken prerequisite, while independent predicates are unaffected. Repairs update these weights based on any errors that remain; invalidated predicates need re-verification to regain credit. A fixed potential converts these annotations into signed per-step rewards. A common reward and annotation interface allows DARS to integrate with a range of reasoning and agentic training methods, such as GiGPO and ARPO/AEPO, without changing their rollout strategies or optimizers. Across five task families and models from 1.5B to 8B, DARS improves success by up to 10 points over GiGPO trained with the same budget and harness (ALFWorld), raises the WebShop task score and Search-R1 QA accuracy, complements AEPO's entropy-based training on AIME24/25 with a Python interpreter, and exceeds OmniOPD in controlled tool-free reasoning comparisons at 1.7B and 4B. Ablations show that step-level credit, dependency attenuation, and graph topology each contribute. On ALFWorld, a distilled 8B annotator matches the API annotator, enabling DARS to run efficiently without a frontier judge. Code is available at https://github.com/JianhuiWei7/DARS.

Abstract

Explore similar work

Dependency-Aware Reward Shaping for Agentic Reinforcement Learning

Uncovering and Mitigating Aggregation-Induced Reward Hacking in Multi-Reward Reinforcement Learning

SMOPD: Multi-Reward Reinforcement Learning via Specialize-and-Merge Online Policy Distillation