SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility

Xuyang Zhi; Peilun zhou; Chengqiang Lu; Hang Lv; Yiwei Liang; Rongyang Zhang; Yan Gao; YI WU

cs.AIApr 9, 2026

SPARD: Self-Paced Curriculum for RL Alignment via Integrating Reward Dynamics and Data Utility

Authors: Xuyang Zhi, Peilun zhou, Chengqiang Lu, Hang Lv, Yiwei Liang, Rongyang Zhang, Yan Gao, YI WU,

Organizations: University of Science and Technology of China · Xiaohongshu Inc.

Abstract

The evolution of Large Language Models (LLMs) is shifting the focus from single, verifiable tasks toward complex, open-ended real-world scenarios, imposing significant challenges on the post-training phase. In these settings, the scale and complexity of reward systems have grown significantly, transitioning toward multi-objective formulations that encompass a comprehensive spectrum of model capabilities and application contexts. However, traditional methods typically rely on fixed reward weights, ignoring non-stationary learning dynamics and struggling with data heterogeneity across dimensions. To address these issues, we propose SPARD, a framework that establishes an automated, self-paced curriculum by perceiving learning progress to dynamically adjust multi-objective reward weights and data importance, thereby synchronizing learning intent with data utility for optimal performance. Extensive experiments across multiple benchmarks demonstrate that SPARD significantly enhances model capabilities across all domains.

Figures & tables

Figure 1: Illustration of the standard Multi-Reward RL loop and examples of training characteristics across diverse data types. The upper panel depicts the workflow of generating multi-reward via an LLM judge and aggregating them for policy updates. The lower panel highlights data heterogeneity , demonstrating that different types of input data differentially impact specific reward dimensions during training.

Figure 2: The framework of SPARD, which consists of two main synergistic mechanisms: (1) Progress-Aware Weight Adaptation dynamically adjusts reward weights ( $\mathbf{w}^{r}$ ) based on the reliability of performance gains, and (2) Reward-Attributed Data Rebalancing computes data weights ( $\mathbf{w}^{d}$ ) by aggregating reward importance via a reward-attribute matrix derived from score dispersion. These components jointly guide the optimization to prioritize current learning objectives and leverage the most efficient data.

Qwen2.5-7B-Instruct
Methods		General Capability			Creative Writing		Chat		AVG
Methods		IFEval	GPQA	LCB	Arena-Hard	CW	MT-Bench	WildBench	AVG
	Base	70.79	33.84	39.75	10.70	48.09	77.93	41.72	46.12
	+ SFT	59.70	32.83	35.00	12.50	41.85	77.56	24.05	40.50
	+ DPO	74.67	33.54	39.50	12.70	50.49	78.37	43.60	47.55
	+ $\text{GRPO}_{\text{rm}}$	73.56	34.85	39.75	11.20	50.17	78.12	41.77	47.06

Table 1: Overall performance comparison on multiple benchmarks. The bold font indicates the best results and an underline indicates the second-best results.

Figure 3: Training trajectories for Qwen2.5-7B-Instruct. To facilitate a clearer comparison of long-term trends and to reduce the visual impact of short-term fluctuations, all curves are smoothed using an exponential moving average (EMA).

Qwen2.5-7B-Instruct
Method	IF	GPQA	LCB	CW	MT
SPARD	75.78	38.38	41.75	52.49	81.38
w/o PAWA	74.86	37.88	41.75	51.24	80.25
w/o RADR	73.56	36.36	40.00	51.87	80.93
Qwen3-8B
SPARD	88.17	49.49	54.75	73.95	78.31

Table 2: Ablation study on the core mechanisms in SPARD. The bold font indicates the best results and an underline indicates the second-best results.

	Method	IF	GPQA	LCB	CW	MT
3b	Base	60.07	27.78	27.00	40.17	73.37
	+Avg	64.51	30.81	26.75	43.41	73.68
	+Ours	65.24	31.31	28.50	44.16	74.25
14b	Base	78.03	46.97	46.50	55.80	79.84
	+ Avg	79.48	45.45	47.50	60.02	82.25
	+ Ours	80.96	46.97	47.25	60.63	84.88

Table 3: Model Performance Across Different Sizes. Avg refers to the method

\text{GRPO}_{\text{avg}}

, which averages rewards across dimensions.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Distribution of selected data sources in the WildChat-IF dataset.

Figure 5: Training dynamics for Correctness, Detail, Fluent, Instruction Following.

Figure 6: Training dynamics for Logic, Relevant, Structure, and Tune.

Figure 7: Data Weight Change During Training Process

Figure 8: Reward Weight Change During Training Process

Hyperparameter	Value
Batch size	32
Epochs	1
Learning rate	5e-6
Warmup ratio	0.05
Weight decay	0.1
Adam betas	(0.9, 0.95)

Appendix

Table 4: Supervised fine-tuning (SFT) hyperparameters used in our experiments.

Hyperparameter	Value
Batch size	32
Epochs	1
Learning rate	1e-6
DPO $\beta$	0.1
Warmup ratio	0.05
Weight decay	0.1

Appendix

Table 5: Direct Preference Optimization (DPO).

Hyperparameter	Value
Batch size	32
Epochs	1
Learning rate	1e-6
Warmup ratio	0.05
Weight decay	0.1
Adam betas	(0.9, 0.95)

Appendix

Table 6: hyperparameters of GRPO and our method

RL Post-Training

Reinforcement learning post-training adapts pretrained models with reward or feedback signals to improve reasoning, alignment, and interactive behavior.

See all 424 papers

SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Models
Jun 5, 2026Yuchen He, Baolong Bi, Shenghua Liu +7Multi-Objective Reinforcement Learning RL for Language Models
Institute of Computing Technology, Chinese Academy of Sciences · University of Electronic Science and Technology of China
RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models
Feb 28, 2026Andrew Zhuoer Feng, Cunxiang Wang, Bosi Wen +4RL for Language Models Reward Design for RL
Tsinghua Univeristy · Peking University
Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL
Jul 31, 2026Ruiming Liang, Yi Zhong, Yizhen Yuan +6Multi-Objective Reinforcement Learning RL for Language Models
1Fundation Model Research Center, CASIA · School of Artificial Intelligence, UCAS · Institute for AI Industry Research (AIR), Tsinghua University +1

Jun 5, 2026J/K move · Enter open · S save

SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Models

Yuchen He, Baolong Bi, Shenghua Liu, Huaming Liao+6

Institute of Computing Technology, Chinese Academy of Sciences · University of Electronic Science and Technology of China

Multi-Objective Reinforcement Learning RL for Language Models Multi-Objective Language Model Alignment

Open paper AskPDF arXiv

Although multi-objective reinforcement learning (MORL) is central to aligning large language models with complex human preferences, the prevailing practice of static weighted summation overlooks a more fundamental phenomenon: reward learning is markedly asynchronous across objectives. Well-learned dimensions quickly produce homogeneous, low-variance signals whose residual noise contaminates the aggregated reward (in GRPO) or occupies a fixed share of the advantage budget (in GDPO), interfering with the scarce yet high-value signals carried by under-learned dimensions. To address this asynchrony, we propose Stage-Aware Dynamic Weighting (SAW), a lightweight, algorithm-agnostic dynamic weighting mechanism. SAW utilizes the coefficient of variation (CV) as a scale-invariant proxy for real-time informativeness, reweighting each dimension's reward or advantage contribution by its relative informativeness within the batch. Unlike gradient-based methods that require multiple forward and backward passes, SAW relies solely on batch-level statistics, introducing nearly negligible computational overhead. Experiments on tool-calling and text summarization tasks demonstrate that SAW consistently improves both training efficiency and final performance under both GRPO and GDPO frameworks, confirming it as a general-purpose plug-in for multi-reward LLM alignment. Our code is available at https://github.com/Zhaolutuan/SAW

Abstract

RL Post-Training

Explore similar work

SAW: Stage-Aware Dynamic Weighting for Multi-Objective Reinforcement Learning in Large Language Models

RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models

Don't Mix Rewards, Mix Policies: Policy Decomposition and Optimization for Multi-Reward RL