Reward Inflation: A Healthy Stimulus for Reinforcement Learning

Ganghun Lee; Minji Kim; Minsu Lee; Byoung-Tak Zhang

cs.LGOct 1, 2026

Reward Inflation: A Healthy Stimulus for Reinforcement Learning

Authors: Ganghun Lee, Minji Kim, Minsu Lee, Byoung-Tak Zhang

Organizations: AIIS, Seoul National University · Department of Computer Science, Seoul National University · School of AI Convergence, Sungshin Women’s University

Abstract

Reward serves as the primary learning signal in reinforcement learning (RL). However, while reward magnitudes are typically held fixed throughout training, their temporal modulation remains underexplored. In this paper, we propose reward inflation, a gradual scaling of rewards over the course of training, and show that it can act as a healthy stimulus for RL. Theoretically, reward inflation induces an implicit recency weighting that upweights recent transitions during policy updates, enabling faster adaptation. We further show that, by sustaining gradient signals as the policy saturates, reward inflation suppresses the emergence of dormant neurons and helps preserve plasticity. Empirical results on ALE games and MuJoCo tasks corroborate these findings, showing that an appropriate level of reward inflation benefits a broad range of tasks. Finally, we introduce Fed, an adaptive variant that adjusts the inflation level on the fly, and find that it often improves upon fixed inflation.

Figures & tables

Metric	0%	2%	Fed (2%)
# Winning Tasks	2	16	12
Mean HNS	219.5	244.8	247.2

Table 1: Summary of performance comparison on 30 ALE tasks.

Method	HalfCheetah	Walker2d	Humanoid	Swimmer	Hopper	Ant	Pusher	Reacher	NS
-2%	$14020{\scriptstyle\pm 1082}$	$4342{\scriptstyle\pm 461}$	$5300{\scriptstyle\pm 176}$	$92{\scriptstyle\pm 43}$	$2421{\scriptstyle\pm 460}$	$3929{\scriptstyle\pm 1090}$	- $24.25{\scriptstyle\pm 4.10}$	- $3.46{\scriptstyle\pm 0.23}$	61.7
0%	15241 ±734	4399 ±527	5826 ±313	116 ±36	2869 ±299	4763 ±1642	-23.28 ±2.89	-3.41 ±0.23 †	81.9
1%	15707 ±715 †	5007 ±560	5859 ±363	121 ±25	2821 ±406	5245 ±1312 †	-22.71 ±2.87	-3.42 ±0.23	88.2
2%	15607 ±372	5102 ±520 †	5861 ±199	136 ±11 †	3040 ±499	4346 ±1267	-22.07 ±1.58 †	-3.42 ±0.24	91.3 †
3%	15091 ±927	4690 ±624	5873 ±306 †	126 ±11	3035 ±445	4392 ±1298	-22.58 ±1.58	-3.44 ±0.24	83.4
5%	14484 ±996	4615 ±518	5545 ±492	131 ±12	3161 ±346	4795 ±1089	-22.20 ±0.89	-3.45 ±0.23	82.8

Table 2: Performance comparison under varying inflation rates (upper rows) and other methods (lower rows) across MuJoCo tasks. The dagger (

\dagger

) marks the best value among fixed inflation rates. The last column reports the average normalized score across tasks.

Figure 1: (a) The nominal and real performance growth over training time under $2\%$ inflation. (b) The inflation rate and inflation level $I(g)$ over the training time. (c) The TD error w.r.t. the transition age in replay buffer, across the inflation rates. (d) The changes of gradient norms during training.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Total timesteps	3M
Replay buffer size	1M
Target smoothing coef. ( $\tau$ )	0.005
Batch size	256
Warm-up steps	100K
Learning rate ( $Q_{\theta}$ )	0.001

Appendix

Table 3: SAC hyperparameter configuration.

Task	Discount factor ( $\gamma$ )
Hopper-v4	0.995
Ant-v4	0.975
Others	0.99 (default)

Appendix

Table 4: Discount factor (

\gamma

) specification.

Task	0%	2%	Fed (2%)	Task	0%	2%	Fed (2%)
Alien	1250 $\pm$ 352	1513 $\pm$ 124	1482 $\pm$ 96	Frostbite	2732 $\pm$ 364	3377 $\pm$ 543	3172 $\pm$ 579
Amidar	296 $\pm$ 33	468 $\pm$ 64	456 $\pm$ 83	Gopher	12595 $\pm$ 3870	16160 $\pm$ 4948	15954 $\pm$ 5352
Assault	3654 $\pm$ 252	3814 $\pm$ 449	3401 $\pm$ 396	Hero	2758 $\pm$ 1779	2731 $\pm$ 624	2549 $\pm$ 215
Asterix	22178 $\pm$ 7214	31424 $\pm$ 5245	29492 $\pm$ 3913	Jamesbond	486 $\pm$ 63	601 $\pm$ 154	567 $\pm$ 120
BankHeist	650 $\pm$ 251	776 $\pm$ 189	797 $\pm$ 109	Kangaroo	7821 $\pm$ 1762	10885 $\pm$ 2742	11983 $\pm$ 2016
BattleZone	22659 $\pm$ 1074	22195 $\pm$ 4406	24501 $\pm$ 945	Krull	7584 $\pm$ 2605	7969 $\pm$ 2602	8799 $\pm$ 419

Appendix

Table 5: Performance comparison of DQN agents in ALE games under reward inflation (0%, 2%, and Fed (2%)).

Figure 4: Learning curves of DQN agents across all ALE games under different reward inflation settings ( $0\%$ , $2\%$ , and Fed with $2\%$ target).

Figure 5: Mean absolute deviation curves of inflation rate under various Fed hyperparameter settings. Specifically, Larger $K$ , $\lambda_{1}$ , and $\lambda_{2}$ , and smaller $\tau_{\rho}$ lead to more aggressive fluctuations in the inflation rate. For $\beta_{s}$ and $\beta_{l}$ , smaller values and larger gaps between them amplify variations in gradient-norm measurements, which in turn induce greater inflation-rate volatility. Note that target inflation $\rho^{\ast}$ does not impact volatility.

Figure 6: Learning curves of Fed varying 5 sets of hyperparameters. Default values are marked with $\ast$ . Episode rewards are normalized across 4 tasks.

Panel of Figure 7 — Figure 7: Task-wise bootstrap 95% CI over Atari tasks. (Left) Baseline vs. 2% inflation rate. (Right) Baseline vs. Fed.

Panel of Figure 8 — Figure 8: Task-wise bootstrap 95% CI over MuJoCo tasks. (Left) Baseline vs. the best-performing inflation rate identified in Table 1 of the main paper. (Right) Baseline vs. Fed.

Panel of Figure 9 — Figure 9: Additional results of dormant (dead) neuron ratios across different tasks under varying reward inflation rates (Fig. 2(a) in the main paper corresponds to HalfCheetah-v4).

Panel of Figure 10 — Figure 10: Dormant (dead) neuron ratios across different tasks under varying reward inflation rates, using LeakyReLU instead of ReLU . Neurons whose negative activation ratio exceeded 99% across inputs were counted as dormant.

Panel of Figure 11 — Figure 11: Dormant (dead) neuron ratios across different tasks under varying reward inflation rates, using SiLU instead of ReLU . Neurons whose activation variance across inputs was below 0.0001 were counted as dormant.

Figure 12: Non-inflation (0%) vs. inflation (2%) on more uncertain conditions.

Reward Design for RL

Reward design for reinforcement learning studies how to specify, learn, and adapt signals that guide agents toward intended behavior.

See all 261 papers

Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
Sep 29, 2026Zhiwei Wang, Yanxi Chen, Yaliang Li +1Policy Gradient Methods Off-Policy RL
Department of Mathematical Sciences Tsinghua University · Alibaba Group
Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning
Mar 11, 2026Zichao Li, Jie Lou, Fangchen Dong +8RL for Language Models RL Post-Training
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Xiaohongshu Inc.
When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?
Oct 14, 2026Kaifei Wang, Zikang Shan, Liwei Wang +2Bandit Learning Momentum Methods
Peking University · Shanghai Jiao Tong University, SIMIS, and Stanford University · Shanghai Jiao Tong University

Sep 29, 2026J/K move · Enter open · S save

Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness

Zhiwei Wang, Yanxi Chen, Yaliang Li, Bolin Ding

Department of Mathematical Sciences Tsinghua University · Alibaba Group

Policy Gradient Methods Off-Policy RL Multi-Armed Bandits

Open paper AskPDF arXiv

We study the learning dynamics of fine-tuning a policy model on self-generated and reward-weighted data, with particular focus on a generalized version of REINFORCE -- referred to as RE(S) -- that updates the rollout distribution once every

S \ge 1

gradient steps. Prior work in bandits and reinforcement learning has developed rich theory for policy gradient methods, and on-policy sampling (i.e., a small

S

, ideally

1

) is often viewed as crucial to their success; yet in prominent application like post-training large language models, reward-guided self-training has proved to be effective even when the rollout distribution is updated infrequently, but theoretical understanding remains limited for the convergence properties of these off-policy methods. To bridge these gaps, we develop a unified theory for RE(S) that covers the full spectrum of

S \ge 1

: it can be interpreted as a stage-wise optimization process, where each stage takes

S

gradient steps for minimizing the Kullback-Leibler distance to a fixed reward-weighted rollout distribution. For multi-arm bandits with softmax policies, our in-depth analysis and numerical experiments reveal three key findings: (1) for any fixed

S

, RE(S) enjoys global convergence to the optimal policy as the number of rollout distribution updates

B = \lfloor T / S \rfloor \rightarrow \infty

, where

T

denotes the number of gradient steps; (2) we prove tight two-sided bounds showing that the suboptimality gap of RE(S) achieves an asymptotic

Θ(1 / T)

convergence rate, while

S

only affects the length of a burn-in phase; (3) when initialized at a weak policy with a small optimal-action probability, RE(1) gets trapped around suboptimal policies for a long period, whereas RE(S) with a suitable

S

avoids the detour and achieves significantly faster convergence to the global optimum, highlighting the benefits of off-policyness in this case.

Abstract

Reward Design for RL

Explore similar work

Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness

Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning

When Does Momentum Help in Reinforcement Learning with Verifiable Rewards?