Action Shaping: Policies Absorb What They Can Express

Yanjun Chen; Jinghan Wang; Xiaoyu Shen; Wenjie Li; Wei Zhang

cs.AISep 26, 2026

Action Shaping: Policies Absorb What They Can Express

Authors: Yanjun Chen, Jinghan Wang, Xiaoyu Shen, Wenjie Li, Wei Zhang

Organizations: Eastern Institute of Technology · The Hong Kong Polytechnic University · Harbin Institute of Technology

Abstract

Reward shaping has a theorem: a potential-based term can be removed without changing the optimal policy. The same practice on the action channel, an offset added in training and dropped at deployment, has no theorem. Nothing cancels an action offset, so the correction is kept at deployment or removed without a guarantee. We call it action shaping and state its principle. A trainable policy absorbs an offset its own output layer can reproduce exactly, which is what we mean by express; what is absorbed can be removed with the return intact. Its minimal instance is a zero-initialized linear head behind a learnable gate, added to an actor that trains through a learned action-value function, with no penalty or schedule. The gate rises and then falls on its own, for deterministic and stochastic actors alike, and on 20 tasks removing the head costs almost nothing. The condition is exact reproduction, not capacity: a nonlinear head with more parameters is not absorbed, and in a paired control, one linear path added to a nonlinear base head restores absorption. Exact reproduction gives the loss a flat direction that gradient noise drifts along, and the offset's amplitude indicates, before removal, what dropping the head will cost. Action shaping thus gains the counterpart of the shaping theorem, a condition for absorption, together with the mechanism behind it and a diagnostic that reads it. Policies absorb what they can express, and only that.

Figures & tables

Figure 1: A trainable policy absorbs a training-time action offset its own output layer can reproduce, and only that. Reward shaping adds a term in training and a theorem says which terms leave the optimal policy unchanged; action shaping adds an offset

\delta=\alpha h

to the action, and nothing cancels it. What removes it for free is absorption: when the base head

\mu

can reproduce

\mu+\alpha h

with its own weights,

W_{\mu}+\alpha W_{h}

, and is free to train, the gate

\alpha

falls back toward zero and removing the head costs almost nothing. On a frozen base, or for a head that

\mu

cannot reproduce, it is not absorbed. Right: the shaped actor in training and the base actor alone at deployment.

Figure 2: The removal cost follows the residual, and is large on a frozen base. (a) Removal cost in normalized return against the terminal residual

r

of § 2.2 , one point per environment and algorithm (20 environments under TD3 and under SAC, mean over 5 seeds); the dashed line marks the threshold

r=0.10

. (b) IQM of the per-run removal cost with its 95% stratified bootstrap interval, for the shared head under TD3, SAC and PPO and for the frozen base under TD3 and SAC.

Figure 3: The shared amplitude rises and falls, further under SAC, while the frozen and independent ones stay high.

\|\delta_{t}\|

per environment on the mechanism suite, median over 5 seeds with the interquartile range shaded, log scale shared within a row. On Humanoid, three of the five frozen-base SAC runs diverge by 220K steps, and that line is the median of the other two.

Checkpoint	TD3		SAC
rising	$+$ 0.015	[0.001, 0.032]	$-$ 0.003	[ $-$ 0.009, 0.002]
peak	$+$ 0.090	[0.051, 0.134]	$+$ 0.045	[0.009, 0.094]
post-peak	$+$ 0.056	[0.028, 0.091]	$+$ 0.109	[0.041, 0.193]
final	$+$ 0.017	[ $-$ 0.005, 0.049]	$-$ 0.012	[ $-$ 0.032, 0.002]

Table 1: Removal cost is largest at or just after the amplitude peak. The head is removed at four points of each run, located by a rule that reads only the amplitude in action space (Appendix H ), on the mechanism suite; mean with 95% bootstrap CI over 30 runs per algorithm.

Figure 4: Absorption requires expressibility and a trainable base. Terminal residual

r

(a) and removal cost (b) for each arm of § 2.3 on the mechanism suite, averaged over seeds and environments; the last row adds a zero-initialized linear path to the MLP base head. Shared and Detach are pooled over algorithms, and the MLP-head row is the mean over widths 16, 64, and 256, which agree within 0.11 in

r

\dagger

: for the frozen base, the residual is the mean over the runs that do not diverge (27 of 30 under SAC), and the cost is the IQM over the 20 environments of § 4.1 with its 95% stratified bootstrap interval.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

	N0: full batch			N1: minibatch			N2: minibatch + target noise
Arm	$r$	done	$h$ kept	$r$	done	$h$ kept	$r$	done	$h$ kept
Shared, linear head	0.997	0/3	0.84	0.695	0/3	0.96	0.0014	3/3	0.14
Detach	0.996	0/3	0.85	0.687	0/3	0.98	0.0012	3/3	0.13
Independent trunk	0.999	0/3	0.66	0.996	0/3	0.80	0.979	0/3	0.99
Frozen base	1.000	0/3	0.64	0.993	0/3	0.51	0.993	0/3	0.51
MLP head, linear base	0.995	0/3	0.60	0.777	0/3	0.87	0.086	2/3	0.13

Appendix

Table 2: Minimal model. Terminal residual

r

(median over 3 seeds), number of seeds completing (

r\leq 0.10

), and the fraction of its peak norm that the head retains at the end (

\|h\|_{\text{end}}/\|h\|_{\text{peak}}

, median), for each configuration under three noise levels; Adam in every row but the last, which uses plain SGD (learning rate

10^{-2}

Figure 5: Without gradient noise nothing returns. Minimal model under Adam. Sign-aligned

\alpha

(top) and

\|\delta\|

on a log scale (bottom) over 100K steps under the three noise levels; color by whether the base head can reproduce the offset, green where it can, orange where it cannot, pink where the base is frozen, with the line style separating arms inside a color; thin lines are seeds and thick lines the median. Under N0 nothing returns; under N1 the shared and detached heads turn over; under N2 they return to zero, the MLP head on the linear base collapses, and the linear head on the MLP base stays.

	Terminal residual $r$				Removal cost
	TD3		SAC		TD3		SAC
Environment	base	$+$ path	base	$+$ path	base	$+$ path	base	$+$ path
Hopper	0.476	0.005	0.902	0.183	$+$ 0.786	$-$ 0.004	$+$ 0.559	$-$ 0.001
HalfCheetah	0.765	0.006	0.954	0.008	$+$ 0.367	$+$ 0.003	$+$ 0.401	$+$ 0.002
Humanoid	0.915	0.081	0.765	0.016	$+$ 0.947	$-$ 0.002	$+$ 0.572	$-$ 0.001
finger/spin	0.173	0.003	0.556	0.351	$+$ 0.012	0.000	$+$ 0.004	$-$ 0.001

Appendix

Table 3: A linear path lowers the residual in every cell and the removal cost to within 0.004 of zero. Terminal residual

r

and removal cost per (environment, algorithm) cell, each the mean over 5 seeds; base is the one-hidden-layer base head and

+

path the same base head with a zero-initialized linear map added to its output. Cost is the normalized return of the shaped actor minus the base actor alone, as in § 2.2 .

	TD3			SAC
Environment	$r$ 1M	$r$ 3M	cost 3M	$r$ 1M	$r$ 3M	cost 3M
Hopper	0.597	0.735	$+$ 0.820	0.798	0.721	$+$ 0.706
HalfCheetah	0.700	0.829	$+$ 1.221	0.858	0.782	$+$ 1.337
Humanoid	0.851	0.848	$+$ 0.790	0.952	0.916	$+$ 1.177
finger/spin	0.382	0.467	$+$ 0.702	0.783	0.672	$+$ 1.092
walker/run	0.382	0.437	$+$ 0.976	0.798	0.609	$+$ 0.948

Appendix

Table 4: The MLP shaping head stays unabsorbed at three million steps. Residual

r

of the same runs read at one million and at three million steps, and removal cost at three million, per (environment, algorithm) cell for the width-64 head, each the mean over 3 seeds. Cost is the normalized return of the shaped actor minus the base actor alone, as in § 2.2 .

Environment	1M	3M	1M	3M	$\\|\delta\\|$
	Residual $r$		Removal cost		Peak
manipulator/bring_ball	0.827	0.732	$+$ 0.935	$+$ 0.056	20395
acrobot/swingup	0.488	0.215	$+$ 0.033	$-$ 0.038	22.1
finger/spin	0.321	0.190	$+$ 0.155	$+$ 0.009	160
humanoid/walk	0.206	0.176	$+$ 0.045	$+$ 0.004	285
InvertedDoublePendulum	0.490	0.056	$+$ 0.950	$+$ 0.489	21.1

Appendix

Table 5: Three times the budget lowers the residual

r

in each of six unfinished TD3 arcs. Residual

r

and removal cost of the same runs read at one million and at three million steps, and the run’s peak amplitude

\|\delta\|

(pre-activation, § 2.2 ), each the mean over 5 seeds; cost is the normalized return of the shaped actor minus the base actor alone.

Environment	Critic	$B$	$T$
HalfCheetah	TD3	64	520
		256	490
		1024	705
HalfCheetah	SAC	64	330
		256	355
		1024	440

Appendix

Table 6: The gate across batch sizes. Closing time

T

from the peak of the amplitude (K steps), median over 5 seeds; definition in the text.

Figure 6: The gate rises and falls at every batch size.

|\alpha_{t}|

during training, median over 5 seeds, log scale; rows labelled TD3 and SAC; black

B=64

, grey dashed

B=256

, light grey dash-dot

B=1024

. The closing time

T

of Table 6 is read from the amplitude rather than from these curves.

Environment	gain	shared	gain	shared	gain	shared	gain	shared	gain	gain
	Terminal residual $r$				Removal cost				$\mathrm{sdi}$
	TD3		SAC		TD3		SAC		TD3	SAC
Hopper	0.368	0.226	0.837	0.026	$+$ 0.018	$+$ 0.011	$-$ 0.030	$+$ 0.001	1.09	0.58
HalfCheetah	0.532	0.007	0.949	0.002	$+$ 0.651	$+$ 0.002	$+$ 0.298	$+$ 0.002	0.94	0.36
Humanoid	0.819	0.020	0.926	0.013	$+$ 0.005	$+$ 0.007	$-$ 0.014	$+$ 0.011	0.67	0.09
finger/spin	0.211	0.420	0.779	0.324	$+$ 0.099	$+$ 0.232	$+$ 0.010	$+$ 0.014	0.74	0.59

Appendix

Table 7: A linear head composed multiplicatively stays. Terminal residual

r

, removal cost and state-dependence index of the gain arm against the paired shared arm on the mechanism suite at 1M steps. Gain cells give the median over three seeds (the mean for the index) and control cells the maximum over seeds; the cost is in normalized return.

	TD3		SAC
Convention	completed	Spearman ( $p$ )	completed	Spearman ( $p$ )
Per-run $r$ , mean over seeds	7	0.614 (0.0040)	13	0.636 (0.0026)
Seed-mean trajectory, last 20 evaluations over peak	8	0.608 (0.0045)	13	0.608 (0.0045)
Seed-mean trajectory, last point over peak	8	0.565 (0.0094)	14	0.571 (0.0085)
Per-run $r$ , median over seeds	10	0.626 (0.0032)	13	0.671 (0.0012)

Appendix

Table 8: Residual conventions. Each row is one way of forming an environment’s terminal residual from the per-run amplitude trajectories. Completed : environments with

r\leq 0.10

, out of 20. Spearman : rank correlation between an environment’s residual and the magnitude of its removal cost, with its

p

-value. The first row is the convention used in the paper, defined in equation 2 ; it counts no more environments as completed than any other row, and its correlation is the one reported in § 4.3 .

Environment	AS-TD3	AS-TD3-Fr	AS-SAC	AS-SAC-Fr	AS-PPO
InvertedPendulum	1.009 $\pm$ 0.006	0.912 $\pm$ 0.109	1.004 $\pm$ 0.024	1.000 $\pm$ 0.015	1.018 $\pm$ 0.030
InvertedDoublePendulum	1.001 $\pm$ 0.012	0.980 $\pm$ 0.032	1.005 $\pm$ 0.022	0.036 $\pm$ 0.003	0.763 $\pm$ 0.127
Reacher	1.004 $\pm$ 0.005	0.950 $\pm$ 0.007	1.020 $\pm$ 0.004	0.588 $\pm$ 0.183	0.969 $\pm$ 0.008
Swimmer	0.805 $\pm$ 0.122	0.594 $\pm$ 0.031	0.909 $\pm$ 0.144	0.649 $\pm$ 0.055	0.873 $\pm$ 0.397
Hopper	0.971 $\pm$ 0.097	0.050 $\pm$ 0.008	1.017 $\pm$ 0.139	0.961 $\pm$ 0.121	1.057 $\pm$ 0.214
HalfCheetah	0.995 $\pm$ 0.128	0.461 $\pm$ 0.053	1.023 $\pm$ 0.020	0.425 $\pm$ 0.073	0.568 $\pm$ 0.059

Appendix

Table 9: Per-environment terminal IQM of the shaped actors (20 environments, 5 seeds, shaped evaluation, min-max normalized);

\pm

is the half-width of the 95% stratified bootstrap CI. AS-TD3-Fr and AS-SAC-Fr are the frozen arms.

Figure 7: The removal-cost interval contains zero in 15 of the 20 environments. Per-environment removal cost (shaped minus base-only): the mean over the pooled runs with its 95% bootstrap CI. The shaped TD3 and SAC actors pooled, 5 seeds each, min-max normalized, sorted by magnitude. The interval of manipulator/bring_ball,

[-0.20,+1.58]

, is clipped to the axis.

Figure 8: Amplitude decomposition: the fall of the amplitude comes from the gate (6 environments, 5 seeds, default configuration). Upper block:

|\alpha|

over training. Lower block:

\|h(s)\|

over training. TD3 in blue, SAC in orange; lines are the interquartile mean over seeds (median of 2,000 bootstrap resamples) and shading is the 95% stratified bootstrap CI.

UTD = 1
Environment	TD3-REDQ	AS-TD3-REDQ	SAC-REDQ	AS-SAC-REDQ
Hopper	0.983 $\pm$ 0.087	1.007 $\pm$ 0.083	0.900 $\pm$ 0.234	0.827 $\pm$ 0.146
HalfCheetah	0.962 $\pm$ 0.099	0.994 $\pm$ 0.039	1.058 $\pm$ 0.044	1.090 $\pm$ 0.035
Humanoid	0.963 $\pm$ 0.022	0.962 $\pm$ 0.021	1.017 $\pm$ 0.033	1.048 $\pm$ 0.064
walker/run	0.911 $\pm$ 0.178	0.814 $\pm$ 0.152	0.690 $\pm$ 0.215	0.928 $\pm$ 0.165
Aggregate	0.963 $\pm$ 0.037	0.962 $\pm$ 0.020	0.970 $\pm$ 0.058	0.999 $\pm$ 0.052

Appendix

Table 10: Update-to-data scaling, per-environment IQM (4 environments, 5 seeds, shaped evaluation). Min-max normalized;

\pm

half-width of the 95% stratified bootstrap CI. Last row of each group: aggregate IQM. All four configurations use the same ensemble critic (

N=10

M=2

	TD3		SAC
Environment	twin	ensemble	twin	ensemble
Hopper	0.2112	0.0022	0.0087	0.0044
HalfCheetah	0.0016	0.0087	0.0015	0.0019
Humanoid	0.0185	0.0098	0.0102	0.0076
walker/run	0.0235	0.0246	0.0019	0.0038
Mean	0.0637	0.0113	0.0056	0.0044

Appendix

Table 11: Terminal-to-peak ratio of

|\alpha|

under the twin and the ensemble critic (4 environments, 5 seeds, shaped actors). Per environment, the mean over seeds of

|\alpha_{t}|

is averaged over the last 20 evaluations and divided by its maximum over training; twin is the critic of the main experiments and ensemble the critic of Table 10 (

N=10

M=2

) at an update-to-data ratio of 1. The ensemble critic brings the one TD3 ratio that the twin critic leaves above 0.10, Hopper, to 0.0022.

Figure 9: The one constant the method adds needs no tuning across a tenfold range of

\varepsilon

(4 environments, 5 seeds, shaped evaluation, raw return). Rows: shaped TD3 (top) and shaped SAC (bottom).

\varepsilon=0.01

(solid, circles): the value used everywhere else in the paper.

\varepsilon=0.1

(dashed, stars): a larger initial gate. Lines are the interquartile mean over seeds (median of 2,000 bootstrap resamples) and shading is the 95% stratified bootstrap CI.

	$\\|\delta_{t}\\|$ at the checkpoint nearest a share of the 1M-step budget
Game	5%	10%	25%	50%	75%	100%	$r$	Arc
Seaquest	0.0003	0.0035	0.0007	0.0002	$<$ 0.0001	0.0001	0.016	completed
Alien	0.0002	0.0019	$<$ 0.0001	$<$ 0.0001	$<$ 0.0001	0.0001	0.030	completed
Boxing	0.0001	0.0011	$<$ 0.0001	$<$ 0.0001	$<$ 0.0001	0.0001	0.057	completed
BattleZone	0.0003	0.0006	0.0149	0.0254	0.0206	0.0086	0.34	active
ChopperCommand	0.0002	0.0018	0.0477	0.0349	0.0328	0.0291	0.49	active

Appendix

Table 12: Atari amplitude ladder for the seven games in which the gate engages. Seed-averaged

\|\delta_{t}\|

, the expected probability-space L2 distance between the shaped and the base policy on a fixed buffer of 512 frames per run, at the saved checkpoint nearest each share of the 1M-step budget (5 seeds per game); cells below

10^{-4}

are shown as

<

0.0001.

r

is the terminal amplitude divided by the peak over all saved checkpoints of the run (a median of 14), computed from unrounded values; the arc is completed when

r\leq 0.10

	TD3	SAC	PPO
Actor and critic hidden layers	$2\times 256$ , ReLU
Actor output	$\tanh$	$\tanh$ -squashed Gaussian	Gaussian, state-independent $\log\sigma$
Optimizer	Adam
Learning rate	$3\times 10^{-4}$ (actor and critic)	actor, critic, temperature $3\times 10^{-4}$	$3\times 10^{-4}$ , linearly annealed
Discount $\gamma$	0.99
Target update	Polyak $\tau=0.005$ , actor and targets every critic step	Polyak $\tau=0.005$	n/a

Appendix

Table 13: Hyperparameters for continuous control. One value per algorithm unless a row names the experiments it applies to.

Experiment	Setting	Value
Batch size (Appendix F )	batch size	64, 256, 1024
Update-to-data ratio (Appendix L )	updates per environment step	1, 5, 20; 10 critics, minimum over 2
Gate initialization (Appendix M )	$\varepsilon$	0.01, 0.1

Appendix

Table 14: Hyperparameters that differ between experiments. Every other setting is as in Table 13 .

Setting	Value
Preprocessing	frame skip 4, $84\times 84$ grayscale, 4 stacked frames, up to 30 no-ops
Episode end	life loss in training, game over in evaluation
Network	convolutions $32{\times}8{\times}8/4$ , $64{\times}4{\times}4/2$ , $64{\times}3{\times}3/1$ , fully connected 512
Optimizer	Adam with numerical constant $10^{-4}$ ; policy, $Q$ and temperature learning rates $3\times 10^{-4}$
Discount $\gamma$	0.99
Batch size	64

Appendix

Table 15: Hyperparameters for Atari (discrete SAC).

Explore similar work

A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning
Aug 8, 2026Fouad BahrpeymaPotential-Based Reward Shaping Unified Framework
Industrial AI Group Professorship for Information Management Faculty of Informatics / Mathematics Dresden University of Applied Sciences Dresden, Germany
Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control
Sep 28, 2026Yisheng Zhang, Tao Wang, Sicun GaoPotential-Based Reward Shaping Policy Gradient
University of California San Diego, La Jolla, CA, USA · Tsinghua University, Beijing, China
Learning Not to Optimize: Physics-Informed Action-Space Reshaping for Intent-Based Network Control
Aug 2, 2026Zuyuan Zhang, Vaneet Aggarwal, Tian LanOptimal Policies Optical Networks
The George Washington University · Purdue University

Aug 8, 2026, cs.AIJ/K move · Enter open · S save

A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning

Fouad Bahrpeyma

Industrial AI Group Professorship for Information Management Faculty of Informatics / Mathematics Dresden University of Applied Sciences Dresden, Germany

Potential-Based Reward Shaping Unified Framework

Open paper AskPDF arXiv

Sparse, delayed, and weakly informative rewards remain central obstacles to efficient reinforcement learning. Reward shaping addresses these limitations by supplementing the task reward with an auxiliary signal that can accelerate learning while, in the classical setting, the original objective remains the evaluation criterion. Established theory guarantees safety for fixed shaping signals: potential-based reward shaping preserves optimal policies when the auxiliary term is the discounted difference of a time-invariant potential. In contemporary reinforcement learning systems, however, both the learner and the information available for guidance evolve during training: value estimates improve, novelty diminishes, feedback shifts, and predictive models are refined. Adaptive reward mechanisms occur across exploration, Bayesian inference, human-in-the-loop learning, automated reward design, and foundation-model-based approaches. This study introduces a unified analytical framework for comparing dynamic reward shaping and neighbouring adaptive reward mechanisms. The proposed framework distinguishes parametric revision from state-dependent variation, separates additive shaping from reward replacement and reward-adjacent guidance, and organises existing methods along temporal, informational, and theoretical dimensions. Using this framework, twelve method families are comparatively analysed. The framework further highlights the conditions under which optimality guarantees survive contemporary deep reinforcement learning pipelines, replay buffers, bootstrapped critics, and reward normalisation, while exposing the unresolved relationship between adaptation rate and learner stability.

Abstract

Explore similar work

A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning

Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control

Learning Not to Optimize: Physics-Informed Action-Space Reshaping for Intent-Based Network Control