Hybrid Energy-Aware Reward Shaping: A Unified Lightweight Physics-Guided Methodology for Policy Optimization

Qijun Liao; Jue Yang; Yiting Kang; Xinxin Zhao; Yong Zhang; Mingan Zhao

cs.LGMar 12, 2026

Hybrid Energy-Aware Reward Shaping: A Unified Lightweight Physics-Guided Methodology for Policy Optimization

Authors: Qijun Liao, Jue Yang, Yiting Kang, Xinxin Zhao, Yong Zhang, Mingan Zhao

Organizations: School of Mechanical Engineering, University of Science and Technology Beijing, Beijing 100083, China · Jiangsu XCMG Construction Machinery Research Institute Co., Ltd., Jiangsu 221000, China

Abstract

Deep reinforcement learning for continuous control often suffers from high variance, low energy efficiency, and poor generalization under distribution shift, as purely data-driven exploration ignores available physical structure. This paper proposes Hybrid Energy-Aware Reward Shaping (H-EARS), which encodes dominant energy terms -- assumed known a priori -- directly as reward potentials at O(n) per-step computation. H-EARS decomposes the shaping potential into task-oriented and energy-based components, supplemented by an action regularization term that deliberately modifies the optimization objective to enforce energy-efficient control. A complete theoretical foundation is established: functional independence of shaping and regularization, energy-based gradient enrichment under positive-definite Hessian conditions, convergence guarantees under function approximation, and approximate potential error bounds. Across four continuous control benchmarks and four baseline algorithms, H-EARS achieves consistent gains in convergence speed, policy stability, and final performance. High-fidelity vehicle simulations validate applicability in safety-critical settings under extreme road conditions.

Figures & tables

Method Type	Complexity	Known Structure?	Generalization	Application Scenario
Lagrangian RL [ 3 ]	$O(n^{3})$	No (learns from data)	Low	Unknown-dynamics systems
Hamiltonian RL [ 4 ]	$O(n^{2})$	No (learns from data)	Low	Unknown conservative systems
Graph Networks [ 17 ]	$O(n^{2})$	No (learns from data)	Medium	Multi-body simulation
Neural ODE [ 18 ]	$O(n^{2})$	No (learns from data)	Medium	Temporal dynamics modeling
Pure Model-free RL	$O(1)$	No	High	Large data available
H-EARS (Ours)	$O(n)$	Partial (dominant terms)	High	Rapid prototyping + deployment

TABLE I: Characteristic Comparison of Different Physics-Guided Methods ‡

Symbol	Definition
$\mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{P},R,\gamma)$	Original MDP
$\mathcal{M}_{\lambda}$	Regularized MDP: reward $R_{\lambda}=R-\lambda\mathcal{E}(a)$
$J(\pi)$ , $J_{\mathcal{M}}(\pi)$	Cumulative return; $J_{\mathcal{M}}$ emphasizes the original reward $R$
$\Phi(s)$	Dual potential: $\alpha_{\text{task}}\Phi_{\text{task}}(s)+\alpha_{\text{energy}}\Phi_{\text{energy}}(s)$
$\Phi_{\text{task}}(s)$	Task-oriented potential (e.g., goal proximity, tracking error)
$\Phi_{\text{energy}}(s)$	Energy-based potential; $\Phi_{\text{energy}}(s)=-E(q,\dot{q})$

TABLE II: Principal Notation

Fig. 1: H-EARS construction pipeline. Left branch (action domain): action regularization defines the regularized MDP, intentionally modifying the optimization objective to trade task performance for energy efficiency (Proposition II.2 ). Right branch (state domain): physical energy knowledge and task specification are combined into the dual potential $\Phi$ ; gradient enrichment under locally positive-definite $\partial^{2}E/\partial q^{2}$ is established in Theorem II.3 . Potential-based reward shaping (PBRS) is applied to regularized MDP using $\Phi$ , yielding $R_{\text{H-EARS}}$ . Since the two branches act on disjoint argument domains, $\alpha_{\text{task}}$ , $\alpha_{\text{energy}}$ , and $\lambda$ serve as independently tunable design parameters (Lemma II.1 ).

Panel of Figure 4 — Fig. 2: Benchmark performance comparison across four environments and four baseline algorithms. SAC and TD3 results validate Theorem II.3 (energy-based gradient enrichment under positive-definite Hessian accelerates convergence) and Lemma II.1 (independent task and energy contributions). DDPG degradation in Ant-v5 instantiates the oscillatory-artifact failure mode analyzed in Proposition II.2 : DDPG’s OU exploration is suppressed by energy regularization, creating local-minima traps in the 8-DOF shaped landscape.

Algorithm	Ant-v5	Hopper-v5	LunarLander-v3	Humanoid-v5
SAC - Vanilla	$3157\pm 182$	$2520\pm 405$	$268\pm 30$	$4988\pm 502$
SAC - H-EARS	$\mathbf{4183\pm 174}$	$\mathbf{3354\pm 354}$	$\mathbf{289\pm 19}$	$\mathbf{5228\pm 470}$
TD3 - Vanilla	$3570\pm 391$	$2364\pm 302$	$\mathbf{279\pm 15}$	$4964\pm 357$
TD3 - H-EARS	$\mathbf{4125\pm 398}$	$\mathbf{2641\pm 298}$	$277\pm 9$	$\mathbf{5190\pm 325}$
PPO - Vanilla	$376\pm 281$	$\mathbf{1621\pm 255}$	$235\pm 24$	$\mathbf{197\pm 89}$
PPO - H-EARS	$\mathbf{501\pm 127}$	$1568\pm 203$	$\mathbf{258\pm 14}$	$178\pm 120$

TABLE III: Comprehensive Performance Comparison across Standard Environments

	Ant-v5			Hopper-v5
Variant	$\alpha_{\text{task}}$	$\alpha_{\text{energy}}$	$\lambda$	$\alpha_{\text{task}}$	$\alpha_{\text{energy}}$	$\lambda$
SAC - Vanilla	0	0	0	0	0	0
Energy Only	0	3e-2	0	0	1e-3	0
Task Only	5e-3	0	0	5e-1	0	0
Regularization Only	0	0	1e-2	0	0	5e-4
Without Regularization	5e-3	3e-2	0	5e-1	1e-3	0

TABLE IV: Hyperparameters for Ablation Variants in Ant-v5 and Hopper-v5

Panel of Figure 7 — Fig. 3: Ablation study in Ant-v5 and Hopper-v5 (SAC backbone). TaskOnly and EnergyOnly converge independently without gradient interference, supporting Lemma II.1 . The Without-Regularization gap validates Proposition II.2 .

Panel of Figure 8 — Fig. 4: Dual-decomposition ablation: H-EARS vs. Combined-Single- $\Phi$ (same total weight $\alpha_{\text{task}}+\alpha_{\text{energy}}$ , single potential), Task $\Phi$ Only, and Energy $\Phi$ Only. SAC backbone, 5 seeds; shading = $\pm$ s.e. Combined-Single- $\Phi$ underperforms H-EARS by $\approx$ 33% (Ant) and $\approx$ 21% (Hopper) despite identical total weighting, validating Proposition II.4 . In Hopper-v5 it falls below Energy $\Phi$ Only, confirming that potential merging suppresses rather than combines individual contributions.

Environment	$\partial^{2}E/\partial q^{2}\succ 0$	$\rho_{\text{info}}$ (est.)	Primary gain mechanism	Perf. gain (%)	Theory consistent?
Ant-v5	Global (rigid-body $M(q)\succ 0$ ; 8-DOF)	$\sim 10^{2}$ – $10^{3}$	Gradient enrichment	$+32.5\%$	✓
Hopper-v5	Local near upright equilibrium	$\sim 10^{1}$ – $10^{2}$	Stability regularization	$+33.1\%$	✓
LunarLander-v3	Partial (non-mechanical energy proxy)	$\sim 10^{0}$ – $10^{1}$	Action regularization	$+7.8\%$	✓(partial)
Humanoid-v5	Global ( $M(q)\succ 0$ ; 21-DOF)	$\sim 10^{1}$ – $10^{2}$ †	Stability (approx. gap limits perf.)	$+4.8\%$	✓

TABLE V: Hessian Condition Validity and Observed H-EARS Gains (SAC backbone)

Environment	$\Delta\mu$	$95\%$ CI	$p$ -value	Sig.
Ant-v5	$+1026$	$[766,\;1286]$	$<0.0001$
Hopper-v5	$+834$	$[278,\;1390]$	$0.0087$	**
* $p<0.001$ ; $p<0.01$ (two-tailed Welch’s $t$ -test, $\alpha=0.01$ ).

TABLE VI: Statistical Significance (Welch’s

t

-test)

Panel of Figure 11 — Fig. 5: Hyperparameter sensitivity analysis (SAC backbone). Each row tests one parameter ( $\alpha_{\text{task}}$ , $\alpha_{\text{energy}}$ , $\lambda$ from top to bottom); each column corresponds to one environment; each colored line corresponds to one tested value (1e0–1e-5 as shown in the legend). Shaded bands show $\pm$ 1 s.d. over 5 seeds. Optimal values for each environment are listed in Table X . The smooth, bounded response in panels a–k and the complete failure of $\lambda{=}$ 1e-5 in panel l are both consistent with Theorem II.6 and Proposition II.2 respectively.

Fig. 6: Vehicle simulation model in Trucksim

Parameter	Symbol/Unit	Value
Total Vehicle Mass	$m$ /kg	2100
Yaw Moment of Inertia	$I_{z}$ /kg·m 2	4116
Center of Mass Height	$h$ /mm	710
Distance from CoG to Front Axle	$a$ /mm	1350
Distance from CoG to Rear Axle	$b$ /mm	1450
Front Track Width	$d_{f}$ /mm	1800

TABLE VII: Vehicle Body Parameter Settings

Panel of Figure 14 — Fig. 7: Training road height and adhesion coefficient variation parameter settings

Panel of Figure 15 — Fig. 8: Test road height and adhesion coefficient variation parameter settings

Fig. 9: Training return across four variants

Algorithm	Episodes to Threshold	Avg. Return (Stable)	CV (%)
H-EARS SAC ( $\varepsilon_{\text{approx}}{=}10\%$ )	179	357.0 $\pm$ 14.0	3.8
H-EARS SAC ( $\varepsilon_{\text{approx}}{=}15\%$ )	281	311.0 $\pm$ 12.0	4.0
H-EARS SAC ( $\varepsilon_{\text{approx}}{=}30\%$ )	348	280.0 $\pm$ 22.0	5.8
SAC	376	275.0 $\pm$ 18.0	8.3

TABLE VIII: Training Performance Comparison Across Approximation Levels

Metric	H-EARS ( $\varepsilon{=}10\%$ )	H-EARS ( $\varepsilon{=}15\%$ )	H-EARS ( $\varepsilon{=}30\%$ )	SAC
Avg Speed Error (m/s)	$2.51\pm 0.32$	$\mathbf{0.23\pm 0.12}$	$0.35\pm 0.20$	$2.9\pm 0.19$
Max Sideslip (°)	$\mathbf{0.30}$	$0.52$	$0.82$	$1.03$
Max $\|r\|$ (°/s)	$\mathbf{0.51}$	$1.18$	$2.71$	$3.24$
CV of Speed (%)	$\mathbf{2.8}$	$5.2$	$7.1$	$7.8$

TABLE IX: Vehicle Control Performance Under Extreme Conditions: Cross-Approximation Comparison

Fig. 10: Longitudinal speed tracking error variation with position for each controller

Panel of Figure 20 — Fig. 11: Sideslip angle and yaw rate variation with position for each controller

Hyperparameters	Value
Shared
Optimizer	Adam ( $\beta_{1}=0.9,\beta_{2}=0.999$ )
Actor learning rate	$3\mathrm{e}{-}4$
Critic learning rate	$3\mathrm{e}{-}4$
Discount factor ( $\gamma$ )	0.99
Policy update interval	5000

TABLE X: Detailed hyperparameters

RL Control

Reinforcement learning control uses reward-driven learning to make decisions in dynamic systems, from robots and vehicles to energy and infrastructure.

See all 272 papers

Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control
Sep 28, 2026Yisheng Zhang, Tao Wang, Sicun GaoReward Shaping Robotic RL
University of California San Diego, La Jolla, CA, USA · Tsinghua University, Beijing, China
Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients
May 14, 2026Matias Alvo, Daniel Russo, Yash KanoriaRL Control Reinforcement Learning
Graduate School of Business Columbia University
A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning
Aug 8, 2026Fouad BahrpeymaReward Shaping Reward Design for RL
Industrial AI Group Professorship for Information Management Faculty of Informatics / Mathematics Dresden University of Applied Sciences Dresden, Germany

Sep 28, 2026J/K move · Enter open · S save

Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control

Yisheng Zhang, Tao Wang, Sicun Gao

University of California San Diego, La Jolla, CA, USA · Tsinghua University, Beijing, China

Reward Shaping Robotic RL Feedback Control Policy Gradient Methods Reward Design for RL

Open paper AskPDF arXiv

Reward shaping is fundamental to modern robotic control with deep reinforcement learning (RL), yet practitioners still rely heavily on heuristic principles borrowed from classical optimal control and trajectory optimization. Existing methods rarely distinguish reward terms that are intrinsic to the control objective from numerical regularizers, leading to brittle hyperparameter tuning. To determine which quantities a reward must contain, we study the stabilization control problem with a focus on zeroth-order (configuration) and first-order (velocity) information. We theoretically and empirically demonstrate that policy gradient methods can successfully solve stabilization tasks without first-order reward terms, adding such terms can instead introduce severe sensitivity as their scale grows. Conversely, our findings confirm that reward functions must be zeroth-order complete over goal-relevant coordinates, while the first-order state remains necessary in the policy observation under our low-dissipation assumptions. Overall, these results provide actionable and principled guidance for reward design in robotic RL.

Abstract

RL Control

Explore similar work

Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control

Policy Optimization in Hybrid Discrete-Continuous Action Spaces via Mixed Gradients

A Unified Framework for Dynamic Reward Shaping in Reinforcement Learning