cs.ROSep 28, 2026

Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control

Authors: Yisheng Zhang, Tao Wang, Sicun Gao

Organizations: University of California San Diego, La Jolla, CA, USA · Tsinghua University, Beijing, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reward shaping is fundamental to modern robotic control with deep reinforcement learning (RL), yet practitioners still rely heavily on heuristic principles borrowed from classical optimal control and trajectory optimization. Existing methods rarely distinguish reward terms that are intrinsic to the control objective from numerical regularizers, leading to brittle hyperparameter tuning. To determine which quantities a reward must contain, we study the stabilization control problem with a focus on zeroth-order (configuration) and first-order (velocity) information. We theoretically and empirically demonstrate that policy gradient methods can successfully solve stabilization tasks without first-order reward terms, adding such terms can instead introduce severe sensitivity as their scale grows. Conversely, our findings confirm that reward functions must be zeroth-order complete over goal-relevant coordinates, while the first-order state remains necessary in the policy observation under our low-dissipation assumptions. Overall, these results provide actionable and principled guidance for reward design in robotic RL.

Figures & tables

Figure 1: A velocity penalty has task-dependent effects in direct trajectory optimization, shown by the mean stable dwell for IPOPT-optimized Acrobot and Pendubot trajectories. Moderate velocity regularization helps in different ranges for the two plants, while an overly large weight suppresses successful motion in both.
TaskZeroth-order term r0(q)r_{0}(q)Velocity error ev(v)e_{v}(v)
Acrobot1−3[ec(θ1)+ec(θ2,a)]1-3[e_{c}(\theta_{1})+e_{c}(\theta_{2,a})]θ˙12+θ˙22\dot{\theta}_{1}^{2}+\dot{\theta}_{2}^{2}
Pendubot1−3[ec(θ1)+ec(θ2,a)]1-3[e_{c}(\theta_{1})+e_{c}(\theta_{2,a})]θ˙12+θ˙22\dot{\theta}_{1}^{2}+\dot{\theta}_{2}^{2}
Cartpole1−3ec(θ)1-3e_{c}(\theta)vc2+θ˙2v_{c}^{2}+\dot{\theta}^{2}
Double cart pendulum2−3[ec(θ1)+ec(θ2,a)]2-3[e_{c}(\theta_{1})+e_{c}(\theta_{2,a})]vc2+θ˙12+θ˙22v_{c}^{2}+\dot{\theta}_{1}^{2}+\dot{\theta}_{2}^{2}
Quadcopter1−3tanh⁡(dp/0.8)−eR1-3\tanh(d_{p}/0.8)-e_{R} , eR=1−(q⊤q∗)2e_{R}=1-(q^{\top}q^{\ast})^{2}∥vbody∥22\lVert v_{\mathrm{body}}\rVert_{2}^{2}
Franka reach1−3tanh⁡(dp/0.8)−3eq1-3\tanh(d_{p}/0.8)-3e_{q} , eq=mean⁡j(qj−qj∗)2e_{q}=\sqrt{\operatorname{mean}_{j}(q_{j}-q_{j}^{*})^{2}}∥q˙∥22\lVert\dot{q}\rVert_{2}^{2}
Table 1: Reward family used in the Isaac Lab sweep. Every row has the form rβ=r0−βevr_{\beta}=r_{0}-\beta e_{v} . The β=0\beta=0 condition removes velocity only from the reward; the policy still receives the full task-defined configuration and velocity observation.
Figure 2: Velocity-reward sweeps with full configuration and velocity observations. Blue solid curves show stable dwell and green dashed curves show success rate; bands are mean ±\pm one population standard deviation over five seeds. Every task learns reaching, braking, and sustained stabilization at β=0\beta=0 , while sufficiently large velocity penalties generally reduce both metrics.
Figure 3: No-velocity-observation ablation at β=0\beta=0 . Each task keeps exactly the same zeroth-order reward and velocity-aware success predicate used in the full-observation experiment. Bars show the five-seed mean and population standard deviation of best dwell (left) and corresponding success rate (right).
Figure 4: Capping only the squared-velocity error prevents the large- β\bm{\beta} performance collapse. Five-seed beta sweeps for Pendubot (left) and Franka (right) compare the uncapped reward with the velocity cap c=4c=4 in Equation 10 . Curves and bands are the mean ±\pm one population standard deviation. The cap leaves the successful low- β\beta regime intact and restores substantial performance where the raw reward fails.
Taskβ\betaRewardDwell (s) ↑\uparrowG^t‾\overline{\widehat{G}_{t}}∣min⁡tG^t∣‾\overline{|\min_{t}\widehat{G}_{t}|}Value MSE ↓\downarrow
Pendubot0Raw6.8731.99.10×1029.10\times 10^{2}3.20×1023.20\times 10^{2}
 1Raw0.018−928-9283.01×1063.01\times 10^{6}2.65×1072.65\times 10^{7}
 1Clipped ( c=4c=4 )5.72−245-2451.15×1031.15\times 10^{3}1.04×1031.04\times 10^{3}
Franka0Raw14.5894.569.10.495
 1Raw0.008−819-8191.91×1031.91\times 10^{3}1.09×1031.09\times 10^{3}
 1Clipped ( c=4c=4 )9.46−99.2-99.21.87×1021.87\times 10^{2}1.49×1021.49\times 10^{2}
Table 2: Reward clipping contracts return tails and critic error. Dwell is the best evaluation duration in each seed-42 run. The remaining columns are means over the final five matched one-iteration probes; G^t\widehat{G}_{t} denotes the scalar return target at time step tt .
Figure 5: Velocity capping contracts critic scale and restores target-centered structure. From left to right: failed raw Pendubot, successful velocity-capped Pendubot, failed raw Franka, and successful velocity-capped Franka, all at β=1\beta=1 and with c=4c=4 in the capped conditions using seed 42. Pendubot fixes angular velocities at zero, while Franka holds all remaining joints at a common target-conditioned reference. Red stars mark the upright or zero-error target.
Figure 6: Configuration completeness determines whether a zeroth-order reward is sufficient. From left to right: best stable dwell (s), success-ever rate, position–orientation pass rate PpRP_{pR} , and joint-velocity pass rate PvP_{v} . Full uses a target joint posture, whereas Partial specifies only the end-effector position and orientation. Curves and bands show the mean ±\pm one population standard deviation over five seeds.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Environments.
Figure 8: Rendered views of the seven environments.
TaskConditions for Ci,t=1C_{i,t}=1
Acrobot / Pendubot∣θ1∣,∣θ2,a∣<0.15|\theta_{1}|,|\theta_{2,a}|<0.15 rad, ∣θ˙1∣,∣θ˙2∣<0.5|\dot{\theta}_{1}|,|\dot{\theta}_{2}|<0.5 rad/s.
Cartpole∣θ∣<0.15|\theta|<0.15 rad, ∣vc∣<0.5|v_{c}|<0.5 m/s, ∣θ˙∣<0.5|\dot{\theta}|<0.5 rad/s.
Double Cart Pendulum∣θ1∣,∣θ2,a∣<0.15|\theta_{1}|,|\theta_{2,a}|<0.15 rad, ∣vc∣<0.5|v_{c}|<0.5 m/s, ∣θ˙1∣,∣θ˙2∣<0.5|\dot{\theta}_{1}|,|\dot{\theta}_{2}|<0.5 rad/s
Quadcopterdp<0.10d_{p}<0.10 m, ∥vbody∥2<0.20\lVert v_{\mathrm{body}}\rVert_{2}<0.20 m/s, ∥ωbody∥2<0.30\lVert\omega_{\mathrm{body}}\rVert_{2}<0.30 rad/s, tilt angle <0.15<0.15 rad.
Franka Reach (Full / Partial)dp<0.05d_{p}<0.05 m, dR<0.20d_{R}<0.20 rad, ∥q˙∥2<0.25\lVert\dot{q}\rVert_{2}<0.25 rad/s.
Humanoiddxy<0.35d_{xy}<0.35 m, ∥vroot∥2<0.35\lVert v_{\mathrm{root}}\rVert_{2}<0.35 m/s, ∥ωroot∥2<0.75\lVert\omega_{\mathrm{root}}\rVert_{2}<0.75 rad/s.
Appendix
Table 3: Instantaneous success predicates for all Isaac Lab evaluations. All conditions in a row must hold simultaneously.
ParameterSix-task shared settingHumanoid setting
Parallel environments40964096
Rollout steps per environment600600
Transitions per PPO update2,457,6002,457,600
Training iterations23001500
Checkpoint interval / final checkpoint150 / 2299150 / 1499
Training seeds0,1,2,3,420,1,2,3,420,1,2,3,420,1,2,3,42
Appendix
Table 4: Rollout, runner, and actor–critic settings.
ParameterSix-task shared settingHumanoid setting
OptimizerAdamAdam
Initial learning rate10−310^{-3}10−410^{-4}
Learning-rate scheduleadaptive KLadaptive KL
Desired KL0.010.008
Policy ratio clip ϵ\epsilon0.20.2
Learning epochs per rollout55
Appendix
Table 5: PPO objective and optimization settings.
Figure 9: Training return ( Equation 19 ) across the seven Isaac Lab tasks. Each seed is first smoothed with a centered 51-iteration moving average; curves and shaded regions then show the across-seed mean and one standard deviation, respectively.
Figure 10: Training value loss ( Equation 20 ) across the seven Isaac Lab tasks. Each seed is first smoothed with a centered 51-iteration moving average; curves and shaded regions then show the across-seed mean and one standard deviation, respectively.
Figure 11: toy example

Explore similar work

CardsList

  1. Model-Free Output Feedback Stabilization via Policy Gradient Methods

    Jan 27, 2026Ankang Zhang, Ming Chi, Xiaoling Wang +1Robust ControlStabilization

    School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China · College of Automation, Nanjing University of Posts and Telecommunications, Nanjing 210023, China

  2. Zero-shot Transfer of Reinforcement Learning Control Policies for the Swing-Up and Stabilization of a Cart-Pole System

    Jun 20, 2026Nikki Xu, Hien TranReinforcement Learning ControlGenerative Control Policies

    Department of Mathematics, Box 8205, NC State University, Raleigh, NC 27695

  3. Stability of Control Lyapunov Function Guided Reinforcement Learning

    May 3, 2026Zachary Olkin, William D. Compton, Aaron D. AmesReinforcement Learning ControlLyapunov Function

    The authors are with the Department of Control and Dynamical Systems at the California Institute of Technology.