Higher-Order Action Supervision Makes A Strong Policy Class

Peng Cheng; Yunxian Hou; Zhi Zhou; Qian Zhang; Chang Huang; Xianyuan Zhan

cs.ROOct 8, 2026

Higher-Order Action Supervision Makes A Strong Policy Class

Authors: Peng Cheng, Yunxian Hou, Zhi Zhou, Qian Zhang, Chang Huang, Xianyuan Zhan

Organizations: Institute for AI Industry Research (AIR), Tsinghua University · Zenfex AI Lab · University of Electronic Science and Technology of China · Horizon Robotics

Abstract

Modern data-driven decision-making methods, such as imitation learning (IL) and reinforcement learning (RL), have achieved great success in solving many complex tasks. However, these methods often suffer from serious control instability and robustness issues when applied in real-world applications such as robotics and autonomous driving, posing notable challenges for their practical deployment. We argue that this instability issue stems largely from their limitations in solely supervising and optimizing zeroth-order actions (i.e., the action labels), failing to account for higher-order action dynamics and temporal consistency. In this paper, we show that simultaneously supervising both zeroth- and first-order actions can dramatically enhance policies' performance and control robustness. To achieve this, we introduce a novel and elegant loss scheme supported by formal theoretical guarantees that can equip any off-the-shelf policy model (e.g., deterministic, stochastic, or flow policies) with the capability for higher-order action supervision, without requiring any structural modifications. Moreover, our proposed method can serve as a lightweight plug-and-play module that seamlessly integrates with a broad spectrum of existing offline RL frameworks. Extensive evaluations on OGBench and D4RL demonstrate that our approach yields substantial performance and robustness improvements across a wide range of continuous control environments. Notably, our method can also enhance policies' out-of-distribution (OOD) generalization capability in the challenging low-data regime, making it an ideal tool in tackling many real-world control problems.

Figures & tables

Figure 1: Trajectories and zeroth-/first-order actions of policies learned on the maze2d task with 20 successful trajectories and 500k training steps. Green arrows denote the zeroth-order action $a$ (joint torque), while red arrows denote the first-order action $\dot{a}$ (time derivative of joint torque, computed as $a^{\prime}-a$ ). (a) Standard behavior cloning (BC) without $\dot{a}$ supervision, which yields a suboptimal trajectory with highly noisy and temporally inconsistent first-order actions. (b) Our method with higher-order action supervision, which yields an optimal navigation trajectory with much smoother and more stable zeroth- and first-order actions.

	Det. Policy		Sto. Policy (mean sup.)		Sto. Policy (standard)		Flow Policy
Task Category (Avg. of 5 tasks)	TD3+BC	TD3+BC+ours	ReBRAC	ReBRAC+ours	IQL	IQL+ours	IFQL	IFQL+ours
antmaze-large-navigate	$83{\scriptstyle\pm 4}$	$98{\scriptstyle\pm 3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!15})$	$81{\scriptstyle\pm 5}$	$98{\scriptstyle\pm 2}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!17})$	$53{\scriptstyle\pm 3}$	$68{\scriptstyle\pm 5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!15})$	$28{\scriptstyle\pm 13}$	$36{\scriptstyle\pm 14}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!8})$
antmaze-giant-navigate	$42{\scriptstyle\pm 8}$	$65{\scriptstyle\pm 10}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!23})$	$26{\scriptstyle\pm 8}$	$61{\scriptstyle\pm 12}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!35})$	$4{\scriptstyle\pm 1}$	$9{\scriptstyle\pm 1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!5})$	$3{\scriptstyle\pm 2}$	$4{\scriptstyle\pm 3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1})$
humanoidmaze-medium-navigate	$34{\scriptstyle\pm 8}$	$42{\scriptstyle\pm 5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!8})$	$22{\scriptstyle\pm 8}$	$55{\scriptstyle\pm 10}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!33})$	$33{\scriptstyle\pm 2}$	$36{\scriptstyle\pm 3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!3})$	$60{\scriptstyle\pm 16}$	$64{\scriptstyle\pm 13}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!4})$
humanoidmaze-large-navigate	$2{\scriptstyle\pm 1}$	$4{\scriptstyle\pm 1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!2})$	$2{\scriptstyle\pm 1}$	$4{\scriptstyle\pm 2}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!2})$	$2{\scriptstyle\pm 1}$	$3{\scriptstyle\pm 1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1})$	$11{\scriptstyle\pm 2}$	$12{\scriptstyle\pm 3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1})$
antsoccer-arena-navigate	$13{\scriptstyle\pm 3}$	$16{\scriptstyle\pm 3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!3})$	$0{\scriptstyle\pm 0}$	$16{\scriptstyle\pm 4}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!16})$	$8{\scriptstyle\pm 2}$	$23{\scriptstyle\pm 3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!15})$	$33{\scriptstyle\pm 6}$	$40{\scriptstyle\pm 13}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!7})$

Table 1: Average normalized scores on 10 OGBench task categories, each aggregating 5 sub-tasks over 5 random seeds. Values after

\pm

denote standard deviations. The

{\color[rgb]{0.0195,0.6523,0.082}\uparrow}

marks indicate improvements greater than

1.0

, whereas

{\color[rgb]{0.5,0.5,0.5}\uparrow}

denotes minor improvements. Full per-task results are presented in Table 4 .

Figure 2: Comparison of vanilla deterministic BC and our $1^{\mathrm{st}}$ -order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation $||\dot{a}||=||a^{\prime}-a||$ ( $a$ is joint torque) are aggregated from 10 trajectories.

Figure 3: Comparison of TD3+BC and our $1^{\mathrm{st}}$ -order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation $||\dot{a}||=||a^{\prime}-a||$ ( $a$ is joint torque) are aggregated from 10 trajectories.

Task	TD3+BC	+CAPS	+L2C2	+LipsNet	+Extra $\dot{a}$ head	TD3+BC+Ours
Hopper-m	$40.1{\scriptstyle\pm 18.6}$	$42.6{\scriptstyle\pm 5.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!2.5})$	$42.7{\scriptstyle\pm 7.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!2.6})$	$43.2{\scriptstyle\pm 3.6}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!3.1})$	$41.5{\scriptstyle\pm 7.8}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1.4})$	$44.1{\scriptstyle\pm 12.7}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!\textbf{4.0}})$
Hopper-me	$17.8{\scriptstyle\pm 7.9}$	$39.6{\scriptstyle\pm 8.2}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!21.8})$	$39.8{\scriptstyle\pm 8.3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!22.0})$	$32.8{\scriptstyle\pm 18.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!15.0})$	$26.6{\scriptstyle\pm 8.6}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!8.8})$	$42.3{\scriptstyle\pm 10.8}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!\textbf{24.5}})$
Hopper-e	$23.2{\scriptstyle\pm 18.2}$	$57.1{\scriptstyle\pm 29.7}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!33.9})$	$55.2{\scriptstyle\pm 34.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!32.0})$	$16.9{\scriptstyle\pm 2.8}({\color[rgb]{1,0,0}\downarrow\!6.3})$	$30.6{\scriptstyle\pm 15.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!7.4})$	$93.5{\scriptstyle\pm 13.6}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!\textbf{70.3}})$
HalfCheetah-m	$16.4{\scriptstyle\pm 10.2}$	$29.6{\scriptstyle\pm 5.6}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!13.2})$	$30.3{\scriptstyle\pm 7.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!13.9})$	$6.6{\scriptstyle\pm 3.0}({\color[rgb]{1,0,0}\downarrow\!9.8})$	$28.1{\scriptstyle\pm 4.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!11.7})$	$31.5{\scriptstyle\pm 2.4}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!\textbf{15.1}})$
HalfCheetah-me	$15.4{\scriptstyle\pm 10.7}$	$19.8{\scriptstyle\pm 3.4}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!4.4})$	$28.6{\scriptstyle\pm 3.8}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!13.2})$	$3.3{\scriptstyle\pm 2.2}({\color[rgb]{1,0,0}\downarrow\!12.1})$	$20.6{\scriptstyle\pm 3.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!5.2})$	$35.6{\scriptstyle\pm 2.4}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!\textbf{20.2}})$
HalfCheetah-e	$1.7{\scriptstyle\pm 3.3}$	$1.7{\scriptstyle\pm 1.6}$	$2.4{\scriptstyle\pm 1.9}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.7})$	$0.8{\scriptstyle\pm 0.6}({\color[rgb]{1,0,0}\downarrow\!0.9})$	$0.2{\scriptstyle\pm 1.8}({\color[rgb]{1,0,0}\downarrow\!1.5})$	$3.9{\scriptstyle\pm 2.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!\textbf{2.2}})$

Table 3: Comparison with action-smoothness approaches and explicit first-order action prediction on D4RL MuJoCo locomotion tasks with reduced-size (10k) datasets. All methods use TD3+BC as the shared backbone. Arrows show score differences relative to TD3+BC on the same task:

{\color[rgb]{0.0195,0.6523,0.082}\uparrow}

indicates a gain of at least

1.0

{\color[rgb]{0.5,0.5,0.5}\uparrow}

a smaller positive gain, and

{\color[rgb]{1,0,0}\downarrow}

a decrease. No arrow indicates no change.

Figure 4: Ablation study on value regularization weight $\lambda_{Q}$ and $1^{\text{st}}$ -order action supervision weight $\lambda_{\dot{a}}$ of IFQL.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Task Category	TD3+BC	TD3+BC+ours	ReBRAC	ReBRAC+ours	IQL	IQL+ours	IFQL	IFQL+ours
	Det. Policy		Sto. Policy (mean sup.)		Sto. Policy (standard)		Flow Policy
antmaze-large-navigate-singletask-task1-v0 ( $\ast$ )	$92{\scriptstyle\pm 8}$	$99{\scriptstyle\pm 2}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!7})$	$91{\scriptstyle\pm 10}$	$100{\scriptstyle\pm 1}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!9})$	$48{\scriptstyle\pm 9}$	$80{\scriptstyle\pm 10}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!32})$	$24{\scriptstyle\pm 17}$	$56{\scriptstyle\pm 14}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!32})$
antmaze-large-navigate-singletask-task2-v0	$85{\scriptstyle\pm 3}$	$91{\scriptstyle\pm 4}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!6})$	$88{\scriptstyle\pm 4}$	$92{\scriptstyle\pm 4}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!4})$	$42{\scriptstyle\pm 6}$	$42{\scriptstyle\pm 15}$	$8{\scriptstyle\pm 3}$	$16{\scriptstyle\pm 11}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!8})$
antmaze-large-navigate-singletask-task3-v0	$72{\scriptstyle\pm 10}$	$100{\scriptstyle\pm 0}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!28})$	$51{\scriptstyle\pm 18}$	$100{\scriptstyle\pm 1}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!49})$	$72{\scriptstyle\pm 7}$	$90{\scriptstyle\pm 6}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!18})$	$52{\scriptstyle\pm 17}$	$72{\scriptstyle\pm 10}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!20})$
antmaze-large-navigate-singletask-task4-v0	$85{\scriptstyle\pm 5}$	$99{\scriptstyle\pm 1}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!14})$	$84{\scriptstyle\pm 7}$	$100{\scriptstyle\pm 1}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!16})$	$51{\scriptstyle\pm 9}$	$52{\scriptstyle\pm 22}\,({\color[rgb]{0.5,0.5,0.5}\uparrow\!1})$	$18{\scriptstyle\pm 8}$	$40{\scriptstyle\pm 17}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!22})$
antmaze-large-navigate-singletask-task5-v0	$82{\scriptstyle\pm 10}$	$99{\scriptstyle\pm 0}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!17})$	$90{\scriptstyle\pm 2}$	$99{\scriptstyle\pm 1}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!9})$	$54{\scriptstyle\pm 22}$	$78{\scriptstyle\pm 9}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!24})$	$38{\scriptstyle\pm 18}$	$48{\scriptstyle\pm 19}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!10})$

Appendix

Table 4: Average normalized scores on 50 OGBench tasks over 5 random seeds. Values after

\pm

denote standard deviations. The

{\color[rgb]{0.0195,0.6523,0.082}\uparrow}

marks indicate improvements greater than

1.0

, whereas

{\color[rgb]{0.5,0.5,0.5}\uparrow}

denotes minor improvements.

	Det. Policy		Sto. Policy (mean sup.)		Sto. Policy (standard)		Flow Policy
Task Category	TD3+BC	TD3+BC+ours	ReBRAC	ReBRAC+ours	IQL	IQL+ours	IFQL	IFQL+ours
Hopper-m	$55.5{\scriptstyle\pm 7.7}$	$58.9{\scriptstyle\pm 7.4}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!3.4})$	$84.9{\scriptstyle\pm 2.6}$	$85.7{\scriptstyle\pm 1.3}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.8})$	$67.3{\scriptstyle\pm 5.7}$	$68.8{\scriptstyle\pm 5.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1.5})$	$89.5{\scriptstyle\pm 5.7}$	$94.2{\scriptstyle\pm 7.2}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!4.7})$
Hopper-mr	$57.9{\scriptstyle\pm 17.6}$	$58.4{\scriptstyle\pm 9.2}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.5})$	$81.2{\scriptstyle\pm 25.1}$	$94.3{\scriptstyle\pm 10.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!13.1})$	$91.7{\scriptstyle\pm 8.6}$	$94.2{\scriptstyle\pm 5.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!2.5})$	$96.2{\scriptstyle\pm 4.7}$	$96.3{\scriptstyle\pm 2.3}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.1})$
Hopper-me	$89.8{\scriptstyle\pm 15.2}$	$89.9{\scriptstyle\pm 8.4}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.1})$	$106.4{\scriptstyle\pm 10.7}$	$106.6{\scriptstyle\pm 5.4}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.2})$	$91.5{\scriptstyle\pm 14.3}$	$109.2{\scriptstyle\pm 9.8}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!17.7})$	$77.1{\scriptstyle\pm 25.5}$	$89.8{\scriptstyle\pm 8.7}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!12.7})$
Hopper-e	$108.3{\scriptstyle\pm 6.7}$	$110.5{\scriptstyle\pm 0.2}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!2.2})$	$108.5{\scriptstyle\pm 4.1}$	$108.6{\scriptstyle\pm 3.8}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.1})$	$99.3{\scriptstyle\pm 9.2}$	$106.5{\scriptstyle\pm 5.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!7.2})$	$96.8{\scriptstyle\pm 12.2}$	$106.2{\scriptstyle\pm 8.6}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!9.4})$
Halfcheetah-m	$47.8{\scriptstyle\pm 0.4}$	$47.9{\scriptstyle\pm 0.4}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.1})$	$55.4{\scriptstyle\pm 1.0}$	$56.9{\scriptstyle\pm 1.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1.5})$	$47.4{\scriptstyle\pm 0.4}$	$48.6{\scriptstyle\pm 0.2}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1.2})$	$51.1{\scriptstyle\pm 0.6}$	$52.8{\scriptstyle\pm 0.3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1.7})$

Appendix

Table 5: Average normalized scores on D4RL MuJoCo locomotion tasks under full-datasize. Standard deviations are reported after the

\pm

sign. The

{\color[rgb]{0.0195,0.6523,0.082}\uparrow}

symbol indicates a performance increase greater than

1.0

, whereas

{\color[rgb]{0.5,0.5,0.5}\uparrow}

denotes minor improvements. All results and standard deviations (

\pm

) are averaged across 5 random seeds.

	Det. Policy		Sto. Policy (standard)		Flow Policy
Task Category	base	base + $\mathcal{L}_{\dot{a}}^{\text{det}}$	base	base + $\mathcal{L}_{\dot{a}}^{\text{sto}}$	base	base + $\mathcal{L}_{\dot{a}}^{\text{flow}}$
Hopper-e	$21.5{\scriptstyle\pm 13.5}$	$67.3{\scriptstyle\pm 14.3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!45.8})$	$57.5{\scriptstyle\pm 15.2}$	$93.1{\scriptstyle\pm 10.7}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!35.6})$	$72.2{\scriptstyle\pm 9.2}$	$97.2{\scriptstyle\pm 12.3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!25.0})$
Halfcheetah-e	$2.4{\scriptstyle\pm 1.2}$	$5.7{\scriptstyle\pm 1.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!3.3})$	$2.8{\scriptstyle\pm 1.6}$	$16.2{\scriptstyle\pm 10.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!13.4})$	$11.4{\scriptstyle\pm 4.5}$	$17.9{\scriptstyle\pm 6.9}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!6.5})$
Walker2d-e	$26.7{\scriptstyle\pm 12.2}$	$91.1{\scriptstyle\pm 14.7}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!64.4})$	$46.4{\scriptstyle\pm 15.2}$	$91.6{\scriptstyle\pm 9.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!45.2})$	$89.8{\scriptstyle\pm 14.6}$	$91.5{\scriptstyle\pm 6.3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1.7})$

Appendix

Table 6: Imitation learning performance on D4RL MuJoCo locomotion tasks with 10k datasets. Average normalized scores across deterministic, stochastic, and flow-based policy paradigms. The

{\color[rgb]{0.0195,0.6523,0.082}\uparrow}

indicates a performance increase

>1.0

, whereas

{\color[rgb]{0.5,0.5,0.5}\uparrow}

represents marginal gains. All results and standard deviations (

\pm

) are averaged across 5 random seeds.

Figure 5: Comparison of vanilla stochastic BC and our $1^{\mathrm{st}}$ -order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation $||\dot{a}||=||a^{\prime}-a||$ ( $a$ is joint torque) are aggregated from 10 trajectories.

Figure 6: Comparison of vanilla IQL and our $1^{\mathrm{st}}$ -order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation $||\dot{a}||=||a^{\prime}-a||$ ( $a$ is joint torque) are aggregated from 10 trajectories.

	Det. Policy		Sto. Policy (mean sup.)		Sto. Policy (standard)		Flow Policy
Task Category	TD3+BC	TD3+BC+ours	ReBRAC	ReBRAC+ours	IQL	IQL+ours	IFQL	IFQL+ours
Training time (D4RL)	28 min 30 s	33 min 6 s	25 min 8 s	30 min 34 s	34 min 44 s	39 min 56 s	30 min 49 s	35 min 52 s
Training time (OGBench)	28 min 23 s	38 min 56 s	25 min 11 s	31 min 54 s	35 min 7 s	42 min 5 s	32 min 0 s	36 min 51 s

Appendix

Table 7: Training time comparison between each base algorithm and its variant equipped with our first-order action supervision. All runs use the same number of training steps and are conducted on a single GPU.

Environment	Method	100k	300k	500k	700k	1M
Hopper-v2	SAC	$1767.1{\scriptstyle\pm 1005.9}$	$2399.1{\scriptstyle\pm 1391.8}$	$3076.8{\scriptstyle\pm 405.9}$	$\bm{3389.6{\scriptstyle\pm 162.2}}$	$3530.4{\scriptstyle\pm 32.6}$
	SAC + Ours	$\bm{2375.8{\scriptstyle\pm 1036.4}}$	$\bm{3273.9{\scriptstyle\pm 111.6}}$	$\bm{3226.4{\scriptstyle\pm 101.1}}$	$3359.2{\scriptstyle\pm 116.4}$	$\bm{3531.8{\scriptstyle\pm 33.2}}$
HalfCheetah-v2	SAC	$5659.2{\scriptstyle\pm 676.4}$	$8564.9{\scriptstyle\pm 1699.7}$	$9750.7{\scriptstyle\pm 1228.6}$	$10805.1{\scriptstyle\pm 804.7}$	$11557.4{\scriptstyle\pm 493.4}$
	SAC + Ours	$\bm{5790.9{\scriptstyle\pm 553.5}}$	$\bm{8614.4{\scriptstyle\pm 1423.9}}$	$\bm{9954.8{\scriptstyle\pm 1126.3}}$	$\bm{10893.5{\scriptstyle\pm 612.1}}$	$\bm{11858.2{\scriptstyle\pm 414.1}}$
Walker2d-v2	SAC	$702.9{\scriptstyle\pm 343.5}$	$2413.7{\scriptstyle\pm 1150.3}$	$2989.8{\scriptstyle\pm 1820.9}$	$4291.2{\scriptstyle\pm 950.5}$	$4768.9{\scriptstyle\pm 393.1}$
	SAC + Ours	$\bm{869.9{\scriptstyle\pm 310.3}}$	$\bm{3300.3{\scriptstyle\pm 879.6}}$	$\bm{4141.4{\scriptstyle\pm 572.9}}$	$\bm{4805.8{\scriptstyle\pm 184.5}}$	$\bm{4991.1{\scriptstyle\pm 264.5}}$

Appendix

Table 8: Performance snapshots of SAC and SAC+Ours on online MuJoCo benchmarks over 1M training steps. Results are averaged over 5 random seeds, with standard deviations reported after the

\pm

sign. Bold indicates the better result at each snapshot.

Environment (OGBENCH)	IFQL	+Ours	+AK ( $\lambda_{a}=0.5$ )	+AK ( $\lambda_{a}=0.1$ )	+AK ( $\lambda_{a}=1$ )
humanoidmaze-medium-navigate-singletask-task1-v0	$69{\scriptstyle\pm 19}$	$71{\scriptstyle\pm 16}$	$\bm{73.2{\scriptstyle\pm 7.0}}$	$51.2{\scriptstyle\pm 32.4}$	$67.6{\scriptstyle\pm 7.1}$
humanoidmaze-medium-navigate-singletask-task4-v0	$\bm{1{\scriptstyle\pm 1}}$	$\bm{1{\scriptstyle\pm 1}}$	$0.0{\scriptstyle\pm 0.0}$	$0.4{\scriptstyle\pm 0.8}$	$0.0{\scriptstyle\pm 0.0}$
antsoccer-arena-navigate-singletask-task2-v0	$75{\scriptstyle\pm 3}$	$\bm{83{\scriptstyle\pm 12}}$	$66.8{\scriptstyle\pm 15.3}$	$40.4{\scriptstyle\pm 33.6}$	$35.2{\scriptstyle\pm 20.4}$
antsoccer-arena-navigate-singletask-task4-v0	$16{\scriptstyle\pm 9}$	$16{\scriptstyle\pm 8}$	$13.6{\scriptstyle\pm 9.6}$	$\bm{20.4{\scriptstyle\pm 20.8}}$	$6.4{\scriptstyle\pm 7.3}$
cube-single-play-singletask-task2-v0	$73{\scriptstyle\pm 3}$	$\bm{100{\scriptstyle\pm 1}}$	$90.0{\scriptstyle\pm 7.0}$	$27.2{\scriptstyle\pm 33.5}$	$94.0{\scriptstyle\pm 2.8}$
cube-single-play-singletask-task4-v0	$79{\scriptstyle\pm 6}$	$88{\scriptstyle\pm 16}$	$\bm{93.6{\scriptstyle\pm 2.3}}$	$26.8{\scriptstyle\pm 34.2}$	$92.0{\scriptstyle\pm 4.0}$

Appendix

Table 9: Performance on OGBench environments. Results are reported as mean

\pm

standard deviation. Bold indicates the highest mean in each environment.

Metric	Ours	AK	Additional overhead
Computational cost (per update)	2.498 GFLOPs	4.978 GFLOPs	+2.480 GFLOPs (+99.3%)
GPU memory	1.141 GiB	1.266 GiB	+0.125 GiB (+11.0%)

Appendix

Table 10: Computational cost and GPU memory usage of Ours and AK. Additional overhead is measured relative to Ours.

Hyperparameter	Value
Learning rate	$3\times 10^{-4}$
Optimizer	Adam [ 24 ]
Gradient steps	$1{,}000{,}000$
Minibatch size	$256$
MLP dimensions	$[512,512,512,512]$ (default); $[256,256,256,256]$ (IQL)
Nonlinearity	GELU [ 18 ]

Appendix

Table 11: Shared hyperparameters used across all four offline RL algorithms.

Explore similar work

Mar 5, 2026cs.RO

Latent Policy Steering through One-Step Flow Policies

Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration. Yet, offline RL's performance often hinges on a brittle trade-off between (1) return maximization, which can push policies outside the dataset support, and (2) behavioral constraints, which typically require sensitive hyperparameter tuning. Latent steering offers a structural way to stay within the dataset support during RL, but existing offline adaptations commonly approximate action values using latent-space critics learned via indirect distillation, which can lose information and hinder convergence. We propose Latent Policy Steering (LPS), which enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor. By eliminating proxy latent critics, LPS allows an original-action-space critic to guide end-to-end latent-space optimization, while the one-step MeanFlow policy serves as a behavior-constrained generative prior. This decoupling yields a robust method that works out-of-the-box with minimal tuning. Across OGBench and real-world robotic tasks, LPS achieves state-of-the-art performance and consistently outperforms behavioral cloning and strong latent steering baselines.

Hokyun Im, Andrey Kolobov, Jianlong Fu +1

Department of Artificial Intelligence, Yonsei University · Microsoft Research

RL for Robotics Latent Action Learning Offline RL

Ask PDF

Sep 28, 2026cs.RO

FailPatch: Failure Residual Patching for Vision-Language-Action Models

Vision-Language-Action (VLA) policies are typically adapted using successful demonstrations, which provide direct action supervision but rarely cover failure-prone states. Deployment failures expose these states, yet lack the corrective actions needed for conventional supervised learning. We propose FailPatch, a failure-driven residual patching framework that decouples action supervision from execution-reliability supervision. Successful demonstrations ground how the policy should act, while deployment trajectories indicate when its behavior becomes unreliable. We further observe that action hidden representations exhibit clear linear separability between reliable and failure-associated states while directly conditioning action generation. Building on these insights, FailPatch introduces a Null-gated Residual Expert Bank into the action hidden space of a frozen VLA policy. A unified Preserve--Redirect--Trust objective retains the original policy in reliable states, selects residual experts in failure-associated states and redirects representations from failure regions toward success-associated regions under bounded intervention. With only 0.52% trainable parameters, FailPatch improves success rates by 11.0 percentage points on four long-horizon RoboTwin tasks under clean evaluation, 9.5 percentage points under clean-to-random generalization, and 16.7 percentage points over the baseline across three real-world tasks. Project and code: https://github.com/yupeng-2003/FailPatch.

Peng Yu, Jiacheng Wang, Ziheng Zhang +6

Xi’an Jiaotong University · Dexmal · Nanjing University

Robot Policy Learning Robot Policy Adaptation Efficient VLA Models Vision-Language-Action Models Residual Policy Learning Policy Learning

Ask PDF

Sep 28, 2026cs.RO

Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control

Reward shaping is fundamental to modern robotic control with deep reinforcement learning (RL), yet practitioners still rely heavily on heuristic principles borrowed from classical optimal control and trajectory optimization. Existing methods rarely distinguish reward terms that are intrinsic to the control objective from numerical regularizers, leading to brittle hyperparameter tuning. To determine which quantities a reward must contain, we study the stabilization control problem with a focus on zeroth-order (configuration) and first-order (velocity) information. We theoretically and empirically demonstrate that policy gradient methods can successfully solve stabilization tasks without first-order reward terms, adding such terms can instead introduce severe sensitivity as their scale grows. Conversely, our findings confirm that reward functions must be zeroth-order complete over goal-relevant coordinates, while the first-order state remains necessary in the policy observation under our low-dissipation assumptions. Overall, these results provide actionable and principled guidance for reward design in robotic RL.

Yisheng Zhang, Tao Wang, Sicun Gao

University of California San Diego, La Jolla, CA, USA · Tsinghua University, Beijing, China

Reward Shaping Robotic RL Feedback Control Policy Gradient Methods Reward Design for RL

Ask PDF