Higher-Order Action Supervision Makes A Strong Policy Class

Peng Cheng; Yunxian Hou; Zhi Zhou; Qian Zhang; Chang Huang; Xianyuan Zhan

cs.ROOct 8, 2026

Higher-Order Action Supervision Makes A Strong Policy Class

Authors: Peng Cheng, Yunxian Hou, Zhi Zhou, Qian Zhang, Chang Huang, Xianyuan Zhan

Organizations: Institute for AI Industry Research (AIR), Tsinghua University · Zenfex AI Lab · University of Electronic Science and Technology of China · Horizon Robotics

Abstract

Modern data-driven decision-making methods, such as imitation learning (IL) and reinforcement learning (RL), have achieved great success in solving many complex tasks. However, these methods often suffer from serious control instability and robustness issues when applied in real-world applications such as robotics and autonomous driving, posing notable challenges for their practical deployment. We argue that this instability issue stems largely from their limitations in solely supervising and optimizing zeroth-order actions (i.e., the action labels), failing to account for higher-order action dynamics and temporal consistency. In this paper, we show that simultaneously supervising both zeroth- and first-order actions can dramatically enhance policies' performance and control robustness. To achieve this, we introduce a novel and elegant loss scheme supported by formal theoretical guarantees that can equip any off-the-shelf policy model (e.g., deterministic, stochastic, or flow policies) with the capability for higher-order action supervision, without requiring any structural modifications. Moreover, our proposed method can serve as a lightweight plug-and-play module that seamlessly integrates with a broad spectrum of existing offline RL frameworks. Extensive evaluations on OGBench and D4RL demonstrate that our approach yields substantial performance and robustness improvements across a wide range of continuous control environments. Notably, our method can also enhance policies' out-of-distribution (OOD) generalization capability in the challenging low-data regime, making it an ideal tool in tackling many real-world control problems.

Figures & tables

Figure 1: Trajectories and zeroth-/first-order actions of policies learned on the maze2d task with 20 successful trajectories and 500k training steps. Green arrows denote the zeroth-order action $a$ (joint torque), while red arrows denote the first-order action $\dot{a}$ (time derivative of joint torque, computed as $a^{\prime}-a$ ). (a) Standard behavior cloning (BC) without $\dot{a}$ supervision, which yields a suboptimal trajectory with highly noisy and temporally inconsistent first-order actions. (b) Our method with higher-order action supervision, which yields an optimal navigation trajectory with much smoother and more stable zeroth- and first-order actions.

	Det. Policy		Sto. Policy (mean sup.)		Sto. Policy (standard)		Flow Policy
Task Category (Avg. of 5 tasks)	TD3+BC	TD3+BC+ours	ReBRAC	ReBRAC+ours	IQL	IQL+ours	IFQL	IFQL+ours
antmaze-large-navigate	$83{\scriptstyle\pm 4}$	$98{\scriptstyle\pm 3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!15})$	$81{\scriptstyle\pm 5}$	$98{\scriptstyle\pm 2}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!17})$	$53{\scriptstyle\pm 3}$	$68{\scriptstyle\pm 5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!15})$	$28{\scriptstyle\pm 13}$	$36{\scriptstyle\pm 14}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!8})$
antmaze-giant-navigate	$42{\scriptstyle\pm 8}$	$65{\scriptstyle\pm 10}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!23})$	$26{\scriptstyle\pm 8}$	$61{\scriptstyle\pm 12}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!35})$	$4{\scriptstyle\pm 1}$	$9{\scriptstyle\pm 1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!5})$	$3{\scriptstyle\pm 2}$	$4{\scriptstyle\pm 3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1})$
humanoidmaze-medium-navigate	$34{\scriptstyle\pm 8}$	$42{\scriptstyle\pm 5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!8})$	$22{\scriptstyle\pm 8}$	$55{\scriptstyle\pm 10}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!33})$	$33{\scriptstyle\pm 2}$	$36{\scriptstyle\pm 3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!3})$	$60{\scriptstyle\pm 16}$	$64{\scriptstyle\pm 13}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!4})$
humanoidmaze-large-navigate	$2{\scriptstyle\pm 1}$	$4{\scriptstyle\pm 1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!2})$	$2{\scriptstyle\pm 1}$	$4{\scriptstyle\pm 2}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!2})$	$2{\scriptstyle\pm 1}$	$3{\scriptstyle\pm 1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1})$	$11{\scriptstyle\pm 2}$	$12{\scriptstyle\pm 3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1})$
antsoccer-arena-navigate	$13{\scriptstyle\pm 3}$	$16{\scriptstyle\pm 3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!3})$	$0{\scriptstyle\pm 0}$	$16{\scriptstyle\pm 4}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!16})$	$8{\scriptstyle\pm 2}$	$23{\scriptstyle\pm 3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!15})$	$33{\scriptstyle\pm 6}$	$40{\scriptstyle\pm 13}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!7})$

Table 1: Average normalized scores on 10 OGBench task categories, each aggregating 5 sub-tasks over 5 random seeds. Values after

\pm

denote standard deviations. The

{\color[rgb]{0.0195,0.6523,0.082}\uparrow}

marks indicate improvements greater than

1.0

, whereas

{\color[rgb]{0.5,0.5,0.5}\uparrow}

denotes minor improvements. Full per-task results are presented in Table 4 .

Figure 2: Comparison of vanilla deterministic BC and our $1^{\mathrm{st}}$ -order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation $||\dot{a}||=||a^{\prime}-a||$ ( $a$ is joint torque) are aggregated from 10 trajectories.

Figure 3: Comparison of TD3+BC and our $1^{\mathrm{st}}$ -order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation $||\dot{a}||=||a^{\prime}-a||$ ( $a$ is joint torque) are aggregated from 10 trajectories.

Task	TD3+BC	+CAPS	+L2C2	+LipsNet	+Extra $\dot{a}$ head	TD3+BC+Ours
Hopper-m	$40.1{\scriptstyle\pm 18.6}$	$42.6{\scriptstyle\pm 5.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!2.5})$	$42.7{\scriptstyle\pm 7.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!2.6})$	$43.2{\scriptstyle\pm 3.6}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!3.1})$	$41.5{\scriptstyle\pm 7.8}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1.4})$	$44.1{\scriptstyle\pm 12.7}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!\textbf{4.0}})$
Hopper-me	$17.8{\scriptstyle\pm 7.9}$	$39.6{\scriptstyle\pm 8.2}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!21.8})$	$39.8{\scriptstyle\pm 8.3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!22.0})$	$32.8{\scriptstyle\pm 18.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!15.0})$	$26.6{\scriptstyle\pm 8.6}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!8.8})$	$42.3{\scriptstyle\pm 10.8}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!\textbf{24.5}})$
Hopper-e	$23.2{\scriptstyle\pm 18.2}$	$57.1{\scriptstyle\pm 29.7}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!33.9})$	$55.2{\scriptstyle\pm 34.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!32.0})$	$16.9{\scriptstyle\pm 2.8}({\color[rgb]{1,0,0}\downarrow\!6.3})$	$30.6{\scriptstyle\pm 15.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!7.4})$	$93.5{\scriptstyle\pm 13.6}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!\textbf{70.3}})$
HalfCheetah-m	$16.4{\scriptstyle\pm 10.2}$	$29.6{\scriptstyle\pm 5.6}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!13.2})$	$30.3{\scriptstyle\pm 7.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!13.9})$	$6.6{\scriptstyle\pm 3.0}({\color[rgb]{1,0,0}\downarrow\!9.8})$	$28.1{\scriptstyle\pm 4.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!11.7})$	$31.5{\scriptstyle\pm 2.4}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!\textbf{15.1}})$
HalfCheetah-me	$15.4{\scriptstyle\pm 10.7}$	$19.8{\scriptstyle\pm 3.4}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!4.4})$	$28.6{\scriptstyle\pm 3.8}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!13.2})$	$3.3{\scriptstyle\pm 2.2}({\color[rgb]{1,0,0}\downarrow\!12.1})$	$20.6{\scriptstyle\pm 3.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!5.2})$	$35.6{\scriptstyle\pm 2.4}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!\textbf{20.2}})$
HalfCheetah-e	$1.7{\scriptstyle\pm 3.3}$	$1.7{\scriptstyle\pm 1.6}$	$2.4{\scriptstyle\pm 1.9}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.7})$	$0.8{\scriptstyle\pm 0.6}({\color[rgb]{1,0,0}\downarrow\!0.9})$	$0.2{\scriptstyle\pm 1.8}({\color[rgb]{1,0,0}\downarrow\!1.5})$	$3.9{\scriptstyle\pm 2.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!\textbf{2.2}})$

Table 3: Comparison with action-smoothness approaches and explicit first-order action prediction on D4RL MuJoCo locomotion tasks with reduced-size (10k) datasets. All methods use TD3+BC as the shared backbone. Arrows show score differences relative to TD3+BC on the same task:

{\color[rgb]{0.0195,0.6523,0.082}\uparrow}

indicates a gain of at least

1.0

{\color[rgb]{0.5,0.5,0.5}\uparrow}

a smaller positive gain, and

{\color[rgb]{1,0,0}\downarrow}

a decrease. No arrow indicates no change.

Figure 4: Ablation study on value regularization weight $\lambda_{Q}$ and $1^{\text{st}}$ -order action supervision weight $\lambda_{\dot{a}}$ of IFQL.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Task Category	TD3+BC	TD3+BC+ours	ReBRAC	ReBRAC+ours	IQL	IQL+ours	IFQL	IFQL+ours
	Det. Policy		Sto. Policy (mean sup.)		Sto. Policy (standard)		Flow Policy
antmaze-large-navigate-singletask-task1-v0 ( $\ast$ )	$92{\scriptstyle\pm 8}$	$99{\scriptstyle\pm 2}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!7})$	$91{\scriptstyle\pm 10}$	$100{\scriptstyle\pm 1}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!9})$	$48{\scriptstyle\pm 9}$	$80{\scriptstyle\pm 10}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!32})$	$24{\scriptstyle\pm 17}$	$56{\scriptstyle\pm 14}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!32})$
antmaze-large-navigate-singletask-task2-v0	$85{\scriptstyle\pm 3}$	$91{\scriptstyle\pm 4}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!6})$	$88{\scriptstyle\pm 4}$	$92{\scriptstyle\pm 4}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!4})$	$42{\scriptstyle\pm 6}$	$42{\scriptstyle\pm 15}$	$8{\scriptstyle\pm 3}$	$16{\scriptstyle\pm 11}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!8})$
antmaze-large-navigate-singletask-task3-v0	$72{\scriptstyle\pm 10}$	$100{\scriptstyle\pm 0}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!28})$	$51{\scriptstyle\pm 18}$	$100{\scriptstyle\pm 1}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!49})$	$72{\scriptstyle\pm 7}$	$90{\scriptstyle\pm 6}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!18})$	$52{\scriptstyle\pm 17}$	$72{\scriptstyle\pm 10}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!20})$
antmaze-large-navigate-singletask-task4-v0	$85{\scriptstyle\pm 5}$	$99{\scriptstyle\pm 1}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!14})$	$84{\scriptstyle\pm 7}$	$100{\scriptstyle\pm 1}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!16})$	$51{\scriptstyle\pm 9}$	$52{\scriptstyle\pm 22}\,({\color[rgb]{0.5,0.5,0.5}\uparrow\!1})$	$18{\scriptstyle\pm 8}$	$40{\scriptstyle\pm 17}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!22})$
antmaze-large-navigate-singletask-task5-v0	$82{\scriptstyle\pm 10}$	$99{\scriptstyle\pm 0}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!17})$	$90{\scriptstyle\pm 2}$	$99{\scriptstyle\pm 1}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!9})$	$54{\scriptstyle\pm 22}$	$78{\scriptstyle\pm 9}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!24})$	$38{\scriptstyle\pm 18}$	$48{\scriptstyle\pm 19}\,({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!10})$

Appendix

Table 4: Average normalized scores on 50 OGBench tasks over 5 random seeds. Values after

\pm

denote standard deviations. The

{\color[rgb]{0.0195,0.6523,0.082}\uparrow}

marks indicate improvements greater than

1.0

, whereas

{\color[rgb]{0.5,0.5,0.5}\uparrow}

denotes minor improvements.

	Det. Policy		Sto. Policy (mean sup.)		Sto. Policy (standard)		Flow Policy
Task Category	TD3+BC	TD3+BC+ours	ReBRAC	ReBRAC+ours	IQL	IQL+ours	IFQL	IFQL+ours
Hopper-m	$55.5{\scriptstyle\pm 7.7}$	$58.9{\scriptstyle\pm 7.4}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!3.4})$	$84.9{\scriptstyle\pm 2.6}$	$85.7{\scriptstyle\pm 1.3}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.8})$	$67.3{\scriptstyle\pm 5.7}$	$68.8{\scriptstyle\pm 5.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1.5})$	$89.5{\scriptstyle\pm 5.7}$	$94.2{\scriptstyle\pm 7.2}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!4.7})$
Hopper-mr	$57.9{\scriptstyle\pm 17.6}$	$58.4{\scriptstyle\pm 9.2}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.5})$	$81.2{\scriptstyle\pm 25.1}$	$94.3{\scriptstyle\pm 10.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!13.1})$	$91.7{\scriptstyle\pm 8.6}$	$94.2{\scriptstyle\pm 5.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!2.5})$	$96.2{\scriptstyle\pm 4.7}$	$96.3{\scriptstyle\pm 2.3}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.1})$
Hopper-me	$89.8{\scriptstyle\pm 15.2}$	$89.9{\scriptstyle\pm 8.4}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.1})$	$106.4{\scriptstyle\pm 10.7}$	$106.6{\scriptstyle\pm 5.4}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.2})$	$91.5{\scriptstyle\pm 14.3}$	$109.2{\scriptstyle\pm 9.8}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!17.7})$	$77.1{\scriptstyle\pm 25.5}$	$89.8{\scriptstyle\pm 8.7}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!12.7})$
Hopper-e	$108.3{\scriptstyle\pm 6.7}$	$110.5{\scriptstyle\pm 0.2}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!2.2})$	$108.5{\scriptstyle\pm 4.1}$	$108.6{\scriptstyle\pm 3.8}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.1})$	$99.3{\scriptstyle\pm 9.2}$	$106.5{\scriptstyle\pm 5.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!7.2})$	$96.8{\scriptstyle\pm 12.2}$	$106.2{\scriptstyle\pm 8.6}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!9.4})$
Halfcheetah-m	$47.8{\scriptstyle\pm 0.4}$	$47.9{\scriptstyle\pm 0.4}({\color[rgb]{0.5,0.5,0.5}\uparrow\!0.1})$	$55.4{\scriptstyle\pm 1.0}$	$56.9{\scriptstyle\pm 1.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1.5})$	$47.4{\scriptstyle\pm 0.4}$	$48.6{\scriptstyle\pm 0.2}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1.2})$	$51.1{\scriptstyle\pm 0.6}$	$52.8{\scriptstyle\pm 0.3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1.7})$

Appendix

Table 5: Average normalized scores on D4RL MuJoCo locomotion tasks under full-datasize. Standard deviations are reported after the

\pm

sign. The

{\color[rgb]{0.0195,0.6523,0.082}\uparrow}

symbol indicates a performance increase greater than

1.0

, whereas

{\color[rgb]{0.5,0.5,0.5}\uparrow}

denotes minor improvements. All results and standard deviations (

\pm

) are averaged across 5 random seeds.

	Det. Policy		Sto. Policy (standard)		Flow Policy
Task Category	base	base + $\mathcal{L}_{\dot{a}}^{\text{det}}$	base	base + $\mathcal{L}_{\dot{a}}^{\text{sto}}$	base	base + $\mathcal{L}_{\dot{a}}^{\text{flow}}$
Hopper-e	$21.5{\scriptstyle\pm 13.5}$	$67.3{\scriptstyle\pm 14.3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!45.8})$	$57.5{\scriptstyle\pm 15.2}$	$93.1{\scriptstyle\pm 10.7}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!35.6})$	$72.2{\scriptstyle\pm 9.2}$	$97.2{\scriptstyle\pm 12.3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!25.0})$
Halfcheetah-e	$2.4{\scriptstyle\pm 1.2}$	$5.7{\scriptstyle\pm 1.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!3.3})$	$2.8{\scriptstyle\pm 1.6}$	$16.2{\scriptstyle\pm 10.1}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!13.4})$	$11.4{\scriptstyle\pm 4.5}$	$17.9{\scriptstyle\pm 6.9}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!6.5})$
Walker2d-e	$26.7{\scriptstyle\pm 12.2}$	$91.1{\scriptstyle\pm 14.7}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!64.4})$	$46.4{\scriptstyle\pm 15.2}$	$91.6{\scriptstyle\pm 9.5}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!45.2})$	$89.8{\scriptstyle\pm 14.6}$	$91.5{\scriptstyle\pm 6.3}({\color[rgb]{0.0195,0.6523,0.082}\uparrow\!1.7})$

Appendix

Table 6: Imitation learning performance on D4RL MuJoCo locomotion tasks with 10k datasets. Average normalized scores across deterministic, stochastic, and flow-based policy paradigms. The

{\color[rgb]{0.0195,0.6523,0.082}\uparrow}

indicates a performance increase

>1.0

, whereas

{\color[rgb]{0.5,0.5,0.5}\uparrow}

represents marginal gains. All results and standard deviations (

\pm

) are averaged across 5 random seeds.

Figure 5: Comparison of vanilla stochastic BC and our $1^{\mathrm{st}}$ -order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation $||\dot{a}||=||a^{\prime}-a||$ ( $a$ is joint torque) are aggregated from 10 trajectories.

Figure 6: Comparison of vanilla IQL and our $1^{\mathrm{st}}$ -order action supervision on small-sample locomotion tasks. Scores are averaged over 5 random seeds. The policy rollout distributions of action variation $||\dot{a}||=||a^{\prime}-a||$ ( $a$ is joint torque) are aggregated from 10 trajectories.

	Det. Policy		Sto. Policy (mean sup.)		Sto. Policy (standard)		Flow Policy
Task Category	TD3+BC	TD3+BC+ours	ReBRAC	ReBRAC+ours	IQL	IQL+ours	IFQL	IFQL+ours
Training time (D4RL)	28 min 30 s	33 min 6 s	25 min 8 s	30 min 34 s	34 min 44 s	39 min 56 s	30 min 49 s	35 min 52 s
Training time (OGBench)	28 min 23 s	38 min 56 s	25 min 11 s	31 min 54 s	35 min 7 s	42 min 5 s	32 min 0 s	36 min 51 s

Appendix

Table 7: Training time comparison between each base algorithm and its variant equipped with our first-order action supervision. All runs use the same number of training steps and are conducted on a single GPU.

Environment	Method	100k	300k	500k	700k	1M
Hopper-v2	SAC	$1767.1{\scriptstyle\pm 1005.9}$	$2399.1{\scriptstyle\pm 1391.8}$	$3076.8{\scriptstyle\pm 405.9}$	$\bm{3389.6{\scriptstyle\pm 162.2}}$	$3530.4{\scriptstyle\pm 32.6}$
	SAC + Ours	$\bm{2375.8{\scriptstyle\pm 1036.4}}$	$\bm{3273.9{\scriptstyle\pm 111.6}}$	$\bm{3226.4{\scriptstyle\pm 101.1}}$	$3359.2{\scriptstyle\pm 116.4}$	$\bm{3531.8{\scriptstyle\pm 33.2}}$
HalfCheetah-v2	SAC	$5659.2{\scriptstyle\pm 676.4}$	$8564.9{\scriptstyle\pm 1699.7}$	$9750.7{\scriptstyle\pm 1228.6}$	$10805.1{\scriptstyle\pm 804.7}$	$11557.4{\scriptstyle\pm 493.4}$
	SAC + Ours	$\bm{5790.9{\scriptstyle\pm 553.5}}$	$\bm{8614.4{\scriptstyle\pm 1423.9}}$	$\bm{9954.8{\scriptstyle\pm 1126.3}}$	$\bm{10893.5{\scriptstyle\pm 612.1}}$	$\bm{11858.2{\scriptstyle\pm 414.1}}$
Walker2d-v2	SAC	$702.9{\scriptstyle\pm 343.5}$	$2413.7{\scriptstyle\pm 1150.3}$	$2989.8{\scriptstyle\pm 1820.9}$	$4291.2{\scriptstyle\pm 950.5}$	$4768.9{\scriptstyle\pm 393.1}$
	SAC + Ours	$\bm{869.9{\scriptstyle\pm 310.3}}$	$\bm{3300.3{\scriptstyle\pm 879.6}}$	$\bm{4141.4{\scriptstyle\pm 572.9}}$	$\bm{4805.8{\scriptstyle\pm 184.5}}$	$\bm{4991.1{\scriptstyle\pm 264.5}}$

Appendix

Table 8: Performance snapshots of SAC and SAC+Ours on online MuJoCo benchmarks over 1M training steps. Results are averaged over 5 random seeds, with standard deviations reported after the

\pm

sign. Bold indicates the better result at each snapshot.

Environment (OGBENCH)	IFQL	+Ours	+AK ( $\lambda_{a}=0.5$ )	+AK ( $\lambda_{a}=0.1$ )	+AK ( $\lambda_{a}=1$ )
humanoidmaze-medium-navigate-singletask-task1-v0	$69{\scriptstyle\pm 19}$	$71{\scriptstyle\pm 16}$	$\bm{73.2{\scriptstyle\pm 7.0}}$	$51.2{\scriptstyle\pm 32.4}$	$67.6{\scriptstyle\pm 7.1}$
humanoidmaze-medium-navigate-singletask-task4-v0	$\bm{1{\scriptstyle\pm 1}}$	$\bm{1{\scriptstyle\pm 1}}$	$0.0{\scriptstyle\pm 0.0}$	$0.4{\scriptstyle\pm 0.8}$	$0.0{\scriptstyle\pm 0.0}$
antsoccer-arena-navigate-singletask-task2-v0	$75{\scriptstyle\pm 3}$	$\bm{83{\scriptstyle\pm 12}}$	$66.8{\scriptstyle\pm 15.3}$	$40.4{\scriptstyle\pm 33.6}$	$35.2{\scriptstyle\pm 20.4}$
antsoccer-arena-navigate-singletask-task4-v0	$16{\scriptstyle\pm 9}$	$16{\scriptstyle\pm 8}$	$13.6{\scriptstyle\pm 9.6}$	$\bm{20.4{\scriptstyle\pm 20.8}}$	$6.4{\scriptstyle\pm 7.3}$
cube-single-play-singletask-task2-v0	$73{\scriptstyle\pm 3}$	$\bm{100{\scriptstyle\pm 1}}$	$90.0{\scriptstyle\pm 7.0}$	$27.2{\scriptstyle\pm 33.5}$	$94.0{\scriptstyle\pm 2.8}$
cube-single-play-singletask-task4-v0	$79{\scriptstyle\pm 6}$	$88{\scriptstyle\pm 16}$	$\bm{93.6{\scriptstyle\pm 2.3}}$	$26.8{\scriptstyle\pm 34.2}$	$92.0{\scriptstyle\pm 4.0}$

Appendix

Table 9: Performance on OGBench environments. Results are reported as mean

\pm

standard deviation. Bold indicates the highest mean in each environment.

Metric	Ours	AK	Additional overhead
Computational cost (per update)	2.498 GFLOPs	4.978 GFLOPs	+2.480 GFLOPs (+99.3%)
GPU memory	1.141 GiB	1.266 GiB	+0.125 GiB (+11.0%)

Appendix

Table 10: Computational cost and GPU memory usage of Ours and AK. Additional overhead is measured relative to Ours.

Hyperparameter	Value
Learning rate	$3\times 10^{-4}$
Optimizer	Adam [ 24 ]
Gradient steps	$1{,}000{,}000$
Minibatch size	$256$
MLP dimensions	$[512,512,512,512]$ (default); $[256,256,256,256]$ (IQL)
Nonlinearity	GELU [ 18 ]

Appendix

Table 11: Shared hyperparameters used across all four offline RL algorithms.

Explore similar work

Latent Policy Steering through One-Step Flow Policies
Mar 5, 2026Hokyun Im, Andrey Kolobov, Jianlong Fu +1RL for Robotics Latent Action Learning
Department of Artificial Intelligence, Yonsei University · Microsoft Research
FailPatch: Failure Residual Patching for Vision-Language-Action Models
Sep 28, 2026Peng Yu, Jiacheng Wang, Ziheng Zhang +6Robot Policy Learning Robot Policy Adaptation
Xi’an Jiaotong University · Dexmal · Nanjing University
Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control
Sep 28, 2026Yisheng Zhang, Tao Wang, Sicun GaoReward Shaping Robotic RL
University of California San Diego, La Jolla, CA, USA · Tsinghua University, Beijing, China

Mar 5, 2026, cs.ROJ/K move · Enter open · S save

Latent Policy Steering through One-Step Flow Policies

Hokyun Im, Andrey Kolobov, Jianlong Fu, Youngwoon Lee

Department of Artificial Intelligence, Yonsei University · Microsoft Research

RL for Robotics Latent Action Learning Offline RL

Open paper AskPDF arXiv

Offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration. Yet, offline RL's performance often hinges on a brittle trade-off between (1) return maximization, which can push policies outside the dataset support, and (2) behavioral constraints, which typically require sensitive hyperparameter tuning. Latent steering offers a structural way to stay within the dataset support during RL, but existing offline adaptations commonly approximate action values using latent-space critics learned via indirect distillation, which can lose information and hinder convergence. We propose Latent Policy Steering (LPS), which enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy to update a latent-action-space actor. By eliminating proxy latent critics, LPS allows an original-action-space critic to guide end-to-end latent-space optimization, while the one-step MeanFlow policy serves as a behavior-constrained generative prior. This decoupling yields a robust method that works out-of-the-box with minimal tuning. Across OGBench and real-world robotic tasks, LPS achieves state-of-the-art performance and consistently outperforms behavioral cloning and strong latent steering baselines.

Abstract

Explore similar work

Latent Policy Steering through One-Step Flow Policies

FailPatch: Failure Residual Patching for Vision-Language-Action Models

Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control