Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control

Yuxuan Gao; Yedong Shen; Shiqi Zhang; Wenhao Yu; Yifan Duan; Jia pan; Jiajia Wu; Jiajun Deng

cs.ROApr 4, 2026

Drift-Based Policy Optimization: Native One-Step Policy Learning for Online Robot Control

Authors: Yuxuan Gao, Yedong Shen, Shiqi Zhang, Wenhao Yu, Yifan Duan, Jia pan, Jiajia Wu, Jiajun Deng,

Organizations: University of Science and Technology of China, Hefei 230026, China. · LYNSENSE, Shanghai, China. · iFLYTEK, Hefei 230088, China.

Abstract

Diffusion policies effectively model multimodal action distributions for robotic manipulation, but their iterative denoising requires tens to hundreds of network function evaluations (NFEs) for each control prediction, limiting their applicability to high-frequency closed-loop control and online reinforcement learning (RL). We present a two-stage framework for native one-step generative policies that transfers iterative refinement from inference to training. First, Drift-Based Policy (DBP) uses a fixed-point drifting objective to internalize corrective dynamics into the model parameters, producing multimodal action chunks with a single network evaluation by design. Second, Drift-Based Policy Optimization (DBPO) augments the pretrained backbone with a stochastic interface that provides exact conditional rollout likelihoods for PPO-style on-policy updates while preserving 1-NFE deployment. On the 12-task Diffusion Policy suite, DBP improves the average success rate from 0.79 to 0.83 while reducing inference from 100 NFEs to 1. Across 37 point-cloud manipulation tasks, DBP achieves an average success rate of 88.4%, surpassing the leading 1-NFE baseline OMP at 82.3%. DBPO further improves pretrained one-step policies through stable online fine-tuning on RoboMimic and D4RL. On a physical dual-arm UR5 platform, DBP achieves 123/150 successes (82%) with an average end-to-end latency of 9.5 ms, compared with MP1's 89/150 successes (59%) under the same setup. Code is available at https://github.com/YuxuanGao0822/DBPO.

Figures & tables

Fig. 1 : Generative policy paradigms for robot control. (a) Multi-step diffusion policies rely on iterative denoising at inference. (b) One-step mean-flow policies generate actions in one pass with auxiliary corrections. (c) Drift-Based Policy internalizes attraction-repulsion refinement during training, yielding a native one-step generator. (d) Our method achieves the best average success rate and control frequency on Adroit and MetaWorld against established generative baselines.

Fig. 2 : Two-stage Drift-Based Policy framework. Stage 1 learns a native one-step generator offline via attraction-repulsion refinement during training. Stage 2 fine-tunes a stochastic actor initialized from the pretrained backbone with on-policy PPO and anchor regularization, while deployment remains one-step (1-NFE).

Fig. 3 : Evolution of the internalized drift manifold. The policy action distribution (blue) progressively aligns with expert modes (peach) during training.

Task	Diffusion Policy (100 NFE)	Ours (1 NFE)
Push-T (Image)	0.91	0.89
Push-T (Low-Dim)	0.85	0.87
BlockPush (P1/P2)	0.24	0.43
RoboMimic (Low-Dim)	0.80	0.92
RoboMimic (Image)	0.91	0.87
Kitchen (P1/P2/P3/P4)	1.00	1.00

TABLE I : Comparison between Diffusion Policy and Ours on the Diffusion Policy suite. BlockPush/Kitchen are phase-averaged. Entries report success rate. Best results are in bold .

		Adroit			Meta-World
Method	NFE	Hammer	Door	Pen	Easy (21)	Medium (4)	Hard (4)	Very Hard (5)	Average
DP	10	16.0 $\pm$ 10.0	34.0 $\pm$ 11.0	13.0 $\pm$ 2.0	50.7 $\pm$ 6.1	11.0 $\pm$ 2.5	5.25 $\pm$ 2.5	22.0 $\pm$ 5.0	35.2 $\pm$ 5.3
DP3	10	100.0 $\pm$ 0.0	56.0 $\pm$ 5.0	46.0 $\pm$ 10.0	87.3 $\pm$ 2.2	44.5 $\pm$ 8.7	32.7 $\pm$ 7.7	39.4 $\pm$ 9.0	68.7 $\pm$ 4.7
Simple DP3	10	98.0 $\pm$ 2.0	40.0 $\pm$ 17.0	36.0 $\pm$ 4.0	86.8 $\pm$ 2.3	42.0 $\pm$ 6.5	38.7 $\pm$ 7.5	35.0 $\pm$ 11.6	67.4 $\pm$ 5.0
Adaflow	—	45.0 $\pm$ 11.0	27.0 $\pm$ 6.0	18.0 $\pm$ 6.0	49.4 $\pm$ 6.8	12.0 $\pm$ 5.0	5.75 $\pm$ 4.0	24.0 $\pm$ 4.8	35.6 $\pm$ 6.1
CP	1	45.0 $\pm$ 4.0	31.0 $\pm$ 10.0	13.0 $\pm$ 6.0	69.3 $\pm$ 4.2	21.2 $\pm$ 6.0	17.5 $\pm$ 3.9	30.0 $\pm$ 4.9	50.1 $\pm$ 4.7

TABLE II : Point-cloud imitation comparison on Adroit and Meta-World under the MP1/OMP protocol. Ours refers to Drift-Based Policy. Entries report success rate (%, mean

\pm

std over 3 seeds). Best results are in bold ; second-best are underlined .

Panel of Figure 6 — Fig. 4 : Online PPO fine-tuning results on RoboMimic and D4RL with anchor ablation (DBPO vs. DBPO w/o anchor). Solid bars denote offline initialization, and hatched bars denote gains after fine-tuning. DBPO achieves the strongest post-fine-tuning performance, while removing the anchor reduces gains over pretrained baselines.

Fig. 5 : Real-world bimanual deployment on the physical UR5 testbed. Drift-Based Policy executes precision Lift, Can, and synchronized bimanual Transport using raw trilateral camera inputs.

Task	DBP (Ours)	MP1 (one-step baseline)
Lift (single-arm)	47/50 (94%)	38/50 (76%)
Can (single-arm)	43/50 (86%)	32/50 (64%)
Transport (dual-arm)	33/50 (66%)	19/50 (38%)
Overall	123/150 (82%)	89/150 (59%)

TABLE III : Additional real-robot dual-arm deployment results on the physical UR5 setup (50 trials per task).

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration	Mean Score	Std Score
single_T0p02 ( $\mathcal{R}=\{0.02\}$ )	0.758	0.031
single_T0p05 ( $\mathcal{R}=\{0.05\}$ )	0.864	0.038
single_T0p20 ( $\mathcal{R}=\{0.2\}$ )	0.873	0.012
multi_default ( $\mathcal{R}=\{0.02,0.05,0.2\}$ )	0.858	0.057
multi_wide ( $\mathcal{R}=\{0.01,0.05,0.2,0.5\}$ )	0.849	0.007
multi_dense ( $\mathcal{R}=\{0.01,0.02,0.05,0.1,0.2,0.4\}$ )	0.817	0.030

Appendix

TABLE IV : Temperature configurations and performance results. Bold indicates the best-performing configuration.

Fig. 10 : Test performance across temperature configurations (mean

\pm

std over 3 seeds, 50 rollouts each). Single temperature

T=0.2

achieves the best performance with the lowest variance.

Mode	Mean Score	Std Score
Chunk mode ( $S=Hd_{a}$ , per_timestep_loss=False )	0.890	0.015
Step-wise mode ( $S=d_{a}$ , per_timestep_loss=True )	0.845	0.021

Appendix

TABLE V : Loss computation mode comparison. Chunk mode achieves

5.3\%

higher performance with lower cross-seed variance.

Method	Training Epochs	Inference NFE
Diffusion Policy	4,500	100
Drift-Based Policy (Ours)	100	1

Appendix

TABLE VI : Training and inference efficiency comparison on the PushT low-dimensional task.

Drift-Field Method
Hyperparameter	Value	Tuned?	Tuning Range
Hypothesis count $G$	4	Yes	$\{1,2,4,8,16\}$
Temperature set $\mathcal{R}$	$\{0.2\}$	Yes	Single: $\{0.02,0.05,0.2\}$
			Multi: various combinations
Negative reference count $C_{n}$	0	No	–
Positive reference count $C_{p}$	1	No	–

Appendix

TABLE VII : DBP hyperparameters. All values are held constant across experiments unless explicitly varied in the corresponding analysis.

Generator (Conditional U-Net 1D)
Component	Configuration
Latent dimension	256
Down-sampling channels	[512, 1024, 2048]
Up-sampling channels	[2048, 1024, 512]
Kernel size	5
Normalization	GroupNorm (8 groups)

Appendix

TABLE VIII : Network architecture hyperparameters. The generator uses a conditional U-Net architecture with 1D convolutions.

Fig. 11 : Drift-Based Policy execution on Adroit and Meta-World tasks (Part 1). Frames show temporally sampled states from successful rollouts under 1-NFE inference, demonstrating smooth manipulation and precise object control.

Fig. 12 : Drift-Based Policy execution on Adroit and Meta-World tasks (Part 2). The temporal sequences demonstrate stable execution and coherent action progression without iterative correction at inference time.

Diffusion Policy

Diffusion policies generate multimodal action sequences by iteratively denoising noise, enabling imitation, planning, and control in robotics.

See all 202 papers

One-Step Flow Policy: Self-Distillation for Fast Visuomotor Policies
Mar 12, 2026Shaolong Li, Lichao Sun, Yongchao ChenVisuomotor Policy Learning One-Step Generative Modeling
CSE Department, University of Michigan · Lehigh University · College of AI, Tsinghua University
Ada3Drift: Adaptive Training-Time Drifting for One-Step 3D Visuomotor Robotic Manipulation
Mar 12, 2026Chongyang Xu, Yixian Zou, Tianyu Yang +4Robot Policy Learning Robot Skill Learning
College of Computer Science, Sichuan University, Chengdu, China · School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu, China
OGPO: One-Step Generative Policy Optimization for Real-Time Robot Control
Jan 28, 2026Guowei Zou, Haitao Wang, Hejun Wu +3Robotic Control Real-Time Robot Control
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China · Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, China

Mar 12, 2026J/K move · Enter open · S save

One-Step Flow Policy: Self-Distillation for Fast Visuomotor Policies

Shaolong Li, Lichao Sun, Yongchao Chen

CSE Department, University of Michigan · Lehigh University · College of AI, Tsinghua University

Visuomotor Policy Learning One-Step Generative Modeling Flow-Based Generative Modeling Robotic Manipulation Self-Distillation

Open paper AskPDF arXiv

Generative flow and diffusion models provide the continuous, multimodal action distributions needed for high-precision robotic policies. However, their reliance on iterative sampling introduces severe inference latency, degrading control frequency and harming performance in time-sensitive manipulation. To address this problem, we propose the One-Step Flow Policy (OFP), a from-scratch self-distillation framework for high-fidelity, single-step action generation without a pre-trained teacher. OFP unifies a self-consistency loss to enforce coherent transport across time intervals, and a self-guided regularization to sharpen predictions toward high-density expert modes. In addition, a warm-start mechanism leverages temporal action correlations to minimize the generative transport distance. Evaluations across 56 diverse simulated manipulation tasks demonstrate that a one-step OFP achieves state-of-the-art results, outperforming 100-step diffusion and flow policies while accelerating action generation by over

100\times

. We further integrate OFP into the

π_{0.5}

model on RoboTwin 2.0, where one-step OFP surpasses the original 10-step policy. These results establish OFP as a practical, scalable solution for highly accurate and low-latency robot control.

Abstract

Diffusion Policy

Explore similar work

One-Step Flow Policy: Self-Distillation for Fast Visuomotor Policies

Ada3Drift: Adaptive Training-Time Drifting for One-Step 3D Visuomotor Robotic Manipulation

OGPO: One-Step Generative Policy Optimization for Real-Time Robot Control