F2F-AP: Flow-to-Future Asynchronous Policy for Real-time Dynamic Manipulation

Haoyu Wei; Xiuwei Xu; Ziyang Cheng; Hang Yin; Angyuan Ma; Jie Zhou; Jiwen Lu; Bingyao Yu

cs.ROApr 2, 2026

F2F-AP: Flow-to-Future Asynchronous Policy for Real-time Dynamic Manipulation

Authors: Haoyu Wei, Xiuwei Xu, Ziyang Cheng, Hang Yin, Angyuan Ma, Jie Zhou, Jiwen Lu, Bingyao Yu

Organizations: Department of Automation, Tsinghua University

Abstract

Asynchronous inference has emerged as a prevalent paradigm in robotic manipulation, achieving significant progress in ensuring trajectory smoothness and efficiency. However, a systemic challenge remains unresolved, as inherent latency causes generated actions to inevitably lag behind the real-time environment. This issue is particularly exacerbated in dynamic scenarios, where such temporal misalignment severely compromises the policy's ability to interpret and react to rapidly evolving surroundings. In this paper, we propose a novel framework that leverages predicted object flow to synthesize future observations, incorporating a flow-based contrastive learning objective to align the visual feature representations of predicted observations with ground-truth future states. Empowered by this anticipated visual context, our asynchronous policy gains the capacity for proactive planning and motion, enabling it to explicitly compensate for latency and robustly execute manipulation tasks involving actively moving objects. Experimental results demonstrate that our approach significantly enhances responsiveness and success rates in complex dynamic manipulation tasks.

Figures & tables

Figure 1: Flow-to-Future Asynchronous Policy (F2F-AP). F2F-AP is an asynchronous robot manipulation policy incorporating future observations. By predicting object flow, it enhances the model’s understanding of the motion trends of interacting objects. F2F-AP is transferable to diverse embodiments and yields significant performance improvements in dynamic tasks.

Figure 2: Decomposition of systemic latency. The total latency ( $\Delta_{o}+\Delta_{i}+\Delta_{c}$ ) results in a misalignment where the initial frames of the predicted trajectory lag behind the real-world state, invalidating them for execution.

Figure 3: Analysis of Asynchronous Inference. Gray and black denote unexecuted and executed actions, respectively. Left : The orange line shows the trajectory gap. Middle : Policy input $o_{2}$ is misaligned with $o_{4}$ at execution time. Right (Ours): Uses predicted, aligned observation $\hat{\mathbf{o}_{4}}$ .

Figure 4: (Left) The model plans from a future state $s_{t_{3}}$ toward the interacting object’s anticipated position $t_{6}$ at timestamp $t_{1}$ , enabling advance motion despite systemic latency. (Middle) Given robot states and multi-frame RGB images, a Flow Predictor extracts object flow to synthesize future observations. The Policy then processes these to generate action chunks. (Right) Contrastive learning minimizes the feature distance between predicted and real future observations. The $\bigstar$ indicates features sharing the same encoder.

Figure 5: Experiment Visualization. (Top) Visualization of the real-world dynamic experiments. (Bottom) The non-linear stochastic trajectories, along with the heatmaps and predicted flow from the robot’s first-person perspective.

Method	Hammer		Banana		Carrot		Pull-Back Car		Chip Can		Bottle		Length Meas.
	SR	ET	SR	ET	SR	ET	SR	ET	SR	ET	SR	ET	SR	ET
Syn. UMI	5/20	5.7s	5/20	3.0s	3/10	2.8s	0/20	-	3/20	4.1s	2/20	9.2s	4/10	2.6s
Asyn. UMI	7/20	6.2s	10/20	2.9s	5/10	2.4s	1/20	1.4s	2/20	4.2s	2/20	9.1s	5/10	2.4s
RTC	12/20	4.1s	12/20	2.4s	6/20	2.0s	0/20	-	3/20	3.9s	3/20	8.2s	5/20	2.2s
VLASH	6/20	3.3s	13/20	2.5s	3/10	1.4s	3/20	1.9s	1/20	3.0s	2/20	7.6s	3/10	1.4s
F2F-AP	15/20	3.4s	13/20	2.5s	6/10	1.6s	8/20	2.5s	9/20	3.5s	11/20	7.9s	7/10	1.9s

Table 1: Quantitative Results. Comparison against baselines in real-world experiments. Length Meas. denotes Collaborative Length Measurement.

Method	SR	ET (s)	Method	SR	ET (s)
Fewer Flow Points	10/20	7.8	MF-UMI	1/20	8.7
Diff. Line Color	11/20	7.8	RTC-MF-UMI	0/20	-
Low Contrastive Wt.	7/20	8.5	Low Freq. Inference	3/20	10.2
w/o Contrastive Loss	8/20	8.5	300ms Latency	12/20	7.5
w/o Future State	3/20	8.3	500ms Latency	10/20	8.5
w/o Heatmap	5/20	9.1	600ms Latency	4/20	11.2

Table 2: Ablation Studies. The results reported correspond to the Grasp Rolling Bottle task (Section 4.2 ).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Hardware Platform. F2F-AP is evaluated on a fixed-base arm and a quadruped manipulator equipped with camera and odometry.

Figure 7: The left panel shows the occlusion scenarios, and the right panel illustrates how to build the ground-truth flow under occlusion. We select a pre-segmented gripper instance and overlay it onto the original fisheye videos to generate the RGB ground truth.

Figure 8: Failure Case Visualization illustrates four representative failure scenarios.

Figure 9: Comparative Visualization of baseline failure modes: (1) Naive Asynchronous Inference, and (2) VLASH.

Figure 10: Visualization of Flow Prediction illustrates the pairs of heatmap and images augmented by predicted flow in different tasks.

Figure 11: (Right) The heatmap indicates cosine similarity in the learned contrastive feature space, with brighter values denoting stronger alignment. The white dashed line marks same-time alignment, while the red dashed line indicates the expected temporal offset corresponding to the prediction horizon. The blue dotted line shows the empirical peak similarity for each predicted feature. (Left) Demonstration of how the flow-augmented observations map to the raw future observations, with the robot’s egocentric observations aligned with the corresponding timestamps in the heatmap.

Explore similar work

ABPolicy: Asynchronous B-Spline Flow Policy for Real-Time and Smooth Robotic Manipulation
Feb 27, 2026Fan Yang, Peiguang Jing, Kaihua Qu +2Robot Policy Learning Robot Motion Generation
The authors are with the School of Electrical and Information Engineering, Tianjin University
FutureRTC: Real-Time Robot Execution with Anticipatory-Conditioned Action Chunking
Jul 27, 2026Hai Jiang, Yixian Zou, Binbin Liang +3Vision-Language-Action Models Action Chunking
School of Aeronautics and Astronautics, Sichuan University · University of Electronic Science and Technology of China · University of Alberta
Reactive Real-Time Flow Policies via Asynchronous Distribution Alignment
Sep 29, 2026Moritz Zoellner, Reece O'Mahoney, Ioannis Havoutis +1Robot Policy Learning Efficient VLA Model Inference
Department of Computer Science, Purdue University · Oxford Robotics Institute, University of Oxford

Feb 27, 2026J/K move · Enter open · S save

ABPolicy: Asynchronous B-Spline Flow Policy for Real-Time and Smooth Robotic Manipulation

Fan Yang, Peiguang Jing, Kaihua Qu, Ningyuan Zhao+1

The authors are with the School of Electrical and Information Engineering, Tianjin University

Robot Policy Learning Robot Motion Generation Robotic Manipulation

Open paper AskPDF arXiv

Robotic manipulation requires policies that are smooth and responsive to evolving observations. However, synchronous inference in the raw action space introduces several challenges, including intra-chunk jitter, inter-chunk discontinuities, and stop-and-go execution. These issues undermine a policy's smoothness and its responsiveness to environmental changes. We propose ABPolicy, an asynchronous flow-matching policy that operates in a B-spline control-point action space. First, the B-spline representation ensures intra-chunk smoothness. Second, we introduce bidirectional action prediction coupled with refitting optimization to enforce inter-chunk continuity. Finally, by leveraging asynchronous inference, ABPolicy delivers real-time, continuous updates. We evaluate ABPolicy across seven tasks encompassing both static settings and dynamic settings with moving objects. Empirical results indicate that ABPolicy reduces trajectory jerk, leading to smoother motion and improved performance. Project website: https://teee000.github.io/ABPolicy/.

Abstract

Explore similar work

ABPolicy: Asynchronous B-Spline Flow Policy for Real-Time and Smooth Robotic Manipulation

FutureRTC: Real-Time Robot Execution with Anticipatory-Conditioned Action Chunking

Reactive Real-Time Flow Policies via Asynchronous Distribution Alignment