Asynchronous inference has emerged as a prevalent paradigm in robotic manipulation, achieving significant progress in ensuring trajectory smoothness and efficiency. However, a systemic challenge remains unresolved, as inherent latency causes generated actions to inevitably lag behind the real-time environment. This issue is particularly exacerbated in dynamic scenarios, where such temporal misalignment severely compromises the policy's ability to interpret and react to rapidly evolving surroundings. In this paper, we propose a novel framework that leverages predicted object flow to synthesize future observations, incorporating a flow-based contrastive learning objective to align the visual feature representations of predicted observations with ground-truth future states. Empowered by this anticipated visual context, our asynchronous policy gains the capacity for proactive planning and motion, enabling it to explicitly compensate for latency and robustly execute manipulation tasks involving actively moving objects. Experimental results demonstrate that our approach significantly enhances responsiveness and success rates in complex dynamic manipulation tasks.
Figures & tables
Figure 1: Flow-to-Future Asynchronous Policy (F2F-AP). F2F-AP is an asynchronous robot manipulation policy incorporating future observations. By predicting object flow, it enhances the model’s understanding of the motion trends of interacting objects. F2F-AP is transferable to diverse embodiments and yields significant performance improvements in dynamic tasks.
Figure 2: Decomposition of systemic latency. The total latency ( Δo+Δi+Δc ) results in a misalignment where the initial frames of the predicted trajectory lag behind the real-world state, invalidating them for execution.
Figure 3: Analysis of Asynchronous Inference. Gray and black denote unexecuted and executed actions, respectively. Left : The orange line shows the trajectory gap. Middle : Policy input o2 is misaligned with o4 at execution time. Right (Ours): Uses predicted, aligned observation o4^ .
Figure 4: (Left) The model plans from a future state st3 toward the interacting object’s anticipated position t6 at timestamp t1 , enabling advance motion despite systemic latency. (Middle) Given robot states and multi-frame RGB images, a Flow Predictor extracts object flow to synthesize future observations. The Policy then processes these to generate action chunks. (Right) Contrastive learning minimizes the feature distance between predicted and real future observations. The ★ indicates features sharing the same encoder.
Figure 5: Experiment Visualization. (Top) Visualization of the real-world dynamic experiments. (Bottom) The non-linear stochastic trajectories, along with the heatmaps and predicted flow from the robot’s first-person perspective.
Method
Hammer
Banana
Carrot
Pull-Back Car
Chip Can
Bottle
Length Meas.
SR
ET
SR
ET
SR
ET
SR
ET
SR
ET
SR
ET
SR
ET
Syn. UMI
5/20
5.7s
5/20
3.0s
3/10
2.8s
0/20
-
3/20
4.1s
2/20
9.2s
4/10
2.6s
Asyn. UMI
7/20
6.2s
10/20
2.9s
5/10
2.4s
1/20
1.4s
2/20
4.2s
2/20
9.1s
5/10
2.4s
RTC
12/20
4.1s
12/20
2.4s
6/20
2.0s
0/20
-
3/20
3.9s
3/20
8.2s
5/20
2.2s
VLASH
6/20
3.3s
13/20
2.5s
3/10
1.4s
3/20
1.9s
1/20
3.0s
2/20
7.6s
3/10
1.4s
F2F-AP
15/20
3.4s
13/20
2.5s
6/10
1.6s
8/20
2.5s
9/20
3.5s
11/20
7.9s
7/10
1.9s
Table 1: Quantitative Results. Comparison against baselines in real-world experiments. Length Meas. denotes Collaborative Length Measurement.
Method
SR
ET (s)
Method
SR
ET (s)
Fewer Flow Points
10/20
7.8
MF-UMI
1/20
8.7
Diff. Line Color
11/20
7.8
RTC-MF-UMI
0/20
-
Low Contrastive Wt.
7/20
8.5
Low Freq. Inference
3/20
10.2
w/o Contrastive Loss
8/20
8.5
300ms Latency
12/20
7.5
w/o Future State
3/20
8.3
500ms Latency
10/20
8.5
w/o Heatmap
5/20
9.1
600ms Latency
4/20
11.2
Table 2: Ablation Studies. The results reported correspond to the Grasp Rolling Bottle task (Section 4.2 ).
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Hardware Platform. F2F-AP is evaluated on a fixed-base arm and a quadruped manipulator equipped with camera and odometry.
Figure 7: The left panel shows the occlusion scenarios, and the right panel illustrates how to build the ground-truth flow under occlusion. We select a pre-segmented gripper instance and overlay it onto the original fisheye videos to generate the RGB ground truth.
Figure 8: Failure Case Visualization illustrates four representative failure scenarios.
Figure 9: Comparative Visualization of baseline failure modes: (1) Naive Asynchronous Inference, and (2) VLASH.
Figure 10: Visualization of Flow Prediction illustrates the pairs of heatmap and images augmented by predicted flow in different tasks.
Figure 11: (Right) The heatmap indicates cosine similarity in the learned contrastive feature space, with brighter values denoting stronger alignment. The white dashed line marks same-time alignment, while the red dashed line indicates the expected temporal offset corresponding to the prediction horizon. The blue dotted line shows the empirical peak similarity for each predicted feature. (Left) Demonstration of how the flow-augmented observations map to the raw future observations, with the robot’s egocentric observations aligned with the corresponding timestamps in the heatmap.
Robotic manipulation requires policies that are smooth and responsive to evolving observations. However, synchronous inference in the raw action space introduces several challenges, including intra-chunk jitter, inter-chunk discontinuities, and stop-and-go execution. These issues undermine a policy's smoothness and its responsiveness to environmental changes. We propose ABPolicy, an asynchronous flow-matching policy that operates in a B-spline control-point action space. First, the B-spline representation ensures intra-chunk smoothness. Second, we introduce bidirectional action prediction coupled with refitting optimization to enforce inter-chunk continuity. Finally, by leveraging asynchronous inference, ABPolicy delivers real-time, continuous updates. We evaluate ABPolicy across seven tasks encompassing both static settings and dynamic settings with moving objects. Empirical results indicate that ABPolicy reduces trajectory jerk, leading to smoother motion and improved performance. Project website: https://teee000.github.io/ABPolicy/.
Fan Yang, Peiguang Jing, Kaihua Qu +2
The authors are with the School of Electrical and Information Engineering, Tianjin University
Real-time deployment of Vision-Language-Action (VLA) policies necessitates asynchronous execution, wherein subsequent action chunks are computed concurrently with the execution of the current chunk, leading to prediction-execution misalignment and manifesting as inter-chunk discontinuities. Existing methods either superficially smooth chunk boundaries, require costly policy optimization, or exclusively forward-predict proprioceptive states yet neglect critical visual observations. In this paper, we propose FutureRTC, a plug-and-play adaptation framework that predicts execution-time observations and states for asynchronous VLA control without modifying the underlying policy. Specifically, FutureRTC features a state correction module to compensate for the discrepancy between rolled-forward and actual execution-time proprioceptive states and an observation prediction module that forecasts execution-time visual representations by leveraging robot motion as an explicit physical prior through motion-aware feature transport and reconstruction. Furthermore, we introduce a policy consistency loss to align the action chunks generated from predicted contexts with those produced under the expected execution-time inputs of the VLA policy. Extensive experiments across simulated and real-world environments demonstrate that FutureRTC achieves superior robustness to inference delays, resulting in smoother trajectories, faster execution, and consistently higher task success rates.
Hai Jiang, Yixian Zou, Binbin Liang +3
School of Aeronautics and Astronautics, Sichuan University · University of Electronic Science and Technology of China · University of Alberta
Generalist robot policies such as vision-language-action models (VLAs) have achieved remarkable generalization, but their inference delays can conflict with the demands of real-time control. Asynchronous execution avoids pauses between action chunks by predicting the next sequence of actions while the robot carries out the previous one. In this paper, we study whether asynchronous execution produces the same action distribution as the original VLA. We find that, for non-Markovian demonstrations, asynchronous execution can produce a fundamentally different action distribution, which can limit the policy's reactivity. In our method, we seek to restore this reactivity by aligning the asynchronously produced action distribution with that of the original VLA through two complementary mechanisms. First, Recursive Flow-Field Distillation trains the asynchronous policy using the VLA's action-generation flow. We characterize the learned distribution theoretically and show experimentally that our asynchronous policy can generate nearly the full range of actions the original VLA would produce, while existing asynchronous methods recover only a fraction of that range. Second, Propose-Resolve prepares multiple action sequences asynchronously and uses the latest observation to select among them based on a lightweight approximation of their likelihood under the VLA's action distribution. Our resulting method matches the original VLA's success on LIBERO and retains about 80% of its success on RoboMimic, about 30 percentage points more than existing asynchronous methods.
Moritz Zoellner, Reece O'Mahoney, Ioannis Havoutis +1
Department of Computer Science, Purdue University · Oxford Robotics Institute, University of Oxford