This paper addresses a fundamental problem of visuomotor policy learning for robotic manipulation: how to enhance robustness in out-of-distribution execution errors or dynamically re-routing trajectories, where the model relies solely on the original expert demonstrations for training. We introduce the Referring-Aware Visuomotor Policy (ReV), a closed-loop framework that can adapt to unforeseen circumstances by instantly incorporating sparse referring points provided by a human or a high-level reasoning planner. Specifically, ReV leverages the coupled diffusion heads to preserve standard task execution patterns while seamlessly integrating sparse referring via a trajectory-steering strategy. Upon receiving a specific referring point, the global diffusion head firstly generates a sequence of globally consistent yet temporally sparse action anchors, while identifies the precise temporal position for the referring point within this sequence. Subsequently, the local diffusion head adaptively interpolates adjacent anchors based on the current temporal position for specific tasks. This closed-loop process repeats at every execution step, enabling real-time trajectory replanning in response to dynamic changes in the scene. In practice, rather than relying on elaborate annotations, ReV is trained only by applying targeted perturbations to expert demonstrations. Without any additional data or fine-tuning scheme, ReV achieve higher success rates across challenging simulated and real-world tasks.
Figures & tables
Figure 1: The role of the proposed Referring-Aware Visuomotor Policy (ReV) in mitigating out-of-distribution failures. In executing the task “Grabbing the steak from the kitchen table” , several challenges can be encountered. Case A : a small covariate shift at tk quickly leads to a compounding misalignment with expert demonstrations, driving the robot into unseen states and resulting in task failure. Case B : an unforeseen obstacle ( e . g ., a pot) emerges at tk to dynamically block the robot’s path, leading to failure as well. Unlike traditional imitation learning methods that often struggle to generalize to unseen states or observations, the proposed ReV employs coupled diffusion heads to online react to external sparse referring provided by a human or a high-level reasoning planner. Without requiring additional training data or complex fine-tuning in post-processing, the model can effectively address Case A and Case B in real-world applications.
Figure 2: Overview of our Referring-Aware Visuomotor Policy. Given the observation O and robot proprioception history A− , the Temporal-Position Prediction Module on the left assign the optimal temporal position k for the referring point P . This temporally-positioned referring point P is then transformed into a temporally-positioned partially-constrained referring action Pa (detailed in Sec. 3.3 ) and injected into the right Policy Model with Coupled Diffusion Heads to generate the final trajectory A satisfying the Manipulation Steering Choreography . Specifically, at step i , the GDH incorporates both anchor history A−′ and partially-constrained referring action Pa via the trajectory-steering strategy to yield a sparse yet globally consistent anchor sequence A′={aj′}j=1N1 , in which {aj′}j=1i−1 is anchor history A−′ , {aj′}j=ii+1 is interpolated by the LDH through a learnable, temporal-dependent interpolation strategy, and {aj′}j=i+1N1 is discarded and will be updated at upcoming inference step. At inference step i+1 , the anchor history A−′ is updated by appending the newly generated anchor ai+1′ at step i , and the entire policy model repeats, continuously updating A .
Figure 3: Temporal-Position Prediction . The slots buffer S covers both retained history and extrapolated future steps. Specifically, at step i , we load the historical action anchors {aj′}j=1i to initialize history part, and the future part is padded by copies of the last action anchor ai′ . Then, the whole slots buffer is augmented with a monotonically-increasing temporal-position embeddings.
Method
Pick Meat-via
Lift Barrier-via
Place Food-via
Camera Alignment-via
RePR( ↑ )
SuR( ↑ )
SmS( ↑ )
RePR( ↑ )
SuR( ↑ )
SmS( ↑ )
RePR( ↑ )
SuR( ↑ )
SmS( ↑ )
RePR( ↑ )
SuR( ↑ )
SmS( ↑ )
ACT
2%
1%
0.9890
1%
1%
0.9904
0%
0%
-
0%
0%
-
DP3
80%
1%
0.9899
99%
25%
0.9945
1%
1%
0.9883
0%
0%
-
CDP
14%
14%
0.9924
99%
99%
0.9933
47%
33%
0.9878
0%
0%
-
OCTO
18%
9%
0.9606
32%
32%
0.9702
1%
1%
0.9597
0%
0%
-
MPD
20%
3%
0.9903
39%
39%
0.9949
3%
3%
0.9887
1%
1%
0.9861
Table 1: Quantitative results on modified simulated benchmark , highlighting the effectiveness of ReV in referring-aware manipulation.
Figure 4: Visualization of the trajectories generated by ReV on Pick-Meat-via and Lift-Barrier-via . Here, we use green bounding boxes to mark the frames in which the end-effector passes through the designated via-point (green ball).
Method
0.1
0.2
0.3
0.4
RePR( ↑ )
SuR( ↑ )
RePR( ↑ )
SuR( ↑ )
RePR( ↑ )
SuR( ↑ )
RePR( ↑ )
SuR( ↑ )
ReV
100%
93%
100%
92%
100%
89%
100%
87%
Table 2: Ablation study on OOD-yet-feasible referring points. 0.1, 0.2, 0.3, and 0.4 indicate the degree of deviation from the center of the expert distribution. Details are provided in Appendix B.4 .
Method
Adroit
DexArt
MetaWorld
RoboFactory
Pen
Door
Laptop
Toilet
Bucket
Reach
Soccer
Sweep Into
Shelf Place
Pick Meat
Lift Barrier
Place Food
Camera Alignment
ACT
47%
66%
35%
8%
6%
21%
28%
23%
43%
91%
40%
13%
21%
DP3
53%
69%
81%
65%
32%
28%
30%
15%
37%
81%
90%
30%
87%
CDP
68%
74%
84%
68%
32%
22%
23%
24%
35%
84%
93%
51%
90%
ReV
73%
79%
87%
71%
61%
36%
33%
27%
47%
94%
99%
57%
94%
Table 3: Quantitative results across simulated benchmarks , highlighting the effectiveness of our Coupled Diffusion Heads architecture.
Method
Single-Agent
Dual-Agent
Collecting Objects-via
Push T-via
Stacking Playing Card-via
Grabbing Rod-via
Handing Eraser Over-via
PeRP ↑
SuR ↑
PeRP ↑
SuR ↑
PeRP ↑
SuR ↑
PeRP ↑
SuR ↑
PeRP ↑
SuR ↑
ACT
3 / 30
1 / 30
5 / 30
3 / 30
3 / 30
0 / 30
2 / 30
0 / 30
3 / 30
1 / 30
DP
6 / 30
5 / 30
12 / 30
9 / 30
9 / 30
3 / 30
3 / 30
1 / 30
6 / 30
2 / 30
ReV
30 / 30
20 / 30
30 / 30
21 / 30
30 / 30
15 / 30
30 / 30
18 / 30
30 / 30
12 / 30
Table 4: Quantitative results on real-world tasks , highlighting the robustness of our ReV in real-world settings.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Demonstration Construction Strategy for GDH and LDH . The orange blocks represent the sparse action anchors. Among them, the lighter orange exemplifies the historical context, while the darker orange exemplifies the future targets. The green blocks denote the fine-grained interpolation actions.
Figure 6: Via-Points Generation. Via-points are generated via Gaussian sampling around a centroid derived from the robot’s initial configuration.
Figure 7: Visualization of ablation studies on key hyperparameters. (a) Total trajectory length N : performance peaks when N matches task complexity, enabling the model to capture task-specific execution patterns. (b) Trajectory-length allocation N1:N2 : balanced ratios ( 1:2∼2:1 ) allow our policy model to learn the entire trajectory most effectively. (c) Update Frequencies of T-P prediction : excessive updates cause anchor drift and degrade performance. Here, success rate is defined as the fraction of trials that simultaneously achieve task completion and pass through the designated referring point, and all ablations were conducted under controlled variables.
Method
Inside Camera
Inside Pot
Under Table
Out of Reach
DP3
0%
0%
0%
0%
CDP
0%
11%
0%
0%
ReV
100%
100%
0%
0%
Appendix
Table 5: Ablation study on infeasible referring points.
Table 6: All Training Settings for simulation experiments.
Benchmark
Task
N
N1
N2
Adroit
Door
54
6
12
Pen
70
6
16
Dexart
Bucket
11
3
8
Laptop
20
4
8
Toilet
70
6
16
MetaWorld
Shelf Place
11
3
8
Appendix
Table 7: Trajectory-Length Allocation for each task in simulation and real-world experiments.
Figure 9: Visualization of the trajectories generated by ReV on Place-Food-via and Camera-Alignment-via . Here, we use green bounding boxes to mark the frames in which the end-effector passes through the designated via-point (green ball).
Figure 10: Real-world Experimental Platform. The setup comprises a dual-robot arm system, a top-down RGB-D camera, and a black table serving as the workspace.
Task
Agent Number
Description
Collecting Objects
1
A doll is placed on the table. The robotic manipulator first grasps it and then transports it into the green box.
Push T
1
A T-shaped object is initially positioned on the table. The robotic arm pushes it into a predefined T-shaped location.
Stacking Playing Card
1
A playing card is placed flat in the central region of the table. The robotic arm first grasps it precisely and then places it vertically into another playing card to achieve a stacking effect.
Grabbing Rod
2
A long rod is placed on a block. The two robotic arms first simultaneously grasp each end of the rod and then collaboratively lift it to a specified height.
Handing Eraser Over
2
One robotic arm first grasps an eraser precisely. It then passes the eraser to another robotic arm through a coordinated handover motion.
Appendix
Table 8: Original Task Descriptions for real-world experiments.
Visuomotor policy learning trains robots to turn camera observations, often with language and proprioception, into actions for manipulation and navigation.
Learning visuomotor policies for long-horizon manipulation remains a fundamental challenge. Recent skill-based imitation learning methods based on discrete quantization have shown promising results by representing complex behaviors as temporally extended skills. However, most existing approaches primarily encode action trajectories into latent skills, yielding weak visual-semantic grounding and limiting the ability to leverage visual observations for skill selection. Moreover, discrete tokenization inevitably incurs precision errors during continuous action generation. To alleviate these issues, we propose Aligned Refinement Policy (ARP), a discrete-skill framework that couples semantic grounding with execution-level refinement. Specifically, ARP introduces (i) a visual--action alignment objective that contrastively aligns visual embeddings with pre-quantized action representations in a shared latent space while preserving a state-independent skill decoder, and (ii) a lightweight Iterative Residual Head (IRH) that performs a two-step refinement to recover fine-grained control for precise execution. Extensive experiments show that ARP achieves state-of-the-art performance on the LIBERO and Meta-World benchmarks. Moreover, real-robot experiments on the Kuavo 4 Pro humanoid platform further validate its effectiveness, yielding consistent performance gains over several baselines on two challenging manipulation tasks.
Yuntian Wang, Zesheng Jia, Yuhui Duan +5
School of Future Science and Engineering, Soochow University · Key Laboratory of General Artificial Intelligence and Large Models in Provincial Universities, Soochow University · College of Mechanical and Electronic Engineering, China Jiliang University +2
Effective robotic manipulation requires policies that can anticipate physical outcomes and adapt to real-world environments. Effective robotic manipulation requires policies that can anticipate physical outcomes and adapt to real-world environments. In this work, we introduce a unified framework, World-Model-Driven Diffusion Policy with Online Adaptive Learning (AdaWorldPolicy) to enhance robotic manipulation under dynamic conditions with minimal human involvement. Our core insight is that world models provide strong supervision signals, enabling online adaptive learning in dynamic environments, which can be complemented by force-torque feedback to mitigate dynamic force shifts. Our AdaWorldPolicy integrates a world model, an action expert, and a force predictor-all implemented as interconnected Flow Matching Diffusion Transformers (DiT). They are interconnected via the multi-modal self-attention layers, enabling deep feature exchange for joint learning while preserving their distinct modularity characteristics. We further propose a novel Online Adaptive Learning (AdaOL) strategy that dynamically switches between an Action Generation mode and a Future Imagination mode to drive reactive updates across all three modules. This creates a powerful closed-loop mechanism that adapts to both visual and physical domain shifts with minimal overhead. Across a suite of simulated and real-robot benchmarks, our AdaWorldPolicy achieves state-of-the-art performance, with dynamical adaptive capacity to out-of-distribution scenarios.
Imitation learning enables robots to acquire manipulation skills from demonstrations by mapping observations to actions. Existing approaches predict either short-horizon continuous action sequences or discrete keyposes. However, continuous prediction methods suffer from compounding errors due to short prediction horizons and struggle with multi-modal action distributions, whereas keypose-based methods necessitate an external planner, constraining real-time applicability. To address these challenges, we introduce SegDiff, a closed-loop visuomotor policy that integrates the strengths of both paradigms. SegDiff decomposes demonstrations into motion segments between keyposes and learns to predict the continuous trajectory from the current state to the next keypose, enabling long-horizon prediction with real-time refinement. Furthermore, we leverage the capability of diffusion models and DDIM inversion to propose a Dynamic Temporal Ensembling mechanism, which allows the policy to efficiently respond to dynamic environments and mitigate discontinuities caused by inconsistent multi-modal sampling. SegDiff demonstrates significant performance gains over existing approaches across various simulated and real-world scenarios, indicating its strong ability to reason over extended temporal dependencies while maintaining real-time adaptability and control stability.
Haidong Cao, Wenjun Cao, Quanhao Li +5
Institute of Trustworthy Embodied AI, Fudan University, China · Shanghai Key Laboratory of Multimodal Embodied AI, China