Achieving precise terminal position and heading alignment remains a significant challenge for learning-based mobile robot control. Recent vision-based learning methods have shown promise for precise target-relative control and generalization to unseen environments with pretrained visual encoders, but general-purpose visual representations are not specifically designed to capture the fine-grained spatial differences between current and goal views required for precise control, and real-world demonstrations combining millimeter-scale terminal accuracy with zero-shot transfer across environments remain limited. To address these challenges, we propose Geometric Representation Conditioning for Precise Visuomotor Policy (GR-VIP), a goal-conditioned visuomotor policy that leverages a visual encoder pretrained for relative camera pose estimation. Rather than using the final relative-pose estimate, GR-VIP conditions action prediction on the dense relational features produced before pose regression, preserving fine-grained cross-view information for learning corrective actions from demonstrations. We evaluate GR-VIP on a real-world docking task and demonstrate high docking success rates and millimeter-scale terminal accuracy in both the training and unseen environments, without additional training for transfer. Furthermore, supplementary experiments illustrate the policy's ability to align with non-docking targets and complete docking under a tested low-light condition.
Figures & tables
Fig. 1: Overview of GR-VIP. (a) GR-VIP performs precise real-world docking with autonomous repositioning to correct misalignment. (b) A policy trained in a single environment transfers to unseen environments without additional training. The same policy also demonstrates alignment to non-docking targets and successful docking under a low-light condition.
Fig. 2: GR-VIP architecture. Dense relational tokens from the frozen Reloc3r capture geometric relationships between each observation and the goal image. The tokens are resampled and fused with wheel-velocity history to condition a diffusion policy that predicts future velocity commands.
Fig. 3: Closed-loop execution of GR-VIP. Multimodal observations condition diffusion-based action generation. Successive action plans are blended to maintain temporal continuity and executed in a receding-horizon manner as new predictions become available. Here, Hm and Ha are both set to 60 steps.
Fig. 4: Experimental setup. (a) Mobile robot platform. (b) Docking setup and initial poses. Dashed arcs indicate 0.5- and 1.0-m approach distances, and purple arrows denote intentionally misaligned initial poses. (c) Marker uncovered for terminal-pose evaluation after policy execution. (d) Training environment (4F-H). (e) Unseen-background environment (4F-I). (f) Environment with an unseen docking station and background (5F-I). (g) Low-light condition. (h, i) Alignment to a non-docking target during policy execution and subsequent terminal-pose evaluation.
Model
Train
Eval setting
Success % (count)
n/N
Lateral (mm)
Longitudinal (mm)
Heading ( ∘ )
(a) Training-environment docking performance
GR-VIP
4F-H
4F-H
100.0 (16/16)
16/16
2.53±2.24
1.95±1.41
2.28±1.64
GC-DP (NoMaD adapted)
56.3 (9/16)
13/16
9.44±6.23
9.86±10.62
2.09±1.17
DINOv3-based
50.0 (8/16)
15/16
12.79±10.40
12.66±12.63
1.81±2.11
(b) Zero-shot transfer and environment-specific baselines
GR-VIP
4F-H
4F-I
100.0 (16/16)
16/16
7.20±4.78
7.15±4.86
3.75±1.71
TABLE I: Quantitative evaluation of docking performance, zero-shot transfer, representation ablations, and alignment to non-docking targets.
Fig. 5: Overview of terminal alignment errors for policies trained on the common 4F-H demonstrations. The top row shows the full error range, while the bottom row provides a detailed view near zero error. n denotes the number of valid error samples. GR-VIP consistently exhibits low terminal alignment errors relative to the other policies across all three environments.
Learning visuomotor policies from scarce expert demonstrations remains a core challenge in robotic manipulation. A primary hurdle lies in distilling high-dimensional RGB representations into control-relevant geometry without overfitting. While using frozen pre-trained Vision Foundation Models (VFMs) improves data efficiency, it also shifts most task adaptation onto a small spatial pooling module, which can latch onto task-irrelevant shortcuts and lose geometric grounding when finetuned with few data samples. More broadly, pre-trained visual representations used for policy learning have been observed to struggle under even minor scene perturbations, highlighting the need for robustness-oriented inductive biases. We propose Geometric Anchor Pre-training (GAP), a simple, action-free warm-up stage that regularizes the spatial adapter before downstream imitation learning. GAP pre-trains the pooling layer on a lightweight simulated proxy task where object masks are available at no cost, encouraging the adapter to produce keypoints that lie on the object, cover its spatial extent, and remain sharp and repeatable over time. This yields stable geometric anchors that provide a reliable coordinate interface for few-shot policy learning, while keeping the VFM frozen. We evaluate GAP on RoboMimic and ManiSkill under severe data scarcity (15-50 demonstrations) and domain shift. A simple adapter regularized with GAP consistently outperforms stronger attention-based poolers and end-to-end fine-tuning, achieving 62% success on RoboMimic Can with 15 demonstrations (+16% over AFA), 63% on the long-horizon high-precision Tool Hang task with 50 demonstrations, and 61% on ManiSkill StackCube with 30 demonstrations (+11% over full fine-tuning). The proxy stage is lightweight and fully decoupled from downstream tasks, making it practical to reuse across environments and manipulation skills.
Davide Buoso, Andrea Protopapa, Stefano Di Carlo +2
Department of Control and Computer Engineering, Polytechnic University of Turin, Italy
Successful mobile manipulation requires coordinated base and arm motion while maintaining accurate spatial positioning. However, demonstration-trained policies can struggle to realise the intended base motion reliably, leading to spatial misalignment and subsequent manipulation failures. We present MAVP (Map-Aware Visuomotor Policies), a framework that improves execution reliability by predicting explicit base-pose targets and tracking them using localisation feedback. MAVP reconstructs a static map from teleoperated demonstrations and expresses demonstrated base trajectories in a shared map frame, providing consistent spatial supervision across demonstrations. At execution time, the policy receives RGB observations, joint states, and the robot's current map-frame base pose, and jointly predicts target base poses, arm actions, and gripper actions. A low-level controller tracks the predicted base targets using feedforward motion and pose error feedback, enabling correction of execution deviations. We additionally use pose-noise augmentation during training to improve robustness to errors in the policy's pose input. Across six real-world manipulation tasks and three policy families, MAVP achieves higher task success rates than unanchored velocity control in all tasks. Videos and additional results are available at https://123qwedsa123.github.io/mavp/.
Jinhe Tang, Ruixiao Dai, Weiming Zhi
School of Computer Science, The University of Sydney, Australia
We present GHOST, a framework for learning visuomotor manipulation policies that generalize beyond the training distribution. GHOST factorizes control into (i) a high-level policy that predicts the next sub-goal as a distribution over 3D end-effector poses from multi-view RGB-D observations, and (ii) a low-level goal-conditioned controller that executes embodiment-specific actions. To condition image-based policies on 3D goals, we introduce a simple spatial interface that projects predicted goals into the image plane and represents them as end-effector heatmaps. Across a suite of manipulation tasks, this hierarchical factorization consistently improves performance and robustness compared to a flat Diffusion Policy. Further, we show that this hierarchical interface also makes it easy to incorporate human demonstrations without relying on (noisy) action retargeting. As sub-goals are largely embodiment-agnostic, we train the high-level policy on human video to specify how learned skills should be applied and composed, while keeping the low-level policy trained purely on robot data. This hierarchy enables adaptation to novel objects and task variations using a small number of human demonstrations.
Sriram Krishna, Ben Eisner, Haotian Zhan +5
Robotics Institute, Carnegie Mellon University · UMass Amherst