While imitation learning has made substantial progress toward generalizable robot manipulation through scaling, viewpoint robustness remains a major bottleneck. Visuomotor policies rely heavily on visual inputs, yet images provide an inherently ambiguous signal under viewpoint change: similar scenes may appear markedly different across viewpoints, while distinct states may share certain visual overlaps. A policy must therefore infer not only the task-relevant state, but also how it is visually projected under a particular camera configuration. This entanglement of viewpoint-dependent appearance and task-relevant state can make learning inefficient and limit generalization to unseen viewpoints. Recent work has begun to address this challenge through camera-ray conditioning and representation learning using geometric cues to induce invariance. However, providing camera information as additional context does not by itself ground visual representations in how the task scene is visually projected, while geometric alignment without camera context can introduce spurious correspondences by ignoring viewpoint-dependent projection. We propose PAIR, a contrastive learning framework that aligns joint image-ray representations using robot proprioception. PAIR encourages observations with similar robot states to remain close in representation space across viewpoints, providing a structured grounding signal for learning viewpoint-robust features. PAIR is a highly adaptable framework: it can be used to train Diffusion Policy (DP) or post-train vision-language-action (VLA) models. In simulated and real-world manipulation benchmarks with camera variation, PAIR consistently outperforms the behavior cloning baseline without ray conditioning by 12.7% and 26.7% points, respectively, demonstrating stronger generalization to unseen viewpoints. Project website: PAIRrobot.github.io.
Figures & tables
Fig. 2 : Overview of PAIR. (A) PAIR introduces two ray representations: camera rays expressed relative to the moving end-effector frame and the robot-base frame, both encoded using Plücker coordinates. Red indicates zero Plücker moment, corresponding to rays that pass through the respective origin. The ray features are injected into the image features through an identity-initialized linear projection, which acts as a selective positional embedding for each vision token. (B) The joint image-ray representations are aligned according to proprioception similarity using supervised contrastive learning, which induces invariance against viewpoint variation.
Fig. 3 : Simulated benchmarks and real-world experiments. PAIR is evaluated across both simulated and real-world manipulation tasks. In simulation, MimicGen and LIBERO use a Franka Panda arm with two side views and one front view, respectively, while AlohaGym uses a bimanual Aloha platform with two front views. In the real world, we evaluate three tabletop tasks using a Franka Research 3 (FR3) arm with two front view cameras, where the camera poses are randomized and estimated using AprilTags [ 13 ] at the start of each rollout.
Name
Value
Training
Train steps
200K
Learning Rate
1 e−4
BC batch size
64
PAIR batch size
192
λq
0.05
TABLE I: Hyperparameters for Diffusion Policy and SmolVLA.
Env.
# Robots
# Cams
View
Multi-task
Horizon
MimicGen
1
2
Side
No
800/800/1000
ALOHAGym
2
2
Front
No
800
LIBERO
1
1
Front
Yes
600
Real-World
1
2
Front
Yes
1000
TABLE II : Comparison of environments, including robot and camera counts, viewpoint, multi-task setting, and task horizon. MimicGen uses separate task horizons for three tasks.
Square D1
3-Stack D1
3-Pc Assembly D1
ALOHA Transfer
Views
Auxillary Objective
w/o Ray
w/ Ray
w/o Ray
w/ Ray
w/o Ray
w/ Ray
w/o Ray
w/ Ray
Single
None (BC)
11.7 ± 1.2
13.4 ± 2.1
20.1 ± 1.1
34.1 ± 0.9
10.2 ± 1.3
18.1 ± 2.1
25.3 ± 1.6
35.2 ± 2.9
Temporal alignment (TCN)
11.7 ± 1.5
16.3 ± 4.5
19.0 ± 2.0
39.0 ± 2.6
12.7 ± 2.1
18.3 ± 2.5
27.3 ± 1.5
35.3 ± 3.5
Proprio alignment (PAIR)
20.7 ± 2.8
24.4 ± 2.9
36.3 ± 2.5
45.1 ± 3.7
20.1 ± 3.5
29.1 ± 1.8
30.0 ± 0.7
42.3 ± 2.1
Multi
None (BC)
26.9 ± 4.6
24.1 ± 1.1
44.1 ± 1.8
49.6 ± 2.6
17.3 ± 1.5
27.1 ± 1.8
55.7 ± 2.1
66.3 ± 2.9
Temporal alignment (TCN)
28.0 ± 2.0
32.3 ± 2.5
47.3 ± 2.1
50.0 ± 3.0
17.7 ± 2.1
25.3 ± 2.1
58.7 ± 3.2
70.0 ± 5.3
TABLE III: Success rate (%) of Diffusion Policy (DP) on three MimicGen tasks and ALOHA Transfer across 3 seeds. For each seed, the maximum success rates from 5 checkpoints is reported.
Object
Goal
Spatial
Long
Average
Auxiliary Objective
w/o Ray
w/ Ray
w/o Ray
w/ Ray
w/o Ray
w/ Ray
w/o Ray
w/ Ray
w/o Ray
w/ Ray
None (BC)
94.6 ± 0.3
92.9 ± 0.6
90.4 ± 0.6
89.2 ± 0.5
82.8 ± 0.9
84.3 ± 1.1
57.9 ± 1.1
66.9 ± 0.9
81.4
83.3
Time-contrastive (TCN)
93.3 ± 0.4
94.1 ± 0.3
88.5 ± 0.6
87.3 ± 0.9
79.5 ± 0.5
84.3 ± 1.1
61.1 ± 1.1
70.4 ± 0.6
80.6
84.0
Proprio-contrastive (PAIR)
92.8 ± 0.5
95.7 ± 0.3
84.7 ± 0.5
90.8 ± 0.6
84.5 ± 0.6
86.8 ± 0.5
73.4 ± 3.1
82.1 ± 0.7
83.9
88.9
TABLE IV: Success rate (%) of SmolVLA on the four LIBERO task suites across 3 seeds. For each seed, the maximum success rates from 5 checkpoints is reported.
Fig. 4 : Success rates (%) of BC (red) and PAIR (blue), with and without ray conditioning, in three real-world tasks. Both subtask completion rates and final task success rates are reported.
Task (View)
PAIR
PAIR w/o rCB
PAIR w/o rCE
Square D1 (Multi)
38.5
28.8 (-9.7)
35.3 (-3.2)
Aloha Transfer (Multi)
74.0
55.2 (-18.8)
67.0 (-7.0)
LIBERO
88.9
87.2 (-1.7)
86.8 (-2.1)
TABLE V : Ablation of end-effector-relative ( rCE ) and robot-base-relative ( rCB ) camera rays in PAIR. Average success rates (%) are reported, with the performance drop shown in parentheses.
Fig. 5 : Policy performance with varied λq and λPA values for Diffusion Policy trained on Square D1 (blue) and SmolVLA trained on LIBERO (red) with the PAIR objective. The baselines report the performance of BC with ray conditioning.
BC
PAIR
Task
w/o Ray
w/ Ray
w/o Ray
w/ Ray
LIBERO
1.28
2.91
1.64
3.52
Real-World
1.17
2.16
2.25
2.67
TABLE VI : Normalized average attention assigned to the visual token containing the ground-truth tool center point. The raw attention score is normalized by the expected attention under a uniform distribution, where a value of 1 denotes the expected value.
Fig. 6 : Visualization of the rollout attention maps of SmolVLA models trained on the LIBERO dataset. The average attention activations are overlaid on the corresponding input images. To ensure the consistency of the rollout, we sample a single trajectory from the training dataset. Without ray input, attention is concentrated on spatially irrelevant regions, even with proprioceptive alignment. In contrast, ray-conditioning leads to stronger attention to the end effector, with proprioceptive alignment further concentrating activation around it and its nearby workspace.
Metric
BC
TCN
PAIR
FLOPs ( ×1013 , ↓ )
1.68 / 1.73
14.1 / 14.7
5.83 / 6.05
Decoded Frames ( ↓ )
64
640
256
TABLE VII : Computational and dataloading cost of training SmolVLA with BC, TCN, and PAIR. FLOPs are reported per forward pass for models without and with ray conditioning, respectively. Decoded Frames denotes the number of camera frames decoded during dataloading for each training batch. Training settings are provided in Table I .B.
Visual imitation learning enables robots to acquire visuomotor skills directly from images, yet RGB observations lack explicit geometric cues, making learned policies brittle to camera perturbations. To address this, we propose Ray-conditioned Vision Transformer Encoder (RayViT), a lightweight architecture that injects camera geometry into pretrained ViT backbones. RayViT represents camera geometry as a Plücker ray map, patchifies it into ray features, and uses gated cross-attention to produce a ray-conditioned class token. These ray features are added as dense positional embeddings, while the ray class token replaces the original ViT class token to provide a geometry-aware summary representation. We combine this approach with an auxiliary cosine similarity loss to consistently improve the performance and robustness for geometry-aware tokens. Experiments on sim- and real-robot tasks demonstrate that RayViT improves robustness by approximately 13 percentage points under camera perturbations in multi-task RoboCasa benchmark and by 1.78 average completed stages in real-world multi-task success rate compared to baselines.
Qian Wang, Longrui Chen, Peiran Sun +8
Karlsruhe Institute of Technology, Germany · University of Leeds, United Kingdom
Proprioception is fundamental to robotic manipulation, yet standard fusion methods often treat it as an isolated vector lacking explicit alignment with visual tokens. Without a direct correspondence between 3D kinematics and 2D feature maps, manipulation policies struggle to ground the robot's state within the scene, frequently underperforming even vision-only baselines. To address this, we introduce GeoProp, a lightweight, plug-and-play adapter that aligns proprioception with vision through explicit geometric grounding and spatial feature sampling. GeoProp projects the robot state onto the image plane to sample localized visual features, constructing a grounded state token. It then injects state-derived spatial priors into the corresponding visual features via FiLM modulation. To capture motion intent, GeoProp further samples features at a short-horizon predicted coordinate derived from recent kinematics, providing look-ahead visual context. Across 67 tasks, GeoProp improves Diffusion Policy by 8.7% on 63 simulation tasks and pi_0 by 4.0% on the RoboTwin subset, and yields a 10.6% average gain across both policy families in the real world, while adding only 2-3% to the parameter count. These results demonstrate that GeoProp is a simple yet high-impact inductive bias for generalist embodied policies. Project page: https://alibaba-damo-academy.github.io/GeoProp/.
Guoyang Zhao, Quanhao Qian, Gongjie Zhang +5
Tongji University · DAMO Academy, Alibaba Group · HuPan Lab +2
Recently, end-to-end robotic manipulation models have gained significant attention for their generalizability and scalability. However, they often suffer from limited robustness to camera viewpoint changes when training with a fixed camera. In this paper, we propose VistaBot, a novel framework that integrates feed-forward geometric models with video diffusion models to achieve view-robust closed-loop manipulation without requiring camera calibration at test time. Our approach consists of three key components: 4D geometry estimation, view synthesis latent extraction, and latent action learning. VistaBot is integrated into both action-chunking (ACT) and diffusion-based (π0) policies and evaluated across simulation and real-world tasks. We further introduce the View Generalization Score (VGS) as a new metric for comprehensive evaluation of cross-view generalization. Results show that VistaBot improves VGS by 2.79× and 2.63× over ACT and π0, respectively, while also achieving high-quality novel view synthesis. Our contributions include a geometry-aware synthesis model, a latent action planner, a new benchmark metric, and extensive validation across diverse environments. The code and models will be made publicly available.