While imitation learning has made substantial progress toward generalizable robot manipulation through scaling, viewpoint robustness remains a major bottleneck. Visuomotor policies rely heavily on visual inputs, yet images provide an inherently ambiguous signal under viewpoint change: similar scenes may appear markedly different across viewpoints, while distinct states may share certain visual overlaps. A policy must therefore infer not only the task-relevant state, but also how it is visually projected under a particular camera configuration. This entanglement of viewpoint-dependent appearance and task-relevant state can make learning inefficient and limit generalization to unseen viewpoints. Recent work has begun to address this challenge through camera-ray conditioning and representation learning using geometric cues to induce invariance. However, providing camera information as additional context does not by itself ground visual representations in how the task scene is visually projected, while geometric alignment without camera context can introduce spurious correspondences by ignoring viewpoint-dependent projection. We propose PAIR, a contrastive learning framework that aligns joint image-ray representations using robot proprioception. PAIR encourages observations with similar robot states to remain close in representation space across viewpoints, providing a structured grounding signal for learning viewpoint-robust features. PAIR is a highly adaptable framework: it can be used to train Diffusion Policy (DP) or post-train vision-language-action (VLA) models. In simulated and real-world manipulation benchmarks with camera variation, PAIR consistently outperforms the behavior cloning baseline without ray conditioning by 12.7% and 26.7% points, respectively, demonstrating stronger generalization to unseen viewpoints. Project website: PAIRrobot.github.io.
Figures & tables
Fig. 2 : Overview of PAIR. (A) PAIR introduces two ray representations: camera rays expressed relative to the moving end-effector frame and the robot-base frame, both encoded using Plücker coordinates. Red indicates zero Plücker moment, corresponding to rays that pass through the respective origin. The ray features are injected into the image features through an identity-initialized linear projection, which acts as a selective positional embedding for each vision token. (B) The joint image-ray representations are aligned according to proprioception similarity using supervised contrastive learning, which induces invariance against viewpoint variation.
Fig. 3 : Simulated benchmarks and real-world experiments. PAIR is evaluated across both simulated and real-world manipulation tasks. In simulation, MimicGen and LIBERO use a Franka Panda arm with two side views and one front view, respectively, while AlohaGym uses a bimanual Aloha platform with two front views. In the real world, we evaluate three tabletop tasks using a Franka Research 3 (FR3) arm with two front view cameras, where the camera poses are randomized and estimated using AprilTags [ 13 ] at the start of each rollout.
Name
Value
Training
Train steps
200K
Learning Rate
1 e−4
BC batch size
64
PAIR batch size
192
λq
0.05
TABLE I: Hyperparameters for Diffusion Policy and SmolVLA.
Env.
# Robots
# Cams
View
Multi-task
Horizon
MimicGen
1
2
Side
No
800/800/1000
ALOHAGym
2
2
Front
No
800
LIBERO
1
1
Front
Yes
600
Real-World
1
2
Front
Yes
1000
TABLE II : Comparison of environments, including robot and camera counts, viewpoint, multi-task setting, and task horizon. MimicGen uses separate task horizons for three tasks.
Square D1
3-Stack D1
3-Pc Assembly D1
ALOHA Transfer
Views
Auxillary Objective
w/o Ray
w/ Ray
w/o Ray
w/ Ray
w/o Ray
w/ Ray
w/o Ray
w/ Ray
Single
None (BC)
11.7 ± 1.2
13.4 ± 2.1
20.1 ± 1.1
34.1 ± 0.9
10.2 ± 1.3
18.1 ± 2.1
25.3 ± 1.6
35.2 ± 2.9
Temporal alignment (TCN)
11.7 ± 1.5
16.3 ± 4.5
19.0 ± 2.0
39.0 ± 2.6
12.7 ± 2.1
18.3 ± 2.5
27.3 ± 1.5
35.3 ± 3.5
Proprio alignment (PAIR)
20.7 ± 2.8
24.4 ± 2.9
36.3 ± 2.5
45.1 ± 3.7
20.1 ± 3.5
29.1 ± 1.8
30.0 ± 0.7
42.3 ± 2.1
Multi
None (BC)
26.9 ± 4.6
24.1 ± 1.1
44.1 ± 1.8
49.6 ± 2.6
17.3 ± 1.5
27.1 ± 1.8
55.7 ± 2.1
66.3 ± 2.9
Temporal alignment (TCN)
28.0 ± 2.0
32.3 ± 2.5
47.3 ± 2.1
50.0 ± 3.0
17.7 ± 2.1
25.3 ± 2.1
58.7 ± 3.2
70.0 ± 5.3
TABLE III: Success rate (%) of Diffusion Policy (DP) on three MimicGen tasks and ALOHA Transfer across 3 seeds. For each seed, the maximum success rates from 5 checkpoints is reported.
Object
Goal
Spatial
Long
Average
Auxiliary Objective
w/o Ray
w/ Ray
w/o Ray
w/ Ray
w/o Ray
w/ Ray
w/o Ray
w/ Ray
w/o Ray
w/ Ray
None (BC)
94.6 ± 0.3
92.9 ± 0.6
90.4 ± 0.6
89.2 ± 0.5
82.8 ± 0.9
84.3 ± 1.1
57.9 ± 1.1
66.9 ± 0.9
81.4
83.3
Time-contrastive (TCN)
93.3 ± 0.4
94.1 ± 0.3
88.5 ± 0.6
87.3 ± 0.9
79.5 ± 0.5
84.3 ± 1.1
61.1 ± 1.1
70.4 ± 0.6
80.6
84.0
Proprio-contrastive (PAIR)
92.8 ± 0.5
95.7 ± 0.3
84.7 ± 0.5
90.8 ± 0.6
84.5 ± 0.6
86.8 ± 0.5
73.4 ± 3.1
82.1 ± 0.7
83.9
88.9
TABLE IV: Success rate (%) of SmolVLA on the four LIBERO task suites across 3 seeds. For each seed, the maximum success rates from 5 checkpoints is reported.
Fig. 4 : Success rates (%) of BC (red) and PAIR (blue), with and without ray conditioning, in three real-world tasks. Both subtask completion rates and final task success rates are reported.
Task (View)
PAIR
PAIR w/o rCB
PAIR w/o rCE
Square D1 (Multi)
38.5
28.8 (-9.7)
35.3 (-3.2)
Aloha Transfer (Multi)
74.0
55.2 (-18.8)
67.0 (-7.0)
LIBERO
88.9
87.2 (-1.7)
86.8 (-2.1)
TABLE V : Ablation of end-effector-relative ( rCE ) and robot-base-relative ( rCB ) camera rays in PAIR. Average success rates (%) are reported, with the performance drop shown in parentheses.
Fig. 5 : Policy performance with varied λq and λPA values for Diffusion Policy trained on Square D1 (blue) and SmolVLA trained on LIBERO (red) with the PAIR objective. The baselines report the performance of BC with ray conditioning.
BC
PAIR
Task
w/o Ray
w/ Ray
w/o Ray
w/ Ray
LIBERO
1.28
2.91
1.64
3.52
Real-World
1.17
2.16
2.25
2.67
TABLE VI : Normalized average attention assigned to the visual token containing the ground-truth tool center point. The raw attention score is normalized by the expected attention under a uniform distribution, where a value of 1 denotes the expected value.
Fig. 6 : Visualization of the rollout attention maps of SmolVLA models trained on the LIBERO dataset. The average attention activations are overlaid on the corresponding input images. To ensure the consistency of the rollout, we sample a single trajectory from the training dataset. Without ray input, attention is concentrated on spatially irrelevant regions, even with proprioceptive alignment. In contrast, ray-conditioning leads to stronger attention to the end effector, with proprioceptive alignment further concentrating activation around it and its nearby workspace.
Metric
BC
TCN
PAIR
FLOPs ( ×1013 , ↓ )
1.68 / 1.73
14.1 / 14.7
5.83 / 6.05
Decoded Frames ( ↓ )
64
640
256
TABLE VII : Computational and dataloading cost of training SmolVLA with BC, TCN, and PAIR. FLOPs are reported per forward pass for models without and with ray conditioning, respectively. Decoded Frames denotes the number of camera frames decoded during dataloading for each training batch. Training settings are provided in Table I .B.