Visual imitation learning is a promising approach to training robot manipulation policies capable of completing a wide variety of tasks. However, policies today remain brittle to viewpoint perturbations, making deployment in diverse environments a challenge. We present a controlled empirical study of which design choices allow visuomotor policies to generalize across viewpoints. We find that viewpoint generalization improves when dense visual tokens are retained and the action head participates in geometric reasoning. On a suite of simulated tasks that span a wide range of camera poses, we show that these design choices yield a policy that remains performant across viewpoints. As a practical consequence, a policy trained with these design choices also transfers zero-shot from simulation to the real world under random camera configurations.
Figures & tables
Fig. 1: Top Left: We sample a wide distribution of camera poses in simulation. Middle: We generate demonstrations under these randomized views and train a policy that grounds visual, proprioception, and action tokens in a shared robot frame. Right: The policy transfers zero-shot to a real robot under random camera configurations.
Fig. 2: Base policy and design axes , shown in the A gp configuration. A set of calibrated RGB views are encoded by a backbone initialized from DINOv2 applying within-view self-attention and cross-view global self-attention, producing dense multiview image tokens ( visual encoder and token compression axes). Every image, proprioception, and action token is assigned a spatiotemporal position (x,y,z,h) and attended over with a 4D rotary positional encoding ( positional encoding axis). An N -block DiT, shared across all variants, cross-attends from proprioception/action tokens to image tokens and self-attends over the action chunk.
Fig. 3: Camera Distribution. We visualize the camera distribution ( green ), the poses of the cameras used in the real world ( blue ) (Sec. V-C ), and the poses used for evaluating out-of-distribution generalization ( orange ) (Sec. V-D ).
Fixed camera
Randomized cameras
Articulated Objects (Norm. Open)
Do You Know Where Your Camera Is? (Success %)
Method
Cabinet
Cabinet
Pick Place Can
Square Assembly
Avg.
Plücker Raymaps
63.6 ± 2.9
39.5 ± 1.4
68.0 ± 0.0
22.7 ± 1.2
43.4
Canonical Views
59.1 ± 3.2 ∗
52.8 ± 3.2
62.0 ± 0.0
32.0 ± 0.0
48.9
4D RoPE ( A gp )
69.1 ± 0.8
64.4 ± 0.8
96.7 ± 3.1
29.3 ± 3.1
63.5
TABLE I: We compare various approaches to incorporating camera information for viewpoint generalization on opening an articulated cabinet and on the “Do You Know Where Your Camera Is?” benchmark. Unless marked as fixed camera, cameras are randomized during training and evaluation. Mean ± std. over 3 seeds. Avg. is computed over Cabinet, Pick Place Can, and Square Assembly. Best: shaded , second-best: underline . ∗ Under the fixed camera setting, the canonical views coincide with the fixed camera views, so Canonical Views receives the same inputs and uses the same architecture as the “w/ Sinusoidal” ablation (Table II ); the two are identical, and we report the same result.
Fixed camera
Randomized cameras
Articulated Objects (Norm. Open)
Do You Know Where Your Camera Is? (Success %)
Method
Cabinet
Cabinet
Pick Place Can
Square Assembly
Avg.
Token compression:
Diffusion Policy
56.1 ± 3.2
33.2 ± 4.3
1.3 ± 1.2
2.7 ± 1.2
12.4
w/ Spatial Softmax
55.2 ± 8.4
33.4 ± 0.2
56.0 ± 5.3
22.7 ± 1.2
37.4
w/ Max Pooling
60.0 ± 3.9
35.1 ± 1.9
61.3 ± 4.2
22.7 ± 3.1
39.7
TABLE II: Ablations of each design choice, varied relative to A gp : visual token compression, visual encoder, and positional encoding. The first task column uses a fixed camera; the remaining task columns use cameras randomized during training and evaluation. Mean ± std. over 3 seeds. Avg. is computed over the three randomized-camera tasks: Cabinet, Pick Place Can, and Square Assembly. A gp retains all image tokens, uses a cross-view finetuned DINOv2 encoder, and uses robot-frame 4D RoPE; rows marked “w/” replace a single component of A gp . Best: shaded , second-best: underline .
Figure 6
Fig. 6: Sim-to-real transfer on microwaves. Policies trained in simulation and evaluated zero-shot in real world under randomly sampled camera poses. Performance averaged over 30 trials across 3 camera configurations (10 trials per camera configuration).
Method
Cabinet (Norm. Open)
Plücker Raymaps
29.8 ± 5.2
Canonical Views
45.6 ± 2.3
A gp
51.4 ± 5.2
TABLE III: Evaluation on the Cabinet task with out-of-distribution cameras . Best: shaded , second-best: underline .
Method
Cabinet (Norm. Open)
Fixed camera
Randomized cameras
DP3 [ 3 ] (point cloud, GT seg.)
50.9 ± 2.0
43.9 ± 1.4
A gp
69.1 ± 0.8
64.4 ± 0.8
TABLE IV: Comparison against DP3 on the Cabinet task. Best: shaded .
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Range / Distribution
Object (microwave) appearance
Material type
50% PBR texture / 50% solid color
Solid color (per channel)
U(0.0,1.0)
Texture scale
U(0.7,5.0)
Roughness
U(0.0,1.0)
Metallic
U(0.0,1.0)
Appendix
TABLE V: Visual domain randomization applied in IsaacLab for sim-to-real transfer. Visual appearance, lighting, and camera intrinsics/extrinsics are randomized on top of the demonstration trajectories.
Parameter
Range / Distribution
Object size ratio
U(0.7,0.9)× base size
Object position (X, Y)
U(−0.1,0.1) m about base, per axis
Object yaw
base +U(−30∘,30∘)
Door initial joint angle
U(0,0.2)× joint range
Robot arm initial joints
per-joint U within Franka limits, shrunk 20% each side
Appendix
TABLE VI: Randomization applied during demonstration generation for the Cabinet (sim) and microwave (real) opening tasks. Object pose, scale, and initial robot/object configurations are sampled per trajectory.
Parameter
Range / Distribution
Distance to target
U(0.8,1.2)m+N(0,0.05)m
Look-at target
(0.7,0.0,0.4)+N(0,0.05) per axis
Shoulder-cam pitch
U(−20∘,20∘)
Shoulder-cam roll
U(−40∘,0∘)
Shoulder-cam yaw
U(−160∘,−110∘) or U(−70∘,−20∘)
Appendix
TABLE VII: Camera pose randomization for both cabinet and microwave tasks. All parameters are sampled per trajectory.
Fig. 7: Visualization of the Cabinet task dataset. Each trajectory is rendered from a random camera pose.
Fig. 8: Visualization of the microwave opening task dataset. Each trajectory is rendered from a random camera pose in IsaacLab.
End-to-end visuomotor policies provide little opportunity for humans to understand or correct the policy's visual attention. We propose GuidedAttention, a visuomotor imitation learning framework that introduces interpretable and correctable visual attention as an explicit intermediate representation. Task-relevant attention keypoints are predicted from camera images and condition a diffusion-based action policy. Users can inspect and optionally correct selected keypoints once at rollout initialization, after which the corrected attention is automatically propagated throughout execution by a tracking module. Experiments in simulation and the real world demonstrate that GuidedAttention consistently improves robot manipulation performance, particularly under positional and appearance out-of-distribution (OOD) conditions. https://mmurooka.github.io/guided-attention-project-page
Masaki Murooka, Ryoichi Nakajo, Keisuke Shirai +4
CNRS-AIST JRL (Joint Robotics Laboratory), IRL, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8560, Japan · Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST), 2-3-26 Aomi, Koto-ku, Tokyo 135-0064, Japan
Visual imitation learning enables robots to acquire visuomotor skills directly from images, yet RGB observations lack explicit geometric cues, making learned policies brittle to camera perturbations. To address this, we propose Ray-conditioned Vision Transformer Encoder (RayViT), a lightweight architecture that injects camera geometry into pretrained ViT backbones. RayViT represents camera geometry as a Plücker ray map, patchifies it into ray features, and uses gated cross-attention to produce a ray-conditioned class token. These ray features are added as dense positional embeddings, while the ray class token replaces the original ViT class token to provide a geometry-aware summary representation. We combine this approach with an auxiliary cosine similarity loss to consistently improve the performance and robustness for geometry-aware tokens. Experiments on sim- and real-robot tasks demonstrate that RayViT improves robustness by approximately 13 percentage points under camera perturbations in multi-task RoboCasa benchmark and by 1.78 average completed stages in real-world multi-task success rate compared to baselines.
Qian Wang, Longrui Chen, Peiran Sun +8
Karlsruhe Institute of Technology, Germany · University of Leeds, United Kingdom
Understanding the generalization capabilities of visuomotor policies is essential in the development of capable robotic agents. Generalizable models learn structures that transfer across domains. However, in practice, visuomotor policies test performance by interpolation on known distributions using unstructured domain shifts (e.g. lighting, clutter, diverse objects). We argue that to measure generalization capabilities we must instead test the inductive capacity of policies on progressively harder, out-of-distribution task variants. We call this inductive generalization, drawing directly on how axis-based evaluation has revealed inherent generalization limitations in language models (e.g. sequence length, counting) arXiv:2502.00197 . We provide a reusable and formal evaluation protocol for measuring inductive generalization in any manipulation policy, and establish baselines showing that existing paradigms fail this test; e.g. SoTA Vision-Language-Action models and find that policies that appear to generalize to prior domain shifts (distractors, etc) fail inductive generalization tests. These results expose a class of learning challenges orthogonal to those addressed by data and model scaling in robot learning, yet are imperative to solve in order to realize general purpose robots.