An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learning

Mino Nakura; Sriram Krishna; Yufei Wang; Shubham Tulsiani; Zackory Erickson; David Held

cs.ROSep 26, 2026

An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learning

Authors: Mino Nakura, Sriram Krishna, Yufei Wang, Shubham Tulsiani, Zackory Erickson, David Held

Organizations: Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA.

Abstract

Visual imitation learning is a promising approach to training robot manipulation policies capable of completing a wide variety of tasks. However, policies today remain brittle to viewpoint perturbations, making deployment in diverse environments a challenge. We present a controlled empirical study of which design choices allow visuomotor policies to generalize across viewpoints. We find that viewpoint generalization improves when dense visual tokens are retained and the action head participates in geometric reasoning. On a suite of simulated tasks that span a wide range of camera poses, we show that these design choices yield a policy that remains performant across viewpoints. As a practical consequence, a policy trained with these design choices also transfers zero-shot from simulation to the real world under random camera configurations.

Figures & tables

Fig. 1: Top Left: We sample a wide distribution of camera poses in simulation. Middle: We generate demonstrations under these randomized views and train a policy that grounds visual, proprioception, and action tokens in a shared robot frame. Right: The policy transfers zero-shot to a real robot under random camera configurations.

Fig. 2: Base policy and design axes , shown in the A gp configuration. A set of calibrated RGB views are encoded by a backbone initialized from DINOv2 applying within-view self-attention and cross-view global self-attention, producing dense multiview image tokens ( visual encoder and token compression axes). Every image, proprioception, and action token is assigned a spatiotemporal position $(x,y,z,h)$ and attended over with a 4D rotary positional encoding ( positional encoding axis). An $N$ -block DiT, shared across all variants, cross-attends from proprioception/action tokens to image tokens and self-attends over the action chunk.

Fig. 3: Camera Distribution. We visualize the camera distribution ( green ), the poses of the cameras used in the real world ( blue ) (Sec. V-C ), and the poses used for evaluating out-of-distribution generalization ( orange ) (Sec. V-D ).

	Fixed camera	Randomized cameras
	Articulated Objects (Norm. Open)		Do You Know Where Your Camera Is? (Success %)
Method	Cabinet	Cabinet	Pick Place Can	Square Assembly	Avg.
Plücker Raymaps	63.6 $\pm$ 2.9	39.5 $\pm$ 1.4	68.0 $\pm$ 0.0	22.7 $\pm$ 1.2	43.4
Canonical Views	59.1 $\pm$ 3.2 ∗	52.8 $\pm$ 3.2	62.0 $\pm$ 0.0	32.0 $\pm$ 0.0	48.9
4D RoPE ( A gp )	69.1 $\pm$ 0.8	64.4 $\pm$ 0.8	96.7 $\pm$ 3.1	29.3 $\pm$ 3.1	63.5

TABLE I: We compare various approaches to incorporating camera information for viewpoint generalization on opening an articulated cabinet and on the “Do You Know Where Your Camera Is?” benchmark. Unless marked as fixed camera, cameras are randomized during training and evaluation. Mean

\pm

std. over 3 seeds. Avg. is computed over Cabinet, Pick Place Can, and Square Assembly. Best: shaded , second-best: underline . ∗ Under the fixed camera setting, the canonical views coincide with the fixed camera views, so Canonical Views receives the same inputs and uses the same architecture as the “w/ Sinusoidal” ablation (Table II ); the two are identical, and we report the same result.

Token compression:
	Fixed camera	Randomized cameras
	Articulated Objects (Norm. Open)		Do You Know Where Your Camera Is? (Success %)
Method	Cabinet	Cabinet	Pick Place Can	Square Assembly	Avg.
Diffusion Policy	56.1 $\pm$ 3.2	33.2 $\pm$ 4.3	1.3 $\pm$ 1.2	2.7 $\pm$ 1.2	12.4
w/ Spatial Softmax	55.2 $\pm$ 8.4	33.4 $\pm$ 0.2	56.0 $\pm$ 5.3	22.7 $\pm$ 1.2	37.4
w/ Max Pooling	60.0 $\pm$ 3.9	35.1 $\pm$ 1.9	61.3 $\pm$ 4.2	22.7 $\pm$ 3.1	39.7

TABLE II: Ablations of each design choice, varied relative to A gp : visual token compression, visual encoder, and positional encoding. The first task column uses a fixed camera; the remaining task columns use cameras randomized during training and evaluation. Mean

\pm

std. over 3 seeds. Avg. is computed over the three randomized-camera tasks: Cabinet, Pick Place Can, and Square Assembly. A gp retains all image tokens, uses a cross-view finetuned DINOv2 encoder, and uses robot-frame 4D RoPE; rows marked “w/” replace a single component of A gp . Best: shaded , second-best: underline .

Fig. 6: Sim-to-real transfer on microwaves. Policies trained in simulation and evaluated zero-shot in real world under randomly sampled camera poses. Performance averaged over 30 trials across 3 camera configurations (10 trials per camera configuration).

Method	Cabinet (Norm. Open)
Plücker Raymaps	29.8 $\pm$ 5.2
Canonical Views	45.6 $\pm$ 2.3
A gp	51.4 $\pm$ 5.2

TABLE III: Evaluation on the Cabinet task with out-of-distribution cameras . Best: shaded , second-best: underline .

Method	Cabinet (Norm. Open)
	Fixed camera	Randomized cameras
DP3 [ 3 ] (point cloud, GT seg.)	50.9 $\pm$ 2.0	43.9 $\pm$ 1.4
A gp	69.1 $\pm$ 0.8	64.4 $\pm$ 0.8

TABLE IV: Comparison against DP3 on the Cabinet task. Best: shaded .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Object (microwave) appearance
Parameter	Range / Distribution
Material type	$50\%$ PBR texture / $50\%$ solid color
Solid color (per channel)	$\mathcal{U}(0.0,1.0)$
Texture scale	$\mathcal{U}(0.7,5.0)$
Roughness	$\mathcal{U}(0.0,1.0)$
Metallic	$\mathcal{U}(0.0,1.0)$

Appendix

TABLE V: Visual domain randomization applied in IsaacLab for sim-to-real transfer. Visual appearance, lighting, and camera intrinsics/extrinsics are randomized on top of the demonstration trajectories.

Parameter	Range / Distribution
Object size ratio	$\mathcal{U}(0.7,0.9)\times$ base size
Object position (X, Y)	$\mathcal{U}(-0.1,0.1)$ m about base, per axis
Object yaw	base $+\ \mathcal{U}(-30^{\circ},30^{\circ})$
Door initial joint angle	$\mathcal{U}(0,0.2)\times$ joint range
Robot arm initial joints	per-joint $\mathcal{U}$ within Franka limits, shrunk $20\%$ each side

Appendix

TABLE VI: Randomization applied during demonstration generation for the Cabinet (sim) and microwave (real) opening tasks. Object pose, scale, and initial robot/object configurations are sampled per trajectory.

Parameter	Range / Distribution
Distance to target	$\mathcal{U}(0.8,1.2)\,\text{m}+\mathcal{N}(0,0.05)\,\text{m}$
Look-at target	$(0.7,0.0,0.4)+\mathcal{N}(0,0.05)$ per axis
Shoulder-cam pitch	$\mathcal{U}(-20^{\circ},20^{\circ})$
Shoulder-cam roll	$\mathcal{U}(-40^{\circ},0^{\circ})$
Shoulder-cam yaw	$\mathcal{U}(-160^{\circ},-110^{\circ})$ or $\mathcal{U}(-70^{\circ},-20^{\circ})$

Appendix

TABLE VII: Camera pose randomization for both cabinet and microwave tasks. All parameters are sampled per trajectory.

Fig. 7: Visualization of the Cabinet task dataset. Each trajectory is rendered from a random camera pose.

Fig. 8: Visualization of the microwave opening task dataset. Each trajectory is rendered from a random camera pose in IsaacLab.

Explore similar work

GuidedAttention: Interpretable and Correctable Visual Attention for OOD-Robust Robot Manipulation via Imitation Learning
Jul 23, 2026Masaki Murooka, Ryoichi Nakajo, Keisuke Shirai +4Visual Attention Robot Skill Learning
CNRS-AIST JRL (Joint Robotics Laboratory), IRL, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8560, Japan · Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST), 2-3-26 Aomi, Koto-ku, Tokyo 135-0064, Japan
RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning
Jul 31, 2026Qian Wang, Longrui Chen, Peiran Sun +8Visual Representation Learning Robot Skill Learning
Karlsruhe Institute of Technology, Germany · University of Leeds, United Kingdom
Inductive Generalization for Robotic Manipulation
Jun 19, 2026Annabella Macaluso, Haochen Zhang, Ishaan Masilamony +2OOD Generalization Generalization in Robotic Manipulation
Carnegie Mellon University

Jul 23, 2026J/K move · Enter open · S save

GuidedAttention: Interpretable and Correctable Visual Attention for OOD-Robust Robot Manipulation via Imitation Learning

Masaki Murooka, Ryoichi Nakajo, Keisuke Shirai, Tomohiro Motoda+3

CNRS-AIST JRL (Joint Robotics Laboratory), IRL, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8560, Japan · Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST), 2-3-26 Aomi, Koto-ku, Tokyo 135-0064, Japan

Visual Attention Robot Skill Learning Generalization in Robotic Manipulation Visuomotor Policy Learning Robotic Manipulation Robot Imitation Learning

Open paper AskPDF arXiv

End-to-end visuomotor policies provide little opportunity for humans to understand or correct the policy's visual attention. We propose GuidedAttention, a visuomotor imitation learning framework that introduces interpretable and correctable visual attention as an explicit intermediate representation. Task-relevant attention keypoints are predicted from camera images and condition a diffusion-based action policy. Users can inspect and optionally correct selected keypoints once at rollout initialization, after which the corrected attention is automatically propagated throughout execution by a tracking module. Experiments in simulation and the real world demonstrate that GuidedAttention consistently improves robot manipulation performance, particularly under positional and appearance out-of-distribution (OOD) conditions. https://mmurooka.github.io/guided-attention-project-page

Abstract

Explore similar work

GuidedAttention: Interpretable and Correctable Visual Attention for OOD-Robust Robot Manipulation via Imitation Learning

RayViT: Ray-Conditioned Visual Representations for Viewpoint-Robust Imitation Learning

Inductive Generalization for Robotic Manipulation