Visuomotor imitation policies can achieve high performance under in-distribution visual conditions yet fail when visually similar objects or receptacles are introduced. We study this behavior as a problem of conditional visual grounding: the visual target required for successful control changes with the manipulation phase and, in more complex tasks, with the observed task state. Using Action Chunking with Transformers (ACT), we systematically introduce distractor objects and receptacles with controlled color and shape similarity and localize failures to picking and placement. We find that distractor sensitivity is specific to both the type of visual similarity and the manipulation stage. Guided by this diagnosis, we evaluate distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting as complementary interventions for improving target selection while preserving spatial information required for control. These interventions substantially improve robustness in simulation and on a physical UR3e. We further examine the same failure pattern in a pretrained vision-language-action policy on a state-conditioned instrument-handling task, where the observed state of a medical instrument determines the correct destination. Together, the results show that visual distractors can cause incorrect object or destination selection even when the underlying manipulation skill remains intact, and that explicitly improving target selection can substantially recover performance across distinct visuomotor policy-learning regimes.
Figures & tables
Fig. 1: ACT-Modified architecture. The full camera observation and phase-selected visual prompt are encoded into the visual memory consumed by the ACT transformer decoder. A phase predictor selects the object or receptacle prompt according to the current manipulation phase, while robot proprioception ( qpos ) provides state conditioning. The decoder combines visual memory, learned queries, cross-attention, and positional embeddings to predict an action sequence. The action-sequence encoder and CLS latent path form the CVAE posterior during training only; during inference, future actions are unavailable and the latent is fixed to the prior mean z=0 .
Simulation (%)
UR3e (%)
Method
T1
T2
T1
T2
Standard ACT
39.5
14.0
0.0
0.0
Augmentation only
100.0
64.0
-
-
ACT-Modified
94.5
88.5
65.0
60.0
TABLE I: End-to-end success under full mixed distractors. Simulation contains 600 rollouts per cell; hardware contains 20 trials per cell. T1 uses a fixed receptacle and T2 a randomized receptacle.
Fig. 2: Robust placement requires both distractor invariance and retained target geometry. The top row compares clean and full-distractor Task 2 place-phase decoder states. The bottom row colors the corresponding representations by the target container’s left/right position, with clean and full conditions shown in a shared within-model projection. Augmentation-only ACT exhibits the strongest clean/full invariance, whereas ACT-Modified preserves substantially clearer container-position structure. The t-SNE projections are illustrative; quantitative comparisons in Table II are computed in the original feature space.
Metric
Standard
DataAug
Modified
Place shift ↓
0.1993
0.0011
0.0125
Phase retained ↑
13.1%
94.8%
108.5%
Container silhouette ↑
0.0095
0.0889
0.2037
Conditional placement ↑
33.3%
66.3%
100.0%
TABLE II: Task 2 decoder-state diagnostics. Shift is the place-phase clean-to-full cosine distance. Phase retention measures full-to-clean pick/place centroid separation. Geometry is the full-distractor container-position silhouette.
End-to-end visuomotor policies provide little opportunity for humans to understand or correct the policy's visual attention. We propose GuidedAttention, a visuomotor imitation learning framework that introduces interpretable and correctable visual attention as an explicit intermediate representation. Task-relevant attention keypoints are predicted from camera images and condition a diffusion-based action policy. Users can inspect and optionally correct selected keypoints once at rollout initialization, after which the corrected attention is automatically propagated throughout execution by a tracking module. Experiments in simulation and the real world demonstrate that GuidedAttention consistently improves robot manipulation performance, particularly under positional and appearance out-of-distribution (OOD) conditions. https://mmurooka.github.io/guided-attention-project-page
Masaki Murooka, Ryoichi Nakajo, Keisuke Shirai +4
CNRS-AIST JRL (Joint Robotics Laboratory), IRL, 1-1-1 Umezono, Tsukuba, Ibaraki 305-8560, Japan · Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST), 2-3-26 Aomi, Koto-ku, Tokyo 135-0064, Japan
Recent visual imitation learning systems have widely adopted multi-camera setups with wrist-mounted cameras as the de facto standard. However, manipulation from a single global view remains challenging, as the policy should capture fine-grained interaction details and identify task-relevant regions without local wrist views. To address this challenge, we present Spatially Conditioned Diffusion Policy (SCDP), a diffusion-based visuomotor policy that achieves precise and robust manipulation in a single-camera setting. Our key idea is that end-effector trajectories can serve as visual attention anchors that reflect task-relevant regions. Building on this idea, SCDP consists of two key components: (i) a visual encoder that produces multi-scale feature maps to capture both broader context and fine-grained visual features, and (ii) a spatial conditioning module that samples point-wise features along intermediate end-effector trajectories in the diffusion loop. Extensive simulation experiments show that SCDP consistently outperforms strong single-view baselines and achieves performance comparable to multi-camera baselines. Real-world experiments further demonstrate precise manipulation and robustness to visual distractors, highlighting the potential of single-camera imitation learning.
Seoyoon Kim, Kanghyun Kim, Dongwoo Ko +2
Korea Advanced Institute of Science and Technology (KAIST) · Neuromeka
Visual imitation learning is a promising approach to training robot manipulation policies capable of completing a wide variety of tasks. However, policies today remain brittle to viewpoint perturbations, making deployment in diverse environments a challenge. We present a controlled empirical study of which design choices allow visuomotor policies to generalize across viewpoints. We find that viewpoint generalization improves when dense visual tokens are retained and the action head participates in geometric reasoning. On a suite of simulated tasks that span a wide range of camera poses, we show that these design choices yield a policy that remains performant across viewpoints. As a practical consequence, a policy trained with these design choices also transfers zero-shot from simulation to the real world under random camera configurations.
Mino Nakura, Sriram Krishna, Yufei Wang +3
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA.