Vision-Language-Action (VLA) models have demonstrated strong performance in robotic manipulation, yet most rely primarily on 2D visual observations and language instructions, leaving scene geometry and obstacle relationships to be inferred implicitly. This limitation is particularly problematic in cluttered environments, where small spatial errors can lead to unintended contacts or collisions. We propose MOSAIC, a modular spatial- and obstacle-aware framework that can be integrated with pretrained VLA policies to improve object-level scene understanding and action generation. Our method augments the VLA backbone with an auxiliary semantic grounding task that predicts role-specific masks for the target object, destination region, and surrounding obstacles. We further combine the target and destination masks with depth estimated from 3D scene reconstruction to construct role-specific geometric tokens that are injected directly into the action expert. In addition, we propose obstacle-aware attention supervision that emphasizes obstacles near the robot body during task execution. The resulting attention features are subsequently injected into later action-expert layers to further guide action generation. Together, these components provide explicit semantic, geometric, and obstacle-aware cues while preserving the underlying pretrained VLA architecture. We evaluate our framework on pretrained VLA model using the RoboPRO benchmark, including visual perturbation tests, and further validate it on real-world manipulation tasks. MOSAIC improves the success rate by 5.1% in cluttered scenes, and by 9.7% under visual perturbations, with gains of up to 25.0% in the most challenging environment. We will release our code and dataset upon acceptance at the following Github page: https://mmahdavian.github.io/MOSAIC/
Figures & tables
Fig. 1 : MOSAIC overview . A pretrained VLA is augmented with role-specific masks, object-level depth tokens, and obstacle-focused attention guidance.
Fig. 2 : MOSAIC Model Structure. (a) A vision–language backbone encodes observations, instructions, and robot state, with role tokens supporting semantic mask prediction. (b) The action expert attends to the shared prefix and produces learned attention maps. (c) Object-level depth is extracted using the VGGT- Ω combined with the predicted masks. The depth tokens are generated and injected into the action expert final layers. (d) Contact-relevant expert targets supervise attention through forward KL divergence during training. (e) Predicted attention maps are averaged, detached, and injected through residual cross-attention into late action layers. (f) Flow matching generates actions for robot execution.
Fig. 3 : The pipeline for groundtruth attention map generation. (a) The original RGB countertop camera view. (b) Per-object contact disk generation. The closest point of each object’s bounding box is projected onto pixel space. Disks are centered at these points to highlight potential contact zones. (c) Proximity weight map β . Higher weight for objects closer to the robot body, less weight for farther objects, zero weight for very distant objects. (d) Obstacle contact mask. Using the proximity weight map β to filter out irrelevant objects. (e) Groundtruth attention heatmap H . Obstacle contact mask convolved with a Gaussian kernel with standard deviation σ to represent the spatial distribution of visual attention. (f) Patch-wise distributed groundtruth attention map. Matches the granularity of the action expert’s attention modules.
Fig. 4 : Task-conditioned visual annotations for (a) Simulation and (b) Real-world data. In (a), rows show RGB images, mask overlays, and binary target (T), destination (D), and obstacle (O) masks. Examples illustrate an object destination (a-1) , anchor-relative placement (a-2) , a table-region destination with 5 cm clearance (a-3) , and pick-only manipulation without a destination (a-4) . Simulation overlays use red/green/blue for T/D/O. In (b), examples (b-1) to (b-4) show four real-world tasks with one to four added distractors.
All
Kitchen-L
Kitchen-S
Office
Study
Scene
π0.5 variant
SR ↑
HSR ↑
CR ↓
SR ↑
HSR ↑
CR ↓
SR ↑
HSR ↑
CR ↓
SR ↑
HSR ↑
CR ↓
SR ↑
HSR ↑
CR ↓
Clean
RoboPRO
66.0
61.0
11.9
65.8
56.0
21.2
68.1
61.5
12.4
54.8
53.2
8.5
75.9
73.6
5.4
MOSAIC (Masks w/o Obst.)
67.8
62.1
16.0
69.0
56.0
27.0
69.6
63.6
13.0
54.2
53.2
14.5
79.0
76.2
9.0
MOSAIC (Masks)
67.0
61.8
16.3
68.2
57.2
26.0
67.5
62.0
14.8
60.2
58.0
16.5
72.1
70.0
7.4
MOSAIC (Masks+Attn)
67.5
62.2
15.6
68.5
57.2
26.5
71.0
64.4
14.8
53.0
51.8
15.0
77.9
75.9
5.9
MOSAIC (Masks+Attn+Depth)
70.1
64.8
14.5
74.0
61.8
23.0
71.8
66.0
12.9
58.2
57.0
16.2
76.7
74.9
5.4
TABLE I: Per-environment ablation results for π0.5 on RoboPRO. We evaluate the contribution of each introduced module. Best per column within each scene in bold , second best underlined . All variants are evaluated on the same episodes.
Fig. 5 : Average attention maps from the first three action-expert layers for the RoboPRO-finetuned base π0.5 model (top) and our final model (bottom) across two manipulation actions. Blue regions indicate higher attention. Our model consistently focuses on potential collision regions, while the base π0.5 exhibits less obstacle-aware attention.
All
Kitchen-L
Kitchen-S
Office
Study
Model
SR ↑
HSR ↑
CR ↓
SR ↑
HSR ↑
CR ↓
SR ↑
HSR ↑
CR ↓
SR ↑
HSR ↑
CR ↓
SR ↑
HSR ↑
CR ↓
π0.5 (RoboPRO)
65.2
63.4
6.6
66.7
61.7
15.0
83.9
83.9
0.0
51.7
50.0
8.3
58.8
58.8
2.0
π0.5 (MOSAIC)
74.9
72.7
6.2
73.3
70.0
11.7
83.9
83.9
0.0
76.7
71.7
10.0
64.7
64.7
2.0
TABLE II: Robustness to visual perturbation (lighting, blur and per-frame pixel shift) on clean RoboPRO scenes. Best results within each model pair are shown in bold , including ties.
Fig. 6 : Real-world experiments. The model is able to perform different tasks while generating the masks and conditioning on them during action generation.
Vision-Language-Action (VLA) models demonstrate impressive zero-shot generalization but frequently suffer from a "Precision-Reasoning Gap" in cluttered environments. This failure is driven by background-induced feature dilution, where high-frequency semantic noise corrupts the geometric grounding required for precise manipulation. To bridge this gap, we propose Concept-Gated Visual Distillation (CGVD), a training-free, model-agnostic inference framework that stabilizes VLA policies. CGVD operates by parsing instructions into safe and distractor sets, utilizing a two-layer target refinement process--combining cross-validation and spatial disambiguation--to explicitly penalize false positives and isolate genuine manipulation targets. We then process the scene via Fourier-based inpainting, generating a clean observation that actively suppresses semantic distractors while preserving critical spatial geometry and visual proprioception. Extensive evaluations in highly cluttered manipulation tasks demonstrate that CGVD prevents performance collapse. In environments with dense semantic distractors, our method significantly outperforms state-of-the-art baselines, achieving a 77.5% success rate compared to the baseline's 43.0%. By enforcing strict attribute adherence, CGVD establishes inference-time visual distillation as a critical prerequisite for robust robotic manipulation in the clutter.
Sangmim Song, Sarath Kodagoda, Marc Carmichael +1
University of Technology Sydney, NSW, Australia · Western Sydney University, NSW, Australia
Vision--Language--Action (VLA) policies have shown strong progress in mapping language instructions and visual observations to robotic actions, yet their reliability degrades in cluttered scenes with distractors. By analyzing failure cases, we find that many errors do not arise from infeasible motions, but from instance-level grounding failures: the policy often produces a plausible grasp trajectory that lands slightly off-target or even on the wrong object instance. To address this issue, we propose TAG (Target-Agnostic Guidance), a simple inference-time guidance mechanism that explicitly reduces distractor- and appearance-induced bias in VLA policies. Inspired by classifier-free guidance (CFG), TAG contrasts policy predictions under the original observation and an object-erased observation, and uses their difference as a residual steering signal that strengthens the influence of object evidence in the decision process. TAG does not require modifying the policy architecture and can be integrated with existing VLA policies with minimal training and inference changes. We evaluate TAG on standard manipulation benchmarks, including LIBERO, LIBERO-Plus, and VLABench, where it consistently improves robustness under clutter and reduces near-miss and wrong-object executions.
Jiaying Zhou, Zhihao Zhan, Ruifeng Zhai +5
Sun Yat-sen University · Guangdong Key Lab of Big Data Analysis & Processing · X-Era AI Lab
Vision-Language-Action (VLA) models leverage the rich world knowledge of pretrained vision-language models (VLMs) to enable instruction-following robotic manipulation. However, the structural mismatch between VLM semantic spaces and embodied control policies often hinders the learning of precise perception-action mappings. To address this challenge, we propose AffordanceVLA, a scalable, unified framework that introduces structured affordance learning as a task-oriented and object-centric intermediate representation to establish a more precise and robust perception-action mapping. Specifically, we progressively model manipulation priors through three complementary components - Which2Act, Where2Act, and How2Act - which jointly produce spatially grounded, semantically conditioned, and action-coupled intermediate representations, thereby naturally bridging vision, language, and action. We integrate these modules into a Mixture-of-Transformer (MoT) architecture and adopt a three-stage progressive training curriculum that leverages 8.8M affordance-centric grounding and pointing samples, translating scalable vision-language supervision into improved robotic manipulation. To overcome the scarcity of dense affordance labels in robotic datasets, we also develop a robust automated data augmentation pipeline. Extensive experiments on ID & OOD simulation and real-world demonstrate that AffordanceVLA outperforms strong VLA baselines across diverse manipulation scenarios, validating structured affordance learning as a scalable and robust bridge that integrates multi-source vision-language supervision while connecting pretrained VLM semantics to embodied robot control.
Qize Yu, Jiadi You, Yuran Wang +12
Peking University · Hong Kong University of Science and Technology (Guangzhou) · The Chinese University of Hong Kong +1