Organizations: Harbin Institute of Technology, Shenzhen, China · DaiMon Robotics, China · Nanjing University, Nanjing, China · Renmin University of China, Beijing, China
Vision-Language-Action (VLA) models improve action generation by conditioning policies on rich vision-language information. However, current auto-regressive policies are constrained by three bottlenecks: (1) architectural bias drives models to overlook visual details, (2) an excessive number of visual tokens makes attention difficult to focus on the correct regions, and (3) task-irrelevant visual information introduces substantial noise - together severely impairing the quality of action. In this paper, we investigate how to effectively utilize different visual representations for action generation. To this end, we first empirically validate the above issues and show that VLA performance is primarily limited by how visual information is utilized, rather than by the quality of visual representations. Based on these insights, we introduce FocusVLA, a novel paradigm that directs the model's attention to task-relevant visual regions to effectively bridge vision to action. Specifically, we first propose Modality Cascaded Attention to eliminate shortcut pathways, thereby compelling VLA models to rely on task-relevant visual details for action generation. Furthermore, we propose Focus Attention, which dynamically selects task-relevant visual patches to control information quantity while explicitly modulating their influence to suppress task-irrelevant noise. Extensive experiments on both simulated and real-world robotic benchmarks demonstrate that FocusVLA not only effectively leverages visual details to perform dexterous manipulations, but also substantially improves performance and accelerates convergence across a variety of tasks.
Figures & tables
Figure 1: Illustration of structural limitations in existing auto-regressive Vision-Language-Action (VLA) models and the motivation of FocusVLA. OpenVLA-OFT suffers from the lack of direct utilization of visual tokens, limiting its ability to ground actions in fine-grained visual details. VLA-Adapter’s mixed attention introduces architectural shortcuts that enable the model to bypass concrete visual details by favoring the easier action-query pathway, leading to imprecise manipulation. Furthermore, it introduces a near-zero gating factor that substantially suppresses visual signals. In contrast, FocusVLA uses Modality Cascaded Attention to enforce sequential modality interaction, ensuring the model relies on task-relevant visual details before action reasoning and eliminating shortcut pathways. In addition, we propose Focus Attention to suppress task-irrelevant information, leading to precise and robust manipulation.
Figure 2: Visualization of attention maps on LIBERO simulation (left) and real-world environments (right). We extract the attention scores from the last layer, average them across attention heads and queries, and project them back onto the image plane for visualization. Specifically, for VLM, we use the attention from the action query to visual tokens, while for the policy, we use the attention from the action latent to visual tokens. VLA-Adapter exhibits highly scattered and distracted attention patterns, largely focusing on task-irrelevant regions due to structural shortcuts, thereby failing to capture concrete visual details. In contrast, FocusVLA produces concentrated and task-aligned attention maps that consistently focus on contact regions and manipulation targets. This targeted attention enables more precise action generation across both simulated and real-world settings.
Figure 3: Architectures of the four proposed policy variants. (a) Vanilla: A baseline utilizing visual tokens without constraints. (b) Pooling: Patch-level optimization by reducing visual token count via 2×2 pooling. (c) 1-param gate: Channel-level optimization that attenuates visual signal intensity using a single-parameter gate. (d) Cascaded attention: Structure-level optimization that alters feature interaction patterns through cascaded multi-head attention.
Figure 4: Success Rates of different policy architectures on the LIBERO-Long. We evaluate four different structures across three distinct visual representations as detailed in Section 3.2 . The experimental results reveal that existing policies often suffer from three critical biases: token quantity imbalance, low signal-to-noise ratio, and structural bias. By systematically addressing these issues through our proposed constraints, we achieve consistent and significant performance improvements across all visual representations. These findings demonstrate that VLA performance is primarily limited by how visual information is utilized, rather than by the quality of visual representations. “VLM” denotes the output visual features from the PrismaticVLM trained on Qwen2.5-0.5B [ 31 ] , while “DS” denotes the combined features of DINOv2 and SigLIP [ 21 , 34 ] . The red dashed line indicates the performance of a variant that does not use VLM visual features, relying solely on action queries.
Figure 5: The architecture of FocusVLA. Our policy explicitly addresses visual utilization bottlenecks through two core components: (1) Cascaded Attention: By enabling the action latent to query each modality independently, the model is forced to focus on and extract only the most task-relevant visual details, ensuring a more targeted feature utilization; and (2) Focus Attention: To resolve the quantity and quality limitations, this component employs patch-level pruning to discard irrelevant tokens and channel-level gated suppression to mitigate feature noise.
Method
Param
Spatial
Object
Goal
Long
Avg.
Multi-weights (one policy per task)
OpenVLA [ 14 ]
7B
84.7
88.4
79.2
53.7
76.5
OpenVLA-OFT [ 13 ]
7B
97.6
98.4
97.9
94.5
97.1
UniVLA [ 5 ]
7B
96.5
96.8
95.6
92.0
95.2
Spatial Forcing [ 16 ]
7B
99.4
99.6
98.8
96.0
98.5
SpatialVLA [ 23 ]
4B
88.2
89.9
78.6
55.5
78.1
Table 1: Comparisons with state-of-the-art methods on LIBERO benchmark.
Figure 6: Comparisons with state-of-the-art methods on RoboTwin 2.0 benchmark.
Attention Pattern
Patch-level Focus
Channel-level Focus
Visual Representation
Spatial SR (%)
Object SR (%)
Goal SR (%)
Long SR (%)
AVG. (%)
Mixed
512
w/o gate
VLM
94.4
95.6
93.2
91.0
93.6
Cascaded
512
w/o gate
VLM
98.0
98.6
96.2
95.0
97.0
Cascaded
128
w/o gate
VLM
96.8
99.0
95.4
93.8
96.3
Cascaded
256
w/o gate
VLM
99.0
99.2
98.2
95.4
98.0
Cascaded
384
w/o gate
VLM
98.4
99.6
97.8
95.2
97.8
Cascaded
512
w/o gate
VLM
98.0
98.6
96.2
95.0
97.0
Table 2: Component Analysis on LIBERO benchmark under the multi-weights setting. *DS denotes the combination of DINOv2 and SigLIP.
Figure 7: Comparison of training efficiency between VLA-Adapter and FocusVLA.
Figure 8: Visualization of Focus Attention. (a) Patch-level focus. In the primary view, background regions are largely removed with key objects retained, maintaining sufficient spatial cues for policy planning. In the wrist view, representations only concentrate on the gripper and target object, enabling precise grasp execution. (b) Channel-level focus. Without gating, noisy information may lead to instruction-inconsistent actions; with gating, irrelevant signals are suppressed, yielding more accurate and instruction-consistent behavior.
Figure 9: Experimental settings and results in real-world scenarios.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Backbone
Qwen2.5-0.5B
Number of layers (τ/M)
24
Hidden size
896
Attention heads
8
Utilization of VLM features
layers 1–24
Utilization of DINOv2 + SigLIP features
last layer
Appendix
Table 3: Implementation details and hyperparameters of FocusVLA.
Suite
1
2
3
4
5
6
7
8
9
10
Avg. ↑
Spatial
100.0
100.0
100.0
100.0
100.0
100.0
100.0
96.0
100.0
100.0
99.6
Object
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Goal
100.0
96.0
98.0
100.0
100.0
100.0
98.0
100.0
98.0
98.0
98.8
Long
96.0
98.0
100.0
100.0
98.0
100.0
94.0
98.0
84.0
94.0
96.2
Appendix
Table 4: Task-level success rates (%) on the LIBERO benchmark.
Task
DP
π0
VLA-Adapter
FocusVLA
Easy
Hard
Easy
Hard
Easy
Hard
Easy
Hard
Beat Block Hammer
42
0
43
21
89
43
93
54
Blocks Ranking RGB
0
0
19
5
35
8
43
11
Click Alarmclock
61
5
63
11
44
4
81
20
Hanging Mug
8
0
11
3
8
1
18
5
Move Pillbottle Pad
1
0
21
1
10
1
20
2
Appendix
Table 5: Detailed success rates (%) on RoboTwin benchmark.
Figure 10: Visualizations of FocusVLA executing tasks in the LIBERO benchmark.
Figure 11: Comparison between FocusVLA and VLA-Adapter on the same LIBERO task.
Figure 12: Examples of unexpected situations during task execution in LIBERO simulation, including two failure trajectories and one successful recovery case.
Figure 13: Visualizations of FocusVLA executing tasks in the RoboTwin benchmark.
Figure 14: Comparison between FocusVLA and VLA-Adapter on the same RoboTwin task.
Figure 15: Failure cases on the RoboTwin benchmark.
Figure 16: Visualizations of FocusVLA executing tasks in the real-world experiments.
Figure 17: Comparison between FocusVLA and VLA-Adapter on the same real-world task.
Figure 18: Failure cases on the real-world experiments.
Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for robotic manipulation, in which reliable action prediction critically depends on accurately interpreting and integrating visual observations conditioned on language instructions. Although recent works have sought to enhance the visual capabilities of VLA models, most approaches treat the LLM backbone as a black box, providing limited insight into how visual information is grounded into action generation. Therefore, we perform a systematic analysis of multiple VLA models across different action-generation paradigms and observe that sensitivity to visual tokens progressively decreases in deeper layers during action generation. Motivated by this observation, we propose DeepVision-VLA, built on a Vision-Language Mixture-of-Transformers (VL-MoT) framework. This framework enables shared attention between the vision foundation model and the VLA backbone, injecting multi-level visual features from the vision expert into deeper layers of the VLA backbone to enhance visual representations for precise and complex manipulation. In addition, we introduce Action-Guided Visual Pruning (AGVP), which leverages shallow-layer attention to prune irrelevant visual tokens while preserving task-relevant ones, reinforcing critical visual cues for manipulation with minimal computational overhead. DeepVision-VLA outperforms prior state-of-the-art methods by 9.0% and 7.5% on simulated and real-world tasks, respectively, providing new insights for the design of visually enhanced VLA models.
Yulin Luo, Hao Chen, Zhuangzhe Wu +10
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The Chinese University of Hong Kong · Simplexity Robotics
A fine-tuned Vision-Language-Action (VLA) policy can easily place the alphabet soup in the basket on command, just as it saw during fine-tuning. However, if the basket is shifted five centimeters to the left, it drops the soup off the table. We developed an interpretability pipeline to investigate what such a policy is computing. Injecting task A's activations into the action expert while the policy looks at task B's scene makes pi0.5 execute task A's motor trajectory in 99.6% of episodes (n=1,968). The injected program is bound to absolute workspace coordinates and ignores the visible scene, which provides a mechanistic account of the perturbation brittleness that concurrent benchmarks report. The binding holds on a physical UR10e: directed replay redirects the commanded task in 26/26 rollouts in one direction and 18/26 in the other, with no-injection and random-direction controls at the floor, and once the source object is moved 10--15,cm the arm grasps and releases at the recorded coordinates in 50/50 trials, picking up the wrong block when one sits there and closing on empty air when none does. Applying the same interventions to six VLAs on LIBERO, MetaWorld, SimplerEnv, and ALOHA over 420{,}000+ rollouts yields three further findings: every architecture encodes the instruction, and four of the five weight it below vision whenever the scene alone identifies the goal; under the standard replacement write-back the SAE pooling that preserves behavior is architecture-specific (pi0.5 per-token, X-VLA mean-pool, SmolVLA indifferent), and goal selection and motor generation separate into VLM and expert pathways wherever the two are distinguishable. We release Action Atlas (https://action-atlas.com), an interactive platform covering all 520+ trained SAEs, 79 identified concepts, and the scored UR10e rollouts. Code: https://github.com/CWRU-AISM/action-atlas
Vision-Language-Action (VLA) models aim for general robot learning by aligning action as a modality within powerful Vision-Language Models (VLMs). Existing VLAs rely on end-to-end supervision to implicitly enable the action decoding process to learn task-relevant features. However, without explicit guidance, these models often overfit to spurious correlations, such as visual shortcuts or environmental noise, limiting their generalization. In this paper, we introduce GuidedVLA, a framework designed to manually guide the action generation to focus on task-relevant factors. Our core insight is to treat the action decoder not as a monolithic learner, but as an assembly of functional components. Individual attention heads are supervised by manually defined auxiliary signals to capture distinct factors. As an initial study, we instantiate this paradigm with three specialized heads: object grounding, spatial geometry, and temporal skill logic. Across simulation and real-robot experiments, GuidedVLA improves success rates in both in-domain and out-of-domain settings compared to strong VLA baselines. Finally, we show that the quality of these specialized factors correlates positively with task performance and that our mechanism yields decoupled, high-quality features. Our results suggest that explicitly guiding action-decoder learning is a promising direction for building more robust and general VLA models.
Xiaosong Jia, Bowen Yang, Zuhao Ge +17
Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI · Shanghai Jiao Tong University +1