Causal Analysis of Robot Foundation Models Enables Efficiency Gains
Authors: Aiden Swann, Mathilde Papillon, Mozes Jacobs, Siddharth Boppana, Tal Haklay, Eric Bigelow, Andrew Lee, Conor Watts, Thomas Icard, Mac Schwager, Owen Lewis, Jack Merullo, Ekdeep Singh Lubana, Atticus Geiger, Thomas Fel, Matthew Kowal
Organizations: •Goodfire · aStanford University · bUniversity of California, Santa Barbara · cHarvard University
Vision-language-action (VLA) models are a promising approach to general purpose robot manipulation. While the internal computation of vision-language models, which compose the backbone of VLAs, has been studied extensively, less is known about how their perceptual and semantic representations are transformed into action in VLA models. We study this question mechanistically in three frontier VLA models (π0.5, MolmoAct2, and MolmoBot) by analyzing the causal pathways through which the input image and the instruction shape actions. Across the three models, activation patching shows that causally relevant information influences the action expert (AE) through a narrow band of mid-to-late cross-attention layers, which we term the action-conditioning interface. The content of this information changes with token and layer within the model. Early-to-mid layer representations of the target noun token determine which object is selected, whereas mid-to-late layers carry the spatial information in text tokens that redirects the movement when patched. Where spatial information is stored in prompt tokens various based on the architecture. In MolmoAct2 the model forms task-conditioned representations at the text positions that follow the instruction, whereas π0.5 additionally stores this information at image token positions. We show that these findings can be translated directly into improving model efficiency. We find that restricting MolmoBot's action expert conditioning to only 3 out of 36 layers reduces inference FLOPs by up to 33% while maintaining comparable task success with the default model across 3,000 evaluation episodes (46.7% vs. 46.9%), and the same principle carries over to real-robot deployment of π0.5.
Figures & tables
Figure 1: Activation patching reveals how a vision-language-action (VLA) model turns perception into action. ( Left ) We copy activations from a donor episode (“Pick up the banana”) into a target episode (“Pick up the apple”) with the position of the objects swapped. Visual information (purple) sits in early image tokens, semantic information (blue) in the early noun token, and geometric information (orange) in later text tokens. The action expert (AE) reads from a narrow band of later layer. ( Top right ) Schematic of these effects by layer. The action expert reads from a narrow band of layers.( Bottom right ) Keeping only this interface in MolmoBot, 3 of 36 connections , cuts compute by 33% with comparable average success (46.7% vs. 46.9%) over 3000 trials.
Figure 2: Action conditioning concentrates in model-specific layer bands. Curves show normalized trajectory shift E toward the reference trajectory (same scene other object requested) after patching one action-expert connection (left) or three consecutive connections (right, indexed by the center layer). The results shown are computed across 144,768 patches over 26 scenes via averaging comparisons B and C ( Section 4.1 ).
Pick
Pick & Place
PnP Next-To
Avg
Cross-attention layers
Success
Oracle
Final
Oracle
Final
(oracle)
All 36 (released MolmoBot-DROID)
62.0
57.4
49.6
21.2
15.1
46.9 ± 1.7
Layers 23–25, pretrained
57.7
59.2
50.7
23.3
18.0
46.7 ± 1.7
Layer 24, pretrained
63.0
53.0
45.7
20.8
15.9
45.6 ± 1.7
Layers 23–25, adapt (50K FT)
56.8
55.4
45.9
23.1
17.8
45.1 ± 1.7
Layer 24, adapt (50K FT)
54.7
54.0
44.9
24.9
19.4
44.5 ± 1.7
Table 1: Keeping only the action-conditioning interface preserves performance Success rate (%) on the MolmoBot paper’s simulation benchmarks (1,000 identical episodes per benchmark). Reading only layer 24 (1 of 36 connections) comes within 1.3 points of the released all-layer model when trained from scratch. Reading only L23–25 comes within 0.2 points and gives the best Pick & Place success of any model. ± is a 95% binomial confidence interval. Final measures success state at end of episode. Oracle terminates at the first success state. Shading ranks each column (darkest = best).
π0.5 policy
Success
Task time (s)
Original (all 18 layers)
64/100 (64%)
34.9
Restricted (layers 14–16)
67/100 (67%)
26.5
Table 2: Real-robot evaluation of π0.5 . Over 100 trials per policy on seven tabletop tasks ( Figure 9 ; per-task results in Table 6 ), restricting action-expert attention to layers 14–16 preserves task success. Task time is the median duration of successful trials for each task, averaged over the seven tasks. It measures full task execution, not inference latency.
Figure 3: Three counterfactual comparisons test hypothesis about target selection We patch activations from a donor run into a recipient run. (a) The input changes in A , B and C produce distinct causal signatures for each interchange intervention type. (b) Green arrows show the unpatched behavior; red arrows show predicted recipient behavior if the patch transfers the labeled information. These labels denote transfer hypotheses , not information uniquely isolated by single comparisons. For example, in A , patching scene-layout information through an image patch can produce the same behavioral shift as patching target-object identity through a noun patch. We therefore interpret effects jointly across the comparisons and patched token positions.
Figure 4: Backbone interventions trace the construction of the geometric variable in MolmoAct2 and π0.5 . Colors show normalized trajectory shift E toward the other-object trajectory in the original scene. Each cell averages 416 interventions: 26 original–counterfactual pairs, each tested in both directions under eight noise seeds. The visual, semantic, and geometric hypotheses predict the signatures in Figure 3(a) . We omit MolmoAct2 layers 27–35 as they have ∣E∣<0.001 .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
A
B
C
Counterfactual layout
V1
V0
V1
Counterfactual request
b
b
a
Clean original output
pL
pL
pL
Clean counterfactual output
pL
pR
pR
Visual intervention YV
pR
pL
pR
Semantic intervention YS
pR
pR
pL
Appendix
Table 3: High-level predictions for the original input (V0,a) . Each entry is the location the actions target. pL preserves the original target ( E=0 ), and pR redirects the reach to the other object’s location in the original scene ( E=1 ). The intervention rows give the signatures in Figure 3(a) .
Figure 5: Complementary and full-trajectory interface patching. (a) Normalized trajectory shift E when patching all connections except one (left) or except three consecutive connections (right). Horizontal axes index the unchanged connection or window center. (b) RMS displacement in millimeters after patching only a single connection (left) or three consecutive connections (right, indexed by their center). Trajectories are reconstructed from predicted actions using forward kinematics. Both panels average comparisons B and C with equal weighting across object-pair families; shading denotes 95% family-bootstrap confidence intervals.
Figure 6: MolmoBot backbone patching across 13 object pairs. Same comparisons, token groups, and metric as Figure 4 . Each cell averages 416 patches: 26 donor–recipient pairs, each patched in both directions under eight noise seeds. Image patches in B are exactly zero because MolmoBot’s image positions cannot attend to the instruction. Omitted layers 29–35 have ∣E∣<0.005 .
Figure 7: Small token sets that carry location in MolmoAct2, across all 13 object pairs. For each layer, filled cells mark the smallest set of text tokens, found by greedy forward selection, whose joint patch keeps at least 75% of the effect of patching all 79 non-state text tokens in comparison C; the right column gives the number of tokens. Effects are averaged over all 52 comparison-C donor–recipient pairs with equal weight per object pair. The sets keep 76 – 89% of the effect on the noise seed used for selection and 72 – 90% on three held-out seeds. The target noun, which differs across object pairs, is labeled <object> ; state tokens were not searched.
Vision
VLM
Action expert
Total
Change
MolmoBot, all 36 layers
2.46
7.25
2.22
11.93
–
MolmoBot, layer 24
2.46
5.04
0.14
7.64
−36%
MolmoBot, layers 23–25
2.46
5.24
0.26
7.96
−33%
π0.5 , all 18 layers
0.66
3.98
0.13
4.76
–
π0.5 , layers 14–16
0.66
3.75
0.11
4.52
−5%
Appendix
Table 4: Estimated TFLOPs per action chunk. Restricted policies stop the VLM after the deepest layer the action expert reads.
MolmoBot
Latency (ms)
Throughput (chunks/s)
Memory (GB)
All 36 layers
311±3
21.5
42.7
Layer 24
204±16 ( −35% )
31.8 ( +48% )
31.7
Layers 23–25
209±2 ( −33% )
30.9 ( +44% )
32.2
Appendix
Table 5: Measured inference cost on one H100 (bfloat16, ten flow-matching steps). Latency is at batch size 1; throughput and peak memory are at batch size 64. Restricted policies stop the VLM after the deepest layer the action expert reads.
Figure 8: Counterfactual scenes. Object pairs used each shown in both layouts with the objects’ left–right positions exchanged. Images show the exterior DROID camera, cropped to the table (during experiments the full frame is used).
Original (all 18 layers)
Restricted (layers 14–16)
Task
Trials
Success
Time (s)
Success
Time (s)
Fruit in basket
25
22
53.6
22
45.4
Marker in mug
20
15
16.1
10
16.4
Stack the bowls
15
8
71.8
11
50.0
Screwdriver in bin
10
5
37.8
7
20.0
Mug in bin
10
3
18.9
8
21.4
Appendix
Table 6: Real-robot results for π0.5 by task. Success counts and median duration of successful trials. The last row gives total successes and the median task time averaged over the seven tasks.
Figure 9: Real-robot tasks for π0.5 . The first exterior-camera frame of each task, with the instruction given to both policies. For the [noun] in bin task,[noun] is replaced with each of the target objects for 10 trials each.
A fine-tuned Vision-Language-Action (VLA) policy can easily place the alphabet soup in the basket on command, just as it saw during fine-tuning. However, if the basket is shifted five centimeters to the left, it drops the soup off the table. We developed an interpretability pipeline to investigate what such a policy is computing. Injecting task A's activations into the action expert while the policy looks at task B's scene makes pi0.5 execute task A's motor trajectory in 99.6% of episodes (n=1,968). The injected program is bound to absolute workspace coordinates and ignores the visible scene, which provides a mechanistic account of the perturbation brittleness that concurrent benchmarks report. The binding holds on a physical UR10e: directed replay redirects the commanded task in 26/26 rollouts in one direction and 18/26 in the other, with no-injection and random-direction controls at the floor, and once the source object is moved 10--15,cm the arm grasps and releases at the recorded coordinates in 50/50 trials, picking up the wrong block when one sits there and closing on empty air when none does. Applying the same interventions to six VLAs on LIBERO, MetaWorld, SimplerEnv, and ALOHA over 420{,}000+ rollouts yields three further findings: every architecture encodes the instruction, and four of the five weight it below vision whenever the scene alone identifies the goal; under the standard replacement write-back the SAE pooling that preserves behavior is architecture-specific (pi0.5 per-token, SmolVLA indifferent), and goal selection and motor generation separate into VLM and expert pathways wherever the two are distinguishable. We release Action Atlas (https://action-atlas.com), an interactive platform covering all 520+ trained SAEs, 79 identified concepts, and the scored UR10e rollouts. Code: https://github.com/CWRU-AISM/action-atlas
Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for robotic manipulation, in which reliable action prediction critically depends on accurately interpreting and integrating visual observations conditioned on language instructions. Although recent works have sought to enhance the visual capabilities of VLA models, most approaches treat the LLM backbone as a black box, providing limited insight into how visual information is grounded into action generation. Therefore, we perform a systematic analysis of multiple VLA models across different action-generation paradigms and observe that sensitivity to visual tokens progressively decreases in deeper layers during action generation. Motivated by this observation, we propose DeepVision-VLA, built on a Vision-Language Mixture-of-Transformers (VL-MoT) framework. This framework enables shared attention between the vision foundation model and the VLA backbone, injecting multi-level visual features from the vision expert into deeper layers of the VLA backbone to enhance visual representations for precise and complex manipulation. In addition, we introduce Action-Guided Visual Pruning (AGVP), which leverages shallow-layer attention to prune irrelevant visual tokens while preserving task-relevant ones, reinforcing critical visual cues for manipulation with minimal computational overhead. DeepVision-VLA outperforms prior state-of-the-art methods by 9.0% and 7.5% on simulated and real-world tasks, respectively, providing new insights for the design of visually enhanced VLA models.
Yulin Luo, Hao Chen, Zhuangzhe Wu +10
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The Chinese University of Hong Kong · Simplexity Robotics
Vision-Language-Action (VLA) models have emerged as a powerful paradigm for open-world robot manipulation, but their practical deployment is often constrained by cost: billion-scale VLM backbones and iterative diffusion/flow-based action heads incur high latency and compute, making real-time control expensive on commodity hardware. We present A1, a fully open-source and transparent VLA framework designed for low-cost, high-throughput inference without sacrificing manipulation success; Our approach leverages pretrained VLMs that provide implicit affordance priors for action generation. We release the full training stack (training code, data/data-processing pipeline, intermediate checkpoints, and evaluation scripts) to enable end-to-end reproducibility. Beyond optimizing the VLM alone, A1 targets the full inference pipeline by introducing a budget-aware adaptive inference scheme that jointly accelerates the backbone and the action head. Specifically, we monitor action consistency across intermediate VLM layers to trigger early termination, and propose Inter-Layer Truncated Flow Matching that warm-starts denoising across layers, enabling accurate actions with substantially fewer effective denoising iterations. Across simulation benchmarks (LIBERO, VLABench) and real robots (Franka, AgiBot), A1 achieves state-of-the-art success rates while significantly reducing inference cost (e.g., up to 72% lower per-episode latency for flow-matching inference and up to 76.6% backbone computation reduction with minor performance degradation). On RoboChallenge, A1 achieves an average success rate of 29.00%, outperforming baselines including pi0(28.33%), X-VLA (21.33%), and RDT-1B (15.00%).