Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Bryce Grant; Xijia Zhao; Peng Wang

cs.ROMar 19, 2026

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Authors: Bryce Grant, Xijia Zhao, Peng Wang

Organizations: Case Western Reserve University

Abstract

A fine-tuned Vision-Language-Action (VLA) policy can easily place the alphabet soup in the basket on command, just as it saw during fine-tuning. However, if the basket is shifted five centimeters to the left, it drops the soup off the table. We developed an interpretability pipeline to investigate what such a policy is computing. Injecting task A's activations into the action expert while the policy looks at task B's scene makes pi0.5 execute task A's motor trajectory in 99.6% of episodes ( $n{=}1{,}968$ ). The injected program is bound to absolute workspace coordinates and ignores the visible scene, which provides a mechanistic account of the perturbation brittleness that concurrent benchmarks report. The binding holds on a physical UR10e: directed replay redirects the commanded task in 26/26 rollouts in one direction and 18/26 in the other, with no-injection and random-direction controls at the floor, and once the source object is moved 10--15,cm the arm grasps and releases at the recorded coordinates in 50/50 trials, picking up the wrong block when one sits there and closing on empty air when none does. Applying the same interventions to six VLAs on LIBERO, MetaWorld, SimplerEnv, and ALOHA over 420{,}000+ rollouts yields three further findings: every architecture encodes the instruction, and four of the five weight it below vision whenever the scene alone identifies the goal; under the standard replacement write-back the SAE pooling that preserves behavior is architecture-specific (pi0.5 per-token, X-VLA mean-pool, SmolVLA indifferent), and goal selection and motor generation separate into VLM and expert pathways wherever the two are distinguishable. We release Action Atlas (https://action-atlas.com), an interactive platform covering all 520+ trained SAEs, 79 identified concepts, and the scored UR10e rollouts. Code: https://github.com/CWRU-AISM/action-atlas

Figures & tables

Figure 1: Three core findings on $\pi_{0.5}$ . Left: Activation injection recovers baseline behavior from null-prompt episodes. Without injection, null prompts drop cosine similarity to 0.775; injecting a single layer (L0) recovers 0.997 and all layers recovers 0.999, demonstrating visual pathway dominance. Middle: Per-token SAE processing is essential. Mean-pooled SAE reconstruction destroys task success (96% $\to$ 8%) despite high explained variance, while per-token processing preserves performance (96% $\to$ 94%). Right: Cross-task injection fails destination tasks (83.3% $\to$ 2.2%) and same-scene injection partially succeeds (35.0%), confirming spatially bound motor programs. These patterns replicate across all six models (Table 6 ).

Figure 2: Methodology overview. Top: activations are recorded from VLA backbone and action expert layers during rollout episodes, then replayed under counterfactual conditions (null prompts, cross-task scenes) to establish causal relationships via behavioral change. Middle: per-token SAEs decompose layer activations into sparse features. Bottom: features are clustered, searched, and causally validated through ablation and steering experiments, with results visualized in Action Atlas.

Model	Params	Layers	Dim	Action Gen.	Pathway	Bench.
$\pi_{0.5}$ ( Physical Intelligence, 2025 )	3B	18	1024	Flow (50 steps)	Dual (PG + Exp.)	LIBERO
OFT ( Kim et al., 2025 )	7B	32	4096	Cont. L1 regr.	Single (Llama-2)	LIBERO
X-VLA ( Zheng et al., 2026 )	$\sim$ 1B	24	1024	Flow matching	Single (Flor.-2)	LIB., SimpE.
SmolVLA ( Shukor et al., 2025 )	450M	32	960/480	Flow matching	Dual (VLM + Exp.)	LIB., MW
GR00T ( Bjorck et al., 2025 )	3B	32	varies	Diff./flow	Triple (D+E+V)	LIBERO
ACT ( Zhao et al., 2023 )	80M	–	–	CVAE	Enc.-Dec.	ALOHA

Table 1: Architectures under study. Five VLAs and ACT (language-free control). PG = PaliGemma, Exp. = Expert, Flor. = Florence, D+E+V = DiT + Eagle + VL-SA, LIB. = LIBERO, MW = MetaWorld, SimpE. = SimplerEnv. Full architecture details in Appendix K.1 .

Model	Episodes	SAEs	Concepts
$\pi_{0.5}$	65,000+	36	43
OpenVLA-OFT	70,700+	32	45
X-VLA	50,000+	96	82
SmolVLA	42,000+	192	45
GR00T N1.5	164,700+	68	36
ACT	1,870	–	–

Table 2: Experimental scale across six models. Episode counts aggregate across applicable experiment types (baselines, counterfactual prompting, cross-task injection, vision perturbation, grid ablation, SAE validation, concept ablation, concept steering, temporal ablation, fraction-to-failure); not all models undergo every type. SAE counts include per-token, mean-pooled, and k-sweep variants.

Model	Condition	Goal	Object	Spatial	10/Long ⋄
$\pi_{0.5}$	Baseline	91.9%	–	77.3%	62.5%
	Null, no injection	0%	–	–	–
	Same-scene inject ALL	91.0%	–	–	75.2%
OFT	Baseline	100%	100%	90%	90%
OFT	Null + zero any layer	14.1%	14.6%	14.4%	13.5%
X-VLA	Baseline	96.7%	100%	90.0%	100%

Table 3: Visual pathway influence across all six architectures, per suite. Baseline : correct prompt, no intervention. Same-scene inject : inject baseline activations into same-task episode. Null source : null prompt with activation injection. Zero : zeroing all activations at a layer. ⋄ GR00T uses libero_long (no libero_10 ).

\pi_{0.5}

baselines from cross-task experiment (

n{=}138/23/24

pairs); same-scene injection: goal

n{=}390

, 10

n{=}330

. OFT:

n{=}120

per layer per suite. GR00T null injection:

n{=}30

per suite (10 tasks

\times

3). SmolVLA: expert-layer ranges reflect L0 (catastrophic) through L31 (near-baseline).

Model	Pairs	Success	Displacement
$\pi_{0.5}$	1,968	2.6%	cos(traj, src) $>$ cos(traj, dst): 99.6%
OFT	1,079	$\sim$ 50% †	cos(traj, src) $>$ cos(traj, dst): 77.9%
X-VLA	3,150	0%	cos $\rightarrow$ src $>$ dst (99.8%)
SmolVLA	732	0%	Expert: 15.8% source-like
GR00T	270	0%	cos $\rightarrow$ src $>$ dst (57.0%)

Table 4: Cross-task injection across five VLA models. Destination task success collapses universally, but displacement analysis reveals architecture-dependent behavioral steering. † OFT: suite-dependent; libero_goal drops 40pp from baseline (

\sim

50%); other suites maintain near-baseline rates due to OFT’s wide 4096-dim representations.

Figure 3: Cross-task displacement override rates. Left: override rate across five models.

\pi_{0.5}

(99.6%,

n{=}1{,}968

) and X-VLA (99.8%,

n{=}3{,}150

) show near-complete source behavior transfer; OFT 77.9% (

n{=}1{,}079

); GR00T 57.0% (

n{=}270

, suite-dependent: goal 85.6%, long 33.3%). Error bars: 95% Wilson CIs. Right: SmolVLA pathway displacement (15.8% expert vs. 9.0% VLM, 732 pairs).

Model / Suite	Baseline	Null	Wrong Obj.
$\pi_{0.5}$ / object	77.4%	77.0%	74.2%
$\pi_{0.5}$ / goal	83.3%	80.0%	76.7%
OFT / object	100%	100%	100%
OFT / goal	100%	10%	10%
OFT / spatial	90%	70%	60%
X-VLA / object	100%	60%	60–90%

Table 5: Counterfactual prompting across models (

n{=}900

OFT,

n{=}3{,}396

\pi_{0.5}

n{=}4{,}800

X-VLA,

n{=}630

GR00T).

\pi_{0.5}

ignores all prompt variations (

p{=}0.247

, ANOVA). OFT mirrors X-VLA’s suite-dependent pattern: object is immune (100%) but goal collapses to 10% under generic/wrong prompts. SmolVLA shows difficulty-dependent sensitivity on MetaWorld. GR00T shows strong suite-dependent sensitivity: libero_goal collapses from 97% to 0% under null and wrong-task prompts, while libero_object retains 50–77% across conditions.

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Phenomenon	$\pi_{0.5}$	OFT	X-VLA	SmolVLA	GR00T	ACT
Visual pathway dominance	Y (73%)	Y (14%)	Y (all layers)	Y	Y	Y
Cross-task failure	Y (2.6%)	Y ( $\sim$ 50%)	Y (99.8% src)	Y (16% src)	Y (57%)	Y (0%)
Language sensitivity	suite-indep.	suite-dep. §	suite-dep. §	partial ∗	suite-dep.	N/A
Pathway specialization	Y	N/A	N/A	Y (2 $\times$ )	Y	N/A
Per-token SAE req.	Y	Y	reversed †	Y	partial ‡	N/A
Causal sensitivity	narrow (54%)	wide (92%)	wide (82%)	narrow (28%)	mixed (59%)	N/A

Appendix

Table 6: Cross-model validation of core findings. Y = confirmed. § libero_goal collapses to 10% under wrong prompts; libero_object immune. ∗ SmolVLA: MetaWorld difficulty-dependent. † X-VLA: mean-pooled SAEs achieve better rollout fidelity. ‡ GR00T: VL-SA layers benefit from mean-pooling. Causal sensitivity spans 28–92% zero-effect rates.

Task	Baseline	Per-Token SAE	Mean-Pooled SAE
0	198/250	103/250	2/250
1	247/250	248/250	0/250
2	203/250	198/250	1/250
3	151/250	248/250	0/250
4	249/250	202/250	3/250
5	249/250	247/250	0/250

Appendix

Table 7: Per-task results for per-token vs. mean-pooled SAE reconstruction on LIBERO-10 (2,500 episodes). Per-token SAE maintains task performance (69.9%) while mean-pooled causes near-complete failure (0.4%).

Concept	Best Layer	2nd Best	3rd Best
Action Concepts
PUT	L17 (133k)	L16 (111k)	L15 (86k)
OPEN	L15 (103k)	L16 (79k)	L13 (71k)
PUSH	L17 (412k)	L13 (275k)	L14 (270k)
INTERACT	L17 (274k)	L12 (261k)	L15 (253k)
Object Concepts

Appendix

Table 8: Layer-wise concept specificity (activation scores in parentheses). Later layers (14-17) show highest concept specificity.

Target Suite	Baseline	After PUT Ablation
Goal	100%	5%
Object	100%	0%
Spatial	100%	11%

Appendix

Table 9: Cross-suite generalization of PUT feature ablation. Caveat: These results use full-layer intervention hooks; corrected MLP-targeted ablation shows no significant effect (see Section F.3 ).

Concept	Tasks Active	Tasks Inactive	Accuracy
PUT	1,3,4,5,6,7,8	0,2,9	100%
OPEN	0,1	2-9	100%
INTERACT	9	0-8	100%

Appendix

Table 10: Concept features show high task discrimination.

Layer	Baseline	SAE Reconstruction
Layer 0-11	80-90%	0-5%
Layer 12-17	80-90%	0-5%
action_out_proj_input	93%	76%

Appendix

Table 11: Mean-pooled SAE reconstruction causes catastrophic failure on all intermediate layers.

Feature Selection	Baseline	With Ablation
Action-correlated	93%	79%
Concept-aligned	93%	93% (no ablation)

Appendix

Table 12: Action-correlated features lack concept specificity.

Intervention	Condition	SR
Recovery	Ablate full episode	0%
	Ablate + steer $\alpha{=}0.5$	0%
	Ablate first 50 + steer	0%
Boosting	Dampen $\alpha{=}{-}0.5$	5.7%
Boosting	Boost $\alpha{=}{+}1.0$	5.7%
Substitution	Ablate OPEN + boost PUT 0.5	0%

Appendix

Table 13: Negative results: ablation recovery, feature boosting, and concept substitution all fail. Baseline: 97–100%. Recovery and substitution use full-layer hooks (Section F.3 ); boosting uses corrected MLP-targeted hooks.

Ablation Window	Avg Effect	Phase
step0	$\sim$ 0%	Episode start
early (0-50)	-56%	Approach/grasp
mid (50-150)	-57%	Manipulation
late (150-300)	-1%	Placement
full (0-300)	-76%	All phases

Appendix

Table 14:

\pi_{0.5}

temporal ablation effects averaged across all concepts tested. Caveat: These results use full-layer intervention hooks; absolute effect magnitudes are inflated (see Section F.3 ).

Suite	Baseline	Full	Early	Mid	Late
LIBERO-Goal	100%	73.1%	73.2%	92.7%	92.4%
LIBERO-Long	100%	53.8%	55.6%	75.3%	75.9%
LIBERO-Object	100%	93.9%	94.0%	95.0%	95.3%
Average	100%	69.5%	70.3%	86.2%	86.4%

Appendix

Table 15: GR00T N1.5 temporal ablation by LIBERO suite (160 conditions across 32 layers). Averages are weighted by number of conditions per suite. Early-phase ablation causes the largest success drop, confirming temporal criticality across architectures.

Features Ablated	Success Rate
0 (baseline)	90%
1 (feature 3259)	0%
2	0%
5	0%

Appendix

Table 16: Single feature ablation causes complete task failure. Caveat: These results use full-layer intervention hooks; corrected MLP-targeted ablation shows no significant effect (see Section F.3 ).

Ablation Timestep	Success Rate
Step 0 only	0%
Step 1 only	100%
Step 2 only	80%
Steps 200+	80%

Appendix

Table 17: Step 0 is uniquely critical for task success.

Concept	Selectivity	Type
PUSH	+8.9%	Task-specific
WINE_BOTTLE	+7.5%	Object-specific
PUT	-6.7%	Motor primitive
OPEN	-10.0%	Motor primitive
INTERACT	-11.1%	Motor primitive

Appendix

Table 18: Selectivity = (effect on concept tasks)

-

(effect on other tasks). Negative selectivity indicates features encode motor primitives affecting all tasks. Caveat: These results use full-layer intervention hooks; absolute magnitudes are inflated (see Section F.3 ).

Figure 4: Concept ablation causal sensitivity across five models. Each bar shows the fraction of concept-task pairs with zero effect (gray), partial effect (blue), and total destruction (

-100

pp, red) under single-feature ablation. SmolVLA (480-dim expert) is the most sensitive at 28% zero-effect rate; OFT (4096-dim) and X-VLA (1024-dim) are the most resilient at 92% and 82% respectively. Causal sensitivity does not follow representation width: X-VLA approaches OFT despite sharing

\pi_{0.5}

’s 1024-dim hidden size.

Figure 5: PUT concept ablation (L8): “Put the cream cheese in the bowl.” Top (green): Baseline. The robot picks up the cream cheese and places it in the bowl (91 steps). Bottom (red): With PUT features zeroed at layer 8, the robot drops the cream cheese into the bowl, knocking it over (300 steps, task failure).

Figure 6: OPEN concept ablation (L8): “Open the middle drawer of the cabinet.” Top (green): Baseline. The robot reaches for the middle drawer handle, grasps it, and pulls it open (140 steps). Bottom (red): With OPEN features zeroed at layer 8 (40% destruction rate), the robot opens the bottom drawer instead of the middle, misdirecting the motor program to the wrong target (300 steps, task failure).

Figure 7: SmolVLA MetaWorld vision perturbation (button-press). Top: baseline (67 steps, success). Bottom: center crop to 50% of the visual field removes spatial context; the robot arm cannot locate the button and times out at 400 steps.

Figure 8: X-VLA SimplerEnv vision perturbation (stack cube). Top (green): Baseline WidowX stacks the cube (100% success). Bottom (red): Grayscale removes color information, causing complete failure (0% success). The robot cannot distinguish the cube from the table surface without color.

Figure 9: $\pi_{0.5}$ temporal ablation effects by episode phase. Feature ablation effects vary by phase using full-layer intervention hooks (see note in Section F.3 ). Corrected MLP-targeted ablation shows no significant temporal pattern ( $p=0.975$ ); the phase-dependence shown here is an artifact of the uncorrected hook rather than a genuine temporal gradient. For valid cross-architecture temporal results, see GR00T temporal ablation in Table 15 .

Figure 10: Language is ignored despite internal distinction. Left: counterfactual prompting across 3,396+ episodes shows no significant behavioral difference (p $>$ 0.24). Right: layer 17 classifiers distinguish prompts with 99.3% accuracy, yet behavior is unchanged.

Figure 11: Steering sensitivity across five models. $\pi_{0.5}$ and GR00T exhibit catastrophic sensitivity to feature steering, while OFT and SmolVLA are comparatively robust. SmolVLA shows non-monotonic dose response (mild amplification at 0.5–2 $\times$ , degradation only at 5 $\times$ ). Error represents mean $\Delta$ SR across all steered features.

Figure 12: SmolVLA language sensitivity varies by MetaWorld task difficulty. Baseline success rates range from 85% (easy) to 62% (hard). Under null prompts, easy tasks drop only 3pp (85%

\to

82%), while hard tasks drop 21pp (62%

\to

41%), showing that language sensitivity scales with task ambiguity rather than architecture.

Figure 13: SmolVLA MetaWorld cross-task displacement and success under injection. Left: source-task override rate by MetaWorld difficulty level: destination alignment decreases as difficulty increases (89.7% on easy, 73.8% on very hard), so harder tasks produce weaker source behavior transfer. Right: destination task success under cross-task injection by difficulty: easy tasks maintain 60.7% success while harder tasks drop to 17–22%, reflecting both the displacement effect and the greater task-specific precision required for harder goals.

Figure 14: X-VLA SAE reconstruction fidelity across environments. Per-token processing achieves 99.2% explained variance on LIBERO but only 36% on WidowX (SimplerEnv), while mean-pooling reduces LIBERO fidelity to 85%. The cross-environment degradation indicates that SAE features trained on LIBERO activations do not fully transfer to the WidowX embodiment distribution.

Figure 15: GR00T N1.5 layer-type contribution profiles. SAE feature ablation across 32 layers (16 DiT + 12 Eagle LM + 4 VL-SA) reveals distinct importance profiles per layer type: DiT layers are the most ablation-sensitive (40–80% success drop), Eagle LM layers show moderate sensitivity, and VL-SA layers are the most resilient. This mirrors the pathway specialization observed in

\pi_{0.5}

and SmolVLA.

Figure 16: Cross-model capability radar. Five VLAs scored on baseline success, visual override strength, language sensitivity, SAE fidelity, cross-task transfer rate, and pathway specialization. OFT lacks pathway specialization (single-pathway architecture); SmolVLA and GR00T show the strongest pathway specialization alongside

\pi_{0.5}

Figure 17: SAE explained variance by architecture and pooling strategy. GR00T DiT layers achieve the highest per-token reconstruction quality (98.3–99.8% EV); GR00T VL-SA layers show the lowest per-token quality (83–89%) but improve to 99% EV with mean-pooling. X-VLA mean-pooled SAEs achieve better rollout fidelity than per-token despite lower EV, likely due to Florence-2’s more uniform token representations. Error bars: range across layers within each model. Dashed line: 90% threshold.

Figure 18: Null injection recovery for

\pi_{0.5}

and OFT. Action cosine similarity (left bars) measures how closely injected actions match baseline; success recovery (right bars) measures task completion under null prompt with injected baseline activations.

\pi_{0.5}

recovers 77% success with 0.999 cosine similarity, showing strong visual pathway dominance. OFT recovers only 14% despite 0.85 cosine similarity; OFT’s representations are more entangled with language context.

Suite	Baseline	Null Layer	Aggregate
libero_goal	$\sim$ 90%	$\sim$ 0%	33/234 (14%)
libero_object	$\sim$ 90%	$\sim$ 0%	43/294 (15%)
libero_spatial	$\sim$ 90%	$\sim$ 0%	31/216 (14%)
libero_10	9/10 tasks	0/10 tasks	–

Appendix

Table 19: OpenVLA-OFT null injection by suite. Zeroing any single layer (tested: 0, 8, 16, 24, 31) destroys task success across all suites.

Suite	Episodes	Aggregate Success
libero_goal	582	87.8%
libero_object	162	77.8%
libero_spatial	192	74.0%
libero_10	582	71.8%

Appendix

Table 20: OpenVLA-OFT same-scene injection by suite (1,518 total episodes).

Window	Task Survival	Surviving Tasks
Baseline	9/10	Tasks 0–5, 7–9
null_early	0/9	None
null_mid	0/9	None
null_late	2/9	Tasks 3, 9

Appendix

Table 21: OpenVLA-OFT temporal injection on libero_10 . Early and mid-episode nulling destroys all tasks; late nulling allows partial recovery.

Suite	Min $R^{2}$	Max $R^{2}$	Mean $R^{2}$	Layers $>$ 0.70
LIBERO-Spatial	0.969	0.994	0.988	32/32 (100%)
LIBERO-Goal	0.845	0.941	0.896	32/32 (100%)
LIBERO-10	0.692	0.837	0.780	30/32 (93.8%)
LIBERO-Object	0.608	0.885	0.813	29/32 (90.6%)
All suites	0.608	0.994	0.869	123/128 (96.1%)

Appendix

Table 22: OpenVLA-OFT multi-layer probing: episode-length

R^{2}

across all 32 layers for four LIBERO suites (

n{=}149

per suite).

Suite	Early (L0–7)	Middle (L8–23)	Late (L24–31)
LIBERO-Spatial	0.984	0.991	0.986
LIBERO-Object	0.719	0.834	0.863
LIBERO-Goal	0.859	0.896	0.933
LIBERO-10	0.733	0.779	0.827

Appendix

Table 23: OpenVLA-OFT multi-layer probing: mean

R^{2}

by layer region.

Model	Probe Target	Metric	Result
$\pi_{0.5}$ (expert)	State prediction	$R^{2}$	0.45
$\pi_{0.5}$ (expert)	Success prediction	AUC	0.93
$\pi_{0.5}$ (PaliGemma)	Goal classification	Accuracy	76.4%
$\pi_{0.5}$ (PaliGemma)	Prompt classification	Accuracy	99.3%
$\pi_{0.5}$ (PaliGemma)	Workspace region (L15)	Accuracy	99.3%
OFT	Task identification	Accuracy	97.8–100%

Appendix

Table 24: Cross-model linear probing summary.

\pi_{0.5}

expert activations encode state and success while PaliGemma encodes prompt/goal semantics despite behavioral invariance. OFT achieves near-perfect task identification and success prediction. SmolVLA expert captures

4.5\times

more state information than VLM (

0.58

vs.

0.13

oracle ratio). GR00T SAE features achieve perfect task identification across all 32 layers. Oracle ratio: probe

R^{2}

/ oracle

R^{2}

(fraction of ground-truth state information linearly decodable from activations).

Suite	Episodes	Categories	ANOVA $p$
libero_object	1,496	22	$>0.24$
libero_spatial	353	9	$>0.067$
libero_goal	380	6	$>0.24$

Appendix

Table 25:

\pi_{0.5}

counterfactual prompting by suite. No suite shows statistically significant behavioral differences across prompt categories.

Suite	Condition	Success	n
GOAL	own_prompt (baseline)	91.5%	236
	cross_prompt, no inject	0.0%	236
	cross_prompt + PaliGemma ALL	0.4%	236
	cross_prompt + Expert L16	1.3%	236
SPATIAL	own_prompt (baseline)	100%	34
	cross_prompt, no inject	64.7%	34

Appendix

Table 26:

\pi_{0.5}

cross-task injection with pathway-specific conditions.

Suite	Baseline	Zero any layer	Null prompt	Zero eff.	$\Delta$ pp
Goal	96.7%	0%	10%	82.6%	$-2.2$
Object	100%	0%	60%	98.2%	$-1.3$
Spatial	90.0%	0%	48%	85.2%	$-2.6$
LIBERO-10	100%	0%	28%	74.7%	$-2.4$

Appendix

Table 27: X-VLA per-suite breakdown. Zero any layer : zeroing any single layer (24 tested) produces 0% success across all suites. Null prompt : success under empty string. Zero eff. : concept ablation zero-effect rate.

\Delta

pp : mean concept ablation delta in percentage points. Counterfactual:

n{=}50

baseline +

n{=}1{,}100

“other” per suite (

n{=}4{,}800

total). Concept ablation:

n{=}2{,}480

pairs total.

Suite	Baseline	Expert zero	Zero eff. (exp)	$\Delta$ pp (exp)
Goal	75.0%	60–83%	14.6%	$-4.7$
Object	78.5%	60–83%	0.0%	$-3.9$
Spatial	67.5%	47–77%	32.7%	$-17.7$
LIBERO-10	40.7%	0–33%	20.3%	$+0.8$

Appendix

Table 28: SmolVLA per-suite breakdown. Expert zero : range of overall success rates when zeroing individual expert layers. Zero eff./

\Delta

pp: concept ablation statistics for the expert pathway (

n{=}1{,}696

pairs total). VLM concept ablation (

n{=}210

pairs): goal

-10.7

pp, object

+6.4

pp, spatial

-8.5

pp, LIBERO-10

+5.9

pp.

Suite	Baseline	Zero DiT	CF “other”	Zero eff.	$\Delta$ pp
Goal	97.0%	0%	18.9%	68.8%	$-5.8$
Object	99.0%	0%	73.3%	69.6%	$-9.9$
Long	75.0%	0%	61.7%	42.2%	$-5.3$

Appendix

Table 29: GR00T N1.5 per-suite breakdown. Zero DiT : zeroing any DiT layer (16 tested) produces 0% on all suites. CF “other” : counterfactual success under non-baseline prompt categories (

n{=}180

per suite). Zero eff./

\Delta

pp : concept ablation across 6,500 pairs total. LIBERO-Long is most sensitive (42.2% zero effect, 11.0% destruction) reflecting its multi-step task complexity.

Figure 19: Action Atlas: Feature Explorer. UMAP ( McInnes et al., 2018 ) projection of 4,096 SAE features from OpenVLA-OFT layer 16, colored by semantic category.

Figure 20: Action Atlas: Video Library. Multi-filter interface for browsing 10,000+ rollout videos.

Figure 21: Action Atlas: Ablation Comparison. Three-column video comparison showing baseline, SAE-reconstructed, and ablated rollouts side-by-side.

Figure 22: Action Atlas: Layer Wires. Interactive visualization of information flow across transformer layers.

Figure 23: Action Atlas: Perturbation Testing. Interface for applying real-time visual perturbations and observing behavioral effects.

Explore similar work

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use
Aug 6, 2026Jiarui Yang, Wen Huang, Jiale Zhang +2ICL for Robotics Continual Learning for VLA Models
Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL)
Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models
Mar 16, 2026Yulin Luo, Hao Chen, Zhuangzhe Wu +10Vision-Language-Action Models Vision-Language Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The Chinese University of Hong Kong · Simplexity Robotics
FocusVLA: Focused Visual Utilization for Vision-Language-Action Models
Mar 30, 2026Yichi Zhang, Weihao Yuan, Yizhuo Zhang +2Vision-Language-Action Models Robotic Manipulation
Harbin Institute of Technology, Shenzhen, China · DaiMon Robotics, China · Nanjing University, Nanjing, China +1

Aug 6, 2026J/K move · Enter open · S save

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use

Jiarui Yang, Wen Huang, Jiale Zhang, Maowei Hu+1

Nankai University · Tsinghua University · Swiss Federal Institute of Technology in Lausanne (EPFL)

ICL for Robotics Continual Learning for VLA Models Robotic Manipulation Vision-Language-Action Models VLMs for Robotics Reasoning for VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have become the dominant recipe for generalist manipulation, yet they are almost universally trained by behavior cloning: a policy imitates expert action chunks conditioned on a static image and a fixed instruction. A natural remedy is to inject explicit reasoning through textual chain-of-thought (CoT). We show, both empirically and analytically, that free-form textual CoT degrades low-level control: the reasoning it produces is ungrounded, its latency breaks closed-loop timing, and, crucially, the reasoning and action tokens are optimized against conflicting objectives so that the policy learns to narrate rather than to act. We argue that what a VLA needs is not the ability to generate language, but the ability to consume grounded language. To this end we introduce \ourmethod{}, a framework that endows a VLA with language competence through (i) in-context post-training, in which perceptual evidence is injected as structured context and the model is supervised only on actions, and (ii) an agentic tool-use interface, in which the policy queries open-vocabulary detectors, monocular depth, and a vision--language model to actively acquire task-relevant information. Rather than emitting a single templated caption, our data engine produces diverse, paraphrased, and evidence-conditioned spatial descriptions, so that the policy learns to interpret language it has never seen verbatim. Across the RoboCasa-GR1, SimplerEnv, and LIBERO simulation benchmarks, together with 8 real-world robot manipulation tasks, our method consistently achieves SOTA results in both performance and efficiency when compared with CoT-based approaches under matched configurations.

Abstract

Explore similar work

In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use

Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models

FocusVLA: Focused Visual Utilization for Vision-Language-Action Models