Sparse Autoencoders Reveal Interpretable and Steerable Features in VLA Models

Aiden Swann; Lachlain McGranahan; Hugo Buurmeijer; Monroe Kennedy; Mac Schwager

cs.ROMar 19, 2026

Sparse Autoencoders Reveal Interpretable and Steerable Features in VLA Models

Authors: Aiden Swann, Lachlain McGranahan, Hugo Buurmeijer, Monroe Kennedy, Mac Schwager

Organizations: Department of Mechanical Engineering, Stanford University · Department of Aeronautics & Astronautics, Stanford University · Department of Computer Science, Stanford University

Abstract

Vision-Language-Action (VLA) models have emerged as a promising approach for general-purpose robot manipulation. However, little research has mechanistically explored when and why they generalize across objects, scenes, and instructions. To probe internal representations, we train Sparse Autoencoders (SAEs) on the VLA's hidden-layer activations. SAEs learn sparse dictionaries over model activations, often revealing features that correspond to interpretable directions in the model's representation space. We identify SAE features corresponding to motion primitives and semantic concepts, including features that are general across episodes and causally steerable. We propose a metric to categorize features as general transferable primitives or episode-specific memorizations, offering a promising glimpse towards VLA generalization. We validate these findings through steering experiments on both the LIBERO simulation benchmark and on real-world DROID hardware. We find that amplifying general and semantic features induces behaviors consistent with their meanings, whereas ablating them destroys model performance. Furthermore, we demonstrate steering as a way to control behavior in unpromptable directions. Together, these results provide mechanistic evidence that VLAs can learn reusable internal features linking perception, language, and action across tasks and scenes. Our project page is located at https://drvla.github.io

Figures & tables

Figure 1: Activations for two general features related to grasping in the $\pi_{0.5}$ model’s PaliGemma Layer 5 over an episode of LIBERO. These features activate generally for grasping and carrying behaviors across the dataset.

Figure 2: We collect internal activations from a VLA, train a Sparse Autoencoder (SAE), and obtain sparse features that represent both memorized and general semantic concepts. We propose metrics to categorize features by generality and offer steering results to validate our features on a DROID setup.

Figure 3: DROID general features across diverse tasks. For each episode, we show evenly spaced frames from the main and wrist cameras from $t_{0}$ to $t_{f}$ , with per-timestep feature activations plotted below. Rows correspond to example tasks, and the color bars map to features F158, F586, F165, and F399 (legend at bottom).

Model	% General
$\pi_{0.5}$ / LIBERO	2.62
$\pi_{0.5}$ / DROID	10.81
OpenVLA / LIBERO Goal	0.45

Table 1: % of SAE features classified as general. Extended version in Table 7

Feature tier	Sponge	Towel	Overall
Unsteered	19/20	20/20	39/40 (97.5%)
Memorized (random)	17/20	20/20	37/40 (92.5%)
General (random)	10/20	16/20	26/40 (65.0%)
Most general (top 4)	0/20	0/20	0/40 (0.0%)

Table 2: Feature ablation on the real-world DROID setup. For each tier, we project out four features and report success rates aggregated across them (four features

\times

five trials

=

20 per task). Removing memorized features has little effect on these held-out tasks, whereas removing general features causes substantial degradation.

Task	$\alpha$	Success	$\Delta_{\text{base}}$	$\alpha$	Success	$\Delta_{\text{base}}$
	SAE (ours)			Haon et al.
Sponge	75	22/25	$+21$	25	5/25	$+4$
Towel	125	9/25	$+9$	25	1/25	$+1$
Toy	75	20/25	$+17$	50	5/25	$+2$
Mug	125	13/25	$+12$	25	5/25	$+4$

Table 3: Target-object grasp success, comparing SAE feature steering with FFN-neuron (Haon et al.) steering. Sponge, Towel, Toy, and Mug are steered with F1784, F8, F1727, and F39, respectively.

\Delta_{\text{base}}

in each column reports the improvement of that method over the no-steering baseline. Full no-steering baseline distributions (

n{=}25

) — Scene 1: 15 Apple, 8 Lid, 1 Sponge, 1 Pot; Scene 2: 20 Apple, 3 Toy, 1 Plate, 1 Mug.

Close ( $\uparrow$ )
Method	Closure
“close the gripper”	0.005
Haon et al.	0.361
SAE F586 (ours)	0.653
Open ( $\downarrow$ )
“open the gripper”	0.575

Table 4: Gripper closure (

0

open,

1

closed), averaged over the first 50 control steps and 5 trials per condition.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Episodes	Tasks	Timesteps
LIBERO	1,693	40	273,465
DROID (2k subset)	2,000	1,545	567,088
LIBERO-Goal (OpenVLA)	428	10	52,042

Appendix

Table 5: Dataset statistics for the models we analyze in this work.

Hyperparameter	Value
Expansion ratio	1
Active features $k$	100
Auxiliary $k_{\mathrm{aux}}$	512
Auxiliary loss coefficient $\lambda_{\mathrm{aux}}$	$1/32$
Learning rate $\eta$	$1\times 10^{-4}$
Optimizer	Adam ( $\beta_{1}=0.9$ , $\beta_{2}=0.999$ )

Appendix

Table 6: Default SAE training hyperparameters. An expansion ratio of

ER=1

means the SAE hidden dimension equals the input dimension

d

, e.g. 2048 features for PaliGemma layers. Expansion-ratio ablations are shown in Figure 4 .

Figure 4: Ablations over SAE expansion ratio and learning rate on $\pi_{0.5}$ PG5 activations. Top row: reconstruction loss versus epoch for DROID (left) and LIBERO (right). Bottom row: dead feature ratio versus epoch. The star marks the setting used throughout this paper, $ER=1$ , chosen as a trade-off between reconstruction loss and dead feature ratio.

Figure 5: Activations from SAEs trained on $\pi_{0.5}$ LIBERO from 6 randomly initialized seeds. We plot the top five features for episode 689; activations are normalized for visualization and ordered by consistency relative to seed 0. The shared patterns across random initializations provide evidence that the learned SAE features arise from the underlying model.

Model	Layer(s)	# Features	# General	# Memorized	% General
$\pi_{0.5}$ – LIBERO
	PG5	2044	32	2012	1.57%
	PG avg (0, 5, 11, 17)	7175	188	6987	2.62%
$\pi_{0.5}$ – DROID
	PG5	2046	104	1942	5.08%
	PG avg (0, 5, 11, 17)	6649	719	5930	10.81%

Appendix

Table 7: Feature classification results across models and layers. We report the number and percentage of SAE features classified as general by the activation-pattern classifier. The remaining features are labeled memorized, reflecting features that are narrower, more episode-specific, or not captured by our generality criteria. Individual layers serve as representative baselines; averages are computed across all analyzed layers within each subnetwork.

Method	Model	Dataset / Layer	$n$ sampled	# interp.	# non-interp.	% interp.
SAE	$\pi_{0.5}$	LIBERO PG5	20	18	2	90%
SAE	$\pi_{0.5}$	LIBERO PG11	20	16	4	80%
SAE	$\pi_{0.5}$	DROID PG5	20	17	3	85%
SAE	$\pi_{0.5}$	DROID PG11	20	14	6	70%
SAE	OpenVLA	LIBERO Goal Layer 8	20	16	$4^{*}$	80%
SAE	OpenVLA	LIBERO Goal Layer 24	20	14	6	70%

Appendix

Table 8: SAE features are frequently interpretable across models, datasets, and layers. As a baseline, we adapt the FFN-neuron approach of Häon et al. [17] to our temporal activation-based feature search pipeline and apply the same 20-feature sampling and labeling protocol. The feature IDs of the random samples for all models are available for cross-referencing on our website .

Figure 6: LIBERO general features across multiple tasks. Each row shows evenly spaced wrist- and main-camera frames from one episode, with per-timestep activations for four SAE features below. The highlighted features activate consistently across scenes, goal objects, and numbers of grasps.

Panel of Figure 7 — Figure 7: Top 15 most active SAE features across layers for single episodes from LIBERO (left), DROID (middle), and OpenVLA (right).

Figure 8: Top 15 SAE features for the $\pi_{0.5}$ embedding layer (top) and the pretrained SigLIP encoder (bottom) for a single episode, Pick up the orange juice and place it in the basket . The SigLIP encoder has not been exposed to any robotics data.

Figure 9: Our experimental setup consists of two scenes shown on the left and right. For each scene several tasks are composed with the available objects, shown in Table 9 .

Scene 1 Tasks	Scene 2 Tasks
T1. Cover the pot with the lid	T4. Pick up the red apple and put it on the plate
T2. Put the sponge on the plate	T5. Put the blue toy on the plate
T3. Pick up the grey towel and put it on the plate	T6. Pick up the mug and put it on the plate

Appendix

Table 9: Real-world tasks used across the two evaluation scenes.

Figure 10: Single-feature ablation on LIBERO. Each black dot is the ablation of a single feature across 150 rollouts. We see a clear pattern that as generality of the ablated feature increases policy performance reduces significantly.

Figure 11: Per-token activations for three features: F225 F1659 and F1881 for $\pi_{0.5}$ DROID PG5. These activations are collected for episode 90 Remove the black ladle from the black pot and put it on the table, put the lid on the black pot . We show the SAE activations for 7 tokens: cam0, cam1, table, pot, black, put, and remove. Additional tokens are shown only for context.

Explore similar work

Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies
May 17, 2026Xinchen Jin, Aditya Chatterjee, Pranav Kumar +1VLM Interpretability Sparse Autoencoders
Department of Computer Science, Purdue University West Lafayette, IN 47907
Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
Mar 19, 2026Bryce Grant, Xijia Zhao, Peng WangVision-Language-Action Models Robotics
Case Western Reserve University
Sparse Feature Policy Unlearning Mitigates State Hallucination in Vision-Language-Action Models
Oct 7, 2026Jiho Lee, Jeongeun Park, Heayoun Choi +2Vision-Language-Action Models Robotic Manipulation
School of Computer Science Engineering, Chung-Ang University, Seoul, 06974, Republic of Korea · School of Interactive Computing, Georgia Institute of Technology, Atlanta, GA, 30308, USA · NAVER AI Lab, Seongnam, 13561, Republic of Korea

May 17, 2026J/K move · Enter open · S save

Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies

Xinchen Jin, Aditya Chatterjee, Pranav Kumar, Rohan Paleja

Department of Computer Science, Purdue University West Lafayette, IN 47907

VLM Interpretability Sparse Autoencoders Mechanistic Interpretability Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) policies translate language and visual inputs into robot actions, where their hidden representations directly shape closed-loop behavior. However, mechanistic interpretability tools from language and vision-language models do not transfer cleanly to VLAs: outputs are robot actions rather than human-readable tokens, and interventions can only be tested via expensive closed-loop rollouts. We propose an event-grounded interpretability pipeline that anchors SAE feature analysis to behavioral events rather than text contexts. End-effector keyframes are clustered within each task using visual, state, and temporal cues, linking SAE features to behaviorally salient events and, via optional VLM annotations, to semantic context. To our knowledge, our pipeline is among the first to ground SAE-based VLA analysis in closed-loop behavioral events. Across two simulation architectures and a real-robot study, event-grounded ranking yields the strongest causal effects on OpenVLA and transfers to the continuous action chunks of

π_{0.5}

. SAE is a sparse but imperfect intervention basis: usability varies with architecture and intervention site, and aggressive intervention reveals safety and interpretability limits. Overall, event-grounded SAE analysis emerges as a practical starting point for behavior-anchored VLA interpretability, motivating future work on SAE features beyond action-aligned coordinates, finer-grained closed-loop evaluation, and safe interventions for high-stakes VLA deployments. Code is available at https://github.com/xc-j/Event-SAE.

Abstract

Explore similar work

Event-Grounded Sparse Autoencoders for Vision-Language-Action Policies

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Sparse Feature Policy Unlearning Mitigates State Hallucination in Vision-Language-Action Models