Can better feature discovery improve control over robot behavior through internal interventions and training data? Recent work in vision and language models reveals multidimensional features, motivating block-sparse featurizers (BSFs), which represent each feature in a low-dimensional subspace. We investigate whether recovering this structure provides an interpretable interface for steering vision-language-action models and shaping the data from which they learn. We show that BSFs recover several multi-dimensional task-relevant features underlying the model's computation (e.g., perception, language, proprioceptive state, episode phase, and control variables). We causally validate these features through fine-grained steering: targeted changes in visual perception, semantic representation, and spatial localization produce predictable changes in downstream action prediction. Finally, we use feature-based data attribution to isolate where a specific behavior (failure recovery) arises in the training dataset, identifying fewer than 1.4% of training tokens whose masking is sufficient to remove the targeted behavior. Overall, our results show how matching featurizers to representation geometry can provide more effective tools for understanding and changing robot behavior.
Figures & tables
Figure 1: Block-sparse featurizers (BSFs) recover multi-dimensional robotics concepts. Examples from vision-language-action models encode end-effector position (top), fruit-and-vegetable semantics (middle), and proprioceptive state (bottom). Each point is a token’s 3D feature code, colored by PCA coordinates, task or semantic labels, and sparse-autoencoder (SAE) assignments across columns. Activations from MolmoAct2 L25, L21, L13 (top to bottom). Left overlays share the PCA colors. SAE features tile distinct regions within each block, showing how BSFs group related variation into a single feature.
Figure 2: BSFs better capture low-dimensional manifolds. (b) One three-coordinate BSF block reconstructs the helix. (c) Many individual SAE features tile the helix.
Figure 3: Reconstruction, dimensionality, and behavioral relevance of SAE and BSF features in MolmoAct2. Left three panels (layer 13): Held-out reconstruction R2 and absolute recorded-action log-likelihood change versus active blocks k (lower change is better; mean ± SEM, 48 episodes), and within-block stable rank versus block size at k=32 (bands: interquartile range; gray: full dimension). Colors denote block size ( b=1 : SAE); solid/dashed lines denote Vanilla/Fisher training. Right two panels (across layers): Three-dimensional Vanilla and Fisher BSFs versus SAEs at approximately matched latent widths and active-coordinate budgets (21 blocks versus 63 latents). Ablation effects measure mean predicted end-effector displacement over the top seven BSF blocks or 21 SAE latents by activation mass. Decodability is the best single-feature ridge-probe R2 on held-out episodes, averaged over ten robot-state and task-progress variables.
Figure 4: Fine-grained steering through three representations. Semantic , visual , and geometric interventions redirect predicted actions from the prompted object to the other object while leaving input images and instructions unchanged. Results cover 52 held-out scene–instruction comparisons with six noise seeds each. # Features : selected-set size, counting three-dimensional BSF blocks or scalar SAE features. Held-out recovery : fraction of the activation-patching effect reproduced by the selected features, averaged over comparisons for which feature discovery finds a steering set. ∗ A steering set is found for only 50/52 comparisons; all other rows cover all 52. Details in Appendix B.1 .
SAE
BSF
Object pair
# Feat.
Select
Grasp
Task
# Feat.
Select
Grasp
Task
Apple / banana
2.0
90%
85%
75%
2.5
100%
95%
90%
Bowl / cable
2.5
100%
95%
85%
2.0
100%
100%
100%
Mouse / towel
3.0
75%
70%
65%
2.5
100%
95%
90%
Total
2.5
88.3%
83.3%
75.0%
2.3
100%
96.7%
93.3%
Table 1: Closed-loop semantic target redirection. The policy is instructed to pick up one object while a fixed semantic feature intervention redirects it to the other. Entries are success rates over 20 trials per object pair (10 per direction; 60 in total): Select, the policy goes for the redirected object; Grasp, it grasps that object; Task, it places that object in the basket. # Feat.: features per direction, averaged. Per-direction counts are in Table 3 .
Figure 5: Visual BSF features capture variation across task stages and robot parts. (left) Marker feature F2973 , with points colored by task completion. Linked images illustrate changes in appearance across a single episode; the larger overlays show the four highest-activating timesteps in the dataset. (right) Robot feature F2571 , with PCA colors and linked images showing how different regions of the block correspond to different robot parts. Both features are Fisher-weighted BSFs from MolmoACT2 layer 7.
Figure 6: A single BSF feature links related container concepts across vision and language. The left column of point clouds shows activations colored by PCA coordinates (top) and token modality (bottom). The right column shows only activating text tokens, colored by token identity (top) and SAE assignment (bottom). The feature nicely represents different types of containers across spatially distinct positions. This feature is Vanilla BSF F1777 from π0.5 layer 3.
Figure 7: A sample of BSF control-relevant features. Left: MolmoACT2 layer 13, F937 , sampled across the dataset. Colorings show PCA coordinates, SAE assignment, image-patch position, and gripper state; linked images illustrate the gripper configurations and locations associated with selected points. Middle: π0.5 layer 7, F2113 , sampled from a single episode and colored by PCA coordinates (top) and end-effector x , y , and z position (bottom). Right: MolmoBot layer 24, F725 , colored by camera side and camera position in the robot frame, with four representative views.
Figure 8: Sampling of vanilla BSFs proprioception features. Top: A MolmoACT2 layer-13 feature shown with PCA coloring (left), quantized state value from 0 to 255 (middle), and state-component index (right).
Method
Reopen (s) ↑
Success (%) ↑
Complete (s) ↓
Unmodified
3.13
90
12.13
Random
4.34
100
10.15
Diff. means
5.01
65
17.49
kNN
3.45
100
12.82
Influence
5.74
100
11.12
Concept (ours)
12.93
100
11.79
Table 2: Behavior shaping. Mean reopening time (15 perturbed trials) and task performance (20 unperturbed trials; 500-step limit). Longer reopening indicates stronger recovery suppression; completion time averages successful trials only.
Figure 9: Feature associated with failure recovery. Left: F1515 block codes colored by PCA coordinates and SAE assignment. Right: Wrist observations, gripper state, and mean feature norm across exterior-image, wrist-image, and text tokens in a DROID demonstration (top) and a real-robot rollout (bottom). Feature activations are measured by replaying the recordings through π0.5 -DROID.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Highest-scoring frames for each labeler. Each column shows five DROID episodes for one method. For the four scored methods, these are the five highest-scoring episodes, and t is the highest-scoring evaluated frame within that method’s selected windows. Random has no score, so its column shows five randomly drawn windows with t at each window’s midpoint; its row order carries no meaning.
Figure 11: Open and closed-loop steering setups . Open loop ( Section B.1 ): we find candidate semantic features and evaluate them on the mirrored scene for semantic features. Visual edits are instead evaluated under the other object’s instruction, and geometric edits on held-out object pairs. Closed loop : feature sets are discovered on both mirrored layouts, then fixed and tested on new scenes and tasks, “pick up the banana and put it in the basket".
A → B
B → A
Objects (A / B)
Featurizer
# Feat.
Select
Grasp
Successful
# Feat.
Select
Grasp
Successful
Apple / banana
SAE
2
10/10
9/10
7/10
2
8/10
8/10
8/10
BSF
2
10/10
10/10
9/10
3
10/10
9/10
9/10
Cable / bowl
SAE
3
10/10
9/10
8/10
2
10/10
10/10
9/10
BSF
2
10/10
10/10
10/10
2
10/10
10/10
10/10
Towel / mouse
SAE
4
6/10
5/10
4/10
2
9/10
9/10
9/10
Appendix
Table 3: Closed-loop redirection by direction. Trials out of 10. Full version of Table 1 . A → B: instructed to pick up A while steered toward B. Feat.: number of features in the intervention; Grasp counts trials that grasp the redirected object; Successful counts trials that place it in the basket.
Improving pretrained robot action models at test time from only a few rollouts remains challenging. Activation steering offers a lightweight approach by modifying feature activations while keeping model weights fixed. However, effective steering requires identifying which features to modify and determining the timing and strength of modification. We introduce SteerAct, a few-shot temporal steering framework that uses a shared temporal descriptor of sparse-feature dynamics for offline feature selection and online control. A pretrained sparse autoencoder provides candidate features, and our temporal descriptor summarizes the distribution of local activation-order patterns for each feature, enabling comparisons across rollouts without requiring timestep alignment. Before steering begins (offline), we use successful and failed reference rollouts to select a single task-specific feature whose temporal descriptors best distinguish the two outcomes and construct its reference banks. During each new execution (online), we compare the feature's evolving temporal descriptor with these references to determine when to intervene and how strongly to increase or decrease its activation via a steering coefficient. Using only 24 outcome-labeled rollouts per task, SteerAct improves task success by approximately 10% relative to unsteered models in evaluations spanning π0.5, Fast-WAM, RoboTwin 2.0, DexJoCo, and real-world manipulation, without updating action-model weights. Code and demos are available at: https://steeract.github.io/.
Kaichen Zhou, Zhiyan Li, Chongkai Gao +11
Massachusetts Institute of Technology · Harvard University · Stanford University +3
Large pretrained vision-language-action models achieve strong robot-manipulation performance, while compact alternatives have largely pursued efficiency by compressing the prevailing observation-to-action paradigm. We investigate whether predictive sensorimotor modeling can make more effective use of a limited parameter budget than direct observation-to-action mapping. We present PredVLA, a language-conditioned predictive-coding policy with only 0.68 million trainable network parameters and no robot-data pretraining. Its hierarchical recurrent dynamics predict visual features and proprioception, while observations influence latent state only through prediction-error-driven online inference. On LIBERO, PredVLA achieves an 86.9% mean success rate across the three short-horizon suites and 75.4% across all four suites. Under a controlled comparison using the same frozen front end, demonstrations, action decoder, and evaluation protocol, PredVLA achieves 3.7x and 7.4x the three-suite mean success rates of parameter-matched Transformer and LSTM behavior-cloning policies, respectively. A mechanism-by-mechanism transition to the recurrent behavior-cloning baseline shows that replacing the predictive pathway with direct observation input produces the largest single performance drop, accounting for approximately 70% of the endpoint gap. Further ablations identify distinct contributions from training-time latent inference, test-time error regression, hierarchical timescales, and sensory prediction-error channels. Together, these results support predictive sensorimotor modeling as a strong inductive bias for compact language-conditioned robot control.
A fine-tuned Vision-Language-Action (VLA) policy can easily place the alphabet soup in the basket on command, just as it saw during fine-tuning. However, if the basket is shifted five centimeters to the left, it drops the soup off the table. We developed an interpretability pipeline to investigate what such a policy is computing. Injecting task A's activations into the action expert while the policy looks at task B's scene makes pi0.5 execute task A's motor trajectory in 99.6% of episodes (n=1,968). The injected program is bound to absolute workspace coordinates and ignores the visible scene, which provides a mechanistic account of the perturbation brittleness that concurrent benchmarks report. The binding holds on a physical UR10e: directed replay redirects the commanded task in 26/26 rollouts in one direction and 18/26 in the other, with no-injection and random-direction controls at the floor, and once the source object is moved 10--15,cm the arm grasps and releases at the recorded coordinates in 50/50 trials, picking up the wrong block when one sits there and closing on empty air when none does. Applying the same interventions to six VLAs on LIBERO, MetaWorld, SimplerEnv, and ALOHA over 420{,}000+ rollouts yields three further findings: every architecture encodes the instruction, and four of the five weight it below vision whenever the scene alone identifies the goal; under the standard replacement write-back the SAE pooling that preserves behavior is architecture-specific (pi0.5 per-token, SmolVLA indifferent), and goal selection and motor generation separate into VLM and expert pathways wherever the two are distinguishable. We release Action Atlas (https://action-atlas.com), an interactive platform covering all 520+ trained SAEs, 79 identified concepts, and the scored UR10e rollouts. Code: https://github.com/CWRU-AISM/action-atlas