Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models

Aiden Swann; Mozes Jacobs; Mathilde Papillon; Siddharth Boppana; David Chanin; Dron Hazra; Usha Bhalla; Curt Tigges

cs.ROOct 14, 2026

Multi-Dimensional Featurizers Yield Fine-Grained Steering and Data Shaping in Robot Foundation Models

Authors: Aiden Swann, Mozes Jacobs, Mathilde Papillon, Siddharth Boppana, David Chanin, Dron Hazra, Usha Bhalla, Curt Tigges,

Organizations: Goodfire · Stanford University · Harvard University · University of California, Santa Barbara

Abstract

Can better feature discovery improve control over robot behavior through internal interventions and training data? Recent work in vision and language models reveals multidimensional features, motivating block-sparse featurizers (BSFs), which represent each feature in a low-dimensional subspace. We investigate whether recovering this structure provides an interpretable interface for steering vision-language-action models and shaping the data from which they learn. We show that BSFs recover several multi-dimensional task-relevant features underlying the model's computation (e.g., perception, language, proprioceptive state, episode phase, and control variables). We causally validate these features through fine-grained steering: targeted changes in visual perception, semantic representation, and spatial localization produce predictable changes in downstream action prediction. Finally, we use feature-based data attribution to isolate where a specific behavior (failure recovery) arises in the training dataset, identifying fewer than $1.4\%$ of training tokens whose masking is sufficient to remove the targeted behavior. Overall, our results show how matching featurizers to representation geometry can provide more effective tools for understanding and changing robot behavior.

Figures & tables

Figure 1: Block-sparse featurizers (BSFs) recover multi-dimensional robotics concepts. Examples from vision-language-action models encode end-effector position (top), fruit-and-vegetable semantics (middle), and proprioceptive state (bottom). Each point is a token’s 3D feature code, colored by PCA coordinates, task or semantic labels, and sparse-autoencoder (SAE) assignments across columns. Activations from MolmoAct2 L25, L21, L13 (top to bottom). Left overlays share the PCA colors. SAE features tile distinct regions within each block, showing how BSFs group related variation into a single feature.

Figure 2: BSFs better capture low-dimensional manifolds. (b) One three-coordinate BSF block reconstructs the helix. (c) Many individual SAE features tile the helix.

Panel of Figure 3 — Figure 3: Reconstruction, dimensionality, and behavioral relevance of SAE and BSF features in MolmoAct2. Left three panels (layer 13): Held-out reconstruction $R^{2}$ and absolute recorded-action log-likelihood change versus active blocks $k$ (lower change is better; mean $\pm$ SEM, 48 episodes), and within-block stable rank versus block size at $k=32$ (bands: interquartile range; gray: full dimension). Colors denote block size ( $b=1$ : SAE); solid/dashed lines denote Vanilla/Fisher training. Right two panels (across layers): Three-dimensional Vanilla and Fisher BSFs versus SAEs at approximately matched latent widths and active-coordinate budgets (21 blocks versus 63 latents). Ablation effects measure mean predicted end-effector displacement over the top seven BSF blocks or 21 SAE latents by activation mass. Decodability is the best single-feature ridge-probe $R^{2}$ on held-out episodes, averaged over ten robot-state and task-progress variables.

Figure 4: Fine-grained steering through three representations. Semantic , visual , and geometric interventions redirect predicted actions from the prompted object to the other object while leaving input images and instructions unchanged. Results cover 52 held-out scene–instruction comparisons with six noise seeds each. # Features : selected-set size, counting three-dimensional BSF blocks or scalar SAE features. Held-out recovery : fraction of the activation-patching effect reproduced by the selected features, averaged over comparisons for which feature discovery finds a steering set. ∗ A steering set is found for only 50/52 comparisons; all other rows cover all 52. Details in Appendix B.1 .

	SAE				BSF
Object pair	# Feat.	Select	Grasp	Task	# Feat.	Select	Grasp	Task
Apple / banana	2.0	90%	85%	75%	2.5	100%	95%	90%
Bowl / cable	2.5	100%	95%	85%	2.0	100%	100%	100%
Mouse / towel	3.0	75%	70%	65%	2.5	100%	95%	90%
Total	2.5	88.3%	83.3%	75.0%	2.3	100%	96.7%	93.3%

Table 1: Closed-loop semantic target redirection. The policy is instructed to pick up one object while a fixed semantic feature intervention redirects it to the other. Entries are success rates over 20 trials per object pair (10 per direction; 60 in total): Select, the policy goes for the redirected object; Grasp, it grasps that object; Task, it places that object in the basket. # Feat.: features per direction, averaged. Per-direction counts are in Table 3 .

Figure 5: Visual BSF features capture variation across task stages and robot parts. (left) Marker feature F2973 , with points colored by task completion. Linked images illustrate changes in appearance across a single episode; the larger overlays show the four highest-activating timesteps in the dataset. (right) Robot feature F2571 , with PCA colors and linked images showing how different regions of the block correspond to different robot parts. Both features are Fisher-weighted BSFs from MolmoACT2 layer 7.

Figure 6: A single BSF feature links related container concepts across vision and language. The left column of point clouds shows activations colored by PCA coordinates (top) and token modality (bottom). The right column shows only activating text tokens, colored by token identity (top) and SAE assignment (bottom). The feature nicely represents different types of containers across spatially distinct positions. This feature is Vanilla BSF F1777 from $\pi_{0.5}$ layer 3.

Figure 7: A sample of BSF control-relevant features. Left: MolmoACT2 layer 13, F937 , sampled across the dataset. Colorings show PCA coordinates, SAE assignment, image-patch position, and gripper state; linked images illustrate the gripper configurations and locations associated with selected points. Middle: $\pi_{0.5}$ layer 7, F2113 , sampled from a single episode and colored by PCA coordinates (top) and end-effector $x$ , $y$ , and $z$ position (bottom). Right: MolmoBot layer 24, F725 , colored by camera side and camera position in the robot frame, with four representative views.

Figure 8: Sampling of vanilla BSFs proprioception features. Top: A MolmoACT2 layer-13 feature shown with PCA coloring (left), quantized state value from 0 to 255 (middle), and state-component index (right).

Method	Reopen (s) $\uparrow$	Success (%) $\uparrow$	Complete (s) $\downarrow$
Unmodified	3.13	90	12.13
Random	4.34	100	10.15
Diff. means	5.01	65	17.49
kNN	3.45	100	12.82
Influence	5.74	100	11.12
Concept (ours)	12.93	100	11.79

Table 2: Behavior shaping. Mean reopening time (15 perturbed trials) and task performance (20 unperturbed trials; 500-step limit). Longer reopening indicates stronger recovery suppression; completion time averages successful trials only.

Figure 9: Feature associated with failure recovery. Left: F1515 block codes colored by PCA coordinates and SAE assignment. Right: Wrist observations, gripper state, and mean feature norm across exterior-image, wrist-image, and text tokens in a DROID demonstration (top) and a real-robot rollout (bottom). Feature activations are measured by replaying the recordings through $\pi_{0.5}$ -DROID.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Highest-scoring frames for each labeler. Each column shows five DROID episodes for one method. For the four scored methods, these are the five highest-scoring episodes, and $t$ is the highest-scoring evaluated frame within that method’s selected windows. Random has no score, so its column shows five randomly drawn windows with $t$ at each window’s midpoint; its row order carries no meaning.

Figure 11: Open and closed-loop steering setups . Open loop ( Section B.1 ): we find candidate semantic features and evaluate them on the mirrored scene for semantic features. Visual edits are instead evaluated under the other object’s instruction, and geometric edits on held-out object pairs. Closed loop : feature sets are discovered on both mirrored layouts, then fixed and tested on new scenes and tasks, “pick up the banana and put it in the basket".

Objects (A / B)	Featurizer	# Feat.	Select	Grasp	Successful	# Feat.	Select	Grasp	Successful
		A $\rightarrow$ B				B $\rightarrow$ A
Apple / banana	SAE	2	10/10	9/10	7/10	2	8/10	8/10	8/10
	BSF	2	10/10	10/10	9/10	3	10/10	9/10	9/10
Cable / bowl	SAE	3	10/10	9/10	8/10	2	10/10	10/10	9/10
	BSF	2	10/10	10/10	10/10	2	10/10	10/10	10/10
Towel / mouse	SAE	4	6/10	5/10	4/10	2	9/10	9/10	9/10

Appendix

Table 3: Closed-loop redirection by direction. Trials out of 10. Full version of Table 1 . A

\rightarrow

B: instructed to pick up A while steered toward B. Feat.: number of features in the intervention; Grasp counts trials that grasp the redirected object; Successful counts trials that place it in the basket.

Robot Foundation Models

Robot Foundation Models are pretrained systems that combine perception, reasoning, and action to support diverse tasks and robot embodiments.

See all 138 papers

SteerAct: Unlocking Action Models through Few-Shot Temporal Steering
Oct 19, 2026Kaichen Zhou, Zhiyan Li, Chongkai Gao +11Data-Efficient Robot Learning Test-Time Adaptation of VLA Models
Massachusetts Institute of Technology · Harvard University · Stanford University +3
PredVLA: Predictive Sensorimotor Modeling for Sub-Million-Parameter Robot Manipulation
Aug 27, 2026Hiroki Sawada, Shunichi KasaharaLanguage-Conditioned Robot Control Efficient VLA Models
Sony Computer Science Laboratory Tokyo, Japan
Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
Mar 19, 2026Bryce Grant, Xijia Zhao, Peng WangVision-Language-Action Models Robotics
Case Western Reserve University

Oct 19, 2026J/K move · Enter open · S save

SteerAct: Unlocking Action Models through Few-Shot Temporal Steering

Kaichen Zhou, Zhiyan Li, Chongkai Gao, Yuhan Wang+10

Massachusetts Institute of Technology · Harvard University · Stanford University+3

Data-Efficient Robot Learning Test-Time Adaptation of VLA Models

Open paper AskPDF arXiv

Improving pretrained robot action models at test time from only a few rollouts remains challenging. Activation steering offers a lightweight approach by modifying feature activations while keeping model weights fixed. However, effective steering requires identifying which features to modify and determining the timing and strength of modification. We introduce SteerAct, a few-shot temporal steering framework that uses a shared temporal descriptor of sparse-feature dynamics for offline feature selection and online control. A pretrained sparse autoencoder provides candidate features, and our temporal descriptor summarizes the distribution of local activation-order patterns for each feature, enabling comparisons across rollouts without requiring timestep alignment. Before steering begins (offline), we use successful and failed reference rollouts to select a single task-specific feature whose temporal descriptors best distinguish the two outcomes and construct its reference banks. During each new execution (online), we compare the feature's evolving temporal descriptor with these references to determine when to intervene and how strongly to increase or decrease its activation via a steering coefficient. Using only 24 outcome-labeled rollouts per task, SteerAct improves task success by approximately 10% relative to unsteered models in evaluations spanning

π_{0.5}

, Fast-WAM, RoboTwin 2.0, DexJoCo, and real-world manipulation, without updating action-model weights. Code and demos are available at: https://steeract.github.io/.

Abstract

Robot Foundation Models

Explore similar work

SteerAct: Unlocking Action Models through Few-Shot Temporal Steering

PredVLA: Predictive Sensorimotor Modeling for Sub-Million-Parameter Robot Manipulation

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models