SteerAct: Unlocking Action Models through Few-Shot Temporal Steering

Kaichen Zhou; Zhiyan Li; Chongkai Gao; Yuhan Wang; Wenlong Huang; Maggie Wang; Fangneng Zhan; Manling Li

cs.ROOct 19, 2026

SteerAct: Unlocking Action Models through Few-Shot Temporal Steering

Authors: Kaichen Zhou, Zhiyan Li, Chongkai Gao, Yuhan Wang, Wenlong Huang, Maggie Wang, Fangneng Zhan, Manling Li,

Organizations: Massachusetts Institute of Technology · Harvard University · Stanford University · Imperial College London ·

Abstract

Improving pretrained robot action models at test time from only a few rollouts remains challenging. Activation steering offers a lightweight approach by modifying feature activations while keeping model weights fixed. However, effective steering requires identifying which features to modify and determining the timing and strength of modification. We introduce SteerAct, a few-shot temporal steering framework that uses a shared temporal descriptor of sparse-feature dynamics for offline feature selection and online control. A pretrained sparse autoencoder provides candidate features, and our temporal descriptor summarizes the distribution of local activation-order patterns for each feature, enabling comparisons across rollouts without requiring timestep alignment. Before steering begins (offline), we use successful and failed reference rollouts to select a single task-specific feature whose temporal descriptors best distinguish the two outcomes and construct its reference banks. During each new execution (online), we compare the feature's evolving temporal descriptor with these references to determine when to intervene and how strongly to increase or decrease its activation via a steering coefficient. Using only 24 outcome-labeled rollouts per task, SteerAct improves task success by approximately 10% relative to unsteered models in evaluations spanning $π_{0.5}$ , Fast-WAM, RoboTwin 2.0, DexJoCo, and real-world manipulation, without updating action-model weights. Code and demos are available at: https://steeract.github.io/.

Figures & tables

Figure 1: Steering one feature at the right time. Using just 24 rollouts per task, SteerAct improves robot behavior by steering a single feature activation, without retraining the action model. As the robot acts, SteerAct compares the patterns of how this feature activation changes with patterns from those existing rollouts to decide when to steer and how to steer. (a) Unsteered and steered real-robot executions illustrate how SteerAct improves task completion. (b) SteerAct improves task success across all 18 evaluated RoboTwin 2.0 and DexJoCo tasks with $\pi_{0.5}$ . (c) On DexJoCo with $\pi_{0.5}$ , SteerAct achieves the highest mean success among the evaluated methods.

Figure 2: Temporal evidence guides when and how to steer. (a) The selected feature’s activation traces compare a successful reference, an unsteered rollout, and rollouts with adaptive steering. (b) The temporal-support score measures whether the observed activation history of current rollout provides sufficient evidence for comparison with patterns from successful and failed reference rollouts. (c) The steering coefficient $\alpha_{t}$ determines the direction and strength of the activation edit; $\alpha_{t}=0$ indicates no intervention at that decision.

Figure 3: SteerAct overview. Offline, an SAE fitted to frozen-policy activations exposes candidate directions. Outcome-labeled rollouts determine a feature and its success/failure banks of ordinal dynamics. Online, supported failure-like prefixes enter a signed coefficient search; other prefixes retain $\alpha_{t}=0$ . The selected decoder direction modifies one configured residual stream, after which the frozen policy generates actions.

	RoboTwin 2.0										DexJoCo
Method	Bottles in Dustbin	Stack Three Blocks	Rank Blocks by Size	Dual Shoes	Dump Bin into Big Bin	Open Microwave	Turn Switch	Mouse Pad	Diverse Bottles	Mean	Hammer Nail	Click Mouse	Pick Bucket	Pinch Tongs	Fold Glasses	Water Plant	Hanoi Bi	Microwave Bi	Photo Bi	Mean	Time $\downarrow$ (ms/decision)
Backbone: $\pi_{0.5}$
Base	74	72	43	73	89	84	47	62	77	69.0	82	67	83	22	71	92	15	77	52	62.3	207.9
DSRL-SAC	80	78	40	57	49	100	65	71	100	71.1	80	60	70	18	74	96	10	69	47	58.2	3582.7
DrVLA	62	84	60	86	72	91	48	68	72	71.4	85	76	87	26	68	94	15	84	55	65.6	221.6
COAST	62	74	49	75	88	86	40	66	76	68.4	87	75	79	33	69	93	19	74	59	65.3	214.4

Table 1: SteerAct improves success across backbones and manipulation benchmarks. Among the evaluated methods, SteerAct (Enc) achieves the highest mean success in all four benchmark–backbone settings with modest inference overhead. Success rates are reported in percent (

\uparrow

); each Mean averages nine tasks. Time reports per-decision inference latency (ms,

\downarrow

), shared across datasets. Enc/Dec denote encoder-branch/action-expert steering. Gray highlights SteerAct (Enc); bold / underline mark the best/second-best distinct success rates within each backbone, including ties.

Task	Base	COAST	DrVLA	SteerAct Dec	SteerAct Enc
Folding Cloth	33.3%	26.7%	40.0%	36.7%	53.3%
Cup Ping-Pong	40.0%	36.7%	56.7%	43.3%	63.3%
Push Cube	20.0%	0%	23.3%	20.0%	40.0%
Average	31.1%	21.1%	40.0%	33.3%	52.2%

Table 2: SteerAct improves success on all three real-robot tasks. With

\pi_{0.5}

frozen, SteerAct (Enc) achieves the highest success on every evaluated task, reaching 52.2% average success compared with 31.1% for Base and 40.0% for the strongest baseline, DrVLA. Values are success rates (%,

\uparrow

). Enc/Dec denote encoder-branch/action-expert steering. Gray highlights SteerAct (Enc); bold / underline mark the best/second-best distinct results in each row.

Figure 4: Single-feature steering improves task completion across manipulation settings. Representative unsteered and steered rollouts on RoboTwin 2.0, DexJoCo, and real robots illustrate how SteerAct corrects execution errors and enables successful task completion while keeping the underlying action model frozen.

Method	Base	Marginals	Increments	Events	Autocorr.	Raw NN	Raw DTW	Dec	Enc
Success	69.0	71.1	70.0	70.1	73.9	72.2	71.7	70.9	77.7

Table 3: Ablations on RoboTwin 2.0 with

\pi_{0.5}

. Values are mean success rates (%) across nine tasks. Gray marks the default single-feature Enc variant (24 total source rollouts). Panels (b)–(d) use Enc; (c) counts successful source rollouts only. Full results are in Appendix Table 5 .

Setting	Frozen	Delay 10	Delay 20	Without support gate	Without failure gate	Without gates	Ours
Mean	23.63	24.63	24.88	23.13	29.88	22.38	36.13

Table 4: Mean success rates (%,

\uparrow

) across eight VLABench tasks, with 200 evaluation episodes per task and setting (

\pi_{0.5}

with SteerAct(Enc)). Full results are in Appendix Table 6 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method / Setting	Put Bottles in Dustbin	Stack Three Blocks	Rank Blocks by Size	Place Dual Shoes	Dump Bin into Big Bin	Open Microwave	Turn Switch	Place Mouse Pad	Pick Diverse Bottles	Mean
(a) Representation and steering ablations
Base	74	72	43	73	89	84	47	62	77	69.0
Marginals	76	76	48	77	88	77	49	67	82	71.1
Increments	78	76	39	78	89	83	43	61	83	70.0
Events	76	78	45	77	89	82	50	54	80	70.1
Autocorrelation	82	83	49	83	93	86	49	56	84	73.9

Appendix

Table 5: Ablations on RoboTwin 2.0 with

\pi_{0.5}

: (a) temporal descriptors, matching rules, feature counts, steering strengths, and intervention sites; (b) total source rollouts; (c) successful source rollouts. (d) feature count and coefficient adaptation. Entries are success rates (%,

\uparrow

); Mean averages over the nine displayed tasks. Matching controls use Enc and retain the ordinal-based support gate. Gray highlights the full method. Bold / underline mark the best/second-best distinct results per column within each group, including ties.

Method	Painting	Book	Drinking	Chemistry Tube	Poker	Mahjong	Toy	Fruit	Mean
Frozen	17	46	15	47	18	6	7	33	23.63
Delay 10	17	51	22	47	13	13	5	29	24.63
Delay 20	17	44	19	52	16	11	12	28	24.88
Without support gate	15	49	23	49	12	8	5	24	23.13
Without failure gate	13	48	27	53	22	18	25	33	29.88
Without gates	17	43	24	42	22	9	3	19	22.38

Appendix

Table 6: Timing and gating comparisons on VLABench. Success rates (%,

\uparrow

) use 200 evaluation episodes per task and reported setting; Mean is the task macro-average. Gray highlights maturity-based steering. Bold / underline indicate the best/second-best distinct reported results per column, including ties.

Parameter	Symbol	Value
Support samples	$M$	500
Support-gate threshold	$\tau_{m}$	0.65
Maximum coefficient magnitude	$A_{\max}$	10
Coefficient grid spacing	$\delta_{\alpha}$	0.5

Appendix

Table 7: Shared SteerAct control hyperparameters.

Panel of Figure 6 — Figure 6: Paired simulated rollouts on RoboTwin 2.0 and DexJoCo (continued). The unsteered and steered rows begin from matched initial scenes, and frames advance from left to right.

Data-Efficient Robot Learning

Data-efficient robot learning develops ways for robots to acquire reliable skills with fewer demonstrations, interactions, and human annotations.

See all 188 papers

WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time
Jul 8, 2026Yusen Feng, Bingchen Han, Jiangran Lyu +13Test-Time Adaptation World Models for Robotics
Peking University · Galbot · CASIA +1
STEAM: Self-Supervised Temporal Ensemble Advantage Modeling for Real-World Robot Learning
Jun 29, 2026Zhihao Liu, Qiuyi Gu, Yitao Wang +16Robot Policy Learning Robot Skill Learning
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · Zhongguancun Academy +8
Steering Autoregressive Vision-Language-Action Policies via Action Token Intervention
Jun 12, 2026Jason Chan, Jonathan C. KaoRobotic Manipulation Vision-Language-Action Models
Department of Computer Science University of California, Los Angeles · Departments of Electrical and Computer Engineering, Computer Science, and Neurobiology University of California, Los Angeles

Jul 8, 2026J/K move · Enter open · S save

WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time

Yusen Feng, Bingchen Han, Jiangran Lyu, Kai Liu+12

Peking University · Galbot · CASIA+1

Test-Time Adaptation World Models for Robotics Robot Foundation Models Cross-Embodiment Robot Learning Robot Imitation Learning

Open paper AskPDF arXiv

Steering robot foundation models (RFMs) toward new task variants or user-preferred behaviors remains challenging, often requiring additional robot demonstrations, task-specific fine-tuning, or long-context conditioning. We present WAM-TTT, a test-time training framework for steering world action models from raw human videos. Rather than treating human videos as trajectories to imitate, WAM-TTT absorbs them into a lightweight adaptive memory inside a frozen WAM through self-supervised video prediction. To make this memory useful for control, we introduce a meta-training stage that aligns human demonstrations with robot behaviors using paired human-robot data and a key--value memory reconstruction objective. At test time, only unlabeled human videos are required to adapt the memory, while the pretrained WAM remains frozen. This enables efficient and reusable steering without robot actions, human-side annotations, or task-specific fine-tuning, while preserving the generalization ability of the foundation model. Extensive experiments show that WAM-TTT consistently outperforms in-context human-video conditioning baselines across diverse manipulation tasks and generalization settings.

Abstract

Data-Efficient Robot Learning

Explore similar work

WAM-TTT: Steering World-Action Models by Watching Human Play at Test Time

STEAM: Self-Supervised Temporal Ensemble Advantage Modeling for Real-World Robot Learning

Steering Autoregressive Vision-Language-Action Policies via Action Token Intervention