STAR: Steering Action Representations during Inference

Guangyu Zhao; Yongchang Zhang; Haoxuan Ru; Kaichen He; Haoran Wen; Kun Zhan; Zeyu Zhang; Yin Li

cs.ROOct 29, 2026

STAR: Steering Action Representations during Inference

Authors: Guangyu Zhao, Yongchang Zhang, Haoxuan Ru, Kaichen He, Haoran Wen, Kun Zhan, Zeyu Zhang, Yin Li,

Organizations: Peking University · Li Auto · BIGAI · University of Wisconsin–Madison

Abstract

Vision-language-action models (VLAs) map language instructions and visual observations to robot actions, showing strong capabilities across robotic manipulation tasks. However, in our probing experiment, we find that action representations near the action output do not fully preserve task-relevant visual grounding. Motivated by this late-layer grounding gap, we introduce STAR, for STeering Action Representations, a training-free inference method for pre-trained VLAs. STAR applies an exponential moving average over late-layer residual updates, allowing later action representations to inherit object-aware update directions from earlier layers without model updates or architectural changes. It further uses recent proprioceptive states to adapt the mixing strength, using conservative mixing when execution dynamics change rapidly. Experiments on robot manipulation benchmarks show that STAR improves late-layer object awareness and increases average task success rates by 4.0 percentage points. Our code is available at https://github.com/PKU-VaLuE-Lab/STAR.

Figures & tables

Figure 1: A case study of layer-wise object awareness in VLA inference. (a) Original and object-masked RGB observations in LIBERO-PRO, where task-relevant object regions are replaced with the mean RGB values of the image. (b) A Transformer layer in a VLA, mapping the action hidden state from $h^{k}$ to $h^{k+1}$ through a residual update. (c) Object awareness of $\pi_{0.5}$ layer outputs across depth under native inference and our method STAR. (d) Success rates of native inference and our method STAR on LIBERO-PRO for $\pi_{0.5}$ and GR00T-N1.5, respectively.

Figure 2: Overview of action generation at step

t

. STAR estimates a step-level EMA coefficient

\beta_{t}

from proprioceptive history and applies it to layer-wise updates. Blue arrows denote the EMA coefficient

\beta_{t}

derived from proprioceptive history, while pink arrows denote the cross-layer update momentum used in the EMA mixing.

Backbone	Method	SimplerEnv Google Robot Tasks						Avg.
Backbone	Method	Open Drawer	Place in Drawer	Pick Coke Can	Move Near	Close Drawer	Pick Object	Avg.
GR00T-N1.5 [ 33 ]	Native	26.7%	4.0%	69.3%	68.7%	45.0%	32.3%	41.0%
	DTP	26.7%	4.0%	59.3%	68.7%	41.7%	39.3%	40.0%
	STAR (Ours)	32.0%	4.7%	77.3%	76.7%	47.3%	40.0%	46.3%
OpenVLA [ 18 ]	Native	7.4%	0.0%	65.3%	56.7%	22.2%	–	30.3%
	DTP	11.1%	0.0%	69.3%	69.0%	22.2%	–	34.3%

Table 1: Main results on SimplerEnv. For OpenVLA, the average is computed over the available tasks. STAR improves frozen VLA policies on both backbones and achieves the best average success rate, outperforming both native inference and DTP while remaining single-pass.

Backbone	Method	LIBERO-PRO Perturbations on LIBERO-10					Avg.
Backbone	Method	lan	object	swap	task	env	Avg.
GR00T-N1.5 [ 33 ]	Native	81.2%	40.4%	0.0%	6.0%	73.8%	40.3%
	DTP	82.0%	42.2%	0.0%	1.8%	75.6%	40.3%
	STAR (Ours)	82.8%	48.0%	2.2%	6.8%	76.4%	43.2%
$\pi_{0.5}$ [ 39 ]	Native	92.8%	66.2%	9.2%	6.8%	65.8%	48.2%
	DTP	90.4%	64.6%	8.0%	7.8%	67.6%	47.7%

Table 2: Main results on LIBERO-PRO. STAR improves the average success rate on both GR00T-N1.5 and

\pi_{0.5}

, while DTP gives limited or mixed gains on this long-horizon benchmark.

Variant	Proprioceptive Evidence				SimplerEnv Google Robot Tasks					Avg.
	$c_{t}$	$v_{t}$	$j_{t}$	$\omega_{t}$	Open Drawer	Place in Drawer	Pick Coke Can	Move Near	Close Drawer	Avg.
Full STAR (Ours)	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	18.5%	3.7%	70.0%	61.7%	25.9%	36.0%
w/o direction consistency	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	14.8%	0.0%	64.0%	56.7%	14.8%	30.1%
w/o translation scale	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	11.1%	0.0%	66.7%	55.0%	22.2%	31.0%
w/o jerk	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	14.8%	3.7%	65.3%	58.3%	14.8%	31.4%
w/o rotation change	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	14.8%	0.0%	66.7%	58.3%	18.5%	31.7%

Table 3: Ablation on proprioceptive evidence. Using all four proprioceptive cues gives the best average success rate, showing that mixed updates benefit from both steady-progress and transition-sensitive signals.

Variant	SimplerEnv Google Robot Tasks					Avg.
	Open Drawer	Place in Drawer	Pick Coke Can	Move Near	Close Drawer	Avg.
Fixed- $\beta=0.25$	11.1%	0.0%	68.0%	56.7%	22.2%	31.6%
Fixed- $\beta=0.35$	14.8%	0.0%	65.3%	56.7%	14.8%	30.3%
Fixed- $\beta=0.45$	9.3%	1.9%	65.7%	58.3%	18.5%	30.7%
Fixed- $\beta=0.55$	14.8%	0.0%	64.0%	60.0%	14.8%	30.7%
Dynamic- $\beta$ $(0.15,0.55)$	14.8%	0.0%	64.0%	55.0%	22.2%	31.2%

Table 4: Ablation on beta scheduling. A proprioceptively scheduled

\beta

with a moderate range outperforms both fixed cross-layer mixing strengths and poorly calibrated dynamic ranges.

Variant	SimplerEnv Google Robot Tasks					Avg.
	Open Drawer	Place in Drawer	Pick Coke Can	Move Near	Close Drawer	Avg.
$\rho=0$	14.8%	0.0%	49.3%	53.3%	11.1%	25.7%
$\rho=1/3$	11.1%	0.0%	55.0%	68.3%	3.7%	27.6%
$\boldsymbol{\rho}$ $\mathbf{=2/3}$ (Ours)	18.5%	3.7%	70.0%	61.7%	25.9%	36.0%
$\rho=1$ (Native)	7.4%	0.0%	65.3%	56.7%	22.2%	30.3%

Table 5: Ablation on start layer. Applying mixed updates only to the final third of layers gives the best average success rate, supporting late residual updates as the effective intervention target.

Method	Successes / Trials	Success Rate
Native	$17/40$	42.5%
STAR (Ours)	$19/40$	47.5%

Table 6: Real-robot results. Success rates on the pear-to-bowl task with

\pi_{0.5}

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Object-awareness and attention-map diagnostics for native inference and STAR. (a) Visualization of the three masks, including the task-specific object mask $m_{o}$ , randomly scattered pixels $m_{r}$ with the same pixel count, and a randomly sampled rectangular background region $m_{b}$ of similar area. (b) Comparison of awareness scores for native inference and STAR under object masking and control masking at layer 18. (c) Attention-map visualization at layers 13, 16, and 18, showing that native inference and STAR produce similar attention patterns. (d) Attention mass on the task-object region across layers. Native inference and STAR exhibit close attention-mass curves, indicating that attention-based object focus does not clearly reveal the hidden-state intervention effect.

Figure 4: Action-gradient scores on LIBERO-PRO. (a) Mean normalized input-gradient evidence contained in the task-object region for native inference and STAR across the five LIBERO-PRO evaluation splits. Higher values indicate that the predicted action depends more strongly on the key object. STAR consistently increases the score on all five splits. (b) Distribution of paired score differences, computed as STAR minus native inference. Most of the mass lies to the right of zero, showing that the increase is consistent across evaluation samples rather than being driven only by split-level averages.

Figure 5: Qualitative action-gradient saliency maps. The left column shows the RGB observations with task-relevant objects marked by cyan boxes. The middle and right columns show saliency under native inference and STAR, respectively. Warmer pixels exert a stronger local influence on the predicted action. Compared with native inference, STAR concentrates more action-relevant evidence within the marked object regions across diverse manipulation scenes.

Backbone	Benchmark	$\boldsymbol{\beta}_{\min}$	$\boldsymbol{\beta}_{\max}$	$\boldsymbol{w}_{c}$	$\boldsymbol{w}_{v}$	$\boldsymbol{w}_{j}$	$\boldsymbol{w}_{\omega}$	$\boldsymbol{\kappa}$
$\pi_{0.5}$	LIBERO-PRO	0.25	0.55	0.25	1.0	$-0.125$	$-0.25$	2.30
GR00T-N1.5	LIBERO-PRO	0.20	0.50	1.0	1.0	$-0.375$	$-2.0$	3.75
GR00T-N1.5	SimplerEnv	0.20	0.50	1.0	1.0	$-0.375$	$-2.0$	4.00
OpenVLA	SimplerEnv	0.25	0.55	1.0	1.0	$-0.25$	$-0.75$	4.75

Appendix

Table 7: Hyperparameter settings for each backbone and benchmark.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,202 papers

StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models
Aug 12, 2026Siyu Xu, Yunke Wang, Zijian Wang +6Reasoning for VLA Models Continual Learning for VLA Models
StellarEdge AI Technical Team
StarVLA- $α$ : Reducing Complexity in Vision-Language-Action Systems
Apr 13, 2026Jinhui Ye, Ning Gao, Senqiao Yang +7Robot Foundation Models Robotic Manipulation
HKUST · XJTU · CUHK +3
Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models
Aug 5, 2026Xingyu Ding, Yuzhong Zhao, Yang Wu +4Robotic Manipulation Vision-Language-Action Models
Nanjing University, Nanjing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China

Aug 12, 2026J/K move · Enter open · S save

StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models

Siyu Xu, Yunke Wang, Zijian Wang, Dihao Zhu+5

StellarEdge AI Technical Team

Reasoning for VLA Models Continual Learning for VLA Models Robotic Manipulation Instruction Generalization in VLA Models Robustness of VLA Models Vision-Language-Action Models+3

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models can follow instructions and manipulate objects, but their performance often collapses out of distribution (OOD), when the scene, viewpoint, or object differs from training. Adapting to each new situation typically requires collecting more data and fine-tuning. We present StellaVLA, a framework that instead adapts at test time by conditioning on a single retrieved demonstration. The key idea is to move beyond imitating what an expert did and instead convey why: an automated offline pipeline converts each raw trajectory into a structured demonstration, e.g., a task plan, sub-goal descriptions, and verbalized 3D motion, at zero human-annotation cost. Provided as in-context guidance, this structured demonstration lets the policy reason about the task rather than mimic a pixel trajectory, which also makes it transferable across embodiments (real-robot, human-hand, or XR demonstrations). A parallel dual-training design internalizes this reasoning during training through a joint action-and-language objective, while inference uses the action expert alone, preserving real-time, high-frequency control with no added latency. On the VLA-Arena leaderboard(Aug 1, 2026), StellaVLA ranks first with an overall score of 0.63, versus 0.44 and 0.22 for the strong prior models (

π_{0.5}

and LingBot-VLA), and it further leads on LIBERO with 98.8% average success rate and LIBERO-Plus with 85.1% success rate. Our real-robot benchmark demonstrates that StellaVLA can use both human/robot demos and human-to-robot (XR) demos as in-context structured demonstration to help VLA model adapt to OOD tasks.

Abstract

Vision-Language-Action Models

Explore similar work

StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models

StarVLA-ααα: Reducing Complexity in Vision-Language-Action Systems

Mind-VLA: Instruction-Aware Spatial Representation Alignment for Vision-Language-Action Models

StarVLA- $α$ : Reducing Complexity in Vision-Language-Action Systems