From Pixels to Actions - Towards Native Vision-Language-Action Policies

Weichen Fan; Haiwen Diao; Penghao Wu; Siyuan Hu; Ziqi Huang; Tianqi Liu; Mingze Sun; Ziyi Wang

cs.CVNov 4, 2026

From Pixels to Actions - Towards Native Vision-Language-Action Policies

Authors: Weichen Fan, Haiwen Diao, Penghao Wu, Siyuan Hu, Ziqi Huang, Tianqi Liu, Mingze Sun, Ziyi Wang,

Organizations: S-Lab, Nanyang Technological University · Ropedia · SenseTime Research

Abstract

Vision-language-action models are typically built on a simple premise: perception is first compressed by visual encoders, and action reasoning begins only afterwards. Yet manipulation often hinges on precisely the visual signals most vulnerable to this abstraction, e.g., local geometry, boundaries, depth, and subtle state changes. This raises a fundamental question: should action policies inherit a fixed visual endpoint, or directly access evolving visual computation? We introduce NEO-VLA, a structurally encoder-free vision-language-action framework. Images are tokenized by lightweight patch convolutional layers and evolve jointly with language throughout the backbone, allowing a flow-matching action expert to draw from instruction-conditioned visual representations at different depths. This simple change reveals a consistent pattern. Native architectures outperform encoder-based counterparts under controlled comparisons; full fine-tuning reduces but does not erase the gap, and additional scale alone cannot recover it. More strikingly, layer-wise probing shows that the action-critical signals often peak well before the final visual representation. Across diverse manipulation settings and distribution shifts, NEO-VLA consequently achieves stronger and more robust control. Together, these results suggest that embodied control benefits from access to visual representations across the computational hierarchy, rather than a fixed encoder output.

Figures & tables

Figure 1: Three VLA designs : encoder-based vs. encoder-free backbones, and discrete vs. continuous actions. (a) A separate visual encoder converts pixels into image features, which are consumed by the language backbone together with text tokens; actions are generated as discrete tokens. (b) The same encoder-based visual interface is retained, while a separate action head predicts continuous actions conditioned on the backbone. (c) NEO-VLA removes the separately pretrained visual encoder: a lightweight patch embedder maps pixels directly into the backbone token space, while text tokens enter the backbone jointly. A narrow action expert then reads cached keys and values from selected backbone layers and predicts a flow-matching velocity field.

Figure 2: NEO-VLA: pixels in, continuous actions out. Previous and current multi-view observations preserve their aspect ratios within a fixed token budget and enter patch embedder shared across time and views. The visual path uses no vision transformer, autoencoder, or vector-quantised codebook. Shared MLPs embed camera intrinsics and rotations, while state MLPs embed previous and current robot states. These tokens join a structured prompt describing control frequency, embodiment, frame reference, and task instruction. A separate, narrow, randomly initialised action expert jointly attends to cached prefix keys and values from selected backbone layers and its own action-token keys and values. Its suffix embeddings combine projected noisy actions and flow time $\tau$ with additive step, frame, and identity embeddings. Both arm blocks and the body block share temporal rotary offsets $0$ – $29$ , with identity embeddings distinguishing the blocks. Trained by flow matching, the expert predicts a velocity field that covers both arm and body action chunks.

Panel of Table 1 — Table 1: Pretraining corpus. Nine public dataset families contribute 2,231,583 episodes and 9,927 hours. As held is the share of stored rows; As sampled is the share of robot-row draws during training.

Figure 3: Data quality filtering. Illustrative checks cover implausible joint readings, unreliable channels, action outliers, corrupt episodes, and camera calibration, removing only affected data. Joint and projection checks are source-specific; episode retention is determined in the final corpus pass.

Backbone	Size	Adaptation	Spatial	Object	Goal	Long	Average
Qwen3-VL	2B	Frozen	$71.4$	$89.8$	$92.8$	$64.0$	$79.5$
		Full FT	$92.2$	$96.6$	$95.8$	$78.6$	$90.8$
		Encoder unfrozen	$91.8$	$92.0$	$94.0$	$79.0$	$89.2$
Gemma-3	4B	Frozen	$80.2$	$93.6$	$92.6$	$71.6$	$84.5$
		Full FT	$89.0$	$97.6$	$94.8$	$82.8$	$91.0$
NEO-VLA	2B	Frozen	$89.2$	$95.8$	$93.2$	$85.4$	$90.9$

Table 2: Backbone comparison on LIBERO. Success rates (%) at the final 30,000-step checkpoint, averaged over 2,000 episodes. Size refers to the backbone, excluding the action expert. Frozen updates only the expert; full FT updates the whole stack. Native-input rows use the linear-depth layer map. Encoder unfrozen updates the vision encoder and action expert with other layers frozen.

Adaptation	Original interface	Encoder access
Frozen backbone	79.5	$64.9$
Full FT	$90.8$	90.9

Table 3: Direct encoder access in Qwen3-VL. Average LIBERO success (%) with the same setting. Encoder access adds no parameters.

Adaptation	Layer map	Depth	LIBERO suites $\uparrow$				Average
Adaptation	Layer map	Depth	Spatial	Object	Goal	Long	Average
Frozen	first-28 $[0..27]$	28	94.8	98.4	97.6	88.8	94.9
	skip-28 $[\mathrm{uniform}]$	40	89.2	95.8	93.2	85.4	90.9
	end-28 $[12..39]$	40	78.4	93.6	94.2	80.4	86.7
Frozen , disjoint	first-14 $[0..13]$	14	95.6	97.2	96.2	85.4	93.6
Frozen , disjoint	last-14 $[26..39]$	40	92.2	95.2	89.4	81.8	89.6

Table 4: Layer-map ablation on LIBERO. Success rate (%) over 2,000 rollouts per configuration at training step 30k. Depth is the number of backbone layers retained. skip-28 and end-28 both use a 40-layer backbone and a 28-layer expert. first-28 and first-14 also shorten the backbone, so the disjoint first-14 / last-14 comparison changes backbone depth as well as readout position. Higher is better.

Figure 4: Interventions at the expert’s attachment points. (A) Translation error after swapping one attachment’s prefix keys and values with another observation’s (purple), adding matched-magnitude residual noise (orange), or making no change (gray). (B) Action change when masking the wrist image or instruction; filled and open markers denote separate even- and odd-attachment runs. (C) Wrist-pose masking in the robot-frame model from B and a wrist-frame model.

Model	Size	Open	Camera	Robot	Language	Light	Background	Noise	Layout	Total
$\pi_{0}$	3B	$\checkmark$	$13.8$	$6.0$	$58.8$	$85.0$	$81.4$	$79.0$	$68.9$	$53.6$
StarVLA	4B	$\checkmark$	$52.5$	$49.8$	$88.5$	$95.7$	$95.7$	$73.0$	$76.9$	$74.1$
ABot-M0	4B	$\checkmark$	$60.4$	$67.9$	$86.4$	$96.2$	$91.6$	$86.4$	$82.6$	$80.5$
Cosmos-Policy	2B	$\checkmark$	$75.8$	$63.3$	$81.7$	$96.5$	$88.9$	$92.7$	$82.2$	$82.2$
$\pi_{0.5}$	3B	$\checkmark$	$78.4$	$73.6$	$80.8$	$96.2$	$94.1$	$89.0$	$84.5$	$84.4$
Qwen-RobotManip-scratch	4B	$\times$	$70.4$	$44.9$	$88.1$	$95.8$	$95.5$	$84.4$	$79.1$	$78.3$

Table 5: LIBERO-plus by perturbation axis. Open indicates whether both code and weights are publicly available, other results follow Yuan et al. (2026) . The total is task-weighted rather than the mean of the seven axes; bold marks the best value in each column.

	Baseline	+ Co-training	+ Action-only
MMB	82.7	81.2	–
RWQA	66.8	68.4	–
AI2D	82.9	82.3	–
TextVQA	76.6	76.4	–
ChartQA	80.8	80.4	–
RoboTwin C2R	–	69.4	10.7

Table 6: Preserving multimodal understanding enables robust VLA post-training. Understanding scores (%) and RoboTwin 2.0 clean-to-randomized (C2R) success (%) before post-training (Baseline), after co-training with diverse understanding corpus (+ Co-training), and after action-only fine-tuning (+ Action-only). The action-only model cannot produce valid text, while co-training keeps understanding capability and gives the best manipulation policy.

Model	Size	Open	Spatial	Object	Goal	Long	Average
OpenVLA	7B	$\checkmark$	$84.7$	$88.4$	$79.2$	$53.7$	$76.5$
$\pi_{0.5}$	3B	$\checkmark$	$98.8$	$98.2$	$98.0$	$92.4$	$96.8$
MMaDA-VLA	8B	$\checkmark$	$98.8$	$99.8$	$98.0$	$95.2$	$98.0$
EO-1	3B	$\checkmark$	99.7	99.8	99.2	$94.8$	$98.4$
Qwen-RobotManip	4B	$\times$	—	—	—	—	99.1
NEO-VLA	2B	$\checkmark$	$98.4$	$99.4$	$95.6$	96.2	$97.4$

Table 7: LIBERO . NEO-VLA is evaluated over 2,000 episodes per suite at step 27,500. Size denotes parameter scale. Open indicates public weights.

Model	Size	Object	Position	Semantic	Task	Avg.
OpenVLA	7B	$93.0$	$0.0$	$97.3$	$0.0$	$47.6$
$\pi_{0}$	3B	$90.5$	$0.0$	$90.5$	$0.0$	$45.3$
$\pi_{0.5}$	3B	96.0	20.8	$95.8$	$0.8$	53.3
X-VLA	0.9B	$79.0$	$0.8$	$96.8$	6.8	$45.8$
NEO-VLA	2B	$84.3$	$3.0$	98.0	$6.3$	$47.9$

Table 8: LIBERO-Pro. Evaluation of VLA models under Object, Position, Semantic, and Task perturbations. Size denotes parameter scale.

Model	Size	Open	Clean	Clean2Rand
FastWAM	6B	$\checkmark$	$77.80$	$1.90$
StarVLA-OFT	3B	$\checkmark$	$46.52$	$3.16$
$\pi_{0.5}$	3B	$\checkmark$	$70.70$	$46.00$
OLA-Sem	8B	$\times$	$75.12$	$67.56$
Qwen-RobotManip	4B	$\times$	89.58	$68.12$
NEO-VLA	2B	$\checkmark$	$75.78$	69.40

Table 9: RoboTwin 2.0. We report the official setting with 2.5K episodes. Size denotes parameter scale. Open indicates public weights.

Model	Size	Task1	Task2	Task3
StarVLA	3B	1/20	2/20	0/20
$\pi_{0.5}$	3B	10/20	14/20	13/20
NEO-VLA	2B	12/20	17/20	13/20

Table 10: Real-World evluation. We report task success rate over 20 episodes per task, with 10 trials in domain and the other 10 trials out-of-domain.

Figure 5: Real-world manipulation benchmark. (a) Place a tennis ball on a can. (b) Place a tape roll in a bowl, then place a tennis ball inside the roll. (c) Reorient a horizontal can and insert it upright into a tape roll. The tasks test placement precision, multi-step execution, and object reorientation. Each model is fine-tuned on 50 episodes per scene and evaluated over 20 trials.

Figure 6: Real-world NEO-VLA demonstrations. (a) Pick up a paper cup and pour water into a glass. (b) Place a tennis ball on a can. (c) Place a tape roll in a bowl, then place the ball inside the roll. (d) Reorient a horizontal can and insert it upright into a tape roll. (e) Pick four gunkan sushi pieces, one at a time, from a table of mixed sushi and place them on a plate. Frames are arranged chronologically from left to right, with dashed circles highlighting key interaction moments and critical manipulation steps.

Policy
Stage	10 steps	2 steps
Patch embedder	$3.1$	$2.7$
Backbone prefix (40 layers)	$84.8$	$84.7$
Action expert sampling	$163.4$	$32.4$
Forward pass, eager	$515.4$	$210.7$
Forward pass, compiled	$282.5$	$147.8$

Table 11: Inference latency summary. We report milliseconds per query on one RTX 3090 in BF16, with raw 30-step chunks.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,207 papers

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
Mar 19, 2026Bryce Grant, Xijia Zhao, Peng WangVision-Language-Action Models Robotics
Case Western Reserve University
Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models
Mar 16, 2026Yulin Luo, Hao Chen, Zhuangzhe Wu +10Spatial Grounding for VLA Models Vision-Language-Action Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The Chinese University of Hong Kong · Simplexity Robotics
APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
Jun 10, 2026Kechun Xu, Zhenjie Zhu, Anzhe Chen +2Visuomotor Policy Learning Vision-Language-Action Models
Zhejiang University · Zhejiang Humanoid Robot Innovation Center

Mar 19, 2026J/K move · Enter open · S save

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Bryce Grant, Xijia Zhao, Peng Wang

Case Western Reserve University

Vision-Language-Action Models Robotics

Open paper AskPDF arXiv

A fine-tuned Vision-Language-Action (VLA) policy can easily place the alphabet soup in the basket on command, just as it saw during fine-tuning. However, if the basket is shifted five centimeters to the left, it drops the soup off the table. We developed an interpretability pipeline to investigate what such a policy is computing. Injecting task A's activations into the action expert while the policy looks at task B's scene makes pi0.5 execute task A's motor trajectory in 99.6% of episodes (

n{=}1{,}968

). The injected program is bound to absolute workspace coordinates and ignores the visible scene, which provides a mechanistic account of the perturbation brittleness that concurrent benchmarks report. The binding holds on a physical UR10e: directed replay redirects the commanded task in 26/26 rollouts in one direction and 18/26 in the other, with no-injection and random-direction controls at the floor, and once the source object is moved 10--15,cm the arm grasps and releases at the recorded coordinates in 50/50 trials, picking up the wrong block when one sits there and closing on empty air when none does. Applying the same interventions to six VLAs on LIBERO, MetaWorld, SimplerEnv, and ALOHA over 420{,}000+ rollouts yields three further findings: every architecture encodes the instruction, and four of the five weight it below vision whenever the scene alone identifies the goal; under the standard replacement write-back the SAE pooling that preserves behavior is architecture-specific (pi0.5 per-token, SmolVLA indifferent), and goal selection and motor generation separate into VLM and expert pathways wherever the two are distinguishable. We release Action Atlas (https://action-atlas.com), an interactive platform covering all 520+ trained SAEs, 79 identified concepts, and the scored UR10e rollouts. Code: https://github.com/CWRU-AISM/action-atlas

Abstract

Vision-Language-Action Models

Explore similar work

Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models

Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models

APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies