VLMs are Native Action Models: Unleashing the Manipulation Capabilities of VLMs with Explicit Action-Space Grounding

Jingkun Yi; Shujie Zhang; Zirui Zhou; Haoxiao Wang; Haiyang Sun; Yuhang Dong; Xudong Xu; Xing Shen

cs.ROOct 9, 2026

VLMs are Native Action Models: Unleashing the Manipulation Capabilities of VLMs with Explicit Action-Space Grounding

Authors: Jingkun Yi, Shujie Zhang, Zirui Zhou, Haoxiao Wang, Haiyang Sun, Yuhang Dong, Xudong Xu, Xing Shen,

Organizations: Zhejiang University · Shanghai Artificial Intelligence Laboratory · Tsinghua University

Abstract

Pretrained vision-language models (VLMs) already exhibit strong visual, spatial and reasoning capabilities relevant to robotic manipulation. The key challenge in turning pretrained VLMs into generalist manipulation policies is to expose these capabilities through an appropriate action interface. Vision-language-action (VLA) models provide a unified trainable policy, but typically introduce learned action tokens, action heads, or action experts whose semantics must be learned from robot data pretraining. In contrast, modular foundation-model-based manipulation methods can exploit pretrained VLM capabilities for zero-shot manipulation, but rely on external planning and control scaffolds that are not end-to-end trainable as a unified continuous-action policy. In this work, we introduce EASG, an Explicit Action-Space Grounding framework that bridges these two paradigms. By projecting a tool center point (TCP)-centered action frame into the wrist camera observation and expressing actions in the form of VLM's native textual generation interface, EASG makes the semantics of a continuous robot action space directly observable in the VLM's perceptual context. This allows pretrained VLMs to directly generate executable robot actions using their existing visual-spatial reasoning capabilities, without robot-action pretraining or additional action modules. Besides, the same action interface supports direct fine-tuning with robot trajectories, allowing the models to continuously improve beyond their manipulation capabilities unleashed by EASG. Overall, EASG enables zero-shot manipulation and generalization across object positions, categories, and robot embodiments, while supporting effective fine-tuning in the same continuous action space. These results demonstrate that, given an appropriate action interface, pretrained VLMs can themselves serve as native action models for robotic manipulation.

Figures & tables

Figure 1: VLMs are native action models. With explicit action space grounding, pretrained VLMs can not only exhibit zero-shot manipulation capabilities without action pretraining, but also be further fine-tuned to improve existing skills and acquire capabilities for manipulation tasks beyond their zero-shot capabilities.

Figure 2: Overview of EASG . EASG explicitly grounds robot-action semantics in the VLM’s perceptual context and expresses actions through its native textual generation interface, enabling both zero-shot manipulation and direct robot-data fine-tuning. Actions are generated in the TCP-centered frame(Action frame) and converted into low-level joint commands.

Type	Task	Task description
Rigid	1.1.1	Grasp the cucumber
	1.1.2	Pick up the block
	1.1.3	Pick up the banana and put it on the plate
Articulated	1.2.1	Close the laptop
	1.2.2	Push the drawer closed
Deformable	1.3	Fold the towel

Table 1: Real-world task setups. See Appendix C.1 for the full object group we included.

Task	Ours	$\bm{\pi}_{0.5}$			$\bm{\pi}_{0}$ -FAST			GR00T N1.6			OpenVLA-OFT
	Zero	10	30	50	10	30	50	10	30	50	10	30	50
1.1.1	93.3	70.0	100.0	100.0	53.3	90.0	100.0	66.7	100.0	100.0	66.7	100.0	100.0
1.1.2	90.0	66.7	100.0	100.0	53.3	86.7	100.0	60.0	100.0	100.0	66.7	93.3	100.0
1.1.3	66.7	30.0	80.0	100.0	0.0	46.7	73.3	10.0	70.0	100.0	20.0	53.3	93.3
1.2.1	100.0	50.0	80.0	100.0	46.7	53.3	100.0	50.0	53.3	100.0	53.3	66.7	100.0
1.2.2	93.3	46.7	66.7	100.0	30.0	53.3	100.0	46.7	50.0	100.0	40.0	70.0	100.0

Table 2: Robot data efficiency comparison between EASG and fine-tuned VLAs. Ours uses no task-specific demonstrations. VLA baselines are fine-tuned with 10/30/50 demonstrations per task.

	Ours (Zero)	$\pi_{0.5}$ (50)	$\pi_{0}$ -FAST (50)	GR00T N1.6 (50)	OpenVLA-OFT (50)
Randomized	77.3 $\pm$ 5.9	40.2 $\pm$ 8.7	25.5 $\pm$ 9.5	37.8 $\pm$ 10.6	28.1 $\pm$ 10.9
Original	79.4 $\pm$ 2.5	96.7 $\pm$ 1.7	85.6 $\pm$ 2.5	95.6 $\pm$ 2.5	88.9 $\pm$ 2.5

Table 3: Generalization under object and position randomization. Randomized results are reported as mean

\pm

Std. over three randomized configurations.

Method	1.1.1		1.1.2		1.1.3		1.2.1		1.2.2		1.3		2.1	2.2	2.3.1	2.3.2	2.4.1	2.4.2	Avg.
	Zero	FT	Zero	FT	Zero	FT	Zero	FT	Zero	FT	Zero	FT	FT	FT	FT	FT	FT	FT	FT
Ours	93.3	100.0	90.0	100.0	66.7	70.0	100.0	100.0	93.3	100.0	33.3	33.3	80.0	63.3	56.7	43.3	76.7	83.3	75.5
$\pi_{0}$ -FAST	–	100.0	–	100.0	–	73.3	–	100.0	–	100.0	–	40.0	83.3	70.0	53.3	50.0	80.0	83.3	77.8
OpenVLA	–	100.0	–	100.0	–	60.0	–	100.0	–	100.0	–	23.3	70.0	46.7	43.3	20.0	73.3	76.7	67.8
Octo	–	100.0	–	100.0	–	63.3	–	100.0	–	100.0	–	30.0	73.3	40.0	46.7	26.7	80.0	76.7	69.7

Table 4: Results on harder real-world tasks. FT denotes fine-tuning with teleoperated data. Gray cells indicate unavailable results, as pretrained VLA models generally need additional adaptation.

Method	Spatial	Object	Goal	10	Average
Ours(0.8B)	92.5 $\pm$ 2.08	93.8 $\pm$ 1.91	83.1 $\pm$ 2.96	63.7 $\pm$ 3.80	83.3 $\pm$ 1.40
$\pi_{0}$ -FAST	96.4	96.8	88.6	60.2	85.5
Octo	78.9	85.7	84.6	51.1	75.1
OpenVLA	84.7	88.4	79.2	53.7	76.5

Table 5: Fine-tuning VLM against pretrained VLAs in simulation.

	w/ Thinking							w/o Thinking
Task	1.1.1	1.1.2	1.1.3	1.2.1	1.2.2	1.3	Avg.	1.1.1	1.1.2	1.1.3	1.2.1	1.2.2	1.3	Avg.
w/ Memory	93.3	90.0	66.7	100.0	93.3	33.3	79.4	13.3	16.7	0.0	16.7	23.3	0.0	11.7
w/o Memory	90.0	83.3	0.0	100.0	93.3	20.0	64.4	10.0	6.7	0.0	16.7	20.0	0.0	8.9

Table 7: Effect of memory and thinking on VLM-native zero-shot manipulation.

Figure 5: Zero-shot capability frontier across VLM families and scales.

	w/ Metric Depth							w/o Metric Depth
Task	1.1.1	1.1.2	1.1.3	1.2.1	1.2.2	1.3	Avg.	1.1.1	1.1.2	1.1.3	1.2.1	1.2.2	1.3	Avg.
w/ Frame	93.3	90.0	66.7	100	93.3	33.3	79.4	0.0	0.0	0.0	0.0	0.0	0.0	0.0
w/o Frame	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0

Table 8: Effect of depth and frame grounding on real-world zero-shot manipulation. The table reports performance on individual tasks under the same setup as experiments in Sec. 4.1 .

Method	Spatial	Object	Goal	10	Average
qwen3.5-0.8B	92.50 $\pm$ 2.08	93.75 $\pm$ 1.91	83.13 $\pm$ 2.96	63.75 $\pm$ 3.80	83.28 $\pm$ 1.40
qwen3.5-0.8B (w.o. GCD)	81.25 $\pm$ 3.09	78.75 $\pm$ 3.23	72.50 $\pm$ 3.53	58.13 $\pm$ 3.90	72.66 $\pm$ 1.73

Table 9: Effect of grammar constrained decoding on simulation performance.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1 : Comparison with the most related methods. We contrast EASG with representative prior methods to highlight the core differences in action representation, execution interface, action pretraining, zero-shot manipulation, and policy fine-tuning.

Method	Action Interface	Execution Pipeline	Further Improvement
Code as Policies [ 38 ]	Executable policy programs	• Perception APIs • Geometric operations • Robot-control APIs	No foundation-model training
VoxPoser [ 28 ]	3D affordance and constraint maps	• RGB-D grounding • 3D value-map construction • Model-based motion planning	No foundation-model training
CoPa [ 25 ]	Part-level spatial constraints and EEF poses	• Object-part grounding • Grasp proposal • Geometric constraint solving	No foundation-model training
ReKep [ 27 ]	Relational keypoint constraints	• Keypoint perception • Constraint generation • Hierarchical SE(3) optimization	No foundation-model training
CaP-X [ 16 ]	Programs over perception/control primitives	• Perception/control tools • Program synthesis • Execution feedback	Skills & coding-agent RL
VIA [ 24 ]	Visual tool calls and target-gripper waypoints	• RGB-D 3D interface • Target-gripper control • Low-level execution	No foundation-model training

Appendix

Table A1 : Comparison with representative code-as-policy and Robot-Use Agent methods. Prior approaches connect foundation-model reasoning to physical control through explicit intermediate representations or external tools, whereas EASG directly grounds continuous robot poses into the VLM native interface.

Figure A2 : Objects used in our real-world manipulation experiments.

Type	Task	Task description	Success criterion
Rigid	1.1.1	Grasp the cucumber	The cucumber is successfully grasped and held by the robot.
	1.1.2	Pick up the block	The block is lifted off the table and held by the robot.
	1.1.3	Pick up the banana and put it on the plate	The banana is lifted from the table and placed on the plate.
Articulated	1.2.1	Close the laptop	The laptop lid reaches the closed position.
	1.2.2	Push the drawer closed	The drawer is fully closed.
Deformable	1.3	Fold the towel	The towel is folded with partial overlap between folded regions.

Appendix

Table A2 : Task Group 1: Zero-shot evaluation tasks (Easy).

Figure A3 : Demonstration of successful zero-shot rollout cases.

Method	Demos.	1.1.1	1.1.2	1.1.3	1.2.1	1.2.2	1.3	Avg.
Ours	Zero	$93.3\pm 5.8$	$90.0\pm 10.0$	$66.7\pm 15.3$	$100.0\pm 0.0$	$93.3\pm 5.8$	$33.3\pm 15.3$	$79.4\pm 2.5$
	10	$70.0\pm 10.0$	$66.7\pm 15.3$	$30.0\pm 10.0$	$50.0\pm 10.0$	$46.7\pm 15.3$	$0.0\pm 0.0$	$43.9\pm 2.5$
	20	$86.7\pm 15.3$	$83.3\pm 15.3$	$56.7\pm 15.3$	$66.7\pm 15.3$	$56.7\pm 15.3$	$13.3\pm 15.3$	$60.6\pm 2.5$
$\pi_{0.5}$	30	$100.0\pm 0.0$	$100.0\pm 0.0$	$80.0\pm 10.0$	$80.0\pm 10.0$	$66.7\pm 15.3$	$30.0\pm 10.0$	$76.1\pm 2.5$
	40	$100.0\pm 0.0$	$100.0\pm 0.0$	$93.3\pm 5.8$	$93.3\pm 5.8$	$90.0\pm 10.0$	$66.7\pm 15.3$	$90.6\pm 2.5$
	50	$100.0\pm 0.0$	$100.0\pm 0.0$	$100.0\pm 0.0$	$100.0\pm 0.0$	$100.0\pm 0.0$	$80.0\pm 10.0$	$96.7\pm 1.7$

Appendix

Table A3 : Full comparison with different amounts of robot data. We report mean success rates

\pm

standard deviations for VLA baselines fine-tuned with 10/20/30/40/50 demonstrations per task. Ours uses no task-specific demonstrations.

Figure A4 : Randomization settings for real-world generalization experiments. For each task, we vary both the object instance and its initial position to evaluate the robustness of EASG beyond the original zero-shot setting.

Type	Task	Task description	Success criterion
Deformable	2.1	Fold the T-shirt in half.	The T-shirt is folded with the two sides partially aligned.
Articulated	2.2	Open the laptop.	The laptop lid is opened beyond 60° and remains stable.
Precise	2.3.1	Pick up the ring and put it on one of the pegs.	The ring is placed onto a peg.
	2.3.2	Hang the cup on the peg.	The cup is stably hung on the peg.
Long-horizon	2.4.1	Open the drawer, then put the Rubik’s Cube in the drawer.	The drawer is partially opened and the cube is placed inside.
	2.4.2	Put the grapes in the drawer, then close the drawer.	The grapes are placed inside the drawer, and the drawer is closed within 2 cm of full closure.

Appendix

Table A4 : Task Group 2: Capability extension tasks (Hard). See Appendix C.1 for the full object group we included.

Method	2.1	2.2	2.3.1	2.3.2	2.4.1	2.4.2	Avg.
Ours	80.0 $\pm$ 10.0	63.3 $\pm$ 11.5	56.7 $\pm$ 15.3	43.3 $\pm$ 15.3	76.7 $\pm$ 5.8	83.3 $\pm$ 11.5	67.2 $\pm$ 5.1
$\pi_{0}$ -FAST	83.3 $\pm$ 5.8	70.0 $\pm$ 10.0	53.3 $\pm$ 15.3	50.0 $\pm$ 10.0	80.0 $\pm$ 10.0	83.3 $\pm$ 5.8	70.0 $\pm$ 3.3
OpenVLA	70.0 $\pm$ 10.0	46.7 $\pm$ 15.3	43.3 $\pm$ 11.5	20.0 $\pm$ 10.0	73.3 $\pm$ 11.5	76.7 $\pm$ 5.8	55.0 $\pm$ 3.3
Octo	73.3 $\pm$ 5.8	40.0 $\pm$ 10.0	46.7 $\pm$ 11.5	26.7 $\pm$ 15.3	80.0 $\pm$ 10.0	76.7 $\pm$ 15.3	57.2 $\pm$ 4.2

Appendix

Table A5 : Fine-tuning results on harder real-world manipulation tasks. Each entry reports the mean task success rate

\pm

standard deviation over three independent evaluation rounds with 10 trials per round.

Figure A5 : Representative inference trajectories after EASG fine-tuning. We show one successful rollout for each Task Group 2 task using the fine-tuned Qwen-3.8-27B. These tasks are beyond the model’s native zero-shot capability and are solved after continuous-action fine-tuning with EASG.

Figure A6 : Robot platforms used for cross-embodiment evaluation. We compare the UR3 platform used for fine-tuning with a Flexiv robot arm equipped with the same Robotiq 2F-85 gripper. The two robots differ in embodiment and kinematics, while evaluation is initialized from matched end-effector poses. The external camera is only used for recording, not for inference .

Embodiment	2.1	2.2	2.3.1	2.3.2	2.4.1	2.4.2	Avg.
UR3	76.7 $\pm$ 11.5	60.0 $\pm$ 10.0	53.3 $\pm$ 11.5	36.7 $\pm$ 15.3	76.7 $\pm$ 5.8	80.0 $\pm$ 10.0	63.9 $\pm$ 5.1
Flexiv	73.3 $\pm$ 11.5	56.7 $\pm$ 15.3	50.0 $\pm$ 10.0	40.0 $\pm$ 17.3	73.3 $\pm$ 5.8	80.0 $\pm$ 17.3	62.2 $\pm$ 3.5

Appendix

Table A6 : Cross-embodiment transfer of the same fine-tuned EASG checkpoint. We report mean

\pm

standard deviation over 3 evaluation rounds, with 10 trials per task in each round. No additional fine-tuning is performed for the Flexiv embodiment.

Figure A7 : Representative synchronized rollouts across robot embodiments. We visualize the same manipulation task executed by UR3 and Flexiv using the identical fine-tuned EASG checkpoint. Starting from matched end-effector poses, both robots receive pose predictions from the same VLM, while embodiment-specific inverse kinematics and low-level controllers execute the predicted task-space actions.

Figure A8 : Qualitative comparison of action-frame grounding. We visualize representative rollouts of the full TCP-centered grounding and two alternative coordinate-frame choices on the same zero-shot manipulation task.

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 627 papers

Mar 16, 2026

Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models

Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for robotic manipulation, in which reliable action prediction critically depends on accurately interpreting and integrating visual observations conditioned on language instructions. Although recent works have sought to enhance the visual capabilities of VLA models, most approaches treat the LLM backbone as a black box, providing limited insight into how visual information is grounded into action generation. Therefore, we perform a systematic analysis of multiple VLA models across different action-generation paradigms and observe that sensitivity to visual tokens progressively decreases in deeper layers during action generation. Motivated by this observation, we propose DeepVision-VLA, built on a Vision-Language Mixture-of-Transformers (VL-MoT) framework. This framework enables shared attention between the vision foundation model and the VLA backbone, injecting multi-level visual features from the vision expert into deeper layers of the VLA backbone to enhance visual representations for precise and complex manipulation. In addition, we introduce Action-Guided Visual Pruning (AGVP), which leverages shallow-layer attention to prune irrelevant visual tokens while preserving task-relevant ones, reinforcing critical visual cues for manipulation with minimal computational overhead. DeepVision-VLA outperforms prior state-of-the-art methods by 9.0% and 7.5% on simulated and real-world tasks, respectively, providing new insights for the design of visually enhanced VLA models.

Yulin Luo, Hao Chen, Zhuangzhe Wu +10

State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University · The Chinese University of Hong Kong · Simplexity Robotics

Spatial Grounding for VLA Models Vision-Language-Action Models Vision-Language Models Robotic Manipulation Visual Token Pruning

Ask PDF

Sep 29, 2026

MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation

Vision-language-action (VLA) models have advanced robotic manipulation, but their zero-shot generalization in new tasks and environments remains limited, and their reliance on specialized training keeps them from benefiting directly from rapidly advancing general-purpose vision-language models (VLMs). In parallel, recent agentic robotic systems leverage VLMs for high-level reasoning or coding agents for robot control, but often depend on extensive external models and tools, introducing additional complexity and cost. This motivates us to ask: Can a general-purpose VLM itself operate a robot more like the human teleoperator by reasoning directly from observations, issuing actions, and continuously adapting to execution feedback, without relying on external models such as learned action experts, coding agents or grounding tools like SAM3? In this work, we introduce MotorMind, a robot manipulation harness that connects VLM-proposed mid-level actions to deterministic robot control and feedback, with asynchronous monitoring and background memory updates. Without task-specific policy training, coding agents, or additional grounding tools such as SAM3, MotorMind achieves 66.7% success on the base LIBERO-PRO suites and 53.8% under perturbations, compared with at most 13.3% and 19.2%, respectively, for the prior zero-shot methods we evaluate. The same interface reaches 95% average success on a real xArm6 robot across direct manipulation and human-perturbation settings. Replacing the backbone with a stronger VLM further improves performance, while the remaining failures - primarily due to visual grounding, embodied reasoning, and action knowledge - decrease as VLM capability improves. These results show that a general-purpose VLM, when equipped with an appropriate mid-level action representation and asynchronous execution harness, can perform effective zero-shot robotic manipulation.

Bingxuan Li, Siqi Song, Yizhuo Wu +3

University of Illinois Urbana-Champaign

Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation Robotic Manipulation VLMs for Robotics Reasoning for VLA Models

Ask PDF

Sep 24, 2026

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal

General-purpose vision-language models (VLMs) bring broad knowledge and spatial reasoning to robot manipulation, yet existing systems either use them indirectly, to predict constraints or write programs, or give them a view of the scene rather than a world in which to act. We present World Action Agent (WAA), a multi-agent harness through which VLMs pilot robots with basic tools, making every decision within a visual action workspace. The workspace has three properties. Contact views, selected automatically from the scene geometry, present the scene around the current interaction. Action rehearsal turns each action into an editable proposal that the agent, alone or through an Imagination Agent, previews and revises against planning feedback before execution. In-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the view where it observes them. Through the same workspace, WAA acquires embodied procedural knowledge in two ways: it evolves multimodal skills from expert videos and human teaching under evidence-based review and consults them through a Skill Agent, and its interaction traces train smaller VLMs to pilot the same harness. On LIBERO-Pro, WAA with skills evolved only from LIBERO-90 reaches a state-of-the-art 75.6% average success, outperforming end-to-end VLAs, code-as-policy agents, and a visual-harness baseline with the same backbone; the same skills remain effective on robosuite without further learning. Fine-tuning Qwen3.5-9B on harness traces raises its out-of-domain success from 1.7% to 43.3%.

Yehang Zhang, Haojian Huang, Yifan Chang +13

HKUST(GZ) · Knowin AI · CUHK

Tool-Using Vision-Language Agents Robotic Manipulation Robot Imitation Learning VLMs for Robotics

Ask PDF

Abstract

VLMs for Robotics

Explore similar work

Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models

MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation

World Action Agent: Harnessing VLMs for Robot Manipulation via World Action Rehearsal