How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective

Bo Peng; Pi Bu; Keyu Pan; Xinrun Xu; Yinxiu Zhao; Miao Chen; Yang Du; Lin Li

cs.AIFeb 24, 2026

How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective

Authors: Bo Peng, Pi Bu, Keyu Pan, Xinrun Xu, Yinxiu Zhao, Miao Chen, Yang Du, Lin Li,

Organizations: University of Science and Technology of China · Alibaba Group · Independent Researcher · Institute of Software, Chinese Academy of Sciences

Abstract

Recent advances in vision-language models (VLMs) have shown promise for human-level embodied intelligence. However, existing benchmarks for VLM-driven embodied agents often rely on high-level commands or discretized action spaces, which are non-native settings that differ markedly from real-world control. In addition, current benchmarks focus primarily on high-level tasks and lack joint evaluation and analysis at both low and high levels. To address these limitations, we present NativeEmbodied, a challenging benchmark for VLM-driven embodied agents that uses a unified, native low-level action space. Built on diverse simulated scenes, NativeEmbodied includes three representative high-level tasks in complex scenarios to evaluate overall performance. For more detailed analysis, we further decouple the skills required by complex tasks and construct four types of low-level tasks, each targeting a fundamental embodied skill. This joint evaluation across task and skill granularities enables fine-grained assessment of embodied agents. Experiments with state-of-the-art VLMs reveal clear deficiencies in several fundamental embodied skills, and further analysis shows that these bottlenecks significantly limit performance on high-level tasks. NativeEmbodied highlights key challenges for current VLM-driven embodied agents and provides insights to guide future research.

Figures & tables

BenchMark	Size	Task Level	Fine-Grained	Multimodal	Native	Decoupled
ALFRED ( Shridhar et al. 2020a )	3,062	High	✗	✔	✗	✗
ALFWorld ( Shridhar et al. 2020b )	274	High	✗	✗	✗	✗
VLMbench ( Zheng et al. 2022 )	4,760	Low	✗	✔	✔	✗
Behavior-1k ( Li et al. 2023 )	1,000	High	✗	✔	✗	✗
Lota-bench ( Choi et al. 2024 )	308	High	✗	✗	✔	✗
GOAT-bench ( Khanna et al. 2024 )	3,919	Low	✗	✔	✔	✗

Table 1: Comparisons between our NativeEmbodied and previous benchmarks .

Figure 1: Our NativeEmbodied benchmark includes four fundamental low-level tasks (i.e., Perception, Alignment, Navigation and Planning) and three integrated high-level tasks (i.e., Exploration, Interaction, and Search).

Figure 2: Sample distribution of NativeEmbodied.

Model	Exploration			Search				Interaction
Model	Acc $\uparrow$	AS $\downarrow$	WAS $\downarrow$	SR $\uparrow$	ACPD $\downarrow$	AS $\downarrow$	WAS $\downarrow$	SR $\uparrow$	AS $\downarrow$	WAS $\downarrow$
Closed-Source Large Vision Language Models
GPT-4o	36.89	12.32	24.11	0.65	131.29	25.00	30.96	22.28	12.25	26.84
GPT-4v	36.89	10.42	23.41	3.27	112.53	12.60	30.35	37.31	12.07	24.03
GPT-o3	52.43	11.06	20.54	34.64	32.94	15.60	25.67	38.34	13.35	24.25
GPT-o4-mini	40.78	5.48	20.59	17.64	37.93	13.07	27.84	26.42	13.33	28.27

Table 2: Performance comparison of closed-source and open-source LVLMs on the three high-level tasks: Exploration, Search and Interaction. For metrics,

\uparrow

\downarrow

mean ”higher is better” / ”lower is better”.

Closed-Source Large Vision Language Models
Model	Perception			Spatial Alignment				Navigation				Planning
Model	P $\uparrow$	R $\uparrow$	F1 $\uparrow$	SR $\uparrow$	ACPD $\downarrow$	AS $\downarrow$	WAS $\downarrow$	SR $\uparrow$	ACD $\downarrow$	AS $\uparrow$	WAS $\downarrow$	SR $\uparrow$	AS $\downarrow$	WAS $\downarrow$
GPT-4o	75.14	73.15	74.28	7.51	86.85	3.91	15.07	50.00	2.16	6.87	11.42	58.55	9.63	14.82
GPT-4v	79.51	78.11	78.83	6.94	66.81	3.23	15.12	55.56	2.23	7.81	11.43	62.18	9.25	14.04
GPT-o3	83.15	84.51	83.97	64.16	22.73	7.28	10.4	63.19	2.02	8.34	11.08	72.54	10.71	13.87
GPT-o4-mini	74.67	75.16	74.92	45.09	27.45	6.34	11.57	35.42	2.68	8.11	13.24	66.32	10.23	14.36

Table 3: Performance of selected LVLMs on four low-level tasks: Perception, Spatial Alignment, Navigation and Planning.

\uparrow

\downarrow

denote “higher is better” / “lower is better”.

Figure 3: Results from the skill-oriented ablation study, aimed to precisely identify the key atomic skills that limit model performance.

Figure 4: Results of think mode ablation study, aimed to explore the capabilities of reasoning models.

Figure 5: Case study of common error trajectories .

Embodied Agent Benchmarks

See all 24 papers

Hy-Embodied-VLM-1.0: Efficient Physical-World Agents
Jul 14, 2026Ziyi Wang, Xumin Yu, Yongming Rao +19Efficient VLA Models Multimodal Foundation Models
Tencent Robotics X Hy Vision Team Futian Laboratory
EmbodiedSkills: World Action Model-augmented Cascaded Skills for Vision-Language-Action Agent
Sep 1, 2026Wei Wang, Wenqiao Zhang, Yutong Lin +14Robotic Control Large Language Model-Based Robot Planning
College of Computer Science and Technology, Zhejiang University · Nanjing University of Aeronautics and Astronautics · Cornell University +3
Embodied Skill Bench: Benchmarking How Well Agents with Embodied Skills Perform on Physical Tasks
Oct 12, 2026Zhi Han, Zihan Guo, Chenxi Zeng +6Embodied Agent Benchmarks Embodied AI
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Sun Yat-Sen University +2

Jul 14, 2026J/K move · Enter open · S save

Hy-Embodied-VLM-1.0: Efficient Physical-World Agents

Ziyi Wang, Xumin Yu, Yongming Rao, Yonggen Ling+18

Tencent Robotics X Hy Vision Team Futian Laboratory

Efficient VLA Models Multimodal Foundation Models Embodied AI VLMs for Robotics Embodied Reasoning

Open paper AskPDF arXiv

Building capable embodied agents requires not only multimodal perception and understanding, but also agentic capabilities for reasoning about actions, adapting to evolving situations, and interacting with the physical world. In this report, we introduce Hy-Embodied-VLM-1.0, an efficient and powerful embodied foundation model specifically designed for embodied agents operating in the physical world. To cultivate such capabilities from the pre-training stage onward, we define an action-centric capability taxonomy comprising three progressive dimensions: Action-Relevant State Understanding, Action-Transition Reasoning, and Sequential and Adaptive Reasoning. Guided by this taxonomy, we develop a systematic data pipeline and curate data mixtures spanning both pre-training and post-training. To deliver strong physical-world understanding and interaction capabilities while supporting latency-sensitive deployment, we build our model on the Hy3-A3B language backbone and the Hy-ViT2 vision encoder. Its efficient Mixture-of-Experts architecture combines strong model capacity with high inference efficiency. We evaluate Hy-Embodied-VLM-1.0 on a comprehensive suite of 38 benchmarks covering embodied perception, physical-world understanding, and embodied reasoning. The model achieves the best performance among similarly sized models on 19 of the 38 benchmarks and substantially outperforms strong competitors, including Qwen3.6-A3B and Cosmos 3. Compared with the previous-generation Hy-Embodied-0.5 MoT-2B, Hy-Embodied-VLM-1.0 improves average performance by 8.4%. Despite activating only 3B parameters, it achieves performance close to that of the previous-generation model with 32B activated parameters. Beyond static benchmark evaluation, Hy-Embodied-VLM-1.0 also demonstrates strong performance on embodied agentic tasks requiring multi-turn interaction and long-horizon reasoning.

Abstract

Embodied Agent Benchmarks

Explore similar work

Hy-Embodied-VLM-1.0: Efficient Physical-World Agents

EmbodiedSkills: World Action Model-augmented Cascaded Skills for Vision-Language-Action Agent

Embodied Skill Bench: Benchmarking How Well Agents with Embodied Skills Perform on Physical Tasks