VLMs for Robotics

VLM: Vision-Language Model

This field studies how robots can use visual observations and language instructions to understand scenes, plan tasks, and act in the physical world. Common approaches adapt pretrained vision-language models with geometric or 3D grounding, maps and affordances, action heads, skill libraries, or feedback loops; some separate slow semantic planning from fast control. Researchers train and test these systems using robot demonstrations, human videos, simulation, and real-robot tasks, with benchmarks such as LIBERO, RLBench, and Habitat. A central challenge is turning plausible language-based reasoning into reliable, metrically grounded action despite occlusion, changing scenes, latency, and safety constraints.

Latest papers

664 papers117 papers in the last four weeks, up 10% on the four weeks before. 0.7% of all new papers.

RoboAgent: Chaining Basic Capabilities for Embodied Task Planning
Apr 9, 2026Peiran Xu, Jiaqi Zheng, Yadong MuRobot Task Planning Embodied Reasoning
Peking University, Beijing, China · XYZ Embodied AI, Beijing, China
Vision-Language Navigation for Aerial Robots: Towards the Era of Large Language Models
Apr 9, 2026Xingyu Xia, Lekai Zhou, Yujie Tang +3Vision-Language Navigation Aerial VLN
Defense Innovation Institute, Chinese Academy of Military Sciences, Beijing 10071, China · Intelligent Game and Decision Laboratory, Beijing 100071, China · Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing 100094, China
HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents
Apr 8, 2026Tencent Robotics X, HY Vision Team, : +20Mixture-of-Transformers Robot Foundation Models
KITE: Keyframe-Indexed Tokenized Evidence for VLM-Based Robot Failure Analysis
Apr 8, 2026Mehdi Hosseinzadeh, King Hang Wong, Feras DayoubRobot Failure Detection VLMs for Robotics
Australian Institute for Machine Learning (AIML), Adelaide University, Australia
ExpressMM: Expressive Mobile Manipulation Behaviors in Human-Robot Interactions
Apr 7, 2026Souren Pashangpour, Haitong Wang, Matthew Lisondra +1Human-Robot Interaction Human-Robot Collaboration
Autonomous Systems and Biomechatronics Laboratory (ASBLab), Department of Mechanical and Engineering, University of Toronto, Toronto, ON M5S 3G8, Canada
ROSClaw: A Hierarchical Semantic-Physical Framework for Heterogeneous Multi-Agent Collaboration
Apr 6, 2026Rongfeng Zhao, Xuanhao Zhang, Zhaochen Guo +4VLMs for Robotics Cross-Embodiment Robot Learning
Shanghai Research Institute for Intelligent Autonomous Systems, Shanghai 201210, China · National Key Laboratory of Autonomous Intelligent Unmanned Systems (Tongji University), Shanghai 201210, China · Frontiers Science Center for Intelligent Autonomous Systems, Shanghai 201210, China +1
Learning Structured Robot Policies from Vision-Language Models via Synthetic Neuro-Symbolic Supervision
Apr 3, 2026Alessandro Adami, Tommaso Tubaldo, Marco Todescato +2Robotics VLMs for Robotics
University of Padova, Dept. of Information Engineering, Italy. · Polytechnic of Bari Dept. of Electrical and Information Engineering, Italy. · Fraunhofer Italia Research, 39100, Bozen, Italy.
Tex3D: Objects as Attack Surfaces via Adversarial 3D Textures for Vision-Language-Action Models
Apr 2, 2026Jiawei Chen, Simin Huang, Jiawei Du +5Adversarial Attacks on VLA Models Adversarial Examples
East China Normal University · Zhongguancun Academy · CFAR, A*STAR, Singapore +2
A ROS 2 Wrapper for Florence-2: Multi-Mode Local Vision-Language Inference for Robotic Systems
Apr 1, 2026J. E. Domínguez-VidalVLMs for Robotics VLM Inference
PRISM: A Multi-View Multi-Capability Retail Video Dataset for Embodied Vision-Language Models
Mar 31, 2026Amirreza Rouhi, Parikshit Sakurikar, Satya Sai Reddy +6VLMs for Robotics VLM Reasoning
DreamVu
LatentPilot: Scene-Aware Vision-and-Language Navigation by Dreaming Ahead with Latent Visual Reasoning
Mar 31, 2026Haihong Hao, Lei Chen, Mingfei Han +5Vision-Language Navigation VLMs for Robotics
University of Science and Technology of China · MBZUAI · Stanford University +2
DRIVE-Nav: Directional Reasoning, Inspection, and Verification for Efficient Open-Vocabulary Navigation
Mar 30, 2026Maoguo Gao, Zejun Zhu, Zhiming Sun +6Embodied Navigation Open-Vocabulary Object Navigation
1Beijing Institute of Technology · 2DeepBlue College · 3The National Key Laboratory of Autonomous Intelligent Unmanned Systems (KAIUS), School of Automation
ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation
Mar 30, 2026Yu Sun, Meng Cao, Yang Ping +24Generalization in Robotic Manipulation Robotic Manipulation
Sun Yat-sen University · X Square Robot · MBZUAI +2
Learning Multi-View Spatial Reasoning from Cross-View Relations
Mar 30, 2026Suchae Jeong, Jaehwi Song, Haeone Lee +9Multimodal Spatial Reasoning Visual Spatial Reasoning
KAIST · Config · Hanyang University +2
Realtime-VLA V2: Learning to Run VLAs Fast, Smooth, and Accurate
Mar 27, 2026Chen Yang, Yucheng Hu, Yunchao Ma +3Robotic Control VLMs for Robotics
Dexmal
Fast-dVLA: Accelerating Discrete Diffusion VLA to Real-Time Performance
Mar 26, 2026Wenxuan Song, Jiayi Chen, Shuai Chen +8VLA Model Training Vision-Language-Action Models
The Hong Kong University of Science and Technology (Guangzhou) · ShanghaiTech University · Shanghai Institute of Technical Physics, CAS +3
ThermoAct:Thermal-Aware Vision-Language-Action Models for Robotic Perception and Decision-Making
Mar 26, 2026Young-Chae Son, Dae-Kwan Ko, Yoon-Ji Choi +1Robotic Perception Language-Conditioned Robot Control
Dongguk University
3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models
Mar 25, 2026Bin Yu, Shijie Lian, Xiaopeng Lin +8VLMs for Robotics Vision-Language-Action Models
HIT · ZGCA · HUST +5
Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning
Mar 25, 2026Tommaso Galliena, Stefano Rosa, Tommaso Apicella +3Memory-Augmented VLMs VLMs for Robotics
Italian Institute of Technology, Genoa, Italy · University of Genoa, Genoa, Italy
ReMemNav: Memory-Based Decision Correction and Target Verification for Zero-Shot Object Navigation
Mar 25, 2026Feng Wu, Wei Zuo, Wenliang Yang +3Robot Navigation Vision-Language Navigation
Fudan University · Tongji University
ELITE: Experiential Learning and Intent-Aware Transfer for Self-improving Embodied Agents
Mar 25, 2026Bingqing Wei, Zhongyu Xia, Dingai Liu +3Lifelong Learning Agents VLMs for Robotics
Wangxuan Institute of Computer Technology, Peking University, Beijing, China
Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation
Mar 24, 2026Anupam Pani, Yanchao YangVLMs for Robotics Robotic Manipulation
Institute of Data Science, University Of Hong Kong
TRAP: Hijacking VLA CoT-Reasoning via Adversarial Patches
Mar 24, 2026Zhengxian Huang, Wenjun Zhu, Haoxuan Qiu +2Adversarial Attacks on VLA Models Robustness of VLA Models
Zhejiang University, Hangzhou, China · Harbin Institute of Technology, Harbin, China
VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents
Mar 24, 2026Pengsen Liu, Maosen Zeng, Nan Tang +4Offline RL VLMs for Robotics
National Key Laboratory for Novel Software Technology, Nanjing University, China & School of Artificial Intelligence, Nanjing University, China · Computational Intelligence Center (CIC), School of Computer and Artificial Intelligence, Shandong Jianzhu University, Jinan, 250101, China
Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models
Mar 24, 2026Ruixing Jin, Zicheng Zhu, Ruixiang Ouyang +4Domain Randomization Robotics Simulation
School of Data Science, The Chinese University of Hong Kong, Shenzhen · Shenzhen Loop Area Institute
CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Models
Mar 24, 2026Youzhi Liu, Li Gao, Liu Liu +2Game-Theoretic RL Reinforcement Learning
Amap, Alibaba Group
CATNAV: Cached Vision-Language Traversability for Efficient Zero-Shot Robot Navigation
Mar 24, 2026Aditya Potnis, Francisco Affonso, Shreya Gummadi +2Robot Traversability Estimation Robot Navigation
Field Robotics Engineering and Science Hub (FRESH), Illinois Autonomous Farm, University of Illinois at Urbana-Champaign (UIUC), IL
ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Model
Mar 23, 2026Haichao Zhang, Yijiang Li, Shwai He +5Latent World Models Trajectory Prediction
Northeastern University · University of California San Diego · University of Maryland +2
DualCoT-VLA: Visual-Linguistic Chain of Thought via Parallel Reasoning for Vision-Language-Action Models
Mar 23, 2026Zhide Zhong, Junfeng Li, Junjie He +10VLMs for Robotics Reasoning for VLA Models
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Foundation Model Department
UniDex: A Robot Foundation Suite for Universal Dexterous Hand Control from Egocentric Human Videos
Mar 23, 2026Gu Zhang, Qicheng Xu, Haozhe Zhang +16Robotic Manipulation Cross-Embodiment Robot Learning
Tsinghua University · Shanghai Qizhi Institute · Sun Yat-sen University +1

Apr 9, 2026J/K move · Enter open · S save

RoboAgent: Chaining Basic Capabilities for Embodied Task Planning

Peiran Xu, Jiaqi Zheng, Yadong Mu

Peking University, Beijing, China · XYZ Embodied AI, Beijing, China

Robot Task Planning Embodied Reasoning VLMs for Robotics

Open paper AskPDF arXiv

This paper focuses on embodied task planning, where an agent acquires visual observations from the environment and executes atomic actions to accomplish a given task. Although recent Vision-Language Models (VLMs) have achieved impressive results in multimodal understanding and reasoning, their performance remains limited when applied to embodied planning that involves multi-turn interaction, long-horizon reasoning, and extended context analysis. To bridge this gap, we propose RoboAgent, a capability-driven planning pipeline in which the model actively invokes different sub-capabilities. Each capability maintains its own context, and produces intermediate reasoning results or interacts with the environment according to the query given by a scheduler. This framework decomposes complex planning into a sequence of basic vision-language problems that VLMs can better address, enabling a more transparent and controllable reasoning process. The scheduler and all capabilities are implemented with a single VLM, without relying on external tools. To train this VLM, we adopt a multi-stage paradigm that consists of: (1) behavior cloning with expert plans, (2) DAgger training using trajectories collected by the model, and (3) reinforcement learning guided by an expert policy. Across these stages, we exploit the internal information of the environment simulator to construct high-quality supervision for each capability, and we further introduce augmented and synthetic data to enhance the model's performance in more diverse scenarios. Extensive experiments on widely used embodied task planning benchmarks validate the effectiveness of the proposed approach. Our codes will be available at https://github.com/woyut/RoboAgent_CVPR26.