VLMs for Robotics

VLM: Vision-Language Model

Ask about this topic

Overview

This field studies how robots can use visual observations and language instructions to understand scenes, plan tasks, and act in the physical world. Common approaches adapt pretrained vision-language models with geometric or 3D grounding, maps and affordances, action heads, skill libraries, or feedback loops; some separate slow semantic planning from fast control. Researchers train and test these systems using robot demonstrations, human videos, simulation, and real-robot tasks, with benchmarks such as LIBERO, RLBench, and Habitat. A central challenge is turning plausible language-based reasoning into reliable, metrically grounded action despite occlusion, changing scenes, latency, and safety constraints.

Latest papers

CardsList

664 papers117 papers in the last four weeks, up 10% on the four weeks before. 0.7% of all new papers.

  1. Critic in the Loop: A Tri-System VLA Framework for Robust Long-Horizon Manipulation

    Mar 5, 2026Pengfei Yi, Yingjie Ma, Wenjiang Xu +4Long-Horizon Robotic ManipulationVLMs for Robotics

    Institute of Automation, Chinese Academy of Sciences · the School of Artificial Intelligence, University of Chinese Academy of Sciences

  2. VehAnchor: Metadata-Free Metric Scale Recovery from Vehicle Cues in Aerial Imagery

    Mar 4, 2026Yifei Chen, Chenqian Le, Jiayi Cheng +1VLMs for Robotics

    Dimension Gate, Haidian District, Beijing 100083, China. · New York University, New York, NY, USA.

  3. ArthroCut: Autonomous Policy Learning for Robotic Bone Resection in Knee Arthroplasty

    Mar 4, 2026Xu Lu, Yiling Zhang, Wenquan Cheng +3Surgical RoboticsRobot Learning

    School of Biomedical Engineering, Tsinghua University, Beijing 100084, China. · Longwood Valley MedTech, Beijing 100176, China. · School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China

  4. Lightweight Visual Reasoning for Socially-Aware Robots

    Mar 4, 2026Alessio Galatolo, Ronald Cumbal, Alexandros Rouchitsas +3VLM ReasoningVLMs for Robotics

    Department of Information Technology, Uppsala University, Sweden.

  5. DISC: Dense Integrated Semantic Context for Large-Scale Open-Set Semantic Mapping

    Mar 4, 2026Felix Igelbrink, Lennart Niecksch, Martin Atzmueller +1Semantic MappingRobotic Perception

    the German Research Center for Artificial Intelligence (DFKI), Osnabrück, Germany · the Department of Computer Sciene, Osnabrück University, Osnabrück, Germany

  6. Act-Observe-Rewrite: Multimodal Coding Agents as In-Context Policy Learners for Robot Manipulation

    Mar 3, 2026Vaishak KumarCode Generation for RoboticsRobot Policy Learning

    General Bionix, Inc.

  7. Uni-Skill: Building Self-Evolving Skill Repository for Generalizable Robotic Manipulation

    Mar 3, 2026Senwei Xie, Yuntian Zhang, Ruiping Wang +1Robot Skill LearningRobot Learning

    Key Laboratory of AI Safety of CAS, Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China · University of Chinese Academy of Sciences, Beijing, 100049, China

  8. LiteVLA-Edge: Quantized On-Device Multimodal Control for Embedded Robotics

    Mar 3, 2026Justin Williams, Kishor Datta Gupta, Roy George +1Efficient VLA Model InferenceOn-Device Inference

    Clark Atlanta University, Atlanta, GA, USA · Siemens Corporation, Princeton, NJ, USA

  9. TacMamba: A Tactile History Compression Adapter Bridging Fast Reflexes and Slow VLA Reasoning

    Mar 2, 2026Zhenan Wang, Yanzhe Wang, Meixuan Ren +8Tactile Sensing for Robotic ManipulationVisuo-Tactile Representation Learning

    Zhejiang University, Hangzhou, China · GigaAI, Beijing, China · Harbin Institute of Technology, Harbin, China

  10. Monocular 3D Object Position Estimation with VLMs for Human-Robot Interaction

    Mar 1, 2026Ari Wahl, Dorian Gawlinski, David Przewozny +3VLMs for Robotics3D Visual Grounding

    Fraunhofer HHI, Berlin, Germany · Berliner Hochschule für Technik (BHT), Berlin, Germany

  11. TGM-VLA: Task-Guided Mixup for Sampling-Efficient and Robust Robotic Manipulation

    Feb 28, 2026Fanqi Pu, Lei Jiang, Wenming YangRobot Imitation LearningData-Efficient Robot Learning

    Shenzhen International Graduate School, Tsinghua University, Shenzhen, China. · The National and Local Co-Build Humanoid Robotics Innovation Center, Shanghai, China.

  12. I-Perceive: A Foundation Model for Vision-Language Active Perception

    Feb 28, 2026Yongxi Huang, Zhuohang Wang, Wenjing Tang +33D Scene UnderstandingRobot Foundation Models

    Shanghai Jiao Tong University · Shanghai Innovation Institute · Beihang University

  13. Zero-Shot Robotic Manipulation via 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation

    Feb 28, 2026Zilong Xie, Jingyu Gong, Xin Tan +2Zero-Shot Robotic ManipulationGeneralization in Robotic Manipulation

    School of Computer Science and Technology, East China Normal University, Shanghai, China · Chongqing Key Laboratory of Precision Optics, Chongqing Institute of East China Normal University, Chongqing, China · Shanghai Key Laboratory of Computer Software Evaluating and Testing, Shanghai, China

  14. TMR-VLA:Vision-Language-Action Model for Magnetic Motion Control of Tri-leg Silicone-based Soft Robot

    Feb 28, 2026Ruijie Tang, Chi Kit Ng, Kaixuan Wu +5Language-Conditioned Robot ControlSoft Robotics

  15. When to Act, Ask, or Learn: Uncertainty-Aware Policy Steering

    Feb 25, 2026Jessie Yuan, Yilin Wu, Andrea BajcsyRobot Policy AdaptationVLMs for Robotics

    Carnegie Mellon University

  16. Are Foundation Models the Route to Full-Stack Transfer in Robotics?

    Feb 25, 2026Freek Stulp, Samuel Bustamante, João Silvério +3Transfer LearningRobot Foundation Models

    Institute of Robotics and Mechatronics, German Aerospace Center (DLR), Wessling, Germany. · Stanford AI Lab, Stanford University, Palo Alto, USA.

  17. LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies

    Feb 25, 2026Yue Yang, Shuo Cheng, Yu Fang +4Long-Horizon Robotic ManipulationRobotic Manipulation

    University of North Carolina at Chapel Hill · Georgia Institute of Technology · Carnegie Mellon University

  18. VLA Knows Its Limits: Adaptive Execution Horizons for Robot Policies

    Feb 24, 2026Haoxuan Wang, Gengyu Zhang, Yan Yan +2Flow-Based VLA ModelsVision-Language-Action Models

    University of Illinois Chicago · Cisco Research

  19. HALO: A Unified Vision-Language-Action Model for Embodied Multimodal Chain-of-Thought Reasoning

    Feb 24, 2026Quanxin Shou, Fangqi Zhu, Shawn Chen +9Vision-Language-Action ModelsVLMs for Robotics

    The Hong Kong University of Science and Technology

  20. IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulation

    Feb 24, 2026Zhian Su, Weijie Kong, Haonan Dong +1RL Fine-TuningVLMs for Robotics

    Grasp Laboratory, Mechanical Engineering Department, Zhejiang University, Hangzhou 310027, China. · Torch Kernel Co., Ltd., Hangzhou 310000, China.

  21. Recursive Belief Vision Language Action Models

    Feb 24, 2026Vaidehi Bagaria, Bijo Sebastian, Nirav Kumar PatelVision-Language-Action ModelsLong-Horizon Robotic Manipulation

    Department of Engineering Design, Indian Institute of Technology, Madras, Chennai, India

  22. Strategy-Supervised Autonomous Laparoscopic Camera Control via Event-Driven Graph Mining

    Feb 24, 2026Keyu Zhou, Peisen Xu, Yahao Wu +3Surgical RoboticsVision-Based Robot Control

    Hangzhou Dianzi University, Hangzhou 310018, China · Zhejiang University, Hangzhou 310058, China

  23. NovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planning

    Feb 23, 2026Jiahui Fu, Junyu Nan, Lingfeng Sun +7Long-Horizon Robotic ManipulationZero-Shot Robotic Manipulation

    Robotics and AI Institute · Carnegie Mellon University · Brown University +1

  24. An Approach to Combining Video and Speech with Large Language Models in Human-Robot Interaction

    Feb 23, 2026Guanting Shen, Zi TianHuman-Robot InteractionLanguage-Conditioned Robot Manipulation

    The authors are with the Dalian University of Technology. Dalian, China.

  25. Seeing Farther and Smarter: Value-Guided Multi-Path Reflection for VLM Policy Optimization

    Feb 22, 2026Yanting Yang, Shenyuan Gao, Qingwen Bu +2VLMs for RoboticsLarge Language Model-Based Robot Planning

    Rutgers University · The Hong Kong University of Science and Technology · The University of Hong Kong

  26. Vid2Sid: Videos Can Help Close the Sim2Real Gap

    Feb 22, 2026Kevin Qiu, Yu Zhang, Marek Cygan +1Robot System IdentificationRobot Dynamics Identification

    University of Warsaw · IDEAS NCBR · EPFL +1

  27. TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics

    Feb 22, 2026Shirui Chen, Cole Harrison, Ying-Chun Lee +6Robot Task Progress EstimationRobot Learning

    University of Washington · Allen Institute for AI · Amazon +1

  28. Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation

    Feb 22, 2026Thanh Nguyen Canh, Thanh-Tuan Tran, Haolan Zhang +3Learning from DemonstrationRobot Imitation Learning

    School of Information Science, Japan Advanced Institute of Science and Technology, Nomi, 923-1211, Ishikawa, Japan. · University of Engineering and Technology, Vietnam National University, 10000, Hanoi, Vietnam. · Department of Robotics, Hanyang University, Ansan, 15588, Gyeonggi, Korea.

  29. Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigation

    Feb 21, 2026Kaiming Jin, Yuefan Wu, Shengqiong Wu +3Vision-Language NavigationVLMs for Robotics

    National University of Singapore · Simon Fraser University · University of Oxford

  30. Zero-shot Interactive Perception

    Feb 20, 2026Venkatesh Sripada, Frank Guerin, Amir GhalamzanRobotic PerceptionZero-Shot Robotic Manipulation

    University of Surrey · University of Sheffield