VLMs for Robotics

VLM: Vision-Language Model

Overview

This field studies how robots can use visual observations and language instructions to understand scenes, plan tasks, and act in the physical world. Common approaches adapt pretrained vision-language models with geometric or 3D grounding, maps and affordances, action heads, skill libraries, or feedback loops; some separate slow semantic planning from fast control. Researchers train and test these systems using robot demonstrations, human videos, simulation, and real-robot tasks, with benchmarks such as LIBERO, RLBench, and Habitat. A central challenge is turning plausible language-based reasoning into reliable, metrically grounded action despite occlusion, changing scenes, latency, and safety constraints.

Latest papers

CardsList

717 papers71 papers in the last four weeks, down 46% on the four weeks before. 0.5% of all new papers.

  1. EmbodyAgent: Post-Training Embodied VLM Agents via Distillation, SFT, and RL

    Nov 19, 2026Wei Dai, Jinkun Liu, Jiayi Chen +3VLMs for RoboticsRL for VLMs

  2. TuringPAI: A Foundation for On-Device Physical AI

    Nov 19, 2026Qiman Wu, Hanlin Chen, Yuhua Wei +25Efficient VLAsVision-Language-Action Models

  3. ActMem: From Latent Memory Reasoning to Grounded Robotic Action

    Nov 19, 2026Linfeng Dong, Jianwei Niu, Jiuyi Zhao +6Memory-Augmented VLMsVLMs for Robotics

  4. Verify2Act: Critic-Guided Latent World Models for Verifying Language-Conditioned Manipulation Plans

    Nov 18, 2026Chrisantus Eze, Christopher CrickWorld Models for VLAsLatent World Models

  5. ZZ-sight: Unifying Vision-Language-Action and Latent World Modeling

    Nov 18, 2026Junjie Ye, Yiqi Zhao, Muchen Xu +12Vision-Language-Action ModelsWorld Models for VLAs

  6. NereidVLA: Learning to Understand and Navigate the Underwater World

    Nov 18, 2026Chenjie Zhao, Zhiliang Zhu, Quan Chen +8VLMs for RoboticsSpatial Grounding for VLAs

  7. SHAPE: Discrete Supervision and Parallel Continuous Streams for Learning Dexterous Manipulation from Egocentric Human Data

    Nov 18, 2026Yaping Li, Zhihong Liu, Jiacheng Li +10Robotic ManipulationVLMs for Robotics

  8. Assessing Vision-Language Models for Early Prediction of Unsafe Situations in Robotics

    Nov 18, 2026Muhammad Farooq, Aitor Arrieta, Paolo Arcaini +2VLMs for RoboticsConfidence Estimation for VLMs

  9. ExecGap: Exploiting Command-Execution Gap for Patch Attacks on Vision-Language-Action Models

    Nov 18, 2026Keke Tang, Guolin Qiu, Tianyu Hao +4Adversarial Attacks on VLAsRobustness of VLAs

  10. ForceHarness: Turning Force into Action and Physics for Frozen Frontier Models

    Nov 16, 2026Ke Sun, Kevin Yuchen Ma, Yike Zhang +2Robotic ManipulationForce-Controlled Robotic Manipulation

  11. MARS: Dynamic Routing of Reinforcement-Learned Atomic Skills for Multi-Stage Robotic Manipulation

    Nov 14, 2026Zequan Zeng, Junqin Yang, Kai WangRobotic ManipulationMulti-Task Robotic Manipulation

  12. RoboCoEvo: Expanding Robot Capabilities through Agent-Policy Co-Evolution

    Nov 14, 2026Ran Qi, Yixin Xiao, Dawei Wang +7Robot LearningRobot Foundation Models

  13. Objectives as Policies: Vision-Language Model Cost Programs for Contact-Rich Manipulation

    Nov 13, 2026Zihao Lu, Trong Gia Hung Nguyen, Fuquan Wang +1Robotic ManipulationContact-Rich Robotic Manipulation

  14. Grounding Object-to-Object Hazard Reasoning into 3D Risk Fields for Task-Conditioned Safety

    Nov 12, 2026Mariana Sosa Guzmán, Yide Song, Amin Abyaneh +2Robot SafetySafe Robot Manipulation

  15. AgriSense: Vision-Language Decision Support and Robotic Intervention in a Virtual Greenhouse Copy title

    Nov 12, 2026Lyes Saad Saoud, Oualid Doukhi, Reza Ghorbani +1Agricultural RoboticsRobotics

  16. NSDL: Neuro-Symbolic Division of Labor for Multi-Robot Task Planning

    Nov 12, 2026Xincheng Tan, Ke Wang, Yining Di +1Neuro-Symbolic PlanningMulti-Robot Task Allocation

  17. I See What You Want: Decoding Hierarchical Operator Intent from Visual Teleoperation History

    Nov 12, 2026Xiangyi Jia, Kaixian Qu, Shengzhi Wang +3VLMs for RoboticsRobot Teleoperation

  18. Low-Cost Mobile Bimanual Manipulation for Long-Horizon Toilet Cleaning

    Nov 11, 2026Sven Schreiber, Bernard Burdiek, Lasse Huber-Saffer +4RoboticsLong-Horizon Robotic Manipulation

  19. Is It Still There? Place-Registered Verification Makes VLMs Reliable Scene-Change Detectors

    Nov 10, 2026Jinjie Xie, Baihua Li, Qinggang Meng3D Scene Change DetectionVLMs for Robotics

  20. Game-Enhanced Embodied Learning with Event and Foresight Pretraining

    Nov 10, 2026Lianlei Shan, Han Jin, Yaunchao Ma +3Robot LearningVLMs for Robotics

  21. VLA-MOPD: On-Policy Distillation for Multi-Task VLA Policy Consolidation

    Nov 9, 2026Yongxuan Lv, Zhen Fang, Lichen Man +10VLMs for RoboticsOn-Policy Distillation

  22. LHNav: Long-Horizon Humanoid Navigation via Multirate Coordination and Physics-Grounded Action Data

    Nov 9, 2026Xueyang Wang, Zehong Shen, Dingkun Wei +6Robot NavigationLanguage-Conditioned Robot Planning

  23. APS-VLA: Action-Guided and Future-Aware Token Pruning for Efficient Vision-Language-Action Models

    Nov 9, 2026Xiaoyang Xing, Chun-Mei Feng, Anthony Cohn +2Efficient VLA InferenceVisual Token Pruning

  24. HINT-VLM: Humanoid Interactive Navigation over Rough Terrain with Embodiment-Grounded Vision-Language Model

    Nov 8, 2026Ziwon Yoon, Seok Hwan Kim, Yiyu Chen +3Robot Obstacle TraversalRobot Motion Planning