VLMs for Robotics

VLM: Vision-Language Model

This field studies how robots can use visual observations and language instructions to understand scenes, plan tasks, and act in the physical world. Common approaches adapt pretrained vision-language models with geometric or 3D grounding, maps and affordances, action heads, skill libraries, or feedback loops; some separate slow semantic planning from fast control. Researchers train and test these systems using robot demonstrations, human videos, simulation, and real-robot tasks, with benchmarks such as LIBERO, RLBench, and Habitat. A central challenge is turning plausible language-based reasoning into reliable, metrically grounded action despite occlusion, changing scenes, latency, and safety constraints.

Latest papers

664 papers117 papers in the last four weeks, up 10% on the four weeks before. 0.7% of all new papers.

Surrogate Uncertainty for Black-box Visual Grounding
Oct 29, 2026Hanbo Zhang, Tianrun Hu, Cedar Site Bai +2Uncertainty Quantification Visual Grounding
National University of Singapore · Shanghai Innovation Institute · Purdue University
From Reasoning to Action: Consequence-Grounded Alignment for Vision-Language-Action Models
Oct 29, 2026Yixue Zhang, Wenke Xia, Kun Wu +6VLMs for Robotics Reasoning for VLA Models
Peking University, Beijing, China. · Beijing Innovation Center of Humanoid Robotics, Beijing, China. · Renmin University of China, Beijing, China. +1
Decoding Geometry, Not Tokens: Efficient VLMs for Robot Affordance and Trajectory Prediction
Oct 29, 2026Akila Ayanthi, Darshana Priyasad, Tharindu Fernando +4VLMs for Robotics Robot Motion Prediction
SAIVT, School of Electrical Engineering and Robotics, Queensland University of Technology, Australia · CSIRO Robotics, CSIRO, Australia
Embodied at Sea: Imitation Learning for Multi-Task Underwater Manipulation
Oct 28, 2026Greta Perez-Haiek, Luca Macesanu, Kevin Angers +3Multi-Task Robotic Manipulation Underwater Robotics
New York University · University of Malta
Which Hand Was Late? Diagnosis for Robot Failure Recovery in Bimanual Tool Use
Oct 27, 2026Baptiste Bonin, Caro Strickland, Audrey DurandRobot Failure Detection Robot Failure Recovery
Universit´e Laval · Mila – Quebec AI Institute · CIFAR AI Chair
SO-101 Bench: Accessible Spatial Manipulation
Oct 27, 2026Truman HickokRobotic Manipulation Robot Manipulation Benchmarks
Independent
Pretraining Knows What, Simulation Teaches How: Compositional Generalization in a Sim-Trained Robot Agent
Oct 26, 2026Benjamin Pikus, Daniel Saunders, Mircea Mironenco +11Generalization in Robotic Manipulation VLMs for Robotics
Luma AI
TrySee: Modular Perception-Enhanced VLM Grounding for Closed-Loop Robot Task Planning in Specialized Domains
Oct 26, 2026Elias Hofele, Thomas Trapp, David Timmermann +3Robot Task Planning VLMs for Robotics
FZI Research Center for Information Technology · Machine Intelligence and Robotics Lab (MaiRo), Karlsruhe Institute for Technology (KIT)
Robot-Aligned 3D Gaussian Splatting for Precise Reaching with Wrist-Camera VLA Policies
Oct 26, 2026Zuoxu Wang, Xiao LiangRobotic Manipulation Synthetic Data Generation for Robotics
Zachry Department of Civil and Environmental Engineering, Texas A&M University, College Station, TX 77843 USA
RoboReason: Boosting Robotics-Relevant VLM Reasoning Through Scalable Data Generation
Oct 24, 2026Sagnik Anupam, Utkarsh Lal, Aneesh Edara +5VLMs for Robotics Visual Reasoning
University of Pennsylvania
RoboJev: An Open Vision-Language Judge for Robot Success Detection
Oct 24, 2026Josef Liyanjun ChenVLMs for Robotics Robot Manipulation Benchmarks
Independent Researcher
Open-World Law-to-Action: Vision-Language Priors and Touch for Parts with Hidden Physics
Oct 24, 2026Mohamed A. MabrokLanguage-Conditioned Robot Manipulation Robot Dynamics Learning
Qatar University
EXE: Egocentric Cross-Embodiment Pretraining via Factorized Latent Actions for VLA Models
Oct 24, 2026Qingyuan Cai, Bo Zhang, Yongzhen Huang +1Cross-Embodiment Robot Learning Robot Learning from Video
ROBOTERA. · Beijing Normal University. · Tsinghua University.
OWLS: Open-world Language-conditioned Speed Fields for Zero-Shot Rover Navigation
Oct 23, 2026Chahyon Ku, Erica Tevere, Georgios Georgakis +2Off-Road Robot Navigation Robot Traversability Estimation
Department of Robotics, University of Michigan, Ann Arbor, MI 48109, USA · Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA 91109, USA
LogLook: Execution-Grounded Memory for Frontier-Model Robot Planners
Oct 23, 2026Tianyi Jiang, Tianyao Tong, Zhenshan BingMemory-Augmented VLMs Large Language Model-Based Robot Planning
State Key Laboratory for Novel Software Technology and School of Intelligence Science and Technology, Nanjing University (Suzhou Campus), Suzhou, China · School of Computation, Information and Technology, Technical University of Munich, Garching, Germany
DuoCook: An Overcooked-Inspired Benchmark for Visual Planning and Coordination
Oct 22, 2026Ziyi Jiao, Heng Zhang, Drishti Regmi +3Multi-Agent Coordination Embodied Agent Benchmarks
Department of Computer Science, University of British Columbia · University of California, San Diego
FastHarness: Experience-Driven Review Scheduling for Vision-Language-Action Policies
Oct 22, 2026Tao Lin, Yuxin Du, Haoxiao Wang +2Agentic AI for Robotics Agent Harness Optimization
School of AI, Shanghai Jiao Tong University · Zhejiang University
ALPS: Visual State Encoders for World Models
Oct 21, 2026Yanqing Liu, Letian Zhang, Chengzhi Liu +5World Model Learning Vision-Language Pretraining
University of California, Santa Cruz · University of California, Santa Barbara · Google
From Reasoning to Action: Benchmarking Frontier AI Models on LIBERO-10 Robot Manipulation Tasks
Oct 21, 2026May Espinola, Qifan Zhang, Yu XiangRobotic Manipulation Zero-Shot Robotic Manipulation
Intelligent Robotics and Vision Lab Department of Computer Science, The University of Texas at Dallas
UniRAP: Towards Unified Part-level Physical Affordance Reasoning and Actionable Perception
Oct 21, 2026Linfei Li, Ruining Hu, Lin Zhang +5Robotic Affordance Inference VLMs for Robotics
School of Computer Science and Technology, Tongji University · School of Electrical Engineering and Computer Science, The University of Queensland · IEIT SYSTEMS (Beijing) Co., Ltd +1
FlowVLA: Learning Flow-Aware Physical Completion for Quantitative Robotic Pouring
Oct 21, 2026Liwen Yang, Yutian Zhang, Jiaqi Zhai +10Robotic Manipulation VLMs for Robotics
Zhejiang University · DEEP Robotics · Shanghai Artificial Intelligence Laboratory +2
From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation
Oct 21, 2026Tingwen Du, Haoyu Guo, Hanxue Zhang +9Robotic Manipulation Long-Horizon Robotic Manipulation
University of Science and Technology of China · Shanghai Artificial Intelligence Laboratory · Shanghai Jiao Tong University +2
Plan First, Act Later: Streaming Reasoning for Executable Memory in Robot Control
Oct 21, 2026Qinghang Liu, Conglin Wang, Zongsheng Li +6Memory-Augmented VLMs Long-Horizon Robotic Manipulation
Peking University · PKU-FirstMoveAI Lab · JD Logistics +2
Failure Modes of Vision-Language Models for Human Activity Understanding in Thermal HRI
Oct 21, 2026Han Hu, He ChenHuman Activity Recognition VLMs for Robotics
Department of Information Engineering, The Chinese University of Hong Kong, Hong Kong SAR, China.
Industrial ReMEmbR: Scaling Spatiotemporal Memory to Robot Fleets
Oct 20, 2026Jad Yammine, Malek Wahidi, Abrar Anwar +3Robotic Spatial Memory Shared-Memory Multi-Agent Systems
Idealworks · NVIDIA
CLOSER: Evidence-Aware Active Visual Verification for Embodied Object-Goal Navigation
Oct 20, 2026Pengying Wu, Mingye Jiang, Yunze Hu +5Object-Goal Navigation Visual Robot Navigation
School of Advanced Manufacturing and Robotics, Peking University, Beijing 100871, China
From Predictive Geometry to Action: Learning Complementary Predictive Representations for Embodied Manipulation
Oct 20, 2026Xiaoxu Xu, Hao Li, Hongtao Zhang +6Robotic Manipulation Predictive Representation Learning
Beihang University · Shanghai Artificial Intelligence Laboratory
Frontier LLMs Enter the World of Robotics: A Cross-Model Evaluation on RoboColiseum
Oct 20, 2026Da Huang, Jichao Wang, Junhui Wu +8Robotic Control Language Model-Based Control
Evaluating Social Reasoning in Embodied Vision-Language Models
Oct 20, 2026Daniel Weiner, Raj KorpanVLMs for Robotics Reasoning for VLMs
The Graduate Center, City University of New York (CUNY), New York, NY, USA · Hunter College and The Graduate Center, City University of New York (CUNY), New York, NY, USA
Where is Physical RSI capability from? An Observational Scaling Law of LLM-as-policy
Oct 19, 2026Zekai Li, Jiayi Yu, Jinlu Zhang +8Self-Improving Agents Embodied AI
Galbot Team

Oct 29, 2026J/K move · Enter open · S save

Surrogate Uncertainty for Black-box Visual Grounding

Hanbo Zhang, Tianrun Hu, Cedar Site Bai, Anxing Xiao+1

National University of Singapore · Shanghai Innovation Institute · Purdue University

Uncertainty Quantification Visual Grounding Visual Grounding for Robotics VLM Calibration VLMs for Robotics

Open paper AskPDF arXiv

Visual grounding is crucial for robots to understand language instructions with visual observations. This paper addresses the problem of visual grounding uncertainties, where a robot uses a black-box vision-language model (VLM) to ground free-form natural-language instructions to target objects. Prevailing methods derive uncertainty from the VLM itself. Yet, outputs of the VLM, e.g., generated uncertainties, are from the same parameters and structures producing the predictions, and inherit the bias. We give a formal Bayesian definition of the problem that treats the VLM prediction as an observation. An external surrogate Bayesian model then quantifies per-query uncertainty over a local neighborhood of a held-out calibration set, decomposed into aleatoric and epistemic components. A hierarchical retrieval across input modalities realizes a compact joint-space neighborhood, mitigating the curse of dimensionality. The estimator further integrates with conformal prediction or model-based planning for active visual grounding, where robots are allowed to take verbal or physical actions to ground targets. Across grounding benchmarks and state-of-the-art VLMs, it consistently improves both calibration and task performance. On a real Fetch robot under open-world instructions, it enables principled trade-offs between confident action and information gathering, substantially improving robustness.