VLMs for Robotics
VLM: Vision-Language Model
Overview
This field studies how robots can use visual observations and language instructions to understand scenes, plan tasks, and act in the physical world. Common approaches adapt pretrained vision-language models with geometric or 3D grounding, maps and affordances, action heads, skill libraries, or feedback loops; some separate slow semantic planning from fast control. Researchers train and test these systems using robot demonstrations, human videos, simulation, and real-robot tasks, with benchmarks such as LIBERO, RLBench, and Habitat. A central challenge is turning plausible language-based reasoning into reliable, metrically grounded action despite occlusion, changing scenes, latency, and safety constraints.
Latest papers
664 papers117 papers in the last four weeks, up 10% on the four weeks before. 0.7% of all new papers.