3D VLMs

VLM: Vision-Language Model

Ask about this topic

Overview

Three-dimensional vision-language models (3D VLMs) study how to connect language with point clouds, RGB-D images, multi-view video, and volumetric scans so models can describe, locate, and reason about objects and scenes. Common approaches align 3D features with pretrained image and text models, then add geometric tokens, scene graphs, spatial maps, or active view selection for tasks such as grounding, question answering, captioning, and navigation. Systems are evaluated on benchmarks including ScanRefer, ScanQA, SQA3D, and VSI-Bench, with growing attention to robot interaction and medical imaging. A central challenge is ensuring models use genuine geometric and viewpoint evidence rather than relying on language or 2D visual shortcuts, while keeping inference efficient and performance reliable across scenes and domains.

Latest papers

CardsList

214 papers24 papers in the last four weeks, up 20% on the four weeks before. 0.1% of all new papers.

  1. Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM

    Mar 29, 2026Haifeng Huang, Yilun Chen, Zehan Wang +23D VLMs3D Scene Understanding

    School of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China. · Shanghai AI Laboratory, Shanghai 200032, China.

  2. SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning

    Mar 28, 2026Jian Zhang, Shijie Zhou, Bangya Liu +23D Spatial Reasoning3D VLMs

    XMU · UCLA · Google +2

  3. Communicating about Space: Language-Mediated Spatial Integration Across Partial Views

    Mar 28, 2026Ankur Sikarwar, Debangan Mishra, Sudarshan Nikhil +2Spatial Reasoning in Language ModelsSpatial Reasoning

    Mila – Quebec AI Institute · Université de Montréal · IIIT Hyderabad +1

  4. 3D-IDE: 3D Implicit Depth Emergent

    Mar 28, 2026Chushan Zhang, Ruihan Lu, Jinguang Tong +23D VLMs3D Scene Understanding

    School of Computing, Australian National University · FreiNexus · School of EECS, The University of Queensland +1

  5. Make Geometry Matter for Spatial Reasoning

    Mar 27, 2026Shihua Zhang, Qiuhong Shen, Shizun Wang +2Visual Spatial Reasoning3D Spatial Reasoning

    National University of Singapore, Singapore

  6. HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models

    Mar 26, 2026Huizhi Liang, Yichao Shen, Yu Deng +53D Spatial Reasoning3D VLMs

    Tsinghua University · Microsoft Research Asia · Xi’an Jiaotong University +1

  7. Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models

    Mar 26, 2026Chengyu Fang, Heng Guo, Zheng Jiang +3Medical VLMs3D VLMs

    Tsinghua University · DAMO Academy, Alibaba Group · Hupan Lab +1

  8. Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Models

    Mar 25, 2026Shengli Zhou, Minghang Zheng, Feng Zheng +13D Spatial ReasoningPositional Encoding

    Department of Computer Science and Engineering, Southern University of Science and Technology · Wangxuan Institute of Computer Technology, Peking University · State Key Laboratory of General Artificial Intelligence, Peking University

  9. 3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models

    Mar 25, 2026Bin Yu, Shijie Lian, Xiaopeng Lin +8VLMs for RoboticsVision-Language-Action Models

    HIT · ZGCA · HUST +5

  10. Grounding Vision and Language to 3D Masks for Long-Horizon Box Rearrangement

    Mar 24, 2026Ashish Malik, Caleb Lowe, Aayam Shrestha +3Robot Task PlanningRobotic Manipulation

    Electrical Engineering and Computer Science, College of Engineering, Oregon State University

  11. 3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding

    Mar 24, 2026Yiping Chen, Jinpeng Li, Wenyu Ke +63D VLMs3D Scene Understanding

    School of Geospatial Engineering and Science, Sun Yat-sen University, Zhuhai, 519082, China. · College of Electronic Science, National University of Defense Technology, Changsha, 410000, China. · Department of Civil and Environmental Engineering, Norwegian University of Science and Technology, Trondheim, Norway. +1

  12. Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps

    Mar 24, 2026Chanyoung Gwak, Yoonwoo Jeong, Byungwoo Jeon +3Multimodal Spatial Reasoning3D Spatial Reasoning

    POSTECH · KAIST · RLWRLD

  13. Adapting Point Cloud Analysis via Multimodal Bayesian Distribution Learning

    Mar 23, 2026Xingyu Zhu, Liang Yi, Shuo Wang +4Bayesian Model AveragingTest-Time Adaptation of VLMs

    MoE Key Lab of BIPC, University of Science and Technology of China · Opus AI Research · Southeast University +1

  14. TSegAgent: Zero-Shot Tooth Segmentation via Geometry-Aware Vision-Language Agents

    Mar 20, 2026Shaojie Zhuang, Lu Yin, Guangshun Wei +3Medical ImagingDental Imaging

    Shandong University, Jinan 250010, China · University of Surrey, Guildford, Surrey, United Kingdom · King’s College London, London, United Kingdom

  15. Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understanding

    Mar 19, 2026Xianjin Wu, Dingkang Liang, Tianrui Feng +53D Spatial Reasoning3D Scene Understanding

    Huazhong University of Science and Technology, China · Baidu Inc., China

  16. Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

    Mar 18, 2026Kevin Qu, Haozhe Qi, Mihai Dusmanu +33D VLMs3D Spatial Reasoning

    Microsoft Spatial AI Lab · ETH Zurich · EPFL

  17. Feeling the Space: Egomotion-Aware Video Representation for Efficient and Accurate 3D Scene Understanding

    Mar 18, 2026Shuyao Shi, Kang G. Shin3D Scene Understanding3D Spatial Reasoning

    Department of Computer Science, University of Michigan, Ann Arbor, MI, USA

  18. IOSVLM: A 3D Vision-Language Model for Unified Dental Diagnosis from Intraoral Scans

    Mar 17, 2026Huimin Xiong, Zijie Meng, Tianxiang Hu +33D VLMsMedical VLMs

    ZJU-UIUC Institute, Zhejiang University, Haining, 314400, China · Stomatology Hospital, School of Stomatology, Zhejiang University School of Medicine, Hangzhou, 310058, China · Angelalign Research Institute, Angel Align Inc., Shanghai, 200011, China

  19. VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations

    Mar 17, 2026Fucai Ke, Zhixi Cai, Boying Li +6Visual ReasoningSpatial Reasoning Benchmarks

    Monash University, Australia · National University of Singapore

  20. GT-PCQA: Geometry-Texture Decoupled Point Cloud Quality Assessment with MLLM

    Mar 16, 2026Guohua Zhang, Jian Jin, Meiqin Liu +33D VLMs

    Beijing Jiaotong University · Nanyang Technological University · University of Science and Technology Beijing

  21. SpatialPoint: Spatial-aware Point Prediction for Embodied Localization

    Mar 16, 2026Qiming Zhu, Zhirui Fang, Tianming Zhang +3VLMs for Robotics3D VLMs

    Visincept Research · Tsinghua University · International Digital Economy Academy (IDEA)

  22. ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulation

    Mar 14, 2026You Wu, Zixuan Chen, Cunxu Ou +9Robotic ManipulationVLMs for Robotics

    School of Computer Science, Nanjing University · School of Intelligence Science and Technology, Nanjing University · School of Electronic Science and Engineering, Nanjing University +2

  23. LoV3D: Grounding Cognitive Prognosis Reasoning in Longitudinal 3D Brain MRI via Regional Volume Assessments

    Mar 12, 2026Zhaoyang Jiang, Zhizhong Fu, David McAllister +2Longitudinal Medical Image AnalysisAlzheimer's Disease Classification

    School of Health & Wellbeing, University of Glasgow, Glasgow, UK · School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China · Institute of Health Informatics, University College London, London, UK

  24. VLM-Loc: Localization in Point Cloud Maps via Vision-Language Models

    Mar 10, 2026Shuhao Kang, Youqi Liao, Peijie Wang +5Map-Based Localization3D VLMs

    VCIP, CS, Nankai University · Wuhan University · CASIA +6

  25. X-GS: An Extensible Framework for Perceiving and Thinking with 3D Gaussian Splatting

    Mar 10, 2026Yueen Ma, Zenglin Xu, Irwin King3D Gaussian Splatting3DGS SLAM

    The Chinese University of Hong Kong · Shanghai Academy of AI for Science · Fudan University

  26. Point Cloud as a Foreign Language for Multi-modal Large Language Model

    Mar 10, 2026Sneha Paul, Zachary Patterson, Nizar Bouguila3D VLMsPoint Cloud Learning

    Concordia University, Canada

  27. UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing

    Mar 9, 2026Jiaxi Zhang, Yunheng Wang, Wei Lu +83D Visual Grounding3D VLMs

    The Hong Kong University of Science and Technology (Guangzhou) · JD Explore Academy · Zhejiang University

  28. Toward Unified Multimodal Representation Learning for Autonomous Driving

    Mar 9, 2026Ximeng Tao, Dimitar Filev, Gaurav PandeyMultimodal PretrainingMultimodal Contrastive Learning

    J. Mike Walker ’66 Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA · The Department of Engineering Technology and Industrial Distribution Texas A&M University, College Station, TX 77843, USA

  29. Do 3D Large Language Models Really Understand 3D Spatial Relationships?

    Mar 6, 2026Xianzheng Ma, Tao Sun, Shuai Chen +73D Spatial Reasoning3D VQA

    VGG, University of Oxford · AVL, University of Oxford · Stanford University +2

  30. Hierarchical Collaborative Fusion for 3D Instance-aware Referring Expression Segmentation

    Mar 6, 2026Keshen Zhou, Runnan Chen, Mingming Gong +13D Referring SegmentationReferring Expression Segmentation

    The University of Sydney · The University of Melbourne · Mohamed bin Zayed University of Artificial Intelligence