3D VLMs

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

14 papers in the last four weeks, up 133% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 118

All topics

Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
Aug 4, 2026Lucy Lin, Ayush Jain, Yifan Liu +13D Foundation Models 3D VQA
Carnegie Mellon University
Hunyuan3D-Buffalo 1.0: A Unified Multimodal Model for Scalable 3D Generation, Understanding, and Editing
Aug 3, 2026Junliang Ye, Kenkun Liu, Guocun Wang +133D Foundation Models Part-Level 3D Generation
Tencent Hunyuan
3DZip: Spatial-Aware Feature Diversity-Guided Token Compression for 3D Question Answering
Aug 2, 2026Changwoo Baek, Kyeongbo KongMultimodal Token Compression 3D VQA
Department of Electrical and Electronics Engineering, Pusan National University, Busan, Republic of Korea
Distill What RGB Can Recover: Privileged 3D Evidence for RGB-Only Vision-Language Models
Jul 31, 2026Yanbin Hu, Jin Cui, Jun Ye +43D Spatial Reasoning VLM Distillation
School of Software, Xi’an Jiaotong University · School of Artificial Intelligence, Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University +1
On the Efficacy of Self-Supervised Point Cloud Encoders for Efficient 3D Large Language Models
Jul 31, 2026Yao Zheng, Tian ZhangSelf-Supervised Pre-Training Point Cloud Learning
Beijing University of Posts and Telecommunications
MedARC: Training-Free Adaptive Redundancy Compression of Visual Tokens for 3D Medical Vision-Language Models
Jul 29, 2026Yitao Zhu, Mengjun Liu, Yingji Fu +2Efficient VLM Inference Medical VLMs
Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong · School of Integrated Circuits and Electronics, Beijing Institute of Technology, China · 3Mental Health Research Center, The Hong Kong Polytechnic University, Hong Kong +1
ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding
Jul 27, 2026Hangjie Yuan, Yichen Qian, Zhiwei Tang +21Medical Report Generation Medical VQA
1DAMO Academy, Alibaba Group, Hangzhou, China · 3Hupan Laboratory, Hangzhou, China · 2DAMO Academy, Alibaba Group, Beijing, China +7
RadSight: Towards Perceptually Reliable Multimodal Radiology Image Understanding
Jul 24, 2026Jianqin Liu, Weiwei Cao, Wanxing Chang +7Radiology VLMs Medical Image Grounding
1DAMO Academy, Alibaba Group · University of Electronic Science and Technology of China
Cheap Probes Predict Expensive Training in 3D-CT Vision--Language Models
Jul 24, 2026Renjie LiangVLM Evaluation Medical VLMs
University of Florida, Gainesville, FL, USA
3D-Aware VLMs with Implicit and Explicit Geometries
Jul 23, 2026Wenhao Li, Xueying Jiang, Quanhao Qian +43D Spatial Reasoning 3D Object Detection
1Nanyang Technological University · 2DAMO Academy, Alibaba Group · 3HuPan Lab +1
D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models
Jul 21, 2026Heesang Han, A. Lynn Abbott, Abhijit SarkarVLMs for Autonomous Driving Camera-LiDAR Fusion
Bradley Department of Electrical and Computer Engineering, Virginia Tech, USA · Sanghani Center for Artificial Intelligence and Data Analytics, USA · Virginia Tech Transportation Institute, USA
When Can Test-Time Adaptation Help Zero-Shot CT Vision-Language Models?
Jul 17, 2026Ailar Mahdizadeh, Puria Azadi Moghadam, Xiangteng He +1Zero-Shot Learning Radiology VLMs
University of British Columbia · 2Vector Institute
Multi-LLM Collaborative MRI Report Generation for Visual Instruction Tuning in Brain Oncology
Jul 16, 2026Sinyoung Ra, Jonghun Kim, Hyunjin ParkRadiology Report Generation 3D Medical Imaging
Department of Artificial Intelligence, Sungkyunkwan University, Suwon, Republic of Korea · Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, Republic of Korea
GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understanding
Jul 15, 2026Hao Li, Han Fang, Zixin Pan +83D Spatial Reasoning Latent Visual Reasoning
Shanghai Jiao Tong University · Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) · The Hong Kong University of Science and Technology (Guangzhou) +1
CoSAG: Compact Semantic Anchor Gaussians via Training-Free Rate-Distortion Coding
Jul 11, 2026Yuang Jia, Jinlong Wang, Junhong Lin +23DGS Compression Entropy Coding
SECE, Peking University · University of Electronic Science and Technology of China
ELSA3D: Elastic Semantic Anchoring for Unified 3D Understanding and Generation
Jul 7, 2026Tianjiao Yu, Xinzhuo Li, Yifan Shen +43D Foundation Models Unified Multimodal Models
University of Illinois Urbana-Champaign
CAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Models
Jul 7, 2026He Liang, Chenyang Ma, Yiming Zhang +43D Spatial Reasoning 3D Scene Understanding
University of Oxford · 3Simon Fraser University · 2Microsoft Research
VaseMuseum: Digital Intelligent Museum for Ancient Greek Pottery
Jul 7, 2026Jiazi Wang, Nonghai Zhang, Qiushi Xie +5Multimodal RAG Cultural Heritage
School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China · School of Computer Science, Peking University, Beijing 100871, China · Department of Computer Science and Information Technology, La Trobe University, Melbourne VIC 3000, Australia +1
PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNet
Jul 7, 2026Xiaopei Wu, Chenshu Hou, Liang Peng +93D Scene Understanding 3D VLMs
1SH AI Lab · 2ZJU · 1SH AI Lab 2ZJU +2
SpaR3D-MoE: Adaptive 3D Spatial Reasoning from Sparse Views Meets Geometry-Inductive Mixture-of-Experts
Jul 7, 2026Haida Feng, Hao Wei, Haolin Wang +33D Spatial Reasoning Mixture of Experts
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, China · YUKUN Intelligent World, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, China
Ground3D-LMM: Fine-Grained 3D Point Grounding and Spatial Reasoning with LMM
Jul 6, 2026Amol Harsh, Zongyan Han, Jean Lahoud +53D Spatial Reasoning Multimodal QA
Mohamed bin Zayed University of Artificial Intelligence · The Hong Kong Polytechnic University · Linköping University
G $^2$ TAM: Geometry Grounded Track Anything Model
Jul 4, 2026Chenming Zhu, Peizhou Cao, Jingli Lin +5Video Object Segmentation 3D Reconstruction
HKU · Shanghai AI Lab · BUAA +3
From Geometric Labels to Semantic Understanding of Indoor Building Components Using Multimodal Large Language Models
Jul 4, 2026Shuju Jing, Chao YinPoint Cloud Understanding 3D VLMs
School of Qilu Transportation, Shandong University, Jinan, 250002, China · Guangzhou Institute of Geography, Guangdong Academy of Sciences, 100 Xianlie Middle Road, Guangzhou, 510070, Guangdong, China
SpaceEra++: A Unified Framework Towards 3D Spatial Reasoning in Video
Jul 2, 2026Weili Guan, Haoyu Zhang, Meng Liu +3Visual Spatial Reasoning 3D Spatial Reasoning
School of Information Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, China · Shenzhen Loop Area Institute, Shenzhen 518045, China · School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen 518055, China +4
Disentangling Pictorial Cue Understanding from Language Bias in VLMs via Depth Ordering Task
Jul 1, 2026Yiqian Liu, Iuliia Kotseruba, John K. TsotsosVLM Evaluation Vision-Language Models
York University, Toronto, Canada · University of Guelph, Guelph, Canada
MV-GEL: Language-Driven Multi-View Geometric Entity Localization on Meshes
Jun 30, 2026Kartik Bali, Roland AydinOpen-Vocabulary 3D Segmentation 3D VLMs
Helmholtz Zentrum Hereon, Max-Planck-Strasse 1, 21502 Geesthacht, Germany · Institute for Continuum and Material Mechanics, Hamburg University of Technology, Eissendorfer Strasse 42, 21073 Hamburg, Germany · German Research Center for Artificial Intelligence, Campus D3 2, 66123 Saarbrücken, Germany
PruneGround: Plug-and-play Spatial Pruning for 3D Visual Grounding
Jun 30, 2026Duc Cao Dinh, Khai Le-Duc, Florent Draye +43D Spatial Reasoning Visually Grounded Reasoning
Knovel Engineering Lab · University of Toronto · Vector Institute +2
SAD-GS: Learning Reliable 3D Semantic Gaussian Fields via Dynamic Geo-Semantic Anchoring
Jun 28, 2026Yufei Zhang, Chenlu Zhan, Gaoang Wang +13D Gaussian Splatting Open-Vocabulary 3D Segmentation
Zhejiang University
Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography
Jun 24, 2026Bowen Shi, Weiwei Cao, Ruifeng Yuan +5Contrastive Learning 3D Medical Imaging
DAMO Academy, Alibaba Group · Hupan Lab, 310023, Hangzhou, China · Shanghai Jiao Tong University, China +2
Jolia: Concept-Level Vision-Language Alignment for 3D CT Contrastive Learning
Jun 23, 2026Julien Khlaut, Charles Corbière, Baptiste Callard +93D Foundation Models Medical Imaging Foundation Models
Raidium, Paris, 75014, France · Department of Vascular and Oncological Interventional Radiology,2026 Hôpital Européen Georges Pompidou, AP-HP, Paris, France · Faculté de Santé, Université Paris-Cité, Paris, France +3

Aug 4, 2026, cs.CVJ/K move · Enter open · S save

Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding

Lucy Lin, Ayush Jain, Yifan Liu, Katerina Fragkiadaki

Carnegie Mellon University

3D Foundation Models 3D VQA 3D Instance Segmentation 3D Scene Understanding 3D VLMs 3D Visual Grounding

Open paper AskPDF arXiv

Large Multimodal Models (LMMs) have achieved remarkable success on images and short videos, yet scaling them to long videos remains challenging due to frame-centric tokenization and limited context windows. 3D geometry provides a natural compression mechanism for visual streams: depth and camera pose enable observations from multiple views and time steps to be fused into a persistent, world-aligned representation. While recent 3D LMMs leverage geometry-aware representations to improve spatial reasoning, they continue to lag behind specialist 3D perception systems on grounding and segmentation tasks. We argue that a key limitation is geometry-aware decoding: existing methods communicate 3D predictions through language tokens, proposal selection, or lightweight grounding queries, creating a bottleneck between language reasoning and dense geometric prediction. Building on these insights, we introduce Qwen-3D, a geometry-aware LMM that compresses visual information within the Qwen backbone using multi-view geometric cues, enabling efficient long-horizon visual reasoning over static scenes. Qwen-3D augments visual tokens with 3D Rotary Positional Embeddings, allowing attention to operate directly in 3D scene space rather than across independent image frames and thereby facilitating scalable cross-view and temporal reasoning. To bridge language and geometry, Qwen-3D incorporates a query-based segmentation decoder that grounds language directly in the underlying 3D scene representation, unifying referential grounding, instance segmentation, and visual question answering across both images and videos. Across a diverse set of benchmarks, Qwen-3D surpasses existing 3D LMMs and outperforms several large proprietary 2D models. Notably, Qwen-3D achieves these improvements while maintaining strong performance on standard 2D vision-language benchmarks by jointly training on 2D and 3D data.

Momentum

Related topics

Latest papers 118