3D VLMs

Three-dimensional vision-language models (3D VLMs) study how to connect language with point clouds, RGB-D images, multi-view video, and volumetric scans so models can describe, locate, and reason about objects and scenes. Common approaches align 3D features with pretrained image and text models, then add geometric tokens, scene graphs, spatial maps, or active view selection for tasks such as grounding, question answering, captioning, and navigation. Systems are evaluated on benchmarks including ScanRefer, ScanQA, SQA3D, and VSI-Bench, with growing attention to robot interaction and medical imaging. A central challenge is ensuring models use genuine geometric and viewpoint evidence rather than relying on language or 2D visual shortcuts, while keeping inference efficient and performance reliable across scenes and domains.