An overview of 3D Vision-Language Models

Márcus Lobo; Vitor Matias; Afonso Paiva; Jeová Farias; Tiago Novello; Moacir Ponti

cs.CVSep 4, 2026

An overview of 3D Vision-Language Models

Authors: Márcus Lobo, Vitor Matias, Afonso Paiva, Jeová Farias, Tiago Novello, Moacir Ponti

Organizations: ICMC-USP · Bowdoin College · IMPA

Abstract

Vision-Language Models (VLMs) are reshaping computer vision by aligning visual and textual embeddings, allowing models to recognize visual concepts and reason about them using natural language. Traditional 3D deep-learning models, however, are typically trained for specific tasks, such as classification, segmentation, or detection, and do not naturally support cross-modal retrieval from their embedding spaces using text or images as queries. To address this issue, Contrastive Language-Image Pretraining (CLIP)-based methods align 3D embeddings with pretrained image and text representations, giving rise to 3D Vision-Language Models (3D VLMs) that support zero-shot classification, cross-modal retrieval, and open-vocabulary recognition of 3D shapes. This tutorial provides an overview of 3D VLMs, ranging from basic definitions of 3D representations and their encoding into embeddings to cross-modal contrastive alignment, modern multimodal frameworks, and 3D Vision-Large Language Models (3D VLLMs). We present the main definitions of contrastive learning for multimodal embedding alignment and highlight recent advances in language-guided 3D Gaussian splatting, 3D shape generation, and embodied AI for robotics.

Figures & tables

Fig. 1: Overview of the 3D VLM pipeline. Cross-modal contrastive objectives increase the similarity of matched 3D–Image and 3D–text pairs relative to negative in-batch candidates, producing a shared semantic embedding space.

Figure 2

Method	Rep.	CL	3D VLMs	3D VLLMs	3DGS/Applications
Zhang et al. [ 64 ]	✗	✓	✗	✗	✗
Ma et al. [ 31 ]	✓	$\circ$	$\circ$	✓	$\circ$
Thengane et al. [ 55 ]	$\circ$	✓	✓	✓	✗
Ren et al. [ 45 ]	$\circ$	$\circ$	$\circ$	✓	✗
Li et al. [ 26 ]	$\circ$	$\circ$	$\circ$	✓	✗
Guo et al. [ 11 ]	$\circ$	$\circ$	$\circ$	$\circ$	$\circ$

TABLE I: Scope comparison with related surveys. ✓ : dedicated coverage;

\circ

: partial or task-specific coverage; ✗ : outside the primary scope. Rep. : multiple 3D representations; CL : contrastive learning.

Fig. 2: Examples of geometric representations used by 3D VLMs: point cloud, meshes, 3D Gaussian Splatting, and multi-view renderings.

Fig. 3: Geometric intuition of contrastive alignment. The top panels visualize the shared embedding space, and the bottom panels show its angular interpretation on the unit hypersphere. (a) Before alignment, a 3D anchor may be closer to a negative image sample than to its positive pair. (b) Contrastive optimization changes this relation by increasing positive similarity relative to the remaining in-batch candidates.

Fig. 4: CLIP-style alignment for 3D VLMs. A 3D encoder aligns its embeddings with image and text encoders through 3D-Image and 3D-text similarity matrices. The main diagonal entries define matched positive pairs, whereas off-diagonal entries define the remaining negative candidates for symmetric InfoNCE [ 37 ] loss.

Fig. 5: Main tasks and applications of 3D VLMs.

Explore similar work

3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning
Aug 29, 2026Márcus Lobo, Vitor Matias, Jeová Farias +13D VLMs Multimodal Pretraining
Instituto de Ciências Matemáticas e de Computação Universidade de São Paulo São Carlos, São Paulo, BR · Bowdoin College Brunswick, Maine, USA
VLM3: Vision Language Models Are Native 3D Learners
May 28, 2026Zhipeng Cai, Zhuang Liu, Yunyang Xiong +3Camera Pose Estimation Monocular Depth Estimation
Meta · Princeton University
PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models
Feb 28, 2026Yuanhao Su, Shaofeng Zhang, Xiaosong Jia +13D VLMs Vision-Language Alignment
University of Science and Technology of China · Fuzhou University · Fudan University +1

Aug 29, 2026J/K move · Enter open · S save

3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning

Márcus Lobo, Vitor Matias, Jeová Farias, Moacir Ponti

Instituto de Ciências Matemáticas e de Computação Universidade de São Paulo São Carlos, São Paulo, BR · Bowdoin College Brunswick, Maine, USA

3D VLMs Multimodal Pretraining Point Cloud Learning Matryoshka Representation Learning

Open paper AskPDF arXiv

Vision-Language Models align point clouds with image and text embeddings, enabling zero-shot recognition, retrieval, and open-vocabulary understanding of 3D shapes. Existing multimodal 3D pre-training methods produce fixed-dimensional embeddings, requiring separate models for different computational budgets. We propose 3D Matryoshka Representation Learning (3D-MRL), a multimodal 3D pre-training framework based on Matryoshka Representation Learning. 3D-MRL learns nested 3D representations by aligning point clouds with frozen CLIP image and text embeddings while applying contrastive supervision across multiple embedding dimensions. The Matryoshka objective is applied only to the 3D encoder, allowing a single model to produce representations at different dimensionalities without retraining. Experiments on the Objaverse-LVIS, ModelNet40, and ScanNet datasets show that 3D-MRL achieves competitive performance on zero-shot and few-shot 3D recognition tasks. In addition, the learned representations support retrieval across different embedding dimensions within a single model. On Objaverse-LVIS, 3D-MRL improves Top-1 accuracy from 46.8% to 50.9%. Retrieval experiments further show that different embedding dimensions yield varying levels of semantic and geometric specificity.

Abstract

Explore similar work

3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning

VLM3: Vision Language Models Are Native 3D Learners

PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models