Vision-Language Models (VLMs) are reshaping computer vision by aligning visual and textual embeddings, allowing models to recognize visual concepts and reason about them using natural language. Traditional 3D deep-learning models, however, are typically trained for specific tasks, such as classification, segmentation, or detection, and do not naturally support cross-modal retrieval from their embedding spaces using text or images as queries. To address this issue, Contrastive Language-Image Pretraining (CLIP)-based methods align 3D embeddings with pretrained image and text representations, giving rise to 3D Vision-Language Models (3D VLMs) that support zero-shot classification, cross-modal retrieval, and open-vocabulary recognition of 3D shapes. This tutorial provides an overview of 3D VLMs, ranging from basic definitions of 3D representations and their encoding into embeddings to cross-modal contrastive alignment, modern multimodal frameworks, and 3D Vision-Large Language Models (3D VLLMs). We present the main definitions of contrastive learning for multimodal embedding alignment and highlight recent advances in language-guided 3D Gaussian splatting, 3D shape generation, and embodied AI for robotics.
Figures & tables
Fig. 1: Overview of the 3D VLM pipeline. Cross-modal contrastive objectives increase the similarity of matched 3D–Image and 3D–text pairs relative to negative in-batch candidates, producing a shared semantic embedding space.
Figure 2
Method
Rep.
CL
3D VLMs
3D VLLMs
3DGS/Applications
Zhang et al. [ 64 ]
✗
✓
✗
✗
✗
Ma et al. [ 31 ]
✓
∘
∘
✓
∘
Thengane et al. [ 55 ]
∘
✓
✓
✓
✗
Ren et al. [ 45 ]
∘
∘
∘
✓
✗
Li et al. [ 26 ]
∘
∘
∘
✓
✗
Guo et al. [ 11 ]
∘
∘
∘
∘
∘
TABLE I: Scope comparison with related surveys. ✓ : dedicated coverage; ∘ : partial or task-specific coverage; ✗ : outside the primary scope. Rep. : multiple 3D representations; CL : contrastive learning.
Fig. 2: Examples of geometric representations used by 3D VLMs: point cloud, meshes, 3D Gaussian Splatting, and multi-view renderings.
Fig. 3: Geometric intuition of contrastive alignment. The top panels visualize the shared embedding space, and the bottom panels show its angular interpretation on the unit hypersphere. (a) Before alignment, a 3D anchor may be closer to a negative image sample than to its positive pair. (b) Contrastive optimization changes this relation by increasing positive similarity relative to the remaining in-batch candidates.
Fig. 4: CLIP-style alignment for 3D VLMs. A 3D encoder aligns its embeddings with image and text encoders through 3D-Image and 3D-text similarity matrices. The main diagonal entries define matched positive pairs, whereas off-diagonal entries define the remaining negative candidates for symmetric InfoNCE [ 37 ] loss.
Vision-Language Models align point clouds with image and text embeddings, enabling zero-shot recognition, retrieval, and open-vocabulary understanding of 3D shapes. Existing multimodal 3D pre-training methods produce fixed-dimensional embeddings, requiring separate models for different computational budgets. We propose 3D Matryoshka Representation Learning (3D-MRL), a multimodal 3D pre-training framework based on Matryoshka Representation Learning. 3D-MRL learns nested 3D representations by aligning point clouds with frozen CLIP image and text embeddings while applying contrastive supervision across multiple embedding dimensions. The Matryoshka objective is applied only to the 3D encoder, allowing a single model to produce representations at different dimensionalities without retraining. Experiments on the Objaverse-LVIS, ModelNet40, and ScanNet datasets show that 3D-MRL achieves competitive performance on zero-shot and few-shot 3D recognition tasks. In addition, the learned representations support retrieval across different embedding dimensions within a single model. On Objaverse-LVIS, 3D-MRL improves Top-1 accuracy from 46.8% to 50.9%. Retrieval experiments further show that different embedding dimensions yield varying levels of semantic and geometric specificity.
Márcus Lobo, Vitor Matias, Jeová Farias +1
Instituto de Ciências Matemáticas e de Computação Universidade de São Paulo São Carlos, São Paulo, BR · Bowdoin College Brunswick, Maine, USA
Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.
The development of 3D Vision-Language Models (VLMs), crucial for applications in robotics, autonomous driving, and augmented reality, is severely constrained by the scarcity of paired 3D-text data. Existing methods rely solely on next-token prediction loss, using only language tokens for supervision. This results in inefficient utilization of limited 3D data and leads to a significant degradation and loss of valuable geometric information in intermediate representations. To address these limitations, we propose {\mname}, a novel feature-level alignment regularization method. {\mname} explicitly supervises intermediate point cloud tokens to preserve fine-grained 3D geometric-semantic information throughout the language modeling process. Specifically, we constrain the intermediate point cloud tokens within the LLM to align with visual input tokens via a consistency loss. By training only a lightweight alignment projector and LoRA adapters, {\mname} achieves explicit feature-level supervision with minimal computational overhead, effectively preventing geometric degradation. Extensive experiments on ModelNet40 and Objaverse datasets demonstrate that our method achieves 2.08 pp improvement on average for classification tasks, with a substantial 7.50 pp gain on the challenging open-vocabulary Objaverse classification task and 4.88 pp improvement on 3D object captioning evaluated by Qwen2-72B-Instruct, validating the effectiveness of {\mname}. Code is publicly available at https://github.com/yharoldsu0627/PointAlign.
Yuanhao Su, Shaofeng Zhang, Xiaosong Jia +1
University of Science and Technology of China · Fuzhou University · Fudan University +1