Self-supervised learning has given rise to vision foundation models that learn general-purpose image representations transferable across downstream tasks, yet representation learning for 3D scenes has not achieved comparable success. Motivated by this, we identify key limitations of current 3D representation learning approaches and revisit their core design choices. We first replace the commonly used hierarchical PTv3 backbone with the Volume Transformer, a ViT-like architecture for 3D scenes. We then redesign two core objectives of modern approaches built around student-teacher networks: matching representations across views and predicting masked content from visible context. For the first objective, we separate the roles of teacher and student: the teacher provides clean, consistent targets while the student learns invariance to stronger perturbations. For the second objective, we prevent the student from accessing the masked geometry and instead require it to infer missing representations from visible context only. Finally, we redesign 2D-to-3D knowledge distillation, replacing direct regression of view-dependent image features with view-consistent supervision. Together, these changes form VINO, a framework that learns semantic representations of 3D scenes without labels. VINO substantially improves dense linear probing across indoor and outdoor benchmarks and transfers strongly to out-of-domain settings.
Figures & tables
Figure 1: Self-supervised 3D Representations. We propose VINO, a framework for learning semantic representations of 3D scenes with no labels. Trained on 3D data only, VINO captures semantic similarity better than Sonata ( ? ), as shown in PCA projections (top) and cosine-similarity maps (bottom) . Sonata yields qualitatively similar representations across backbones, whereas VINO better distinguishes semantically different but geometrically similar regions, for example, chair and table legs, and captures semantic similarity across geometrically different instances, as illustrated by one plant strongly activating the other.
Figure 2: Comparison of Training Views. Sonata ( ? ) geometrically corrupts all views and retains masked tokens in the student encoder. In contrast, VINO keeps the teacher’s global views clean, removes masked tokens from the student encoder, and applies geometric corruptions only to local views.
Figure 3: VINO Overview. Overlapping global and local views are sampled from a 3D scene. The EMA teacher encodes clean global views and softly assigns their tokens to learned prototypes, producing balanced targets. The prototypes are optimized independently by encouraging balanced and consistent assignments across global views. The student encodes strongly augmented local views and masked global views. Local-view embeddings, through a projection head, predict the targets of corresponding global-view tokens. Masked view embeddings provide context to a cross-attention-only predictor that predicts the targets at masked locations.
Table 2: Local Prediction. The masked branch is removed. Same evaluation setup as in Tab. 1 .
ScanNet200
Local pred.
32.8
+ Masked pred.
35.6
+ Fix artifacts
35.9
Table 3: Masked Prediction. Same evaluation setup as in Tab. 1 .
Figure 4: Representation Artifacts. We observe high-norm token artifacts in the representations (left). Register tokens largely eliminate these artifacts, but early in training, the learned representations still strongly reflect local voxel occupancy. Averaging voxel features within each patch removes this residual artifact (right).
ScanNet200
Align from start
37.2
Raw embeddings
38.6
Prototypes
40.4
Table 4: 2D–3D Distillation. Linear probing (mIoU) for different 2D–3D distillation objectives.
Figure 5: Consistent 2D Targets. VFM ( ? ) features of the same 3D region vary across views. We learn prototypes in the frozen VFM feature space, optimized such that corresponding tokens across views produce matching soft assignments. These serve as our targets for 2D–3D distillation.
Method
Backbone
Indoor Datasets
Outdoor Datasets
ScanNet200
ScanNet
ScanNet++
nuScenes
Sem.KITTI
Waymo
Supervised †
Volt-B (94.3 M)
36.2
77.7
51.7
81.3
70.6
71.5
3D only
LAM3C ( ? )
PTv3-B (108.4 M)
25.3
66.0
34.2
–
–
–
MSM ( ? )
HUNet (67.6 M)
26.8
68.7
–
–
–
–
Sonata ( ? )
PTv3-B (108.4 M)
29.3
72.5
38.9
66.1
62.0
60.5
PointINS ( ? )
PTv3-B (108.4 M)
29.6
72.4
–
74.6
68.0
–
Table 5: Linear Probing 3D Semantic Segmentation (mIoU). For each dataset, a linear classifier is trained on the training split with the 3D encoder frozen and evaluated on the validation split. Methods are grouped by 3D-only pretraining (top) and pretraining with additional 2D supervision (bottom) . DOS and PointINS outdoor results use PTv3-S. † is trained end-to-end with Volt’s decoder and serves as a reference.
Method
Backbone
Indoor Datasets
Outdoor Datasets
ScanNet200
ScanNet
ScanNet++
nuScenes
Sem.KITTI
Waymo
D-DITR ( ? )
PTv3-S (46.2 M)
37.7
79.2
–
80.7
69.8
72.1
Sonata ( ? )
PTv3-B (124.8 M)
36.8
79.4
49.3
81.7
72.6
72.9
DOS ( ? )
PTv3-B (124.8 M)
37.1
79.7
–
81.8
73.5
73.9
PointINS ( ? )
PTv3-B (124.8 M)
36.6
79.0
–
81.1
72.7
–
Concerto ( ? )
PTv3-B (124.8 M)
39.2
80.7
50.7
82.0
71.2
69.2
Table 6: Fine-tuning 3D Semantic Segmentation (mIoU). Pretrained encoders and task-specific decoders are jointly fine-tuned on the training split and evaluated on the validation split for each dataset.
Method
Backbone
ScanNet200
ScanNet
ScanNet++
mAP 50
mAP
mAP 50
mAP
mAP 50
mAP
3D only
LAM3C ( ? )
PTv3-B (108.4 M)
–
8.3
–
25.1
–
11.3
MSM ( ? )
HUNet (67.6 M)
8.8
–
44.4
–
–
–
Sonata ( ? )
PTv3-B (108.4 M)
21.3
10.9
53.9
30.7
26.5
14.8
DOS ( ? )
PTv3-B (108.4 M)
20.6 ‡
10.9 ‡
54.6
32.5
–
–
PointINS ( ? )
PTv3-B (108.4 M)
24.9 ‡
13.4 ‡
57.6
34.6
–
–
Table 7: Linear Probing 3D Instance Segmentation. PointGroup ( ? ) is trained with a linear semantic head and a linear offset head on top of the frozen 3D encoder. † Reproduced. DOS and PointINS results are taken from ? . ‡ Single-dataset pretraining with PTv3-S encoder.
Method
Backbone
ScanNet200
ScanNet
ScanNet++
mAP 50
mAP
mAP 50
mAP
mAP 50
mAP
Sonata ( ? )
PTv3-B (124.8 M)
35.6
25.4
63.9
42.4
36.5
24.6
DOS ‡ ( ? )
PTv3-S (46.2 M)
33.5
22.5
62.0
40.5
–
–
PointINS ‡ ( ? )
PTv3-S (46.2 M)
34.8
25.1
63.7
41.5
–
–
Concerto ( ? )
PTv3-B (124.8 M)
38.7
27.4
64.9
42.9
38.3
26.0
Utonia † ( ? )
PTv3-B (157.7 M)
36.1
25.4
66.4
42.9
37.1
25.5
Table 8: Fine-tuning 3D Instance Segmentation. The pretrained encoder and PointGroup ( ? ) heads are jointly fine-tuned. † Reproduced. ‡ Taken from ? .
Method
3DMatch
3DLoMatch
FPFH ( ? )
77.5
36.4
Concerto
65.0
22.7
Utonia
76.5
36.3
VINO (Ours)
84.1
44.6
Table 9: Point-Cloud Registration. Registration success rate (%) on 3DMatch and 3DLoMatch using frozen encoder features as descriptors with a shared registration backend ( ? ). FPFH is a hand-crafted descriptor.
Method
Arch.
NYUv2
ScanNet++
DINOv2
ViT-B/14
56.0
34.5
DINOv2
ViT-g/14
59.0
39.3
DINOv3
ViT-B/16
62.0
34.2
Concerto
PTv3-B
54.9
45.6
Utonia
PTv3-B
55.9
44.7
VINO (Ours)
Volt-B
59.2
48.2
Table 10: Cross-Modal Semantic Probing (mIoU). Linear probing with frozen encoders. On NYUv2, single-view RGB-D images are lifted to 3D for the 3D methods. On ScanNet++, 2D features are lifted from posed views and aggregated across views for the 2D methods.
Predictor Design
ScanNet200
Cross-attention only decoder
35.9
Query-to-query attention
34.2
Full self-attention decoder
33.0
Registers as context
35.3
Table 11: Masked Predictor. Semantic segmentation linear-probing performance (mIoU) for different predictor designs.
Block Size
ScanNet200
Mask Ratio
ScanNet200
60 cm
35.7
45%
35.2
80 cm
35.9
55%
35.9
100 cm
34.5
65%
34.8
Table 12: Masking. Linear probing performance across block sizes and mask ratios.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Initialization
ScanNet200
ScanNet
ScanNet++
mAP 50
mAP
mAP 50
mAP
mAP 50
mAP
Supervised
48.4
38.6
78.3
59.2
53.1
34.5
VINO-3D
50.3
39.2
82.9
64.4
54.4
35.8
VINO
55.7
43.3
83.4
64.7
55.8
37.0
Appendix
Table 13: Instance Segmentation with SPFormer. Fine-tuning with different encoder initializations.
ScanNet200
ScanNet++
Sonata † (PTv3)
28.3
38.0
Sonata (Volt)
29.6
39.1
Simpler scheduling
29.5
39.1
No masked loss
28.9
38.4
Asymmetric augm.
31.3
39.3
Decoupled head
32.8
40.9
Appendix
Table 14: Additional Ablations. Semantic segmentation linear-probing performance (mIoU) on ScanNet200 and ScanNet++. † is reproduced. ScanNet200 results correspond to Tabs. 1 , 2 , 3 , and 4 .
Figure 6: Qualitative Transfer to NYUv2. PCA of frozen features for a single NYUv2 image. For the 3D models, the image is lifted to 3D.
Figure 7: Token Norm Histograms. Distribution of token ℓ2 norms, with frequency shown on a logarithmic scale. Registers remove many high-norm outliers; voxel averaging removes most of the remaining tail.
Figure 8: Qualitative results on Replica. VINO features of a large indoor scene from Replica ( ? ). We visualize PCA on the bottom left and cosine similarity maps between tokens marked with a red cross and the other 3D tokens.
Figure 9: Qualitative results on SemanticKITTI. VINO features of an outdoor scene from SemanticKITTI ( ?? ). We visualize PCA (top) and cosine similarity maps between the token marked with a red cross and all other 3D tokens (middle and bottom).
Transformers have become a common foundation across deep learning, yet 3D scene understanding still relies on specialized backbones with strong domain priors. This keeps the field isolated from the broader Transformer ecosystem, limiting the transfer of new advances as well as the benefits of increasingly optimized software and hardware stacks. To bridge this gap, we adapt the vanilla Transformer encoder to 3D scenes with minimal modifications. Given an input 3D scene, we partition it into volumetric patch tokens, process them with full global self-attention, and inject positional information via a 3D extension of rotary positional embeddings. We call the resulting model the Volume Transformer (Volt) and apply it to 3D semantic segmentation. Naively training Volt on standard 3D benchmarks leads to shortcut learning, highlighting the limited scale of current 3D supervision. To overcome this, we introduce a data-efficient training recipe based on strong 3D augmentations, regularization, and distillation from a convolutional teacher, making Volt competitive with state-of-the-art methods. We then scale supervision through joint training on multiple datasets and show that Volt benefits more from increased scale than domain-specific 3D backbones, achieving state-of-the-art results across indoor and outdoor datasets. Finally, when used as a drop-in backbone in a standard 3D instance segmentation pipeline, Volt again sets a new state of the art, highlighting its potential as a simple, scalable, general-purpose backbone for 3D scene understanding.
Kadir Yilmaz, Adrian Kruse, Tristan Höfer +2
RWTH Aachen University · Eindhoven University of Technology
3D Foundation Models (3DFMs) such as VGGT have recently pushed the boundaries of 3D vision by predicting rich unified representations with feed-foward transformers. The scene representations learned by these models enable strong performance on multiple 3D vision tasks. In this paper, we investigate using their internal representations to infer 3D in the scene from new views. Our hypothesis is that in order to solve the task of 3D reconstruction, these models need to learn a representation that includes a large amount of general knowledge about 3D scenes. After showing that it is possible to decode hidden surfaces from internal 3DFM representations, we propose a method, Z3D, that estimates pointmaps in unseen views by doing latent diffusion on 3DFM representation. We show that Z3D can predict realistic depth maps for new views across multiple datasets.
Denis M. Akola, David F. Fouhey
New York University, Tandon School of Engineering, 1 MetroTech Center, Brooklyn, NY 11201, USA · Courant Institute of Mathematical Sciences, 251 Mercer St, New York, NY 10012
Vision Language Models (VLMs) enable a unified model to solve various vision tasks through prompting. They have shown promising performance in semantic understanding. However, 3D understanding still largely relies on expert vision models with complex task-specific designs. The key argument this work wants to make is that VLMs are native 3D learners. Our in-depth large scale study shows that 1) focal length unification, 2) text-based pixel reference and 3) data mixture and scaling, are all you need for effective 3D learning. Model architecture changes, large models, heavy data augmentations, and complex losses including the regression formulation, many of which form the foundation of expert vision models, are actually not necessary conditions. As a result, we propose VLM3, a scalable method with the simplest design that enables standard VLMs to master diverse 3D tasks. VLM3 not only advances the VLM depth estimation accuracy by a large margin (0.84 -> 0.9), but also enables diverse 3D tasks such as pixel correspondence, camera pose estimation and object-level 3D understanding, matching expert vision model accuracy while maintaining standard architectures and text-based training. We believe VLM3 opens up a new paradigm for simple and scalable 3D learning.