point2vec for Self-Supervised Representation Learning on Point Clouds

Karim Knaebel; Jonas Schult; Alexander Hermans; Bastian Leibe

cs.CVMar 29, 2023

point2vec for Self-Supervised Representation Learning on Point Clouds

Authors: Karim Knaebel, Jonas Schult, Alexander Hermans, Bastian Leibe

Organizations: RWTH Aachen University, Germany

Abstract

We present point2vec, a self-supervised method for learning point cloud representations through masked prediction of a teacher network's latent features. Building on data2vec, we investigate a limitation of applying this approach to point clouds: positional information associated with masked patches reveals the overall object shape to the student, weakening the learning task even under heavy masking. point2vec addresses this issue by passing only visible patches to the student encoder and introducing a shallow decoder to predict representations for masked patches. The teacher processes the complete point cloud to provide contextualized training targets. These targets capture global context through the teacher's self-attention, providing supervision beyond the local geometry of individual patches. This design conceals masked-patch positions from the student and reduces its computational cost during pre-training. Our ablations show that the decoder alone provides only part of the improvement; concealing positional information is key. After pre-training on ShapeNet, point2vec outperforms the evaluated self-supervised baselines on shape classification on ModelNet40 and ScanObjectNN and few-shot classification on ModelNet40, while achieving competitive part segmentation results on ShapeNetPart.

Figures & tables

Figure 1: Overview of point2vec. During training, a teacher network predicts latent representations using a complete view of the point cloud. The student network encodes a partial view of the same point cloud. A shallow decoder predicts the teacher’s representations for masked regions , providing the training signal for the student and decoder. The teacher weights are updated using an exponential moving average of the student weights.

Figure 2: point2vec pre-training. We divide the point cloud into patches using farthest point sampling (FPS) and $k$ -NN aggregation. We obtain patch embeddings by applying a mini-PointNet to each point patch (right). The teacher Transformer encoder infers contextualized representations for all patch embeddings, which, after normalization and averaging over the last $K$ Transformer layers, serve as training targets. The student’s input is a masked view of the input data, i.e. we randomly mask out a fraction of patch embeddings and only pass the remaining embeddings into the student Transformer encoder . After applying a shallow decoder to the outputs of the student, padded with learned mask embeddings M , we train the student and decoder to predict the teacher’s latent patch representations.

Method	Overall accuracy		mAcc
Method	Voting	No voting	mAcc
Transf.-OcCo [ 49 ]	$92.1$	—	—
ParAE [ 19 ]	—	$92.9$	—
STRL [ 26 ]	$93.1$	—	—
Point-BERT [ 49 ]	$93.2$	$92.7$	—
PointGLR [ 34 ]	—	$93.0$	—

Table 1: Shape classification on ModelNet40 [ 44 ] . We report the overall accuracy with and without voting, as well as the mean per-class accuracy (mAcc).

Method	mIoU C	mIoU I	Aero	Bag	Cap	Car	Chair	Earph	Guitar	Knife	Lamp	Laptop	Motor	Mug	Pistol	Rocket	Skateb	Table
Transf.-OcCo [ 49 ]	$83.4$	$85.1$	$83.3$	$85.2$	$88.3$	$79.9$	$90.7$	$74.1$	$91.9$	$87.6$	$84.7$	$95.4$	$75.5$	$94.4$	$84.1$	$63.1$	$75.7$	$80.8$
Point-BERT [ 49 ]	$84.1$	$85.6$	$84.3$	$84.8$	$88.0$	$79.8$	$91.0$	$81.7$	$91.6$	$87.9$	$85.2$	$95.6$	$75.6$	$94.7$	$84.3$	$63.4$	$76.3$	$81.5$
MaskPoint [ 28 ]	$84.4$	$86.0$	$84.2$	$85.6$	$88.1$	$80.3$	$91.2$	$79.5$	$91.9$	$87.8$	$86.2$	$95.3$	$76.9$	$95.0$	$85.3$	$64.4$	$76.9$	$81.8$
Point-MAE [ 31 ]	$84.1$	$86.1$	$84.3$	$85.0$	$88.3$	$80.5$	$91.3$	$78.5$	$92.1$	$87.4$	$86.1$	$96.1$	$75.2$	$94.6$	$84.7$	$63.5$	$77.1$	$82.4$
Point-M2AE [ 51 ]	84.9	86.5	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—
from scratch	$84.1$	$85.7$	$84.2$	$84.8$	$87.9$	$80.2$	$91.2$	$79.3$	$91.6$	$87.8$	$85.4$	$96.1$	$75.8$	$94.7$	$85.0$	$64.7$	$75.4$	$81.1$

Table 2: Part segmentation results on ShapeNetPart [ 48 ] . We report category-averaged mIoU C and instance-averaged mIoU I , as well as the mean IoU for each object category.

Panel of Figure 3 — Figure 3: Learning curves for ModelNet40 [ 44 ] . We show the mean (solid line) and the standard deviation (shaded background) over $6$ independent runs of point2vec, data2vec-PC as well as the model trained from scratch on ModelNet40. point2vec consistently outperforms the baselines by a large margin.

Method	Overall accuracy
	OBJ-BG	OBJ-ONLY	PB-T50-RS
Transf.-OcCo [ 49 ]	$84.9$	$85.5$	$78.8$
Point-BERT [ 49 ]	$87.4$	$88.1$	$83.1$
MaskPoint [ 28 ]	$89.3$	$89.7$	$84.6$
Point-MAE [ 31 ]	$90.0$	$88.3$	$85.2$
Point-M2AE [ 51 ]	91.2	$88.8$	$86.4$

Table 3: Shape classification on ScanObjectNN [ 40 ] . We report the overall accuracy over the three subsets OBJ-BG, OBJ-ONLY, and the most challenging variant PB-T50-RS.

Method	5-way		10-way
	10-shot	20-shot	10-shot	20-shot
OcCo [ 43 ]	$91.9(36)$	$93.9(31)$	$86.4(54)$	$91.3(46)$
Transf.-OcCo [ 49 ]	$94.0(36)$	$95.9(23)$	$89.4(51)$	$92.4(46)$
Point-BERT [ 49 ]	$94.6(31)$	$96.3(27)$	$91.0(54)$	$92.7(51)$
MaskPoint [ 28 ]	$95.0(37)$	$97.2(17)$	$91.4(40)$	$93.4(35)$
Point-MAE [ 31 ]	$96.3(25)$	$97.8(18)$	$92.6(41)$	$95.0(30)$

Table 4: Few-shot classification on ModelNet40 [ 44 ] . We report the mean and standard deviation over

10

runs.

Panel of Table 5 — Table 5: Ablation. We find that a deferred shallow decoder (D) ( Fig. 2 , ) predicting the teacher’s representations for masked patches shows consistent improvements but we identify that concealing positional information (No M ) from the student is key.

Panel of Figure 4 — Figure 4: Leakage of positional information. The center points of masked point patches are associated with the masked embeddings ( Fig. 2 , M ). (a) data2vec-PC discloses the positions of masked patches to the student, revealing the chair’s overall shape. (b) point2vec excludes masked embeddings from the student and therefore conceals the positions of the masked patches.

Figure 5: Visualization of learned representations. We use PCA to project the learned representations into RGB space. Both a random initialization and data2vec-PC pre-training show a fairly strong positional bias, whereas point2vec exhibits stronger semantic grouping without being trained on downstream dense prediction tasks.

Panel of Table 6 — Table 6: Masking strategy. We explore two variants for masking the input of the student. For (a) random masking, we uniformly mask out a given ratio of all embeddings. For (b) block masking, we mask out a random embedding and its nearest neighbors. We report the overall accuracy on ModelNet40 and ScanObjectNN.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

	Overall accuracy
Pretext task	ModelNet40		ScanObjNN
	Voting	No voting	PB-T50-RS
none / from scratch	$93.3$	$93.0$	$84.3$
classification (ShapeNet)	$93.2$	$93.0$	$82.9$
point2vec (ModelNet40)	$93.9$	$93.6$	$84.4$
point2vec (ShapeNet)	94.8	94.7	87.5

Appendix

Table 7: Pretext tasks. Pre-training with a classification objective on ShapeNet offers no gains over training from scratch on ModelNet40 and yields worse performance on the PB-T50-RS variant of ScanObjectNN. However, point2vec already brings performance gains when pre-trained with the much smaller ModelNet40 dataset and significant improvements when pre-trained with the large ShapeNet dataset.

	Overall accuracy
$\tau_{n}$	ModelNet40		ScanObjNN
	Voting	No voting	PB-T50-RS
$80$	$94.5$	$94.1$	$86.7$
$160$	$94.6$	$94.2$	$87.4$
$200$	94.8	94.7	87.5
$300$	$94.1$	$94.0$	$87.3$

Appendix

Table 8: EMA rate warm-up. We linearly warm up the EMA rate during the first

\tau_{n}

epochs.

Figure 6: Qualitative results on ShapeNetPart [ 48 ] . point2vec produces well-localized boundaries between parts with minimal semantic errors. In most cases, the differences between the results of point2vec and the ground truth are imperceptible to the human eye. However, the last example shows a failure case where the jet engine is not properly segmented.

$K$	ModelNet40		ScanObjNN
	Overall accuracy
	Voting	No voting	PB-T50-RS
$1$	$94.4$	$94.1$	$87.0$
$3$	$94.7$	$94.3$	$87.1$
$6$	94.8	94.7	$87.5$
$9$	$94.3$	$94.0$	87.6

Appendix

Table 9: Target layer aggregation. We construct training targets by averaging the outputs of the last

K

Transformer blocks of the teacher. We observe that

K=6

is optimal for ModelNet40 and close to optimal for the PB-T50-RS variant of ScanObjectNN.

Parameter	Value
Epochs	$300$
Batch size	$16$
Optimizer	AdamW
Learning rate	$2\times 10^{-4}$
Weight decay	$0.05$
LR schedule	cosine

Appendix

Table 10: Hyperparameters for part segmentation. We use the same hyperparameters when fine-tuning point2vec and data2vec-PC on ShapeNetPart [ 48 ] .

Panel of Figure 7 — Figure 7: Pre-training data efficiency. Irrespective of the quantity of pre-training data used from ShapeNet, point2vec consistently achieves better results than Point-MAE [ 31 ] on ModelNet40 (with voting) and the PB-T50-RS variant of ScanObjectNN.

Table 18

Self-Supervised Point Cloud Learning

See all 22 papers

Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework
May 17, 2026Yumeng Yao, Jingzhi Dong, Haowen Gu +4Point Cloud Learning 3D ViTs
Nanjing University of Science and Technology · Hangzhou City University · East China Normal University +1
Joint-Embedding Prediction of Masked Point Tubes for Self-Supervised Learning on 4D Point Cloud Videos
Aug 25, 2026Jheng-Ling Lee, Shang-Tse ChenSelf-Supervised Point Cloud Learning Self-Supervised Pre-Training
Department of Computer Science and Information Engineering, National Taiwan University
SIEVE: Overcoming Topological Obstruction in Equivariant Self-Supervised Learning
Oct 15, 2026Jongann Lee, Sun Woo Park, Yun Young ChoiSelf-Supervised Point Cloud Learning Point Cloud Learning
Max Planck Institute of Mathematics Bonn, Germany · SolverX Seoul, Republic of Korea

May 17, 2026J/K move · Enter open · S save

Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework

Yumeng Yao, Jingzhi Dong, Haowen Gu, Tao Chen+3

Nanjing University of Science and Technology · Hangzhou City University · East China Normal University+1

Point Cloud Learning 3D ViTs Self-Supervised Pre-Training Sequence Modeling Next-Token Prediction Predictive Representation Learning+1

Open paper AskPDF arXiv

With the rapid progress of multimodal foundation models and predictive pre-training, an important open question is how to equip 3D point clouds with a pre-training paradigm that is better aligned with next-token and next-embedding learning. Existing point-cloud self-supervised methods are largely built on masked reconstruction or explicit geometric generation, and thus remain tied to input recovery rather than predictive dependency modeling. In this paper, we introduce PointNTP, which reformulates point cloud pre-training as a fully causal, decoder-free latent Next-Token Prediction problem. Specifically, each point cloud is first partitioned into local patches and serialized into a structured 3D token sequence according to patch-center geometry. The resulting sequence is then modeled by a causal Transformer under prefix-only conditioning, and trained with a shift-based prediction objective stabilized by stop-gradient targets. This design enables the model to learn structural dependencies directly in latent space, without reconstruction decoders or explicit geometric recovery. Extensive experiments demonstrate that the proposed PointNTP is highly competitive across multiple downstream tasks: it achieves 93.8%(+0.5%), 92.6%(+0.3%), and 89.3%(+1.1%) on OBJ_BG, OBJ_ONLY, and PB_T50_RS of ScanObjectNN, respectively; obtains 85.0%(+0.1%) in Cls.mIoU on ShapeNetPart; and reaches 71.1% mAcc on S3DIS Area 5. Overall, decoder-free causal latent prediction provides a simple, scalable, and potentially modality-agnostic paradigm for point-cloud self-supervised learning, offering a new 3D perspective on foundation-style predictive learning for 3D data.

Abstract

Self-Supervised Point Cloud Learning

Explore similar work

Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework

Joint-Embedding Prediction of Masked Point Tubes for Self-Supervised Learning on 4D Point Cloud Videos

SIEVE: Overcoming Topological Obstruction in Equivariant Self-Supervised Learning