cs.CVMar 11, 2026

Pointy - A Lightweight Transformer for Point Cloud Foundation Models

Authors: Konrad Szafer, Marek Kraft, Dominik Belter

Organizations: Institute of Robotics and Machine Intelligence, Poznan University of Technology

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Foundation models for point cloud data have recently grown in capability, often leveraging extensive representation learning from language or vision. In this work, we take a more controlled approach by introducing a lightweight transformer-based point cloud architecture. In contrast to the heavy reliance on cross-modal supervision, our model is trained only on 39k point clouds - yet it outperforms several larger foundation models trained on over 200k training samples. Interestingly, our method approaches state-of-the-art results from models that have seen over a million point clouds, images, and text samples, demonstrating the value of a carefully curated training setup and architecture. To ensure rigorous evaluation, we conduct a comprehensive replication study that standardizes the training regime and benchmarks across multiple point cloud architectures. This unified experimental framework isolates the impact of architectural choices, allowing for transparent comparisons and highlighting the benefits of our design and other tokenizer-free architectures. Our results show that simple backbones can deliver competitive results to more complex or data-rich strategies. The implementation, including code, pre-trained models, and training protocols, is available at https://github.com/KonradSzafer/Pointy.

Figures & tables

Model#PointsParam. (M) ↓\downarrowFLOPs (G) ↓\downarrowModelNet40ScanObjectNN
OA (%) ↑\uparrowOA (%) ↑\uparrow
PointNet [ 19 ]2k1.78.191.680.3
PointNet++ [ 20 ]2k1.78.190.781.7
DGCNN [ 25 ]2k1.417.890.871.5
PointMLP [ 15 ]2k13.162.991.678.8
Transformer-based
Table 1: Comparison of classification performance across ModelNet40 and ScanObjectNN benchmarks . We report overall accuracy (OA), model parameters, and FLOPs (in G), with all models trained under identical conditions: batch size 16 for 100 epochs, points normalized to the [-1,1] range, and random z-axis rotation. Results demonstrate the competitive performance of our lightweight architecture (3.0M parameters, 16.2G FLOPs) across diverse point cloud datasets. Models marked with [T] denote transformer-based architectures, while those with [ST] incorporate an unchanged transformer block.
Figure 1: Architecture Pointy – transformer backbone for point cloud processing. The model takes raw point cloud data as input, applies patch partitioning using Farthest Point Sampling (FPS), and k-Nearest Neighbors (kNN). Raw point features are preserved through residual connections alongside learned patch embeddings in the embedding layer based on PointNet. The architecture consists of token merging operations between adjacent tokens after each transformer block. This hierarchical design enables local and global feature learning through progressive patch merging. The final output produces P ×\times D dimensional representations, where P is the number of patches and D is the embedding dimension.
Model#PointsParam. (M) ↓\downarrowFLOPs (G) ↓\downarrowObjaverse-LVIS
OA (%) ↑\uparrow
PCT [ 8 ]2k2.94.336.3
PointTransformer [ 31 ]2k9.573.634.1
PointTransformer [ 5 ]2k22.015.28.5
PointMAE [ 18 ]2k21.84.734.9
Our - Small2k3.016.236.4
Table 2: Zero-shot Evaluation of Transformer-based Models Pre-trained on the Objaverse-LVIS Subset. All models are pre-trained on an 85/15 split of LVIS data (85% training, 15% testing) drawn from a curated 39k-sample subset spanning 1,156 LVIS-annotated object categories. Training is performed for 30 epochs under identical conditions—batch size 16, 2k points per sample normalized to the [-1,1] range, and random rotations about the z-axis. The checkpoints obtained after this training are subsequently used for zero-shot evaluation, ensuring a consistent and fair comparison of transferability.
Figure 2: Training dynamics of different transformer-based models on ModelNet40, ScanObjectNN, and Objaverse-LVIS datasets. The plots show overall classification accuracy (%) versus training epochs for our proposed small model compared to existing approaches: PCT [ 8 ] , PointMAE [ 18 ] , and two variants of PointTransformer [ 31 ] and [ 5 ] . Our method demonstrates faster convergence across all datasets while achieving competitive or superior final accuracy. The inset plots for (a) and (b) show detailed performance in later epochs. Notable is the consistent underperformance of PointTransformer [ 5 ] relative to other methods across all datasets.
Figure 3: Classification accuracy on ModelNet40 as a function of input point cloud size. Models were trained for 30 epochs under identical conditions, with results showing the peak accuracy achieved. While PCT demonstrates superior performance in the 256-1024 point range, our architecture achieves competitive results and attains 89.3% for 2048 points.
More in this topic

Point Cloud Learning

Point Cloud Learning develops methods for representing and interpreting unordered 3D point sets from LiDAR, depth sensors, radar, and 3D models.

See all 156 papers

Explore similar work

CardsList

  1. PointTransformerX: Portable and Efficient 3D Point Cloud Processing without Sparse Algorithms

    Apr 27, 2026Laurenz Reichardt, Nikolas Ebert, Oliver WasenmüllerRotary Positional Embeddings3D ViTs

    Mannheim University of Applied Sciences, Germany

  2. Mantis: Mamba-native Tuning is Efficient for 3D Point Cloud Foundation Models

    May 5, 2026Zihao Guo, Yiding Sun, Jihua Zhu +33D Foundation ModelsMamba

    Xi’an Jiaotong University, Xi’an, China · Peking University, Beijing, China · School of Artificial Intelligence and Robotics, Hunan University, China +1

  3. Rethinking Point Clouds as Sequences: A Causal Next-Token Predictive Learning Framework

    May 17, 2026Yumeng Yao, Jingzhi Dong, Haowen Gu +4Point Cloud Learning3D ViTs

    Nanjing University of Science and Technology · Hangzhou City University · East China Normal University +1