LitePT: Lighter Yet Stronger Point Transformer

Yuanwen Yue; Damien Robert; Jianyuan Wang; Sunghwan Hong; Jan Dirk Wegner; Christian Rupprecht; Konrad Schindler

cs.CVDec 15, 2025

LitePT: Lighter Yet Stronger Point Transformer

Authors: Yuanwen Yue, Damien Robert, Jianyuan Wang, Sunghwan Hong, Jan Dirk Wegner, Christian Rupprecht, Konrad Schindler

Organizations: ETH Zurich · University of Oxford · University of Zurich

Abstract

Modern neural architectures for 3D point cloud processing contain both convolutional layers and attention blocks, but the best way to assemble them remains unclear. We analyse the role of different computational blocks in 3D point cloud networks and find an intuitive behaviour: convolution is adequate to extract low-level geometry at high-resolution in early layers, where attention is expensive without bringing any benefits; attention captures high-level semantics and context in low-resolution, deep layers more efficiently, where convolution inflates the parameter count. Guided by this design principle, we propose a new, improved 3D point cloud backbone that employs convolutions in early stages and switches to attention for deeper layers. To avoid the loss of spatial layout information when discarding redundant convolution layers, we introduce a novel, parameter-free 3D positional encoding, PointROPE. The resulting LitePT model has $3.6\times$ fewer parameters, runs $2\times$ faster, and uses $2\times$ less memory than the state-of-the-art Point Transformer V3, but nonetheless matches or outperforms it on a range of tasks and datasets. Code and models are available at: https://github.com/prs-eth/LitePT.

Figures & tables

Figure 2: PTv3 block. The block is composed of a convolutional conditional positional encoding module followed by an attention module.

Panel of Figure 3 — Figure 3: Parameter count and latency. E0-E4 denote encoder stages from shallow to deep, and D3-D0 denote decoder stages from deep to shallow. The length of each bar reflects the relative parameter count or latency of the corresponding module. Top: In PTv3, the positional encoding implemented via a convolution block accounts for the majority of its parameters, particularly in the later stages. In contrast, our PointROPE is parameter-free. Bottom: The PTv3 latency map reveals the significant cost of early-stage attention. LitePT restricts attention to late stages, where it is most effective and less costly.

Figure 4: Representations learnt by the hierarchical U-Net encoder. The hierarchical U-Net encoder exhibits an operator-agnostic feature hierarchy: shallow stages consistently encode local geometric structure, while semantics emerge in deeper stages.

			ScanNet [ 14 ]	nuScenes [ 6 ]
	Model	#Params	mIoU	mIoU
	PTv3 [ 88 ]	46.1M	77.5	80.4
\scriptsize{1}⃝	PTv3 w/o Transformer	32.4M	73.4	76.1
\scriptsize{2}⃝	PTv3 w/o SPConv	15.4M	70.7	74.9

Table 1: Revisiting PTv3. We evaluate two PTv3 variants: in \scriptsize{1}⃝ , the attention and MLP modules are removed, and in \scriptsize{2}⃝ , only the sparse convolution layers are removed.

Figure 5: LitePT-S architecture. Our model comprises five stages, employing convolution blocks in the early stages and PointROPE augmented attention blocks in the later ones. LitePT-S uses a lightweight decoder. Alternatively, adding convolution or attention blocks symmetrically in the decoder produces LitePT-S*.

		Training		Inference
Method	#Params	Latency	Memory	Latency	Memory
MinkUNet [ 12 ]	39.2M	60ms	1.9G	21ms	2.4G
PTv2 [ 87 ]	12.8M	188ms	22.8G	151ms	22.9G
PTv3 [ 88 ]	46.1M	110ms	5.8G	51ms	4.1G
LitePT-S (Ours)	12.7M	72ms	2.3G	21ms	2.0G
LitePT-S* (Ours)	16.0M	81ms	3.3G	26ms	2.0G

Table 2: Efficiency comparison. Results are reported as average over the full ScanNet dataset using a single RTX 4090 GPU. Automatic Mixed Precision (AMP) is enabled for all models during training and disabled during inference. * denotes our variant with a heavier decoder that includes attention or convolutional blocks.

Figure 6: Performance-efficiency trade-off. Left: Progressively dropping attention in more of the early stages. Right: Progressively dropping convolution in more of the late stages.

$L_{c}$	Setting	#Params	Latency	mIoU
0	A-A-A-A-A	11.8M	35.1ms	82.1
1	C-A-A-A-A	11.9M	30.4ms	81.7
2	C-C-A-A-A	12.0M	25.8ms	82.0
3	C-C-C-A-A	12.7M	21.5ms	82.2
4	C-C-C-C-A	18.8M	16.2ms	80.9
5	C-C-C-C-C	26.9M	13.5ms	75.4

Table 3: Effect of

L_{c}

and “hand-over” stage. C: convolutional block; A: attention block; X: both convolution and attention are used at that stage. We compare model variants and report parameter count, latency, and validation mIoU on the nuScenes dataset. The grey-shaded row is our recommended setting.

	Semantic Segmentation (mIoU)				Instance Segmentation ( $\text{mAP}_{50}$ )
Decoder	ScanNet [ 14 ]	Structured3D [ 104 ]	nuScenes [ 6 ]	Waymo [ 72 ]	ScanNet [ 14 ]
LitePT-S	76.5	83.6	82.2	73.1	62.2
LitePT-S*	76.8	83.0	81.8	72.7	64.9

Table 4: Decoder design. We compare two decoder variants: in LitePT-S*, we apply our stage-tailored design symmetrically to the decoder stages, while in LitePT-S, we retain only linear projection layers in all decoder stages.

Method	#Params	Latency	Memory	mIoU
PTv3 [ 88 ]	46.1M	57ms	5.83G	82.4
LitePT-S (Ours)	12.7M	23ms	2.56G	83.6
LitePT-B (Ours)	45.1M	36ms	2.60G	85.1
LitePT-L (Ours)	85.9M	44ms	3.58G	85.4

Table 5: Model scaling on Structured3D dataset. Our model scales efficiently, achieving consistent performance gains from small to large variants with modest increases in latency and memory. Even when scaled to twice the parameters of PTv3, LitePT-L remains more efficient.

	$w/o$ PointROPE	$b=10$	$b=100$	$b=1000$	$b=10000$
mIoU	79.6	81.7	82.2	81.8	81.3

Table 6: PointROPE. Dedicated positional encoding is needed—dropping PointROPE leads to a significant performance drop. PointROPE works similarly well with a wide range of base frequencies, the grey-shaded column is our recommended setting.

		nuScenes [ 6 ]		Waymo [ 72 ]
Method	#Param	mIoU	mAcc	mIoU	mAcc
MinkUNet [ 12 ]	39.2M	73.3	-	65.9	76.6
SPVNAS [ 74 ]	-	77.4	-	-	-
Cylinder3D [ 105 ]	-	76.1	-	-	-
AF2S3Net [ 11 ]	-	62.2	-	-	-
SphereFormer [ 41 ]	-	78.4	-	69.9	-

Table 7: Outdoor semantic segmentation on nuScenes and Waymo validation set. Scores of prior work courtesy of [ 88 , 89 ] .

			Limited Scenes (Pct.)				Limited Annotations (Pts.)
Method	#Params	Full	1 $\%$	5 $\%$	10 $\%$	20 $\%$	20	50	100	200
MinkUNet [ 12 ]	39.2M	72.2	26.0	47.8	56.7	62.9	41.9	53.9	62.2	65.5
PTv2 [ 87 ]	12.8M	75.4	24.8	48.1	59.8	66.3	58.4	66.1	70.3	71.2
PTv3 [ 88 ]	46.1M	77.5	25.8	48.9	61.0	67.0	60.1	67.9	71.4	72.7
LitePT-S (Ours)	12.7M	76.5	27.3	50.6	63.1	67.3	62.5	68.4	70.9	72.8
LitePT-S* (Ours)	16.0M	76.8	27.2	51.6	63.0	67.1	63.2	69.5	72.0	74.2

Table 8: Indoor semantic segmentation on ScanNet validation set. In mean IoU. Scores of prior work courtesy of [ 88 ] .

		Val		Test
Method	#Params	mIoU	mAcc	mIoU	mAcc
MinkUNet [ 12 ]	39.2M	76.4	84.3	77.4	85.5
PTv2 [ 87 ]	12.8M	79.0	86.8	78.5	86.6
PTv3 [ 88 ]	46.1M	82.4	90.3	82.1	90.3
LitePT-S (Ours)	12.7M	83.6	90.7	82.4	90.3

Table 9: Indoor semantic segmentation on Structured3D.

		ScanNet [ 14 ]			ScanNet200 [ 62 ]
PointGroup [ 37 ]	#Params	mAP 25	mAP 50	mAP	mAP 25	mAP 50	mAP
MinkUNet [ 12 ]	39.2M	72.8	56.9	36.0	32.2	24.5	15.8
PTv2 [ 87 ]	12.8M	76.3	60.0	38.3	39.6	31.9	21.4
PTv3 [ 88 ]	46.2M	77.5	61.7	40.9	40.1	33.2	23.1
LitePT-S* (Ours)	16.0M	78.5	64.9	41.7	40.3	33.1	22.2

Table 10: Indoor instance segmentation on ScanNet and ScanNet200 validation set. Scores of prior work courtesy of [ 88 ] .

	Vehicle L2		Pedestrian L2		Cyclist L2		Mean L2
Method	mAP	APH	mAP	APH	mAP	APH	mAPH
PointPillars [ 43 ]	63.6	63.1	62.8	50.3	61.9	59.9	57.8
CenterPoint [ 99 ]	66.7	66.2	68.3	62.6	68.7	67.6	65.5
SST [ 21 ]	64.8	64.4	71.7	63.0	68.0	66.9	64.8
SST-Center [ 21 ]	66.6	66.2	72.4	65.0	68.9	67.6	66.3
VoxSet [ 28 ]	66.0	65.6	72.5	65.4	69.0	67.7	66.2

Table 11: Outdoor object detection on Waymo with single frames input. Scores of prior work courtesy of [ 88 ] .

Panel of Figure 10 — Figure 10: nuScenes semantic segmentation. We present various scenes of the nuScenes validation set: the input point cloud coloured by LiDAR intensity, the semantic segmentation from LitePT-S, and the corresponding ground truth.

Panel of Figure 11 — Figure 11: Waymo semantic segmentation. We present various scenes of the Waymo validation set: the input point cloud coloured by LiDAR intensity, the semantic segmentation from LitePT-S, and the corresponding ground truth.

Panel of Figure 12 — Figure 12: ScanNet semantic segmentation. We present various scenes of the ScanNet validation set: the input point cloud, the semantic segmentation from LitePT-S, and the corresponding ground truth.

Panel of Figure 13 — Figure 13: Structured3D semantic segmentation. We present various scenes of the Structured3D validation set: the input point cloud, the semantic segmentation from LitePT-S, and the corresponding ground truth.

Panel of Figure 14 — Figure 14: ScanNet instance segmentation. We present various scenes of the ScanNet validation set: the input point cloud, the instance segmentation from LitePT-S*, and the corresponding ground truth. Colours for each instance are randomly assigned.

Panel of Figure 15 — Figure 15: Waymo object detection. We present various scenes of the Waymo validation set: the input point cloud, the object detections from LitePT, and the corresponding ground truth.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Detailed architectures. We illustrate the full pipelines of LitePT-S, LitePT-S*, Point Transformer V3 [ 88 ] , and the building blocks of each architecture.

Figure 8: PointROPE attention. We apply PointROPE to query and key before standard scaled dot-product attention.

	LitePT-S	LitePT-S*	LitePT-B	LitePT-L
stem	$\text{C}\!=\!36,\text{K}\!=\!5{\times}5{\times}5$	$\text{C}\!=\!36,\text{K}\!=\!5{\times}5{\times}5$	$\text{C}\!=\!36,\text{K}\!=\!5{\times}5{\times}5$	$\text{C}\!=\!36,\text{K}\!=\!5{\times}5{\times}5$
E0	$\left[\begin{array}[]{c}\text{C}\!=\!36\\ \text{K}\!=\!3{\times}3{\times}3\end{array}\right]\!\times\!2$	$\left[\begin{array}[]{c}\text{C}\!=\!36\\ \text{K}\!=\!3{\times}3{\times}3\end{array}\right]\!\times\!2$	$\left[\begin{array}[]{c}\text{C}\!=\!54\\ \text{K}\!=\!3{\times}3{\times}3\end{array}\right]\!\times\!3$	$\left[\begin{array}[]{c}\text{C}\!=\!72\\ \text{K}\!=\!3{\times}3{\times}3\end{array}\right]\!\times\!3$
E1	pool stride 2	pool stride 2	pool stride 2	pool stride 2
E1	$\left[\begin{array}[]{c}\text{C}\!=\!72\\ \text{K}\!=\!3{\times}3{\times}3\end{array}\right]\!\times\!2$	$\left[\begin{array}[]{c}\text{C}\!=\!72\\ \text{K}\!=\!3{\times}3{\times}3\end{array}\right]\!\times\!2$	$\left[\begin{array}[]{c}\text{C}\!=\!108\\ \text{K}\!=\!3{\times}3{\times}3\end{array}\right]\!\times\!3$	$\left[\begin{array}[]{c}\text{C}\!=\!144\\ \text{K}\!=\!3{\times}3{\times}3\end{array}\right]\!\times\!3$
E2	pool stride 2	pool stride 2	pool stride 2	pool stride 2
E2	$\left[\begin{array}[]{c}\text{C}\!=\!144\\ \text{K}\!=\!3{\times}3{\times}3\end{array}\right]\!\times\!2$	$\left[\begin{array}[]{c}\text{C}\!=\!144\\ \text{K}\!=\!3{\times}3{\times}3\end{array}\right]\!\times\!2$	$\left[\begin{array}[]{c}\text{C}\!=\!216\\ \text{K}\!=\!3{\times}3{\times}3\end{array}\right]\!\times\!3$	$\left[\begin{array}[]{c}\text{C}\!=\!288\\ \text{K}\!=\!3{\times}3{\times}3\end{array}\right]\!\times\!3$

Appendix

Table 12: Detailed architecture specifications. C: channel dimension, K: kernel size in the convolution block, H: number of heads,

b

: base frequency of PointROPE, F: MLP ratio in the FFN module, N: number of points in local group.

	nuScenes [ 6 ]	Waymo [ 72 ]	ScanNet [ 14 ]	Structured3D [ 104 ]
Input feature	XYZ+Intensity		RGB+Normal
Grid size	0.05m		0.02m
Head (framework)	Linear segmentor		Linear segmentor
Loss	CrossEntropy+Lovasz [ 3 ]		CrossEntropy+Lovasz [ 3 ]
Optimizer	AdamW [ 48 ]		AdamW [ 48 ]
Weight decay	0.005		0.05

Appendix

Table 13: Detailed training settings for semantic segmentation.

	ScanNet [ 14 ]	ScanNet200 [ 62 ]
Input feature	RGB+Normal
Grid size	0.02m
Head (framework)	PointGroup [ 37 ]
Loss	As in PointGroup [ 37 ]
Optimizer	AdamW [ 48 ]
Weight decay	0.05

Appendix

Table 14: Detailed training settings for instance segmentation.

	Object Detection
	Waymo [ 72 ]
Input feature	XYZ+Intensity+Elongation
Grid size	(0.32m, 0.32m, 6.0m)
Head (framework)	CenterPoint-Pillar [ 99 ]
Loss	As in CenterPoint-Pillar [ 99 ]
Optimizer	Adam [ 39 ]

Appendix

Table 15: Detailed training settings for object detection.

	mIoU	mAcc
$w/o$ PointROPE	79.6	86.5
Spherical	80.7	87.1
Cartesian	82.2	88.1
$x\!:\!y\!:\!z\!=\!6\!:\!6\!:\!6$	82.2	88.1
$x\!:\!y\!:\!z\!=\!4\!:\!4\!:\!10$	80.3	86.8
$x\!:\!y\!:\!z\!=\!8\!:\!8\!:\!2$	80.3	86.7

Appendix

Table 16: Additional ablation on PointROPE on nuScenes.

	$w/o$ PosEnc	APE (sinusoidal)	RPE	PointROPE
mIoU	79.6	80.2	80.6	82.2
Latency	20.7 ms	23.3 ms	27.6 ms	21.5 ms

Appendix

Table 17: Comparison with other positional encoding schemes on nuScenes.

Method	#Param	mIoU	mAcc
PTv3 [ 88 ]	46.1M	57.1	69.5
LitePT-S	12.7M	65.0	77.2

Appendix

Table 18: Cross-dataset transfer. Models are pretrained on Structured3D and evaluated on ScanNet validation set using linear probing.

	\scriptsize{1}⃝ ModelNet40				\scriptsize{2}⃝ ScanNet
	#Params	Latency	Memory	OA (%)	mIoU
PointMamba	12.3 M	10.5 ms	0.20 G	92.4	60.5
LitePT-S (Ours)	12.7 M	11.7 ms	0.13 G	92.5	76.5

Appendix

Table 19: Comparison with PointMamba.

Figure 9: LitePT-S vs. PointMamba efficiency on ScanNet.

	Semantic Segmentation (mIoU)			Instance Segmentation ( mAP )
	Structured3D	Waymo	nuScenes	ScanNet
#Points (Billion)	9.55	4.15	0.98	0.17
PTv3	82.4	71.3	80.4	40.9
LitePT-S	83.6	73.1	82.2	41.7
LitePT-L	1111 85.4 (+1.8)	1111 74.0 (+0.9)	1111 82.6 (+0.4)	1111 42.3 (+0.6)

Appendix

Table 20: More results for LitePT-L.

		Training		Inference
Method	#Params	Latency	Memory	Latency	Memory	mIoU
PTv3	46.1 M	153 ms	14.1 G	100 ms	7.1 G	71.3
LitePT-S (Ours)	12.7 M	86 ms	9.4 G	47 ms	5.4 G	73.1

Appendix

Table 21: Outdoor Efficiency comparison on Waymo. Results are reported as average over the full Waymo dataset using a single RTX 4090 GPU. Automatic Mixed Precision (AMP) is enabled for all models during training and disabled during inference.

Method	#Param	mIoU	mAcc
PTv3 [ 88 ]	46.1M	80.4	87.2
LitePT-S	12.7M	82.2	88.1
PTv3 [ 88 ] ( $w/o$ chunking and TTA)	46.1M	78.3	86.0
LitePT-S ( $w/o$ chunking and TTA)	12.7M	80.4	86.9

Appendix

Table 22: Semantic segmentation on nuScenes without chunking and TTA.

Point Cloud Learning

Point Cloud Learning develops methods for representing and interpreting unordered 3D point sets from LiDAR, depth sensors, radar, and 3D models.

See all 159 papers

PointTransformerX: Portable and Efficient 3D Point Cloud Processing without Sparse Algorithms
Apr 27, 2026Laurenz Reichardt, Nikolas Ebert, Oliver WasenmüllerRotary Positional Embeddings 3D ViTs
Mannheim University of Applied Sciences, Germany
Pointy - A Lightweight Transformer for Point Cloud Foundation Models
Mar 11, 2026Konrad Szafer, Marek Kraft, Dominik BelterPoint Cloud Learning 3D Foundation Models
Institute of Robotics and Machine Intelligence, Poznan University of Technology
Position Anchor Tuning: Towards Efficient Adaptation of Pre-Trained Point Cloud Transformers
Sep 16, 2026Zheng Liu, Xin Gao, Jinchao Zhu +1Parameter-Efficient Fine-Tuning Efficient Transformer Inference
School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China, and also with the Beijing Engineering Research Center of Industrial Spectrum Imaging, Beijing 100083, China · College of Software, Nankai University, Tianjin 300350, China · Department of Automation, BNRist, Tsinghua University, Beijing 100084, China

Apr 27, 2026J/K move · Enter open · S save

PointTransformerX: Portable and Efficient 3D Point Cloud Processing without Sparse Algorithms

Laurenz Reichardt, Nikolas Ebert, Oliver Wasenmüller

Mannheim University of Applied Sciences, Germany

Rotary Positional Embeddings 3D ViTs Efficient ViTs Point Cloud Understanding

Open paper AskPDF arXiv

3D point cloud perception remains tightly coupled to custom CUDA operators for spatial operations, limiting portability and efficiency on non-NVIDIA, AMD, and embedded hardware. We introduce PointTransformerX (PTX), a fully PyTorch-native vision transformer backbone for 3D point clouds, removing all custom CUDA operators and external libraries while retaining competitive accuracy. PTX introduces 3D-GS-RoPE, a rotary positional embedding that encodes 3D spatial relationships directly in self-attention without neighborhood construction, and further replaces sparse convolutional patch embedding with a linear projection. PTX explores inference-time scaling of attention windows to improve accuracy without retraining. With a redesigned feed-forward network, PTX achieves 98.7% of PointTransformer V3's accuracy on ScanNet with 79.2% fewer parameters and executing 1.6\times faster while requiring just 253 MB memory. PTX runs natively on NVIDIA GPUs, AMD GPUs (ROCm), and CPUs, providing an efficient and portable foundation for point cloud perception.

Abstract

Point Cloud Learning

Explore similar work

PointTransformerX: Portable and Efficient 3D Point Cloud Processing without Sparse Algorithms

Pointy - A Lightweight Transformer for Point Cloud Foundation Models

Position Anchor Tuning: Towards Efficient Adaptation of Pre-Trained Point Cloud Transformers