MixPT: Point Transformer with Interleaved Window and Global Attention

Minkyun Seo; Hyungtae Lim; Jaesik Park

cs.CVOct 22, 2026

MixPT: Point Transformer with Interleaved Window and Global Attention

Authors: Minkyun Seo, Hyungtae Lim, Jaesik Park

Organizations: Seoul National University · Massachusetts Institute of Technology

Abstract

Point cloud transformers aim to capture detailed local geometry and relationships across a scene at a manageable computational cost. Two common approaches are window attention and global attention. They differ in how they enable scene-wide interaction, which refers to information exchange among tokens across a scene. Window attention reduces computational cost by restricting self-attention to groups of tokens called windows; in serialized designs, each window contains a fixed number of consecutive tokens from a spatially ordered sequence. Because each block only directly connects tokens within the same window, information exchange across windows relies on changes in window membership across blocks and hierarchical pooling. Global attention, in contrast, directly connects all tokens within a scene, but computing attention over all token pairs can be costly when the number of tokens is large. To balance computational cost and scene-wide interaction, we introduce MixPT, a point transformer that uses a mix of window and global attention blocks in an interleaved schedule, while retaining a single fixed serialization pattern for its window blocks. For scenes with a large number of tokens, MixPT applies global attention to spatially pooled tokens and returns residual updates to the original-resolution stream. Across indoor and outdoor benchmarks, MixPT achieves competitive accuracy with lower latency than representative window and global attention baselines. We further demonstrate the feasibility of scaling the attention stage in depth by repeating the interleaved pattern of window and global attention blocks.

Figures & tables

Figure 1: Serialize once, mix window and global attention. MixPT-S serializes tokens once ( S ) and reuses that order throughout its WWGWWGWW attention schedule. For the same query, window attention restricts interaction to its fixed window, whereas global attention enables scene-wide interaction. For clarity, the left panel provides a 2D schematic of 3D serialization around the query token.

Panel of Figure 2 — Figure 2: Information propagation through the first five blocks of encoder stage 3 of LitePT on the Waymo validation set ( Sun et al., 2020 ) (stages indexed from zero; blocks indexed from one; random order shuffling disabled). Green marks tokens reachable from the red-star token through attention connections within this stage; gray marks those not yet reached.

Figure 3: Structural reachability across blocks on the Waymo validation set ( Sun et al., 2020 ) . Lines show scene-level means; shading indicates the 10th to 90th percentiles.

Figure 4: Normalized attention entropy within each block’s available support on the Waymo validation set ( Sun et al., 2020 ) . Lines show scene-level means; shading indicates the 10th to 90th percentiles.

Panel of Figure 5 — Figure 5: Attention patterns across Volt ( Yilmaz et al., 2026 ) blocks on the Waymo validation set ( Sun et al., 2020 ) . For a selected query, each token is colored by its relative attention weight (“Rel. attn.”), the ratio of its head-averaged attention weight to the uniform weight over the available keys. Values of 1 and 10 indicate weights equal to and ten times the uniform weight, respectively. Attention is broadly distributed in the first block and more concentrated in several later blocks, particularly in the central regions of the visualized scene.

Figure 6: Block-level overview of MixPT: (a) sparse convolution stages, (b) attention stage, and (c) decoder stages. Our attention stage follows the WWGWWGWW schedule, where W and G denote window and global attention blocks, respectively. (d) LitePT’s attention stages ( Yue et al., 2026 ) . Unlike LitePT’s varying serialization patterns, MixPT uses one fixed serialization within a single attention stage and directly connects tokens across windows through global attention blocks.

Figure 7: Window attention (left) and pooled global attention (right) blocks in MixPT.

Method	Attention	#Params	Sem.KITTI	Waymo	nuScenes	ScanNet
PTv3 ( Wu et al., 2024 )	Window	46.1M	63.6	87.8	38.3	48.1
LitePT-S ( Yue et al., 2026 )	Window	12.7M	36.4	53.0	27.8	25.7
Volt-S ( Yilmaz et al., 2026 )	Global	23.7M	140.3	295.5	40.0	23.0
MixPT-S (Ours)	Window + Global	14.1M	28.2	42.8	19.2	17.5
MixPT-B (Ours)	Window + Global	51.8M	56.4	90.3	33.8	30.2
MixPT-L (Ours)	Window + Global	89.4M	72.4	120.1	45.7	33.4

Table 1: Median latency for a single model forward pass over voxelized validation scenes (ms), measured with batch size one and FP16 autocast on an RTX A6000. For MixPT, -S, -B, and -L denote the small, base, and large variants, respectively.

		Sem.KITTI		Waymo		nuScenes		ScanNet
Method	#Params	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc
PTv3 ( Wu et al., 2024 )	46.1M	69.6	77.2	70.7	81.3	79.6	86.6	76.5	85.3
LitePT-S ( Yue et al., 2026 )	12.7M	69.8	76.8	71.7	82.8	80.3	87.1	75.5	83.4
Volt-S ( Yilmaz et al., 2026 )	23.7M	69.4	77.0	70.5	80.3	80.5	87.5	76.4	84.9
MixPT-S (Ours)	14.1M	70.0	77.3	72.1	83.0	80.6	87.4	76.0	84.3
MixPT-B (Ours)	51.8M	70.4	77.2	72.1	84.1	81.1	87.3	76.4	84.9

Table 2: Semantic segmentation w/o TTA (mIoU/ mAcc, %). Bold and underline indicate the best and second-best results, respectively, among the baselines and MixPT-S.

Panel of Figure 8 — Figure 8: MixPT-S latency breakdown on nuScenes: (a) across model stages and (b) across individual components within the attention stage.

		ScanNet			ScanNet200
Backbone	#Params	mAP 25	mAP 50	mAP	mAP 25	mAP 50	mAP
PTv3 ( Wu et al., 2024 )	46.2M	77.5	61.7	40.9	40.1	33.2	23.1
LitePT-S* ( Yue et al., 2026 )	16.0M	78.5	64.9	41.7	40.3	33.1	22.2
Volt-S ( Yilmaz et al., 2026 )	23.7M	77.4	62.7	40.2	38.2	32.1	21.1
MixPT-S* (Ours)	18.4M	78.9	65.1	41.1	38.5	32.5	22.4

Table 3: Indoor instance segmentation on ScanNet and ScanNet200 (%).We evaluate different backbones using PointGroup ( Jiang et al., 2020 ) .

		Sem.KITTI ( $s=4$ )				ScanNet ( $s=1$ )
Serialization	Block layout	#Params	Latency (ms)	mIoU	mAcc	#Params	Latency (ms)	mIoU	mAcc
Z + TZ + H + TH	WWWWWWWW	13.6M	32.99	68.5	75.6	13.6M	21.28	74.8	84.0
TZ	WWWWWWWW	13.6M	27.88	68.6	76.3	13.6M	17.16	73.7	83.5
TZ	WWGWWGWW	14.1M	28.20	70.0	77.3	13.6M	17.48	76.0	84.3
-	GGGGGGGG	15.6M	27.36	68.0	75.8	13.6M	17.60	75.7	84.0

Table 4: Fixed serialization and global attention ablation (w/o TTA, mIoU/ mAcc, %).Attention-stage depth and width were held fixed across variants.

			nuScenes				ScanNet
Block layout	#W	#G	#Params	Latency (ms)	mIoU	mAcc	#Params	Latency (ms)	mIoU	mAcc
WWGWWGWW	6	2	14.1M	19.16	80.6	87.4	13.6M	17.48	76.0	84.3
WWWGWWWG	6	2	14.1M	19.39	80.4	87.1	13.6M	17.61	75.1	83.9
WWWWGGGG	4	4	14.6M	21.05	80.0	87.2	13.6M	17.44	75.6	83.9
WGWGWGWG	4	4	14.6M	21.46	79.7	86.9	13.6M	17.42	75.6	84.3
WWWGWWWGWW	8	2	17.1M	21.00	80.8	87.6	16.6M	18.54	75.6	83.6

Table 5: Block arrangement ablation for semantic segmentation (w/o TTA, mIoU/ mAcc, %).

Panel of Figure 9 — Figure 9: Effect of global pooling stride $s$ (w/o TTA). (a) Waymo mIoU (gray bars) and full-model latency (orange line), with separate training per stride under the same protocol. (b) Mean latency per global block. (c) Peak allocated memory for global (solid) and window (dashed) attention blocks.

						nuScenes			ScanNet
RoPE	#Heads	Head dim.	$d_{x}$	$d_{y}$	$d_{z}$	#Params	mIoU	mAcc	#Params	mIoU	mAcc
Axial (symmetric)	16	18	6	6	6	10.0M	79.8	87.3	9.7M	75.2	84.0
Axial (symmetric)	16	24	8	8	8	16.5M	80.6	87.3	15.9M	75.7	84.2
Axial (asymmetric)	16	20	6	6	8	12.0M	80.1	86.8	11.6M	75.4	84.3
Axial (asymmetric)	16	22	8	8	6	14.1M	80.6	87.4	13.6M	76.0	84.3
Axial (asymmetric)	16	26	10	10	6	19.1M	80.8	87.2	18.4M	75.5	84.0

Table 6: Axial RoPE allocation ablation (w/o TTA, mIoU/ mAcc, %).

Figure 10: Spatial extents of representative scenes (metres) and serialized windows (grid cells) on ScanNet (left) and nuScenes (right). Dashed lines connect the selected window in each scene to its corresponding window view. Window extents include empty grid cells.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	MixPT-S	MixPT-S*	MixPT-B	MixPT-L
Encoder channels (0–3)	$(36,72,144,352)$	$(36,72,144,352)$	$(54,108,216,512)$	$(72,144,288,672)$
Encoder pooling strides	$(2,2,2)$
Encoder blocks (0–3)	$(2,2,2,8)$	$(2,2,2,10)$	$(3,3,3,14)$	$(3,3,3,14)$
Encoder attention	$(\mathtt{WWG})_{2}\mathtt{WW}$	$(\mathtt{WWG})_{3}\mathtt{G}$	$(\mathtt{WWG})_{4}\mathtt{WW}$	$(\mathtt{WWG})_{4}\mathtt{WW}$
Decoder channels (2–0)	$(144,72,72)$	$(144,72,72)$	$(216,108,108)$	$(288,144,144)$
Decoder conv. blocks (2–0)	$(0,0,0)$	$(2,2,2)$	$(0,0,0)$	$(0,0,0)$

Appendix

Table A1: MixPT model configurations.

Setting	Sem.KITTI	Waymo	nuScenes
Input feature	XYZ + intensity
Grid size	$0.05\;\mathrm{m}$
Global pooling stride ( $s$ )	4	4	2
Head (framework)	Linear segmentor
Loss	Class-weighted cross-entropy + Lovász	Cross-entropy + Lovász	Cross-entropy + Lovász
Optimizer	AdamW

Appendix

Table A2: Outdoor semantic segmentation training and evaluation settings for MixPT-S.

Setting	Semantic segmentation	Instance segmentation
Setting	ScanNet	ScanNet	ScanNet200
Input feature	RGB + normals
Grid size	$0.02\;\mathrm{m}$
Global pooling stride ( $s$ )	1
Head (framework)	Linear segmentor	PointGroup ( Jiang et al., 2020 )	PointGroup ( Jiang et al., 2020 )
Loss	Cross-entropy + Lovász	Cross-entropy + Lovász + offset regression ( $L_{1}$ + cosine)	Cross-entropy + Lovász + offset regression ( $L_{1}$ + cosine)

Appendix

Table A3: Indoor training and evaluation settings for MixPT-S (semantic segmentation) and MixPT-S* (instance segmentation).

Panel of Figure 20 — Figure A1: Fragment counts across TTA views for SemanticKITTI, nuScenes, and ScanNet. Each fragment requires one model forward at batch size one.

			Sem.KITTI		Waymo		nuScenes		ScanNet
	Method	#Params	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc	mIoU	mAcc
w/ TTA	PTv3 ( Wu et al., 2024 )	46.1M	71.0	77.9	71.3	80.5	81.2	87.5	77.6	85.9
	LitePT-S ( Yue et al., 2026 )	12.7M	71.3	77.7	73.1	83.8	82.2	88.1	76.5	84.2
	Volt-S ( Yilmaz et al., 2026 )	23.7M	70.5	77.6	71.5	80.9	81.0	87.7	77.3	85.5
	MixPT-S (Ours)	14.1M	71.2	78.1	72.9	83.4	81.8	87.9	76.9	84.9
	MixPT-B (Ours)	51.8M	71.8	78.1	73.0	84.5	81.7	87.8	77.4	85.4

Appendix

Table A4: Semantic segmentation w/ TTA and w/o TTA (mIoU/ mAcc, %). Bold and underline indicate the best and second-best results, respectively, among the baselines and MixPT-S within each TTA setting.

	Sem.KITTI			nuScenes			ScanNet
Order	Latency (ms)	mIoU	mAcc	Latency (ms)	mIoU	mAcc	Latency (ms)	mIoU	mAcc
Z	27.83	69.2	76.3	18.92	80.4	87.0	17.51	75.4	83.9
TZ	28.20	70.0	77.3	19.16	80.6	87.4	17.48	76.0	84.3
H	30.15	69.1	76.8	21.34	80.4	87.0	18.97	75.5	83.8
TH	30.22	69.2	76.1	21.02	80.3	87.2	18.95	75.2	84.2

Appendix

Table A5: Serialization order ablation of MixPT-S (w/o TTA, mIoU/ mAcc, %).

	Scene [m]			Window extent [cells]			Global extent [cells]
Dataset	$x$	$y$	$z$	$x$	$y$	$z$	$x$	$y$	$z$
Sem.KITTI	157.3	109.8	9.7	79	63	11	99	69	7
Waymo	148.6	132.4	8.5	72	63	14	93	83	6
nuScenes	145.0	188.3	24.3	126	120	31	182	236	31
ScanNet	5.6	5.1	2.5	30	23	16	35	32	16

Appendix

Table A6: Median axis ranges over the complete validation sets. Scene ranges are measured in metres, while window and global ranges are measured in cells of their respective attention grids.

Panel of Figure 24 — Figure A2: Qualitative semantic segmentation on SemanticKITTI. Columns show (a) input point clouds, (b) MixPT-S predictions, and (c) ground truth. Insets highlight representative regions.

Panel of Figure 25 — Figure A3: Qualitative semantic segmentation on ScanNet. Columns show (a) input point clouds, (b) MixPT-S predictions, and (c) ground truth.

Panel of Figure 26 — Figure A4: Qualitative instance segmentation on ScanNet200. Columns show (a) input point clouds, (b) MixPT-S* predictions, and (c) ground truth. Colors distinguish instances rather than semantic classes. For visualization, prediction-GT pairs matched one-to-one within the same class at mask IoU $>0.5$ share colors; unmatched instances receive distinct colors. Gray indicates background or unassigned points.

Point Transformer

See all 10 papers

Dec 15, 2025

LitePT: Lighter Yet Stronger Point Transformer

Modern neural architectures for 3D point cloud processing contain both convolutional layers and attention blocks, but the best way to assemble them remains unclear. We analyse the role of different computational blocks in 3D point cloud networks and find an intuitive behaviour: convolution is adequate to extract low-level geometry at high-resolution in early layers, where attention is expensive without bringing any benefits; attention captures high-level semantics and context in low-resolution, deep layers more efficiently, where convolution inflates the parameter count. Guided by this design principle, we propose a new, improved 3D point cloud backbone that employs convolutions in early stages and switches to attention for deeper layers. To avoid the loss of spatial layout information when discarding redundant convolution layers, we introduce a novel, parameter-free 3D positional encoding, PointROPE. The resulting LitePT model has

3.6\times

fewer parameters, runs

2\times

faster, and uses

2\times

less memory than the state-of-the-art Point Transformer V3, but nonetheless matches or outperforms it on a range of tasks and datasets. Code and models are available at: https://github.com/prs-eth/LitePT.

Yuanwen Yue, Damien Robert, Jianyuan Wang +4

ETH Zurich · University of Oxford · University of Zurich

Point Cloud Learning Point Transformer

Ask PDF

Sep 16, 2026

Position Anchor Tuning: Towards Efficient Adaptation of Pre-Trained Point Cloud Transformers

Parameter-efficient fine-tuning (PEFT) has recently emerged as a pivotal research direction for adapting pre-trained point cloud transformers to diverse downstream tasks. Although existing methods achieve excellent fine-tuning performance with high parameter efficiency, they ignore inference efficiency. To tackle this problem, a novel PEFT method termed position anchor tuning (PAT) is proposed in this paper. As multi-head attention (MHA) and feed-forward network (FFN) are computation-heavy blocks in pre-trained transformers, PAT decreases their computational cost through token aggregation-expansion pairs. Each pair comprises a token aggregation module (TAM) and a token expansion module (TEM). For MHA and FFN blocks, TAMs extract representative tokens from their input tokens based on position anchors in 3D space. These extracted tokens, rather than the original input tokens, are processed by the blocks, thereby reducing the number of tokens involved in computation. Then, TEMs propagate the learned representations back to the original input tokens. Since TAMs are solely responsible for capturing task-specific representations, base-sharing low-rank adaptation (BSLoRA) is further introduced to enable them to learn such representations effectively with only a small number of trainable parameters. Extensive experiments on widely used benchmarks demonstrate that PAT performs comparably to state-of-the-art methods while incurring significantly lower computational overhead and fewer trainable parameters.

Zheng Liu, Xin Gao, Jinchao Zhu +1

School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing 100083, China, and also with the Beijing Engineering Research Center of Industrial Spectrum Imaging, Beijing 100083, China · College of Software, Nankai University, Tianjin 300350, China · Department of Automation, BNRist, Tsinghua University, Beijing 100084, China

Parameter-Efficient Fine-Tuning Efficient Transformer Inference 3D ViTs Token Merging

Ask PDF

Apr 27, 2026

PointTransformerX: Portable and Efficient 3D Point Cloud Processing without Sparse Algorithms

3D point cloud perception remains tightly coupled to custom CUDA operators for spatial operations, limiting portability and efficiency on non-NVIDIA, AMD, and embedded hardware. We introduce PointTransformerX (PTX), a fully PyTorch-native vision transformer backbone for 3D point clouds, removing all custom CUDA operators and external libraries while retaining competitive accuracy. PTX introduces 3D-GS-RoPE, a rotary positional embedding that encodes 3D spatial relationships directly in self-attention without neighborhood construction, and further replaces sparse convolutional patch embedding with a linear projection. PTX explores inference-time scaling of attention windows to improve accuracy without retraining. With a redesigned feed-forward network, PTX achieves 98.7% of PointTransformer V3's accuracy on ScanNet with 79.2% fewer parameters and executing 1.6\times faster while requiring just 253 MB memory. PTX runs natively on NVIDIA GPUs, AMD GPUs (ROCm), and CPUs, providing an efficient and portable foundation for point cloud perception.

Laurenz Reichardt, Nikolas Ebert, Oliver Wasenmüller

Mannheim University of Applied Sciences, Germany

Rotary Positional Embeddings 3D ViTs Efficient ViTs Point Cloud Understanding

Ask PDF

Abstract

Point Transformer

Explore similar work

LitePT: Lighter Yet Stronger Point Transformer

Position Anchor Tuning: Towards Efficient Adaptation of Pre-Trained Point Cloud Transformers

PointTransformerX: Portable and Efficient 3D Point Cloud Processing without Sparse Algorithms