PCT-Prompt: A Prompt-Guided Transformer Framework for Dense Prediction Tasks in Point Clouds

Dejun Zhang; Yanzi Bai; Yiqi Wu

cs.CVAug 17, 2026

PCT-Prompt: A Prompt-Guided Transformer Framework for Dense Prediction Tasks in Point Clouds

Authors: Dejun Zhang, Yanzi Bai, Yiqi Wu

Organizations: China University of Geosciences · Li Auto Inc

Abstract

Standard Transformers have proven effective in point cloud object classification, but their performance in dense prediction tasks within complex scenes is often hindered by weak prior assumptions. To address this challenge, we propose PCT-Prompt, a novel framework that enhances standard Transformers by introducing a prompt-guided feature branch to improve performance in dense prediction tasks. The standard Transformer branch leverages pre-trained models for global feature extraction from point cloud data, serving as the backbone for processing high-level features. Meanwhile, the prompt-guided feature branch consists of two key components: a fine-grained feature extraction block that captures multi-scale geometric features using geometry-sensitive abstraction layer, along with the PnP-3D layer to integrate local context with global regularization. The second component, the prompt-refined feature learning block generates prompt tokens, which are subsequently refined through cross-attention mechanisms. Additionally, we introduce a prompt drop mechanism that progressively removes prompt information across Transformer layers, balancing local details and global consistency. Experimental results on the ShapeNetPart, S3DIS, and DALES datasets demonstrate that PCT-Prompt significantly improves the adaptability of standard Transformers to dense prediction tasks, achieving strong performance in real-world scenarios.

Figures & tables

Figure 1: Overall architecture of PCT-Prompt.

Figure 2: The structure of the fine-grained feature extraction block.

Figure 3: The structure of the prompt-refined feature learning block.

Methods	Type	ins. mIoU	cls. mIoU	aero	bag	cap	car	chr.	e.ph.	gtar	knf.	lmp.	ltp.	m.b.	mg	pstl	s.b.	skte	tbl.
PointNet [ 30 ]	T	83.7	80.4	83.4	78.7	82.5	74.9	89.6	73.0	91.5	85.9	80.8	95.3	65.2	93.0	81.2	57.9	72.8	80.6
PointNet++ [ 1 ]	T	85.1	81.9	82.4	79.0	87.7	77.3	90.8	71.8	91.0	85.9	83.7	95.3	71.6	94.1	81.3	58.7	76.4	82.6
PointCNN [ 3 ]	T	86.1	84.6	84.1	86.5	86.0	80.8	90.6	79.7	92.3	88.4	85.3	96.1	77.2	95.2	84.2	64.2	80.0	83.0
DGCNN [ 2 ]	T	85.2	82.3	84.0	83.4	86.7	77.8	90.6	74.7	91.2	87.5	82.8	95.7	66.3	94.9	81.1	63.5	74.5	82.6
PointMLP [ 45 ]	T	86.1	84.6	83.5	83.4	87.5	80.5	90.3	78.2	92.2	88.1	82.6	96.2	77.5	95.8	85.4	64.6	83.3	84.3
PointASNL [ 46 ]	T	86.1	-	84.1	84.7	87.9	79.7	92.2	73.7	91.0	87.2	84.2	95.8	74.4	95.2	81.0	63.0	76.3	83.2

Table 1: Object part segmentation results on the ShapeNetPart dataset. “T”, “S”, and “V” represent traditional neural network models, standard Transformers, and variant Transformers, respectively.

Methods	Type	mIoU	mAcc	OA	ceil.	flr	wl	bm	col.	wnd.	dr	tbl.	chr	sf.	b.c.	brd	cltr.
PointNet [ 30 ]	T	41.1	49.0	-	88.8	97.3	69.8	0.1	3.9	46.3	10.8	59.0	52.6	5.9	40.3	26.4	33.2
PointCNN [ 3 ]	T	57.3	63.9	85.9	92.3	98.2	79.4	0.0	17.6	22.8	62.1	74.4	80.6	31.7	66.7	62.1	56.7
SPG [ 48 ]	T	58.0	66.5	86.4	89.4	96.9	78.1	0.0	42.8	48.9	61.6	84.7	75.4	69.8	52.6	2.1	52.2
PointWeb [ 49 ]	T	60.3	66.6	87.0	92.0	98.5	79.4	0.0	21.1	59.7	34.8	76.3	88.3	46.9	69.3	64.9	52.5
KPConv [ 31 ]	T	67.1	72.8	-	92.8	97.3	82.4	0.0	23.9	58.0	69.0	81.5	91.0	75.4	75.3	66.7	58.9
PAT [ 50 ]	V	60.1	70.8	-	93.0	98.5	72.3	1.0	41.5	85.1	38.2	57.7	83.6	48.1	67.0	61.3	33.6

Table 2: Semantic segmentation results on S3DIS dataset, tested on Area 5. *means we report the best results among public codebases and our own reproduction.

Figure 4: Visualizing the comparison between Point-BERT and PCT-Prompt on the S3DIS Dataset.

Methods	Type	mIoU	mAcc	OA	Ground	Buildings	Cars	Trucks	Poles	Power Lines	Fences	Vegetation
PointNet++ [ 1 ]	T	68.3	-	95.7	94.1	89.1	75.4	30.3	40.0	79.9	46.2	91.2
SPG [ 48 ]	T	60.6	-	95.5	94.7	93.4	62.9	18.7	28.5	65.2	33.6	87.9
PointCNN [ 3 ]	T	58.4	-	97.2	97.5	95.7	40.6	4.8	57.6	26.7	52.6	91.7
SPT [ 47 ]	V	79.6	-	97.5	96.7	93.1	86.1	52.4	94.0	52.7	65.3	96.7
Point-BERT [ 11 ]	S	72.0	79.6	96.0	92.9	91.9	63.3	27.8	93.3	49.0	65.4	93.1
PCT-Prompt	S	78.5	84.9	97.3	98.8	96.8	91.8	45.9	97.5	68.9	76.6	96.9

Table 3: Comparison of semantic segmentation performance on the DALES dataset.

Transformer	FFE	PFL			mIoU	mAcc
Transformer	FFE	Generator	Refiner	Drop	mIoU	mAcc
$\surd$	$\times$	$\times$	$\times$	$\times$	63.5	75.7
$\times$	$\surd$	$\times$	$\times$	$\times$	55.6	65.7
$\surd$	$\surd$	$\times$	$\times$	$\times$	65.5	75.8
$\surd$	$\surd$	$\surd$	$\times$	$\times$	67.2	76.0
$\surd$	$\surd$	$\times$	$\surd$	$\times$	66.7	76.1

Table 4: Component ablation study of PCT-Prompt on the S3DIS.

Frequency	mIoU	mAcc
$N=0$	63.5	75.7
$N=1$	65.9	76.1
$N=2$	68.4	76.0
$N=4$	68.9	76.2
$N=6$	70.9	78.8
$N=8$	67.4	75.5

Table 5: Quantitative comparisons of different numbers of interactions on the S3DIS.

Methods	Pre-trained Models	mIoU	mAcc
standard Transformer	-	72.1	62.9
	ACT [ 14 ]	63.1	73.2
	Point-MAE [ 12 ]	63.1	72.1
	MaskPoint [ 15 ]	64.4	72.7
	Point-BERT [ 11 ]	63.5	75.7
	ReCon [ 13 ]	64.8	73.3

Table 6: Quantitative comparisons among various pre-trained models.

3D ViTs

3D Vision Transformers model spatial and volumetric data for tasks such as reconstruction, segmentation, detection, and pose estimation.

See all 145 papers

Point Ladder Tuning: Parameter-Efficient Hierarchical Adaptation for 3D Point Cloud Understanding
Jul 21, 2026Junlin Chang, Longhao Zou, Rui Li3D Point Cloud Segmentation Point Cloud Classification
Beihang University, Beijing · Pengcheng Laboratory, Shenzhen
GAPrompt++: Multi-Granular Geometry-Aware Point Cloud Prompt for 3D Vision Model
Sep 17, 2026Zixiang Ai, Zhenyu Cui, Yufei Guo +4Point Cloud Learning Vision Foundation Model Adaptation
Wangxuan Institute of Computer Technology, Peking University, Beijing 100871, China · Intelligent Science and Technology Academy of CASIC, Beijing 100041, China · Institute of Software, Chinese Academy of Sciences, Beijing 100190, China +1
3DTMDet: A Dual-Path Synergy Network of Transformer and SSM for 3D Object Detection in Point Clouds
May 15, 2026Bingwen Qiu, Yuan Liu, Junqi Bai +53D ViTs 3D Object Detection
aSchool of Electronic and Optical Engineering, Nanjing University of Science and Technology, Nanjing, 210094, Jiangsu, China · bThe 28th Research Institute of China Electronics Technology Group Corporation, Nanjing, 210007, Jiangsu, China · cCollege of Astronautics, Nanjing University of Aeronautics and Astronautics, Nanjing, 210016, Jiangsu, China +2

Jul 21, 2026J/K move · Enter open · S save

Point Ladder Tuning: Parameter-Efficient Hierarchical Adaptation for 3D Point Cloud Understanding

Junlin Chang, Longhao Zou, Rui Li

Beihang University, Beijing · Pengcheng Laboratory, Shenzhen

3D Point Cloud Segmentation Point Cloud Classification Multi-Scale Feature Fusion Prompt Tuning Point Cloud Understanding Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Fine-tuning pre-trained point-cloud backbones typically updates all parameters, resulting in substantial computation and memory overhead. More importantly, modern point backbones rely on aggressive tokenization and downsampling, which yields compact global tokens but irreversibly discards fine-grained local geometry, an inherent bottleneck for parameter-efficient adaptation. Consequently, existing PEFT methods that operate only on these coarsened tokens can modulate global semantics but struggle to recover the missing multi-scale locality. We present Point Ladder Tuning (PLT), a locality-aware PEFT framework that performs hierarchical, instance-conditioned adaptation while keeping the backbone frozen. PLT forms a lightweight closed loop: (i) a Hierarchical Ladder Network (HLN) constructs a multi-resolution local feature pyramid directly from raw points; (ii) a Local-Global Fusion (LGF) aligns and fuses local pyramids with intermediate backbone semantics; and (iii) a Dynamic Prompt Generator produces instance-aware multi-scale prompts to modulate the frozen backbone effectively. For dense prediction, we further introduce a lightweight segmentation head that progressively upsamples fused features and leverages backbone priors to refine fine structures. Extensive experiments on classification and dense prediction show that PLT consistently surpasses prior PEFT baselines with minimal tunable parameters. PLT achieves state-of-the-art performance using only 2.71% trainable parameters for classification and 7.69% for dense prediction, and scales favorably to larger backbones, requiring merely 0.36% parameters on PointGPT-L. The code is released at https://github.com/JunLinChang/ECCV2026-PLT.

Abstract

3D ViTs

Explore similar work

Point Ladder Tuning: Parameter-Efficient Hierarchical Adaptation for 3D Point Cloud Understanding

GAPrompt++: Multi-Granular Geometry-Aware Point Cloud Prompt for 3D Vision Model

3DTMDet: A Dual-Path Synergy Network of Transformer and SSM for 3D Object Detection in Point Clouds