Refine Then Fusion: Training-Free 3D Point Cloud Adaptation with Priority Refinement and Multi-Modal Knowledge Fusion

Hang Cheng; Yan Chen; Mingyu Fan; Long Zeng

cs.CVSep 18, 2026

Refine Then Fusion: Training-Free 3D Point Cloud Adaptation with Priority Refinement and Multi-Modal Knowledge Fusion

Authors: Hang Cheng, Yan Chen, Mingyu Fan, Long Zeng

Organizations: Tsinghua Shenzhen International Graduate School, Tsinghua University

Abstract

Recent pre-trained foundation models provide rich multi-modal priors for downstream 3D vision tasks. However, the effectiveness of these representations in few-shot scenarios is limited by two fundamental challenges: High-dimensional features often contain substantial channel redundancy and task-irrelevant noise, while the reliability of different modalities varies across samples. Consequently, direct aggregation of heterogeneous representations overlooks sample-dependent modality reliability and may obscure the discriminative cues essential. To address these limitations, we propose Refine Then Fusion(RTF), a training-free framework for few-shot 3D recognition. RTF first identifies discriminative feature channels by jointly modeling inter-class similarity and intra-class stability, thereby decoupling domain-specific knowledge refinement from the cached representations of pre-trained models. It then introduces a reliability-aware fusion mechanism that estimates sample-wise modality reliability from the distribution shifts induced by feature refinement, enabling adaptive aggregation of multi-modal representations. Furthermore, RTF constructs a memory cache that integrates instance-level support features with class-level prototypes to infer query labels. Extensive experiments on five benchmarks demonstrate that RTF consistently outperforms single-modal baselines, partial-fusion variants, and existing lightweight adaptation methods, achieving state-of-the-art few-shot 3D recognition performance without gradient optimization, additional training data, auxiliary training, or parameter updates.

Figures & tables

Figure 1: Comparison of accuracy and training GFLOPs on 16-shot ModelNet40 classification. Our method achieves competitive accuracy with substantially reduced optimization cost, demonstrating its effectiveness and efficiency for training-free adaptation.

Figure 2: Comparison between previous adaptation paradigms and our RTF framework.

Figure 3: Overview of the proposed RTF framework, consisting of Priority-guided Channel Refinement, Reliability-aware Multi-Modal Fusion, and Hybrid Memory Cache for training-free 3D adaptation.

Method	Conditions (K-shot)	2D-Modal	3D-Modal	ModelNet40	ModelNet10	OBJ_ONLY	OBJ_BG	OBJ_T50RS
PointCLIP	Train-Free (0)	✓	✓	20.18	30.23	19.28	21.34	15.38
CLIP2Point	Train-Free (0)	✓	✓	49.38	66.63	30.46	35.46	23.32
PointCLIP-V2	Train-Free (0)	✓	✓	64.22	73.13	50.09	41.22	35.36
RECON	Train-Free (0)	✓	✓	61.70	75.60	43.70	40.40	30.50
OpenShape	Train-Free (0)	✓	✓	85.30	-	-	56.70	-
VIT-Lens	Train-Free (0)	✓	✓	87.60	-	-	60.10	-

Table 1: Comparison with state-of-the-art methods under few-shot and full-shot 3D recognition settings. “2D-Modal” indicates the use of image inputs during inference, while “3D-Modal” denotes the use of point clouds. ULIP-2 ∗ represents the model pre-trained on the large-scale Objaverse dataset. Best results are highlighted in bold.

Proportion	10%	20%	40%	60%	80%	100%
ModelNet10	89.73	91.25	91.96	92.29	92.62	93.50
ModelNet40	89.71	91.00	91.25	91.57	92.26	92.91
OBJ_ONLY	84.17	85.44	86.06	87.61	88.47	91.57
OBJ_BG	80.72	84.51	87.78	87.44	88.64	90.36
OBJ_T50RS	76.65	78.31	79.98	81.40	80.36	81.85

Table 2: Impact of Different Support Set Sizes

Figure 4: Few-shot performance comparison of different core components ablation on multiple benchmarks.

Modality			Accuracy (%)
3D	2D	Geometry	ModelNet40	ModelNet10	OBJ_ONLY	OBJ_BG	OBJ_T50RS
$\checkmark$	$\times$	$\times$	91.41	92.73	90.19	88.47	77.97
$\times$	$\checkmark$	$\times$	83.14	87.56	70.05	74.18	54.75
$\times$	$\times$	$\checkmark$	18.92	76.98	27.02	28.23	24.50
$\checkmark$	$\checkmark$	$\times$	91.98	92.95	91.22	88.81	78.14
$\checkmark$	$\times$	$\checkmark$	92.54	92.62	90.71	89.62	81.30

Table 3: Ablation study of different modality combinations.

Figure 5: Comparison of different modality fusion strategies.

Figure 6: Comparison of different clustering strategies for class-level memory construction.

Figure 7: Sensitivity analysis of the channel keep ratio

\rho

(left) and similarity-variance trade-off coefficient

\lambda_{s}

(right).

Dataset	ULIP-2	ULIP-2	PointTFA	PointTFA m	Ours
Dataset	(Noise-Free)	(+Noise)	(+Noise)	(+Noise)	(+Noise)
ModelNet40	75.6	43.1	53.1	68.2	69.3
ModelNet10	84.8	75.9	79.6	90.3	90.8
OBJ_ONLY	53.4	31.5	44.0	47.0	49.5
OBJ_BG	48.6	31.0	47.3	49.9	51.7
OBJ_T50RS	40.4	22.7	27.2	32.1	35.6

Table 4: Performance comparison under noise (

\sigma=0.01

) on different datasets.

Few-Shot Learning

Few-shot learning develops methods that recognize or adapt to new tasks from only a handful of labeled examples.

See all 207 papers

CLIPoint3D: Language-Grounded Few-Shot Unsupervised 3D Point Cloud Domain Adaptation
Feb 23, 2026Mainak Singha, Sarthak Mehrotra, Paolo Casari +3Unsupervised Domain Adaptation Few-Shot Domain Adaptation
University of Trento, Italy · MDSR Labs Adobe, India · IIT Bombay, India +1
Learning Hierarchical Orthogonal Prototypes for Generalized Few-Shot 3D Point Cloud Segmentation
Mar 20, 2026Yifei Zhao, Fanyu Zhao, Zhongyuan Zhang +33D Point Cloud Segmentation 3D Semantic Segmentation
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China
PointRFT: Explicit Reinforcement Fine-tuning for Point Cloud Few-shot Learning
Mar 25, 2026Yankai Wang, Yiding Sun, Qirui Wang +3Point Cloud Learning RL Fine-Tuning
School of Software Engineering, Xi’an Jiaotong University, Xi’an 710048, China · International School, Beijing University of Posts and Telecommunications, Beijing 100876, China

Feb 23, 2026J/K move · Enter open · S save

CLIPoint3D: Language-Grounded Few-Shot Unsupervised 3D Point Cloud Domain Adaptation

Mainak Singha, Sarthak Mehrotra, Paolo Casari, Subhasis Chaudhuri+2

University of Trento, Italy · MDSR Labs Adobe, India · IIT Bombay, India+1

Unsupervised Domain Adaptation Few-Shot Domain Adaptation VLM Adaptation 3D VLMs Prompt Tuning

Open paper AskPDF arXiv

Recent vision-language models (VLMs) such as CLIP demonstrate impressive cross-modal reasoning, extending beyond images to 3D perception. Yet, these models remain fragile under domain shifts, especially when adapting from synthetic to real-world point clouds. Conventional 3D domain adaptation approaches rely on heavy trainable encoders, yielding strong accuracy but at the cost of efficiency. We introduce CLIPoint3D, the first framework for few-shot unsupervised 3D point cloud domain adaptation built upon CLIP. Our approach projects 3D samples into multiple depth maps and exploits the frozen CLIP backbone, refined through a knowledge-driven prompt tuning scheme that integrates high-level language priors with geometric cues from a lightweight 3D encoder. To adapt task-specific features effectively, we apply parameter-efficient fine-tuning to CLIP's encoders and design an entropy-guided view sampling strategy for selecting confident projections. Furthermore, an optimal transport-based alignment loss and an uncertainty-aware prototype alignment loss collaboratively bridge source-target distribution gaps while maintaining class separability. Extensive experiments on PointDA-10 and GraspNetPC-10 benchmarks show that CLIPoint3D achieves consistent 3-16% accuracy gains over both CLIP-based and conventional encoder-based baselines. Project page: https://sarthakm320.github.io/CLIPoint3D.

Abstract

Few-Shot Learning

Explore similar work

CLIPoint3D: Language-Grounded Few-Shot Unsupervised 3D Point Cloud Domain Adaptation

Learning Hierarchical Orthogonal Prototypes for Generalized Few-Shot 3D Point Cloud Segmentation

PointRFT: Explicit Reinforcement Fine-tuning for Point Cloud Few-shot Learning