Recent pre-trained foundation models provide rich multi-modal priors for downstream 3D vision tasks. However, the effectiveness of these representations in few-shot scenarios is limited by two fundamental challenges: High-dimensional features often contain substantial channel redundancy and task-irrelevant noise, while the reliability of different modalities varies across samples. Consequently, direct aggregation of heterogeneous representations overlooks sample-dependent modality reliability and may obscure the discriminative cues essential. To address these limitations, we propose Refine Then Fusion(RTF), a training-free framework for few-shot 3D recognition. RTF first identifies discriminative feature channels by jointly modeling inter-class similarity and intra-class stability, thereby decoupling domain-specific knowledge refinement from the cached representations of pre-trained models. It then introduces a reliability-aware fusion mechanism that estimates sample-wise modality reliability from the distribution shifts induced by feature refinement, enabling adaptive aggregation of multi-modal representations. Furthermore, RTF constructs a memory cache that integrates instance-level support features with class-level prototypes to infer query labels. Extensive experiments on five benchmarks demonstrate that RTF consistently outperforms single-modal baselines, partial-fusion variants, and existing lightweight adaptation methods, achieving state-of-the-art few-shot 3D recognition performance without gradient optimization, additional training data, auxiliary training, or parameter updates.
Figures & tables
Figure 1: Comparison of accuracy and training GFLOPs on 16-shot ModelNet40 classification. Our method achieves competitive accuracy with substantially reduced optimization cost, demonstrating its effectiveness and efficiency for training-free adaptation.
Figure 2: Comparison between previous adaptation paradigms and our RTF framework.
Figure 3: Overview of the proposed RTF framework, consisting of Priority-guided Channel Refinement, Reliability-aware Multi-Modal Fusion, and Hybrid Memory Cache for training-free 3D adaptation.
Method
Conditions (K-shot)
2D-Modal
3D-Modal
ModelNet40
ModelNet10
OBJ_ONLY
OBJ_BG
OBJ_T50RS
PointCLIP
Train-Free (0)
✓
✓
20.18
30.23
19.28
21.34
15.38
CLIP2Point
Train-Free (0)
✓
✓
49.38
66.63
30.46
35.46
23.32
PointCLIP-V2
Train-Free (0)
✓
✓
64.22
73.13
50.09
41.22
35.36
RECON
Train-Free (0)
✓
✓
61.70
75.60
43.70
40.40
30.50
OpenShape
Train-Free (0)
✓
✓
85.30
-
-
56.70
-
VIT-Lens
Train-Free (0)
✓
✓
87.60
-
-
60.10
-
Table 1: Comparison with state-of-the-art methods under few-shot and full-shot 3D recognition settings. “2D-Modal” indicates the use of image inputs during inference, while “3D-Modal” denotes the use of point clouds. ULIP-2 ∗ represents the model pre-trained on the large-scale Objaverse dataset. Best results are highlighted in bold.
Proportion
10%
20%
40%
60%
80%
100%
ModelNet10
89.73
91.25
91.96
92.29
92.62
93.50
ModelNet40
89.71
91.00
91.25
91.57
92.26
92.91
OBJ_ONLY
84.17
85.44
86.06
87.61
88.47
91.57
OBJ_BG
80.72
84.51
87.78
87.44
88.64
90.36
OBJ_T50RS
76.65
78.31
79.98
81.40
80.36
81.85
Table 2: Impact of Different Support Set Sizes
Figure 4: Few-shot performance comparison of different core components ablation on multiple benchmarks.
Modality
Accuracy (%)
3D
2D
Geometry
ModelNet40
ModelNet10
OBJ_ONLY
OBJ_BG
OBJ_T50RS
✓
×
×
91.41
92.73
90.19
88.47
77.97
×
✓
×
83.14
87.56
70.05
74.18
54.75
×
×
✓
18.92
76.98
27.02
28.23
24.50
✓
✓
×
91.98
92.95
91.22
88.81
78.14
✓
×
✓
92.54
92.62
90.71
89.62
81.30
Table 3: Ablation study of different modality combinations.
Figure 5: Comparison of different modality fusion strategies.
Figure 6: Comparison of different clustering strategies for class-level memory construction.
Figure 7: Sensitivity analysis of the channel keep ratio ρ (left) and similarity-variance trade-off coefficient λs (right).
Dataset
ULIP-2
ULIP-2
PointTFA
PointTFA m
Ours
(Noise-Free)
(+Noise)
(+Noise)
(+Noise)
(+Noise)
ModelNet40
75.6
43.1
53.1
68.2
69.3
ModelNet10
84.8
75.9
79.6
90.3
90.8
OBJ_ONLY
53.4
31.5
44.0
47.0
49.5
OBJ_BG
48.6
31.0
47.3
49.9
51.7
OBJ_T50RS
40.4
22.7
27.2
32.1
35.6
Table 4: Performance comparison under noise ( σ=0.01 ) on different datasets.
Recent vision-language models (VLMs) such as CLIP demonstrate impressive cross-modal reasoning, extending beyond images to 3D perception. Yet, these models remain fragile under domain shifts, especially when adapting from synthetic to real-world point clouds. Conventional 3D domain adaptation approaches rely on heavy trainable encoders, yielding strong accuracy but at the cost of efficiency. We introduce CLIPoint3D, the first framework for few-shot unsupervised 3D point cloud domain adaptation built upon CLIP. Our approach projects 3D samples into multiple depth maps and exploits the frozen CLIP backbone, refined through a knowledge-driven prompt tuning scheme that integrates high-level language priors with geometric cues from a lightweight 3D encoder. To adapt task-specific features effectively, we apply parameter-efficient fine-tuning to CLIP's encoders and design an entropy-guided view sampling strategy for selecting confident projections. Furthermore, an optimal transport-based alignment loss and an uncertainty-aware prototype alignment loss collaboratively bridge source-target distribution gaps while maintaining class separability. Extensive experiments on PointDA-10 and GraspNetPC-10 benchmarks show that CLIPoint3D achieves consistent 3-16% accuracy gains over both CLIP-based and conventional encoder-based baselines. Project page: https://sarthakm320.github.io/CLIPoint3D.
Mainak Singha, Sarthak Mehrotra, Paolo Casari +3
University of Trento, Italy · MDSR Labs Adobe, India · IIT Bombay, India +1
Generalized few-shot 3D point cloud segmentation aims to adapt to novel classes from only a few annotations while maintaining strong performance on base classes, but this remains challenging due to the inherent stability-plasticity trade-off: adapting to novel classes can interfere with shared representations and cause base-class forgetting. We present HOP3D, a unified framework that learns hierarchical orthogonal prototypes with an entropy-based few-shot regularizer to enable robust novel-class adaptation without degrading base-class performance. HOP3D introduces hierarchical orthogonalization that decouples base and novel learning at both the gradient and representation levels, effectively mitigating base-novel interference. To further enhance adaptation under sparse supervision, we incorporate an entropy-based regularizer that leverages predictive uncertainty to refine prototype learning and promote balanced predictions. Extensive experiments on ScanNet200 and ScanNet++ demonstrate that HOP3D consistently outperforms state-of-the-art baselines under both 1-shot and 5-shot settings. The code is available at https://fdueblab-hop3d.github.io/.
Yifei Zhao, Fanyu Zhao, Zhongyuan Zhang +3
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China
Understanding spatial dynamics and semantics in point cloud is fundamental for comprehensive 3D comprehension. While reinforcement learning algorithms such as Group Relative Policy Optimization (GRPO) have recently achieved remarkable breakthroughs in large language models by incentivizing reasoning capabilities through strategic reward design, their potential remains largely unexplored in the 3D perception domain. This naturally raises a pivotal question: Can RL-based methods effectively empower 3D point cloud fine-tuning? In this paper, we propose PointRFT, the first reinforcement fine-tuning paradigm tailored specifically for point cloud representation learning. We select three prevalent 3D foundation models and devise specialized accuracy reward and dispersion reward functions to stabilize training and mitigate distribution shifts. Through comprehensive few-shot classification experiments comparing distinct training paradigms, we demonstrate that PointRFT consistently outperforms vanilla supervised fine-tuning (SFT) across diverse benchmarks. Furthermore, when organically integrated into a hybrid Pretraining-SFT-RFT paradigm, the representational capacity of point cloud foundation models is substantially unleashed, achieving state-of-the-art performance particularly under data-scarce scenarios.
Yankai Wang, Yiding Sun, Qirui Wang +3
School of Software Engineering, Xi’an Jiaotong University, Xi’an 710048, China · International School, Beijing University of Posts and Telecommunications, Beijing 100876, China