PointRFT: Explicit Reinforcement Fine-tuning for Point Cloud Few-shot Learning

Yankai Wang; Yiding Sun; Qirui Wang; Pengbo Li; Chaoyi Lu; Dongxu Zhang

cs.CVMar 25, 2026

PointRFT: Explicit Reinforcement Fine-tuning for Point Cloud Few-shot Learning

Authors: Yankai Wang, Yiding Sun, Qirui Wang, Pengbo Li, Chaoyi Lu, Dongxu Zhang

Organizations: School of Software Engineering, Xi’an Jiaotong University, Xi’an 710048, China · International School, Beijing University of Posts and Telecommunications, Beijing 100876, China

Abstract

Understanding spatial dynamics and semantics in point cloud is fundamental for comprehensive 3D comprehension. While reinforcement learning algorithms such as Group Relative Policy Optimization (GRPO) have recently achieved remarkable breakthroughs in large language models by incentivizing reasoning capabilities through strategic reward design, their potential remains largely unexplored in the 3D perception domain. This naturally raises a pivotal question: Can RL-based methods effectively empower 3D point cloud fine-tuning? In this paper, we propose PointRFT, the first reinforcement fine-tuning paradigm tailored specifically for point cloud representation learning. We select three prevalent 3D foundation models and devise specialized accuracy reward and dispersion reward functions to stabilize training and mitigate distribution shifts. Through comprehensive few-shot classification experiments comparing distinct training paradigms, we demonstrate that PointRFT consistently outperforms vanilla supervised fine-tuning (SFT) across diverse benchmarks. Furthermore, when organically integrated into a hybrid Pretraining-SFT-RFT paradigm, the representational capacity of point cloud foundation models is substantially unleashed, achieving state-of-the-art performance particularly under data-scarce scenarios.

Figures & tables

Fig. 1: Schematic illustration of our PointRFT. (a) SFT frequently provokes catastrophic forgetting, particularly when knowledge is transferred across domains. Reinforcement fine-tuning mitigates this decay and equips the backbone with broader generalization. (b) Grounded in this behavior, we introduce three reusable paradigms: Pre-S, Pre-R, and Pre-S-R, and benchmark them on multiple few-shot classification tasks using three prevalent 3D foundation models.

Fig. 2: Framework of PointRFT. Following the vanilla point cloud fine-tuning pipeline, the input point cloud is fed into a pre-trained base model and fine-tuned via Eq. 1 . For RL fine-tuning, we treat the base model before the update as the detached reference policy and the updated base model as the actively optimized policy to stabilize training. Building on this, we propose PointRFT, which trains the foundation model to maximize reward via Alg. 1 . At each epoch, the parameters of the old model are updated and incorporated into the loss without gradient backflow. Since RFT and SFT share the same labeled data and inputs, the two paradigms do not conflict. In other words, we can maximize the downstream performance of the base model by first applying SFT and then RFT.

Method	Paradigm	ScanObjectNN
		5-way 1-shot			5-way 5-shot
		Split 1	Split 2	Split 3	Split 1	Split 2	Split 3
Point-MAE [ 24 ]	Pre-S (10%)	27.85 $\pm$ 0.62	29.91 $\pm$ 0.68	33.62 $\pm$ 0.87	63.77 $\pm$ 0.83	65.42 $\pm$ 1.37	67.08 $\pm$ 0.45
	Pre-S	41.08 $\pm$ 1.93	45.56 $\pm$ 1.04	49.33 $\pm$ 1.12	70.19 $\pm$ 1.94	72.64 $\pm$ 1.05	74.55 $\pm$ 1.26
	Pre-R (10%)	28.03 $\pm$ 0.91	29.76 $\pm$ 0.38	36.54 $\pm$ 0.87	67.08 $\pm$ 0.45	67.33 $\pm$ 1.72	68.91 $\pm$ 0.91
	Pre-R	42.00 $\pm$ 1.59	47.22 $\pm$ 0.70	50.68 $\pm$ 1.44	72.64 $\pm$ 1.05	76.25 $\pm$ 0.67	77.48 $\pm$ 1.59

TABLE I: Few-shot learning result on the ScanObjectNN Dataset. We conduct this analysis across three distinct splits for both 5-way 1-shot and 5-way 5-shot scenarios. The performance metrics are reported as accuracy percentages, accompanied by their respective standard deviations. The darker the color, the higher the result.

Method	Paradigm	ModelNet40
Method	Paradigm	10-way 10-shot	10-way 20-shot
Point-MAE [ 24 ]	Pre-S (10%)	76.6 $\pm$ 4.3	79.9 $\pm$ 2.1
	Pre-S	92.6 $\pm$ 4.1	95.0 $\pm$ 3.0
	Pre-R (10%)	83.6 $\pm$ 5.3	84.9 $\pm$ 2.1
	Pre-R	92.3 $\pm$ 4.5	95.0 $\pm$ 3.0
	Pre-S-R	94.2 $\pm$ 3.3	95.6 $\pm$ 2.8

TABLE II: Few-shot learning result on the ModelNet40 Dataset. We conduct this analysis for both 10-way 10-shot and 10-way 20-shot scenarios. We conduct further experiments that use 10% of the training time for comparison.

Method	Paradigm	ShapeNetCore
Method	Paradigm	10-way 1-shot	10-way 5-shot
Point-MAE [ 24 ]	Pre-S (10%)	56.08 $\pm$ 0.85	68.27 $\pm$ 1.36
	Pre-S	59.83 $\pm$ 0.54	72.45 $\pm$ 1.78
	Pre-R (10%)	53.91 $\pm$ 0.42	66.18 $\pm$ 1.09
	Pre-R	66.70 $\pm$ 0.87	71.52 $\pm$ 1.63
	Pre-S-R	70.34 $\pm$ 0.95	73.09 $\pm$ 1.21

TABLE III: Few-shot learning result on the ShapeNetCore Dataset. We conduct this analysis for both 10-way 1-shot and 10-way 5-shot scenarios. We conduct further experiments that use 10% of the training time for comparison.

Dataset	Epsilon	Dispersion Reward	Accuracy
ModelNet40 [ 32 ]	$\varepsilon=0.2$	set to 0	83.4 $\pm$ 4.9
		a=1,b=1	89.2 $\pm$ 3.3
		Eq. 6	92.6 $\pm$ 3.7
ShapeNetCore [ 1 ]	$\varepsilon=0.3$	set to 0	56.12 $\pm$ 1.93
		a=1,b=1	60.66 $\pm$ 1.89
		Eq. 6	65.42 $\pm$ 1.15

TABLE IV: Comparison in terms of accuracy with different dispersion reward function and hyperparameter settings. “set to 0 ” means only an accuracy reward is applied. 10-way 10-shot, 10-way 1-shot, and 5-way 1-shot are tested respectively.

Fig. 3: Comparison of computational costs in different datasets. We report the GLOPs and training time (s) for each epoch. 10-way 10-shot, 10-way 1-shot and 5-way 1-shot are selected respectively.

Point Cloud Learning

Point Cloud Learning develops methods for representing and interpreting unordered 3D point sets from LiDAR, depth sensors, radar, and 3D models.

See all 156 papers

P3T: Prototypical Point-level Prompt Tuning with Enhanced Generalization for 3D Vision-Language Models
Apr 17, 2026Geunyoung Jung, Soohong Kim, Kyungwoo Song +1Point Cloud Classification VLM Adaptation
Department of Artificial Intelligence, University of Seoul, South Korea · Department of Applied Statistics, Yonsei University, South Korea
Refine Then Fusion: Training-Free 3D Point Cloud Adaptation with Priority Refinement and Multi-Modal Knowledge Fusion
Sep 18, 2026Hang Cheng, Yan Chen, Mingyu Fan +1Few-Shot Learning Point Cloud Learning
Tsinghua Shenzhen International Graduate School, Tsinghua University
Partition-Invariant Tuning for 3D Scene Understanding
Sep 11, 2026Hongqiang Lin, Tianle Wang, Shuiwang Li +4Point Cloud Learning Parameter-Efficient Fine-Tuning
ZJU · GUT · THU +3

Apr 17, 2026J/K move · Enter open · S save

P3T: Prototypical Point-level Prompt Tuning with Enhanced Generalization for 3D Vision-Language Models

Geunyoung Jung, Soohong Kim, Kyungwoo Song, Jiyoung Jung

Department of Artificial Intelligence, University of Seoul, South Korea · Department of Applied Statistics, Yonsei University, South Korea

Point Cloud Classification VLM Adaptation Domain Generalization Visual Prompt Learning 3D VLMs

Open paper AskPDF arXiv

With the rise of pre-trained models in the 3D point cloud domain for a wide range of real-world applications, adapting them to downstream tasks has become increasingly important. However, conventional full fine-tuning methods are computationally expensive and storage-intensive. Although prompt tuning has emerged as an efficient alternative, it often suffers from overfitting, thereby compromising generalization capability. To address this issue, we propose Prototypical Point-level Prompt Tuning (P

^3

T), a parameter-efficient prompt tuning method designed for pre-trained 3D vision-language models (VLMs). P

^3

T consists of two components: 1) Point Prompter, which generates instance-aware point-level prompts for the input point cloud, and 2) Text Prompter, which employs learnable prompts into the input text instead of hand-crafted ones. Since both prompters operate directly on input data, P

^3

T enables task-specific adaptation of 3D VLMs without sacrificing generalizability. Furthermore, to enhance embedding space alignment, which is key to fine-tuning 3D VLMs, we introduce a prototypical loss that reduces intra-category variance. Extensive experiments demonstrate that our method matches or outperforms full fine-tuning in classification and few-shot learning, and further exhibits robust generalization under data shift in the cross-dataset setting. The code is available at \textcolor{violet}{https://github.com/gyjung975/P3T}.

Abstract

Point Cloud Learning

Explore similar work

P3T: Prototypical Point-level Prompt Tuning with Enhanced Generalization for 3D Vision-Language Models

Refine Then Fusion: Training-Free 3D Point Cloud Adaptation with Priority Refinement and Multi-Modal Knowledge Fusion

Partition-Invariant Tuning for 3D Scene Understanding