Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perception

Yiding Sun; Jihua Zhu; Haozhe Cheng; Chaoyi Lu; Zhichuan Yang; Lin Chen; Yaonan Wang

cs.CVFeb 26, 2026

Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perception

Authors: Yiding Sun, Jihua Zhu, Haozhe Cheng, Chaoyi Lu, Zhichuan Yang, Lin Chen, Yaonan Wang

Organizations: the School of Software Engineering, Xi’an Jiaotong University, Xi’an 710048, China · the State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University, Xi’an 710048, China · the School of Electrical and Information Engineering, Hunan University, Changsha, 410082, China · the National Engineering Research Center for Robot Visual Perception and Control Technology, Changsha 410082, China

Abstract

Point cloud video understanding is critical for robotics as it accurately encodes motion and scene interaction. We recognize that 4D datasets are far scarcer than 3D ones, which hampers the scalability of self-supervised 4D models. A promising alternative is to transfer 3D pre-trained models to 4D perception tasks. However, rigorous empirical analysis reveals two critical limitations that impede transfer capability: overfitting and the modality gap. To overcome these challenges, we develop a novel "Align then Adapt" (PointATA) paradigm that decomposes parameter-efficient transfer learning into two sequential stages. Optimal-transport theory is employed to quantify the distributional discrepancy between 3D and 4D datasets, enabling our proposed point align embedder to be trained in Stage 1 to alleviate the underlying modality gap. To mitigate overfitting, an efficient point-video adapter and a spatial-context encoder are integrated into the frozen 3D backbone to enhance temporal modeling capacity in Stage 2. Notably, with the above engineering-oriented designs, PointATA enables a pre-trained 3D model without temporal knowledge to reason about dynamic video content at a smaller parameter cost compared to previous work. Extensive experiments show that PointATA can match or even outperform strong full fine-tuning models, whilst enjoying the advantage of parameter efficiency, e.g. 97.21 % accuracy on 3D action recognition, $+8.7 \%$ on 4 D action segmentation, and 84.06% on 4D semantic segmentation.

Figures & tables

Fig. 1: Current 4D PETL methods face two limits. Upper panel : Adapters are feasible (Smiley), yet current methods expose models to severe overfitting (Crying). Lower panel : Cross-modal transfer needs prior alignment. This practice is mature in 2D Vision and NLP. But to our knowledge, no 4D PETL study measures this gap (Crying). Without considering the gap, it will hurt downstream performance undoubtedly (Crying).

Fig. 2: Comparison of 3D full fine-tuning, 4D full fine-tuning, 4D adapter tuning, and our PointATA. Considering that the embedder required for 4D encoding is often heavier than that for 3D, when facing a 4D dynamic perception task, the number of parameters that need to be updated is even greater (over 100%) than the full update for 3D. PointATA saves large amount of resource and time than full fine-tuning and significantly boosts reuse of 3D pre-trained models. It also exploits 3D priors better than 4D adapter tuning and further cuts parameters to curb overfitting.

Fig. 3: PointATA employs a two-stage workflow to quickly adapt large 3D pre-trained models to diverse 4D downstream tasks. In Stage 1, it obtains 4D features via the P4D embedder and learns by minimizing the distribution distance to 3D source features. The weight of P4D embedder is randomly initialized. In Stage 2, it jointly fine-tunes the P4D embedder and the PVA to minimize task loss. The 3D backbone remains frozen throughout.

Supervised Learning
Methods	Reference	8 Frames	12 Frames	16 Frames	24 Frames	32 Frames	36 Frames
MeteorNet [ 43 ]	ICCV2019	81.14	86.53	88.21	88.50	N/A	N/A
PSTNet [ 44 ]	ICLR2021	83.50	87.88	89.90	91.20	N/A	N/A
P4Transformer [ 31 ]	CVPR2021	83.17	87.54	89.56	90.94	87.93	82.81
Kinet [ 45 ]	CVPR2022	83.84	88.53	91.92	93.27	N/A	N/A
PPTr [ 32 ]	ECCV2022	84.02	89.89	90.31	92.33	N/A	N/A

TABLE I: Action recognition accuracy(%)on the MSR-Action3D.

Method	Input	NTU 60		NTU 120
Method	Input	Subject	View	Subject	Setup
NTU Baseline [ 50 ]	D.	N/A	N/A	48.7	40.1
PointNet++ [ 51 ]	P.	80.1	85.1	72.1	79.4
3DV [ 52 ]	V.	84.5	95.4	76.9	92.5
3DV-PN++ [ 52 ]	V+P	88.8	96.3	82.4	93.5
PSTNet [ 44 ]	P.	90.5	96.5	87.0	93.8

TABLE II: 3D Action Recognition Accuracy (%) on NTU RGB+D 60 and NTU RGB+D 120. D. means depth. P. means Point. V. means voxel.

Supervised Learning
Method	Reference	Acc	Edit	F1@50
P4Transformer [ 31 ]	CVPR2021	71.2	73.1	58.2
PPTr [ 32 ]	ECCV2022	77.4	80.1	69.5
MAMBA4D [ 3 ]	CVPR2025	85.5	91.3	85.5
Self-Supervised Learning (End-to-End Fine-Tuning)
P4Trans+C2P [ 33 ]	CVPR2023	73.5	76.8	62.4

TABLE III: Action segmentation on the HOI4D dataset. Acc means Accuracy (%).

Fig. 4: Visualization of action segmentation. P4Transformer has a serious over-segmentation problem.

Method	Reference	Input	Frame	mIoU (%)
3D MinkNet14 [ 19 ]	CVPR2019	voxel	1	76.24
4D MinkNet14 [ 19 ]	CVPR2019	voxel	3	77.46
MeteorNet-M [ 43 ]	ICCV2019	point	2	81.47
MeteorNet-L [ 43 ]	ICCV2019	point	3	81.80
PSTNet [ 44 ]	ICLR2021	point	3	82.24
P4Transformer [ 31 ]	CVPR2021	point	1	82.41

TABLE IV: 4D semantic segmentation results on Synthia 4D dataset.

Supervised Learning
Methods	Reference	Acc.(%)
PLSTM-base [ 54 ]	CVPR2020	87.6
PLSTM-early [ 54 ]	CVPR2020	93.5
PLSTM-PSS [ 54 ]	CVPR2020	93.1
PLSTM-middle [ 54 ]	CVPR2020	94.7
PLSTM-late [ 54 ]	CVPR2020	93.5

TABLE V: Gesture recognition accuracy (%) on the SHREC’17.

Methods	Reference	EPE3D	Acc3DS	Acc3DR	Outliers
FlowNet3D [ 55 ]	CVPR2019	0.169	0.254	0.579	0.789
FLOT [ 56 ]	ECCV2020	0.110	0.419	0.721	0.486
OGSFNet [ 57 ]	CVPR2021	0.075	0.706	0.869	0.327
FESTA [ 58 ]	CVPR2021	0.097	0.449	0.833	N/A
3DFlow [ 59 ]	ECCV2022	0.073	0.819	0.890	0.261
BPF [ 60 ]	ECCV2022	0.065	0.769	0.906	0.264

TABLE VI: Comparison results on KITTI. Metrics are EPE, outlier ratio (

>0.3\mathrm{~m}

5\%

), Acc3DS and Acc3DR.

Fig. 5: 4D semantic segmentation visualization on the Synthia 4D dataset. Key points are demarcated by red dashed circular bounding boxes.

Fig. 6: Ablation studies of PointATA under various experimental settings.

Model	Clip T.	Training T.	Test T.	GPU Hours
PointCSA	189.3	43	34	6.8
PointATA	238.4	36	22	5.6
-Stage 1	89.2	N/A	N/A	N/A
-Stage 2	149.2	N/A	N/A	N/A
$\Delta$	$\color[rgb]{0,0,1}+25.9\%$	$\color[rgb]{0,0,1}-16.3\%$	$\color[rgb]{0,0,1}-35.3\%$	$\color[rgb]{0,0,1}-17.6\%$

TABLE VII: Training and inference advantages of PointATA over baseline. All values are recorded in seconds. Results are averaged over 10 runs. T. means time.

Model	PAE	SCE	PVA	Accuracy (%)
A0	$\times$	$\times$	$\times$	$77.35_{\color[rgb]{1,0,0}(baseline)}$
A1	$\checkmark$	$\times$	$\times$	$84.51_{\color[rgb]{0,0,1}(+7.16)}$
A2	$\checkmark$	$\checkmark$	$\times$	$88.55_{\color[rgb]{0,0,1}(+11.2)}$
A3	$\checkmark$	$\times$	$\checkmark$	$95.12_{\color[rgb]{0,0,1}(+17.77)}$
A4 (ours)	$\checkmark$	$\checkmark$	$\checkmark$	$\textbf{95.62}_{\color[rgb]{0,0,1}(+18.27)}$

TABLE VIII: Ablation studies on architecture designs. PAE denotes Point Align Embedder, and SCE denotes additional Spatial Context Encoders.

Model	OTDD	Epoch Number	Sub-Sampling	Accuracy (%)
B0	$\times$	N/A	N/A	$92.68_{\color[rgb]{1,0,0}(baseline)}$
B1	$\checkmark$	2	16	$93.37_{\color[rgb]{0,0,1}(+0.69)}$
B2	$\checkmark$	4	16	$94.42_{\color[rgb]{0,0,1}(+1.74)}$
B3	$\checkmark$	6	32	$94.77_{\color[rgb]{0,0,1}(+2.09)}$
B4	$\checkmark$	8	32	$95.12_{\color[rgb]{0,0,1}(+2.44)}$
B5	$\checkmark$	10	32	$95.62_{\color[rgb]{0,0,1}(+2.94)}$

TABLE IX: Effects of Epoch Number and Sub-sampling strategy for Class-Weighted Stochastic OTDD on Transfer Learning Performance.

Model	Reference	Points	Clips	Accuracy (%)
PTM [ 66 ]	TCSVT2024	2048	32 F.	95.12
		4096	32 F.	95.47
		2048	36 F.	96.51
		4096	36 F.	96.86
PointM2AE [ 67 ]	NeurIPS2022	2048	32 F.	94.42
		4096	32 F.	94.77

TABLE X: Scaling behavior of 3D backbones with PointATA. F. means frames per clip.

Fig. 7: Training log visualization. We consider 50 epochs for tuning is excessive. For a fair comparison with naive tuning, we shorten the schedule to 10 epochs for the embedding layer followed by 40 epochs of parameter-efficient fine-tuning.

Fig. 8: Different location of Point Video Adapter and Spatial Context Encoder.

Fig. 9: (a) Comparison of different tuning strategies. (b) Discrepancy between training and test accuracy under 16 frames on MSR-Action3D.

Fig. 10: t-SNE visualization showing that aligning static and dynamic clouds with the Point Align Embedder is essential. Smaller indices indicate tighter clusters. After alignment, intra-class distances shrink and inter-class gaps widen.

Explore similar work

Tri-Efficient Transfer Learning for Point Cloud Videos
Jun 23, 2026Yiding Sun, Dongxu Zhang, Jihua Zhu +6Point Cloud Understanding Self-Supervised Visual Representation Learning
School of Software Engineering, Xi’an Jiaotong University · School of Information and Communication Engineering, Xi’an Jiaotong University · SIGS, Tsinghua University
Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning
Jul 1, 2026Trung Thanh Nguyen, Hai Nguyen-Truong, Tu Vo +2Cross-Modal Knowledge Distillation Point Cloud Learning
Nagoya University, Japan · Northeastern University, USA · KC Machine Learning Lab, Korea +2
GATS: Gaussian Aware Temporal Scaling Transformer for Invariant 4D Spatio-Temporal Point Cloud Representation
Mar 17, 2026Jiayi Tian, Jiaze WangPoint Cloud Learning Invariant Representation Learning
State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Harbin Institute of Technology, Shenzhen, Pengcheng Laboratory

Jun 23, 2026J/K move · Enter open · S save

Tri-Efficient Transfer Learning for Point Cloud Videos

Yiding Sun, Dongxu Zhang, Jihua Zhu, Haozhe Cheng+5

School of Software Engineering, Xi’an Jiaotong University · School of Information and Communication Engineering, Xi’an Jiaotong University · SIGS, Tsinghua University

Point Cloud Understanding Self-Supervised Visual Representation Learning Parameter-Efficient Fine-Tuning Data-Efficient Learning

Open paper AskPDF arXiv

While point cloud foundation models have significantly advanced point cloud video understanding, existing parameter-efficient fine-tuning (PEFT) methods still suffer from two critical limitations: prohibitive annotation costs for large-scale point cloud datasets and severe memory bottlenecks. In this paper, we aim to mine richer supervision signals from existing data rather than blindly scaling datasets. A further key principle is that the memory footprint of fine-tuning must be drastically reduced compared to full fine-tuning, which remains elusive for current PEFT techniques. Driven by these challenges, we identify three core desiderata: data-, parameter-, and memory efficiency, and present PoinTriE, a unified framework that excels along all three dimensions. For pre-training, pseudo-motion trajectories are synthesized via rigid transformations, paired with text corpora and 2D projections derived from raw point clouds. We then propose a Geometric-Motion Duality Network optimized via multimodal contrastive learning, rigid rotation prediction, and motion distribution divergence to produce dense self-supervision. During fine-tuning, we freeze the pretrained backbone and only update a lightweight Spatio-temporal Side Network built with LoRA units. Equipped with a gradient flow masking strategy, PoinTriE simultaneously reduces memory consumption and parameter overhead. Extensive experiments confirm that PoinTriE establishes new state-of-the-art results on action recognition and semantic segmentation tasks.

Abstract

Explore similar work

Tri-Efficient Transfer Learning for Point Cloud Videos

Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning

GATS: Gaussian Aware Temporal Scaling Transformer for Invariant 4D Spatio-Temporal Point Cloud Representation