Joint-Embedding Prediction of Masked Point Tubes for Self-Supervised Learning on 4D Point Cloud Videos

Jheng-Ling Lee; Shang-Tse Chen

cs.CVAug 25, 2026

Joint-Embedding Prediction of Masked Point Tubes for Self-Supervised Learning on 4D Point Cloud Videos

Authors: Jheng-Ling Lee, Shang-Tse Chen

Organizations: Department of Computer Science and Information Engineering, National Taiwan University

Abstract

Self-supervised representation learning for 4D point cloud videos is challenging because annotations are costly and reconstruction-based pretraining can overemphasize low-level geometric details. We propose a JEPA-style framework that learns from unlabeled spatiotemporal point clouds through latent point-tube prediction. Instead of reconstructing raw coordinates, the model masks spatiotemporal regions and predicts their target representations from visible context representations in feature space. To stabilize latent prediction, we incorporate Sketched Isotropic Gaussian Regularization, which encourages non-collapsed embeddings without relying on explicit reconstruction targets. This formulation aims to capture both spatial structure and temporal dynamics while keeping the pretraining objective aligned with downstream semantic recognition. Experiments on action and gesture recognition benchmarks show that the learned representations improve downstream fine-tuning, limited-label learning, and cross-dataset transfer. These results suggest that JEPA-style latent prediction is a promising alternative to reconstruction-centered pretraining for 4D point cloud videos.

Figures & tables

Figure 1: Overview of the proposed pretraining framework. Point-tube tokens are split into visible context and masked targets; a predictor infers target latents while SIGReg regularizes the latent distribution.

Figure 2: Illustration of SIGReg. Random one-dimensional projections are drawn from the latent embedding distribution, and each projected distribution is matched to a Gaussian reference.

Supervised Learning Only
Method	Acc.
MeteorNet ( Liu et al. 2019 )	88.50
PSTNet ( Fan et al. 2021b )	91.20
PSTNet++ ( Fan et al. 2022 )	92.68
Kinet ( Zhong et al. 2022 )	93.27
PPTr ( Wen et al. 2022 )	92.33

Table 1: Action recognition accuracy (%) on MSRAction-3D. Baselines are grouped by training setting.

Supervised Learning Only
Method	Full	50% Labels
3DV-Motion ( Wang et al. 2020 )	84.5	–
3DV-PointNet++ ( Wang et al. 2020 )	88.8	–
PSTNet ( Fan et al. 2021b )	90.5	–
PSTNet++ ( Fan et al. 2022 )	91.4	–
Kinet ( Zhong et al. 2022 )	92.3	–

Table 2: Action recognition accuracy (%) on NTU RGB+D under the cross-subject protocol. The semi-supervised setting fine-tunes with 50% labeled training videos.

Initialization	1-shot	3-shot	5-shot
Scratch	$20.07{\pm}10.40$	$49.79{\pm}22.35$	$75.89{\pm}10.32$
Pretrained	$\mathbf{31.05{\pm}9.15}$	$\mathbf{63.41{\pm}5.43}$	$\mathbf{79.76{\pm}4.07}$
Gain	$+10.98$	$+13.62$	$+3.87$

Table 3: All-class few-shot P4Transformer video-level top-1 action recognition accuracy (%) on MSRAction-3D. Results are reported as mean

\pm

standard deviation over ten paired support-set runs.

Method	Epochs	Acc.
Supervised Learning Only
PointLSTM ( Min et al. 2020 )	–	87.6
PointLSTM-PSS ( Min et al. 2020 )	–	93.1
Kinet ( Zhong et al. 2022 )	–	95.2
P4Transformer ( Fan et al. 2021a )	30	87.5
P4Transformer ( Fan et al. 2021a )	50	91.2

Table 4: Transfer learning accuracy (%) on SHREC’17.

Figure 3: Qualitative visualization of attention maps from the MSR-pretrained encoder. Each panel contains the original point cloud sequence, the masked point cloud sequence, and the encoder attention map from top to bottom.

Variant	Acc.
Full model	94.08
w/o SIGReg	89.90
Stop-gradient targets	79.44
w/o target-position query	93.03
EMA target encoder	88.50

Table 5: Component/path ablation on MSRAction-3D.

Figure 4: Target-latent standard deviation over pretraining epochs for representative component-ablation variants on MSRAction-3D.

Setting	Value	Target Latent Std.	Acc.
Mask ratio	0.65	–	88.85
Mask ratio	0.75	–	94.08
Mask ratio	0.85	–	91.99
$\lambda_{\mathrm{SIGReg}}$	0.001	0.870	86.76
$\lambda_{\mathrm{SIGReg}}$	0.01	0.984	94.08
$\lambda_{\mathrm{SIGReg}}$	0.1	0.992	89.55

Table 6: Sensitivity ablations on MSRAction-3D.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Classes	Train/Test	Protocol
MSRAction-3D	20	270 / 297	S1–S5 / S6–S10
NTU RGB+D	60	40,320 / 16,560	Official
SHREC’17	28	1,960 / 840	Official

Appendix

Table 7: Dataset statistics and cross-subject split protocols used in the experiments. Counts are video-level sequence counts under the corresponding protocol.

Protocol	Epochs	Batch	LR	WD	Frames	Pts.	Clip Str.	Frame Str.	$r_{s}$	Sp. Str.	Nbrs.	SIGReg
MSR pretraining	200	96	3e-4	5e-2	24	1024	1	1	0.3	32	32	0.01
NTU pretraining	100	56	3e-4	5e-2	24	1024	2	2	0.1	32	32	0.03
MSR full fine-tuning	50	48	5e-4	1e-4	24	2048	1	1	0.7	32	32	–
NTU full/semi fine-tuning	20	48	5e-4	5e-2	24	2048	2	2	0.1	32	32	–
SHREC transfer fine-tuning	50	24	1e-3	1e-4	16	256	1	1	0.3	16	9	–
MSR few-shot fine-tuning	50	48	5e-4	1e-4	24	2048	1	1	0.7	32	32	–

Appendix

Table 8: Final optimization and point-tube sampling settings. LR and WD denote learning rate and weight decay; Frames denotes frames per input clip; Pts. denotes sampled points per frame; Clip Str. and Frame Str. denote the temporal strides between successive clips and successive frames within a clip, respectively. Sp. Str. and Nbrs. denote P4D spatial stride and neighbor samples. SIGReg weights apply only to self-supervised pretraining.

Setting	MSR	NTU	SHREC	Few-shot
Enc. dim	384	384	384	384
Enc. heads	8	8	8	8
Enc. depth	5	10	10	10
Enc. MLP	4	4	4	4
Pred. dim	192	192	–	–
Pred. heads	6	6	–	–

Appendix

Table 9: Architecture settings. Enc., Pred., and Cls. denote encoder, predictor, and classifier, respectively; dim, heads, depth, MLP, hidden, and dropout report the corresponding architectural hyperparameters. The NTU semi-supervised experiments use the same architecture as the full-data NTU experiments.

Variant	Pred. Loss	Target Std.	Acc.
Full model ( $m=0.75$ , $\lambda_{\mathrm{SIGReg}}=0.01$ )	0.001219	0.984	94.08
w/o SIGReg	0.000000	0.008	89.90
Stop-gradient targets	0.095069	0.993	79.44
w/o target-position query	0.002563	0.942	93.03
EMA target encoder	0.000452	0.388	88.50

Appendix

Table 10: Prediction-loss diagnostics for component/path ablations on MSRAction-3D.

Setting	Value	Pred. Loss	Target Std.	Acc.
Mask ratio $m$	0.65	0.001050	0.984	88.85
Mask ratio $m$	0.75	0.001219	0.984	94.08
Mask ratio $m$	0.85	0.001401	0.985	91.99
$\lambda_{\mathrm{SIGReg}}$	0.001	0.000179	0.870	86.76
$\lambda_{\mathrm{SIGReg}}$	0.01	0.001219	0.984	94.08
$\lambda_{\mathrm{SIGReg}}$	0.1	0.010965	0.992	89.55

Appendix

Table 11: Prediction-loss diagnostics for sensitivity ablations on MSRAction-3D.

	1-shot		3-shot		5-shot
Seed	Scratch	Pretrained	Scratch	Pretrained	Scratch	Pretrained
0	35.889	33.798	62.718	62.718	79.094	77.003
1	19.861	27.526	24.739	65.505	77.700	72.474
2	30.662	38.328	57.840	60.279	84.669	83.275
3	6.969	36.585	54.355	58.537	80.139	81.882
4	13.589	37.631	40.767	64.460	82.927	81.185

Appendix

Table 12: Per-seed all-class few-shot P4Transformer video-level top-1 accuracy (%) on MSRAction-3D. Means are reported with standard deviations over ten run seeds.

Self-Supervised Point Cloud Learning

See all 17 papers

Diffusion Masked Pretraining for Dynamic Point Cloud
May 5, 2026Zhuoyue Zhang, Yiding Sun, Haozhe Cheng +4Self-Supervised Pre-Training Point Cloud Learning
School of Software Engineering, Xi’an Jiaotong University, Xi’an, China · School of Artificial Intelligence and Robotics, Hunan University, Changsha, China · Department of Computer Science and Software Engineering, University of Western Australia, Perth, Australia
Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning
Jul 1, 2026Trung Thanh Nguyen, Hai Nguyen-Truong, Tu Vo +2Cross-Modal Knowledge Distillation Point Cloud Learning
Nagoya University, Japan · Northeastern University, USA · KC Machine Learning Lab, Korea +2
Motion-Saliency Complementary Masked Modeling for Point Cloud Video Understanding
Aug 31, 2026Wei Wang, Yiding Sun, Yuyan Wang +4Point Cloud Learning Generative Modeling
School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, China · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ), Shenzhen, 518107, China · School of Software Engineering, Xi’an Jiaotong University, Xi’an, 710049, China +4

May 5, 2026J/K move · Enter open · S save

Diffusion Masked Pretraining for Dynamic Point Cloud

Zhuoyue Zhang, Yiding Sun, Haozhe Cheng, Chaowei Fang+3

School of Software Engineering, Xi’an Jiaotong University, Xi’an, China · School of Artificial Intelligence and Robotics, Hunan University, Changsha, China · Department of Computer Science and Software Engineering, University of Western Australia, Perth, Australia

Self-Supervised Pre-Training Point Cloud Learning Point Cloud Understanding Diffusion Models Self-Supervised Point Cloud Learning

Open paper AskPDF arXiv

Dynamic point cloud pretraining is still dominated by masked reconstruction objectives. However, these objectives inherit two key limitations. Existing methods inject ground-truth tube centers as decoder positional embeddings, causing spatio-temporal positional leakage. Moreover, they supervise inter-frame motion with deterministic proxy targets that systematically discard distributional structure by collapsing multimodal trajectory uncertainty into conditional means. To address these limitations, we propose Diffusion Masked Pretraining (DiMP), a unified self-supervised framework for dynamic point clouds. DiMP introduces diffusion modeling into both positional inference and motion learning. It first applies forward diffusion noise only to masked tube centers, then predicts clean centers from visible spatio-temporal context. This removes positional leakage while preserving visible coordinates as clean temporal anchors. DiMP also reformulates point-wise inter-frame displacement supervision as a DDPM noise-prediction objective conditioned on decoded representations. This design drives the encoder to target the full conditional distribution of plausible motions under a variational surrogate, rather than collapsing to a single deterministic estimate. Extensive experiments demonstrate that DiMP consistently improves downstream accuracy over the backbone alone, with absolute gains of 11.21% on offline action segmentation and 13.65% under causally constrained online inference.Codes are available at https://github.com/InitalZ/DiMP.git.

Abstract

Self-Supervised Point Cloud Learning

Explore similar work

Diffusion Masked Pretraining for Dynamic Point Cloud

Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning

Motion-Saliency Complementary Masked Modeling for Point Cloud Video Understanding