PT$^3$: Prototype-Guided Self-Supervised Pretraining for Generalizable mmWave Human Point Cloud Representation

Ziyang Liu; Kieren Yu; Xiao Yang; Xin Yang; Kaishun Wu

cs.AIOct 28, 2026

PT $^3$ : Prototype-Guided Self-Supervised Pretraining for Generalizable mmWave Human Point Cloud Representation

Authors: Ziyang Liu, Kieren Yu, Xiao Yang, Xin Yang, Kaishun Wu

Organizations: The Hong Kong University of Science and Technology (Guangzhou)

Abstract

Multi-source millimeter-wave (mmWave) observations offer an opportunity to learn human representations reusable across sensing tasks. However, sparse detections and multipath effects make these observations unstable, challenging existing point cloud pretraining methods. We propose PT $^3$ , a prototype-guided self-supervised pretraining framework for mmWave human point clouds. PT $^3$ represents local states in spatial regions as sparse compositions of shared prototypes and learns their temporal evolution through autoregressive prediction. Observation consistency further stabilizes representations under radar perturbations. After pretraining on unlabeled sequences from multiple sources, the same frozen encoder supports action classification, dense point generation, joint pose estimation, and future motion prediction on datasets not used for pretraining. Across these tasks, PT $^3$ outperforms the compared pretraining baselines and achieves competitive or superior performance to task-specific models.

Figures & tables

Figure 2: Overview of PT 3 . Left: self-supervised learning of regional prototype compositions and their temporal evolution. Right: radar point cloud encoding into spatial slots shared by pretraining and downstream tasks.

Dataset		Modalities			Annotations			Statistics
Name	RF-BW (GHz)	RGB	Depth	mmWave	Actions	Skeleton	Mesh	# Act.	# Sub.	# Seq.	# Frame
M4Human	62–69	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	50	20	999	661K
MARS	77–80.2	–	–	$\checkmark$	$\checkmark$	$\checkmark$	–	10	4	80	40K
milliFlow	62–63.6	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	–	8	12	–	43.2K
MiliPoint	77–81	–	–	$\checkmark$	$\checkmark$	$\checkmark$	–	49	11	–	545K
DGUHA	76–81	–	–	$\checkmark$	$\checkmark$	$\checkmark$	–	7	19	–	–

Table 1: Statistics of the mmWave human datasets used in our experiments. RF-BW reports the radar chirp operation band. The upper block lists pretraining data; the lower block covers downstream evaluation. † UniMM-HAR unifies mRI, MM-Fi, and RadHAR for action classification.

Table 3

	Random		Cross-Subject		Cross-Environment
Method	MPJPE $\downarrow$	PA $\downarrow$	MPJPE $\downarrow$	PA $\downarrow$	MPJPE $\downarrow$	PA $\downarrow$
PointTransformer	73.09	55.60	75.96	58.70	88.28	68.79
DiffPose	73.44	56.83	70.31	54.12	86.35	66.87
mmDiff	68.62	53.11	68.46	52.55	85.63	66.43
PointMAE	70.91	52.71	70.15	53.77	87.48	67.69
PointBERT	70.80	52.90	70.10	53.06	83.32	63.46

Table 4: Joint pose estimation on MM-Fi. We report MPJPE and PA-MPJPE (PA) in mm, under the Random, Cross-Subject and Cross-Environment protocols.

	Chest Expand		Side Twist		Raise Hand		Pickup		Throwing		Kicking		Bowing
Method	ADE $\downarrow$	FDE $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	ADE $\downarrow$	FDE $\downarrow$	ADE $\downarrow$	FDE $\downarrow$
SPGSN	0.402	0.422	0.430	0.445	0.397	0.446	0.595	0.686	0.449	0.517	0.426	0.449	0.336	0.395
HumanMAC	0.353	0.306	0.407	0.392	0.363	0.333	0.547	0.578	0.439	0.471	0.425	0.421	0.304	0.314
mmPred	0.272	0.230	0.315	0.308	0.237	0.218	0.452	0.481	0.371	0.370	0.374	0.368	0.246	0.234
PointMAE	0.328	0.279	0.431	0.393	0.290	0.251	0.460	0.448	0.413	0.410	0.355	0.350	0.248	0.246
PointBERT	0.310	0.263	0.376	0.343	0.275	0.246	0.449	0.447	0.394	0.399	0.356	0.354	0.240	0.242

Table 5: Future motion prediction on MM-Fi for different actions, evaluated by ADE and FDE.

	HAR		Dense Point Gen.		Joint pose estimation (MPJPE $\downarrow$ / PA $\downarrow$ )			Motion Pre.
Variant	C-Sub $\uparrow$	C-Set $\uparrow$	CD $\downarrow$	EMD $\downarrow$	Random	C-Sub	C-Env	ADE $\downarrow$	FDE $\downarrow$
Full model	81.35	85.23	0.91	4.70	66.74 / 50.95	65.75 / 50.92	77.43 / 58.93	0.313	0.299
w/o $L_{\mathrm{con}}$	82.28	81.79	0.95	5.15	91.04 / 73.56	92.30 / 74.70	100.62 / 80.29	0.379	0.364
w/o Prototype (Slot-AR)	78.70	80.66	0.91	5.10	89.26 / 71.35	90.89 / 72.90	98.81 / 78.49	0.367	0.349
w/o $L_{\mathrm{AR}}$	78.53	80.28	0.95	4.83	82.03 / 64.87	83.74 / 65.99	92.59 / 72.04	0.370	0.353

Table 6: Pre-training component ablation across four downstream tasks.

Figure 3: JS divergence between prototype compositions. Left: between actions, averaged over subjects. Right: between subjects, averaged over actions. Darker cells larger differences.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Protocol	Data Sources	Split Criterion	#Act.	#Train	#Test	#Total
C-Sub	mRI, MM-Fi	Subject split (5:5)	25	10,053	9,604	19,657
C-Set	RadHAR, mRI, MM-Fi	Scene split (4:2)	27	19,603	8,437	28,040

Appendix

Table 9: Summary of the two UniMM-HAR evaluation protocols. #Act. denotes the number of action categories, while #Train, #Test, and #Total denote the numbers of training, test, and total samples, respectively.

Input	Accuracy (%)	Weighted F1 (%)	Correct / Total
ZP	91.30	91.19	147 / 161
GN	86.96	86.98	140 / 161
AHC	94.41	94.52	152 / 161

Appendix

Table 10: Action classification on DGUHA with a frozen PT 3 encoder and a Ridge-regression linear probe. ZP, GN, and AHC are the dataset’s three processed point-cloud inputs.

Figure 4: Sensitivity to the number of selected prototypes per slot

k

. The four panels show action classification, dense point generation, joint pose estimation, and future motion prediction from left to right. The vertical axis gives relative improvement (%) over the default

k=4

; positive values indicate better performance.

Figure 5: Sensitivity to prototype bank size

K

K=64

; positive values indicate better performance.

Figure 6: Sensitivity to the number of spatial slots

S

S=12

; positive values indicate better performance.

Figure 7: Action prototype heatmap. Slot–prototype composition of one representative subject on MM-Fi, averaged over the temporal windows of each action and shown relative to the subject’s mean assignment over the six actions. Each panel corresponds to one action, with slots along one axis and prototypes along the other.

Figure 8: Prototype-composition differences measured by Jensen–Shannon (JS) divergence. Left: both axes list 14 action IDs, and each cell averages an action-pair distance over 10 subjects. Right: both axes list 10 subject IDs, and each cell averages a subject-pair distance over 14 actions. Both color bars encode JS divergence: pale yellow indicates smaller differences and dark red indicates larger differences. Diagonal cells are zero because each item is compared with itself.

Figure 9: Qualitative joint pose estimation results on six action categories: A02 chest expand, A04 side twist to the left, A13 raise the left hand, A19 pick up an object, A22 kick to the left, and A27 bow.

Figure 10: Qualitative future motion prediction results on actions A02, A03 and A13. From top to bottom: the input mmWave point cloud, the reference current pose, the predicted future pose, and the reference future pose.

Figure 11: Qualitative future motion prediction results on actions A14, A17 and A21.

Self-Supervised Point Cloud Learning

See all 18 papers

Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation
Jul 21, 2026Bo Liang, Chen Gong, Wei Gao +1Millimeter-Wave Radar mmWave Radar Sensing
School of Computer Science, Peking University · University of Pittsburgh · Key Laboratory of High Confidence Software Technologies, Ministry of Education (PKU)
Diffusion Masked Pretraining for Dynamic Point Cloud
May 5, 2026Zhuoyue Zhang, Yiding Sun, Haozhe Cheng +4Self-Supervised Pre-Training Point Cloud Learning
School of Software Engineering, Xi’an Jiaotong University, Xi’an, China · School of Artificial Intelligence and Robotics, Hunan University, Changsha, China · Department of Computer Science and Software Engineering, University of Western Australia, Perth, Australia
WiCompass: Oracle-driven Data Scaling for mmWave Human Pose Estimation
Feb 21, 2026Bo Liang, Chen Gong, Haobo Wang +8Human Pose Estimation OOD Generalization
School of Computer Science, Peking University · University of Electronic Science and Technology of China · University of Pittsburgh +2

Jul 21, 2026J/K move · Enter open · S save

Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation

Bo Liang, Chen Gong, Wei Gao, Chenren Xu

School of Computer Science, Peking University · University of Pittsburgh · Key Laboratory of High Confidence Software Technologies, Ministry of Education (PKU)

Millimeter-Wave Radar mmWave Radar Sensing 3D Human Pose Estimation Human Pose Estimation

Open paper AskPDF arXiv

Millimeter-wave (mmWave) radar enables privacy-friendly human sensing, but its sparse point clouds are physical measurements of view-dependent electromagnetic reflections and only indirectly characterize body articulation. Recovering a complete 3D pose from such partial, geometry-dependent observations is therefore under-constrained. Existing methods directly regress joint coordinates from paired radar-pose data, relying on the same limited paired supervision to learn radar perception, human-body structure, and their alignment. This coupling can encourage dataset-specific shortcuts under ambiguous radar observations. We propose Wave2Body, a radar-to-body token translation framework that decouples these learning targets using a self-supervised mmWave tokenizer, a pretrained compositional body tokenizer that defines the output space, and a lightweight translator between them. Experiments on M4Human and mmBody show that Wave2Body achieves stronger cross-domain generalization than previous methods while incurring much lower computational costs for training and inference. All the code and experiment results are publicly available at https://github.com/Galaxywalk/Wave2Body.

Abstract

Self-Supervised Point Cloud Learning

Explore similar work

Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation

Diffusion Masked Pretraining for Dynamic Point Cloud

WiCompass: Oracle-driven Data Scaling for mmWave Human Pose Estimation