Expanding mmWave Datasets for Human Pose Estimation with Unlabeled Data and LiDAR Datasets

Zhuoxuan Peng; Boan Zhu; Xingjian Zhang; Wenying Li; S. -H. Gary Chan

cs.CVMar 15, 2026

Expanding mmWave Datasets for Human Pose Estimation with Unlabeled Data and LiDAR Datasets

Authors: Zhuoxuan Peng, Boan Zhu, Xingjian Zhang, Wenying Li, S. -H. Gary Chan

Organizations: The Hong Kong University of Science and Technology

Abstract

Current millimeter-wave (mmWave) datasets for human pose estimation (HPE) are scarce and lack diversity in both point cloud (PC) attributes and human poses, hindering the generalization ability of their trained models. On the other hand, unlabeled mmWave HPE data and diverse LiDAR HPE datasets are readily available. We propose EMDUL, a novel approach to expand the volume and diversity of an existing mmWave dataset using unlabeled mmWave data and LiDAR datasets. EMDUL consists of two independent modules, namely a pseudo-label estimator to annotate unlabeled mmWave data, and a closed-form converter that translates an annotated LiDAR PC to its mmWave counterpart. Expanding the original dataset with both LiDAR-converted and pseudo-labeled mmWave PCs significantly boosts the performance and generalization ability of all the examined HPE models, reducing 15.1% and 18.9% error for in-domain and out-of-domain settings, respectively. Code is available at https://github.com/Shimmer93/EMDUL.

Figures & tables

Panel of Figure 1 — Figure 1: Examples illustrating the effect of dataset expansion. (a) Samples from an mmWave HPE training dataset. (b) Samples from a LiDAR dataset with richer pose diversity used for dataset expansion; (c) An mmWave PC from an unseen scenario. (d) The ground-truth skeleton. (e) The predicted skeleton of SOTA P4T [ 11 ] without expansion. (f) The predicted skeleton of P4T trained on EMDUL-expanded dataset. Joints are colored red for errors $>10\,\text{cm}$ and green otherwise. EMDUL achieves stronger generalization ability than the baseline P4T.

Figure 2: The overview of EMDUL integrating both PC conversion and pseudo-labeling modules.

Figure 3: Illustration of the motion-detection mechanism in mmWave radar using an MM-Fi sample. Joints with high flow (yellow) lie close to detected points, while low-flow joints (dark blue) have no nearby points.

Panel of Figure 4 — Figure 4: Step-by-step visualization of the point-cloud (PC) conversion pipeline Blue joints have lower flow magnitudes and yellow joints higher ones.

Figure 5: Sample point clouds from different mmWave and LiDAR HPE datasets and the standardized 15-keypoint skeleton structure used in this paper.

Method	Backbone	F $\rightarrow$ F		F $\rightarrow$ B		B $\rightarrow$ F		B $\rightarrow$ B
Method	Backbone	MPJPE	PA-MPJPE	MPJPE	PA-MPJPE	MPJPE	PA-MPJPE	MPJPE	PA-MPJPE
10% labeled mmWave only
PT [ 39 ]	PT	15.88	10.62	17.75	14.03	35.22	14.83	13.32	9.78
mmDiff [ 12 ]	mmDiff	15.10	10.73	44.00	21.76	28.12	20.15	14.98	10.69
P4T [ 11 ]	P4T	12.23	7.95	20.78	16.10	33.62	15.85	11.39	8.37
SPiKE [ 3 ]	SPiKE	11.85	7.92	18.70	14.40	37.09	16.30	10.70	7.86

Table 1: Comparison with state-of-the-art mmWave HPE methods. Each model is trained with only 10% labeled data from MM-Fi (F) or mmBody (B). Depending on the setting, methods may additionally use the remaining 90% unlabeled mmWave data and/or the LiDAR dataset HmPEAR. All results are reported in centimeters (cm), and lower is better.

Setting		F $\to$ F		F $\to$ B
Method	LiDAR Dataset	MPJPE	PA- MPJPE	MPJPE	PA- MPJPE
MT	H	10.37	6.80	16.97	12.12
EMDUL	H	10.06	7.01	14.89	11.11
MT	L	10.93	6.85	18.62	16.31
EMDUL	L	10.05	6.93	16.97	12.82
MT	H+L	10.60	6.84	17.04	13.53

Table 2: Comparison with Mean Teacher (MT) pseudo-labeling when expanding MM-Fi (F) with different LiDAR datasets. P4T [ 11 ] serves as the common HPE model.

Point feature	MPJPE	PA-MPJPE
Doppler	25.15	16.25
Height	19.68	15.24

Table 3: Preliminary error comparison of different point features.

Components			F $\to$ B
$L^{\text{lab}}$	$L^{\text{dyn}}$	$L^{\text{sta}}$	MPJPE	PA-MPJPE
			15.22	11.51
✓			15.12	11.44
✓	✓		15.35	11.73
✓		✓	15.53	11.80
✓	✓	✓	14.89	11.11

Table 4: Ablation study on pseudo-labeling of unlabeled data.

Components				F $\to$ B
NPA	FPF	RS	NI	MPJPE	PA-MPJPE
				15.85	12.23
✓				15.80	12.23
	✓			15.63	11.80
✓	✓			15.49	11.74
✓	✓	✓		15.47	11.56

Table 5: Ablation study on PC conversion of a LiDAR dataset.

UTCL Hyperparameters									FPF Hyperparameters
$\mu$	MPJPE	PA- MPJPE	$\eta$	MPJPE	PA- MPJPE	$\rho$	MPJPE	PA- MPJPE	$\gamma$	MPJPE	PA- MPJPE	$\delta$	MPJPE	PA- MPJPE
1	15.37	11.67	1	15.28	11.37	10	15.67	11.90	1	15.43	11.38	2	15.25	11.60
5	14.89	11.11	5	14.89	11.11	20	14.89	11.11	2	14.89	11.11	5	14.89	11.11
10	15.52	11.86	10	15.67	11.74	50	14.93	11.12	5	15.47	11.78	10	15.58	11.48

Table 6: Ablation study on hyperparameters in UTCL and FPF under F

\to

Panel of Figure 6 — Figure 6: Comparison of pseudo-labels generated with and without UTCL. (a) Two consecutive ground-truth skeletons in $D^{\text{unlab}}$ . (b) Pseudo-labels generated without using UTCL, (c) Pseudo-labels generated by EMDUL using UTCL.

Dataset	$D^{\text{mm}}$	$D^{\text{lidar}}$	$D^{\text{conv}}$ w/o FPF	$D^{\text{conv}}$ w/ FPF
Ratio (%)	99.61	2.66	43.06	60.46

Table 7: Ratio of PCs classified as mmWave in a binary classification task distinguishing between and mmWave and LiDAR data.

Setting		F $\to$ F		F $\to$ B
Method	Labeled Ratio	MPJPE	PA- MPJPE	MPJPE	PA- MPJPE
MT	1%	18.40	13.20	19.94	12.23
EMDUL	1%	14.77	10.37	15.95	11.46
MT	10%	10.37	6.80	16.97	12.12
EMDUL	10%	10.06	7.01	14.89	11.11
MT	50%	8.25	5.63	17.02	12.15

Table 8: Performance under different ratios of labeled mmWave data (

D^{\text{lab}}

/ MM-Fi).

Component	Parameters	Explanation
NPA	$\sigma_{1}=2\,\text{cm}$ , $p=0.5$ , $n=32$	$n$ points sampled from $\mathcal{N}(C,\sigma_{1}^{2}I)$ to a portion of $p$ of the LiDAR PCs, where $C$ is the center of the skeleton and $I$ is a $3\times 3$ identity matrix.
FPF	$\gamma=2\,\text{cm}$ , $\delta=10\,\text{cm}$	The flow threshold is sampled from $\text{U}[\gamma,\delta]$ .
RS	$r_{\text{min}}=0.125$ $r_{\text{max}}=1.0$ $m=128$	A fraction of $r\in U[r_{\text{min}},r_{\text{max}}]$ points are randomly sampled from the PC if it contains at least $m$ points.
NI	$\sigma_{2}=5\,\text{cm}$	A noise following $N(O,\sigma_{2}^{2}I)$ is injected to each point, where $O$ is the origin and $I$ is a $3\times 3$ identity matrix.

Table 9: The entire PC conversion process in our EMDUL.

Setting		F $\to$ F		F $\to$ B
Method	Labeled Ratio	MPJPE	PA- MPJPE	MPJPE	PA- MPJPE
MT	10%	11.84	8.13	16.29	12.14
EMDUL	10%	10.59	7.56	15.64	12.07
MT	50%	10.65	7.40	16.26	12.05
EMDUL	50%	10.27	7.19	16.25	12.55
MT	100%	10.37	6.80	16.97	12.12

Table 10: Ablation studies on the LiDAR data size, represented by the ratio of data used in HmPEAR.

Components				F $\to$ B
NPA	FPF	RS	NI	MPJPE	PA-MPJPE
				15.85	12.23
✓				15.80	12.23
	✓			15.63	11.80
		✓		15.63	12.01
			✓	15.84	12.22

Table 11: Ablation study on PC conversion of a LiDAR dataset.

Figure 7: More visualization results of point cloud conversion. Left: original LiDAR PCs. Right: converted mmWave PCs. Joints with high flow values are yellow, while those with low flow values are blue.

Panel of Figure 8 — Figure 8: Visualization results on MM-Fi $\rightarrow$ mmBody (first row) and mmBody $\rightarrow$ MM-Fi (second and third rows). Joints with high error ( $>10\,\text{cm}$ ) are colored red while others are colored green.

Human Pose Estimation

Human pose estimation recovers people’s body keypoints, 3D skeletons, or meshes from images, video, and other sensor data.

See all 79 papers

MAEPose: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on mmWave Video
Apr 30, 2026Xijia Wei, Yuan Fang, Kevin Chetty +2Masked Autoencoders Self-Supervised Learning
University College London London, UK
WiCompass: Oracle-driven Data Scaling for mmWave Human Pose Estimation
Feb 21, 2026Bo Liang, Chen Gong, Haobo Wang +8Human Pose Estimation OOD Generalization
School of Computer Science, Peking University · University of Electronic Science and Technology of China · University of Pittsburgh +2
Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation
Jul 21, 2026Bo Liang, Chen Gong, Wei Gao +1Millimeter-Wave Radar mmWave Radar Sensing
School of Computer Science, Peking University · University of Pittsburgh · Key Laboratory of High Confidence Software Technologies, Ministry of Education (PKU)

Apr 30, 2026J/K move · Enter open · S save

MAEPose: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on mmWave Video

Xijia Wei, Yuan Fang, Kevin Chetty, Youngjun Cho+1

University College London London, UK

Masked Autoencoders Self-Supervised Learning Millimeter-Wave Radar mmWave Radar Sensing 3D Human Pose Estimation Human Pose Estimation

Open paper AskPDF arXiv

Millimetre-wave (mmWave) radar offers a more privacy-preserving alternative to RGB-based human pose estimation. However, existing methods typically rely on pre-extracted intermediate representations such as sparse point clouds or spectrogram images, where the rich spatiotemporal information naturally present in radar video streams is discarded for model learning, while such signal processing adds system complexity. In addition, existing solutions are mainly conducted in an end-to-end supervised manner without leveraging unlabelled raw video streams to learn generalized representations. In this study, we present MAEPose, a masked autoencoding-based human pose estimation approach that operates directly on mmWave spectrogram videos. MAEPose learns spatiotemporal motion-aware generalized representations from unlabelled radar video, and leverages its heatmap decoder for multi-frame pose estimation predictions. We evaluate it across three datasets based on leave-one-person-out cross-validation with rigorous statistical testing. MAEPose consistently outperforms state-of-the-art baselines by up to 22.1% in MPJPE p<0.05, and maintains robust accuracy under zero-shot bystander interference with only a 6.5% error increase. Ablation studies confirm that both the pre-training and the heatmap decoder contribute substantially, while modality analysis indicates that leveraging Range-Doppler video as input achieves better pose estimation performance than Range-Azimuth or their fusion, with lower computational cost.

Abstract

Human Pose Estimation

Explore similar work

MAEPose: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on mmWave Video

WiCompass: Oracle-driven Data Scaling for mmWave Human Pose Estimation

Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation