Radar-Based 3D Human Pose Estimation via Motion-Aware Pose Initialization and Observation-Guided Refinement

Woo-Jin Jung; Chan-Bin Lim; Seung-Hyun Kong

cs.CVOct 20, 2026

Radar-Based 3D Human Pose Estimation via Motion-Aware Pose Initialization and Observation-Guided Refinement

Authors: Woo-Jin Jung, Chan-Bin Lim, Seung-Hyun Kong

Organizations: CCS Graduate School of Mobility, Korea Advanced Institute of Science and Technology, Daejeon 34051, South Korea · Robotics Program, Korea Advanced Institute of Science and Technology, Daejeon 34141, South Korea

Abstract

3D Human Pose Estimation (3D HPE) is used in indoor environments, such as smart homes, to analyze human movement and monitor human states. Millimeter-wave (mmWave) radar is suitable for this task because it is robust to lighting variations and does not capture images of human appearance. However, radar observations often provide insufficient information about certain body parts or contain overlapping reflections from multiple body parts, limiting accurate 3D joint localization. To address these limitations, we propose the Initialization-to-Refinement Pose Framework (I2R-Pose), comprising two stages: Motion-Aware Pose Initialization and Observation-Guided Refinement. In the first stage, joint-wise features are learned by incorporating temporal relationships across consecutive radar observations and human skeletal structure to compensate for insufficient joint information and estimate an initial 3D pose sequence. In the second stage, separately extracted radar observation features predict joint-wise positional corrections that refine the initial pose and generate the final pose sequence. The framework supports individual and fused inputs of time-Doppler/time-range (TD/TR) maps and radar point clouds (PCs). To evaluate performance across indoor activities, we construct the Radar Motion and Spatial Information for 3D Human Pose Estimation (RaMoS-Pose) dataset, providing synchronized TD/TR maps, radar PCs, and 3D joint ground truth (GT). Evaluations on RaMoS-Pose and the publicly available MVDoppler-Pose dataset show that I2R-Pose reduces mean per-joint position error (MPJPE) by 39.50% on MVDoppler-Pose and by 15.13% on RaMoS-Pose with fused inputs, compared with the best-performing existing methods evaluated on each dataset.

Figures & tables

Fig. 1: Conceptual overview of I2R-Pose. In the Motion-Aware Pose Initialization stage, joint-wise features are learned by incorporating temporal relationships across consecutive radar observations (TD/TR maps, PCs, or both) and human skeletal structure to estimate an initial 3D pose sequence. In the subsequent Observation-Guided Refinement stage, radar observation features extracted separately from the same radar input are combined with the joint-wise features used for initial pose estimation to predict joint-wise positional corrections. These corrections are applied to the initial pose to generate the final pose sequence. The refinement process and changes in MPJPE are illustrated using a representative frame, while RGB images are provided solely for visual reference.

Fig. 2: Overview of the I2R-Pose framework. The TD/TR-only configuration with two observation views is illustrated as an example. The Motion Branch learns joint-wise features from radar observations, while the Pose Branch sequentially performs initial pose estimation using these features and pose refinement using separately extracted observation features. The GT-based Keypoint Motion Encoder and Keypoint HPE Head are used only during training, and Cross-Modal Motion Feature Alignment guides the learning of joint-wise radar features. For visual clarity, the feature tensors are illustrated with the temporal dimension ( $T$ ) along the X-axis, the joint or token dimension ( $J$ or $N_{\mathrm{O}}$ ) along the Y-axis, and the channel dimension ( $C$ ) along the Z-axis.

Panel of Figure 3 — Fig. 3: Main feature extraction and representation processes in I2R-Pose. (a) Generation of joint-wise radar motion features through TD/TR feature extraction and Time–Joint Cross-Attention. (b) Generation of PC spatial tokens through voxel-based feature extraction and global and local tokenization. The illustrated motion features are generated in the Radar Motion Encoder, while the same tokenization structure is independently used in the Radar Observation Encoder with separate weights. (c) Construction of joint-wise motion descriptors from GT joint sequences and their transformation into keypoint motion features.

Feature group	Components	Dim.
Body geometry	Root-relative positions and bone vectors	6
Temporal changes	Joint, bone, and parent-relative differences	18
Motion statistics	Motion magnitudes and reference-axis differences	4
Multiscale relations	Root and bone temporal cross-products	24
Total		52

TABLE I: Composition of the 52-dimensional keypoint motion descriptor.

Dataset	Year	Data	Pose and activity coverage	Size	Radar	Main characteristics
MARS [ 38 ]	2021	PC	Arm and leg exercises, squats, lunges, and other rehabilitation activities (10 types)	40,083 frames, approx. 70 min	TI IWR1443BOOST	Rehabilitation-focused; PC representation
mRI [ 1 ]	2022	PC	Squats, lunges, stretching, walking, and other rehabilitation activities (12 types)	160k+ frames	TI IWR1443BOOST	Rehabilitation-focused; PC representation
mmBody [ 31 ]	2022	PC	Static poses, daily activities, exercise, and sports	200k+ frames	Arbe Phoenix	PC-based 3D human body reconstruction
HIBER [ 39 ]	2022	HM	Standing, sitting, walking, and squatting	804,760 frames	TI AWR2243 Cascade $\times 2$	radio-frequency heatmap-based pose and activity observations
MM-Fi [ 32 ]	2023	PC	Squats, lunges, jumping, object picking, and other daily and rehabilitation activities (27 types)	320,760 frames, 1,080 sequences	TI IWR6843AOP	Diverse activities and PC-based HPE
MiliPoint [ 33 ]	2023	PC	Arm- and leg-focused aerobic exercises (49 types)	545,059 frames	TI IWR1843	PC-based evaluation of diverse movements

TABLE II: Comparison of radar-based HPE datasets providing 3D joint GT.

Fig. 4: Data collection area and sensor configuration of RaMoS-Pose. The left panel shows the radar, camera, and LiDAR arrangement and the data collection area, while the right panel shows the actual sensor setup and relative distances between sensors.

Category	Included activities	Sequences	Ratio (%)
Static poses	Standing, sitting, lying, leaning against a wall	335	19.13
Pose transitions	Sitting down, standing up, lying down, getting up from a lying position	60	3.43
Locomotion	Walking, running	296	16.90
Exercise	Exercise-labeled motions, push-ups, squats, jumping	84	4.80
Daily activities	Cleaning, mobile phone use and calls, object manipulation and organization, moving furniture, desk work, drinking, eating	877	50.09
Falls	Simulated falls	27	1.54

TABLE III: Distribution of representative motion categories in RaMoS-Pose. The 1,751 sequences are classified into mutually exclusive categories based on their primary motion, and percentages are calculated based on the number of sequences.

Method			Evaluation metrics
Model	Year	Input	MPJPE (mm) $\downarrow$	$\rho\uparrow$	PCK 150 (%) $\uparrow$
MD-Pose [ 21 ]	2023	TD	113.780	0.173	71.780
MDPose [ 22 ]	2024	TD	122.240	0.482	73.670
MVDoppler-Pose [ 23 ]	2025	TD/TR (1 view)	87.604	0.499	82.920
MVDoppler-Pose [ 23 ]	2025	TD/TR (2 views)	60.650	0.596	92.680
I2R-Pose (Ours)	2026	TD/TR (1 view)	72.942	0.514	87.164

TABLE IV: Quantitative comparison on the MVDoppler-Pose dataset.

Method			Evaluation metrics
Model	Year	Input	MPJPE (mm) $\downarrow$	$\rho\uparrow$	PCK 150 (%) $\uparrow$
MD-Pose [ 21 ]	2023	TD	201.793	0.044	52.358
MDPose [ 22 ]	2024	TD	227.028	0.091	42.774
MVDoppler-Pose [ 23 ]	2025	TD/TR	170.418	0.109	57.126
VCF-Pose [ 16 ]	2026	PC	151.098	0.143	64.290
MS-STPoseNet [ 7 ]	2026	PC	157.680	0.187	61.100

TABLE V: Quantitative comparison on the RaMoS-Pose dataset. TD/TR uses a single radar view, and PC denotes SRS radar point clouds.

Fig. 5: Qualitative comparison of 3D human pose estimation results on (a) MVDoppler-Pose and (b) RaMoS-Pose. Each row shows the ground-truth (GT) pose and the poses estimated by existing methods and I2R-Pose (Ours) for a representative motion. The numbers below each estimated pose indicate the MPJPE (mm) for the corresponding frame, with bold values indicating the lowest MPJPE in each row. RGB images in (b) are provided solely for visual reference.

Configuration		Evaluation metrics
Applied modules	KME input	MPJPE (mm) $\downarrow$	$\rho\uparrow$	PCK 150 (%) $\uparrow$
Baseline	–	60.650	0.596	92.680
+RME	–	39.465	0.734	97.195
+KME	52D	56.804	0.660	93.408
+OG	–	48.901	0.699	95.348
+RME+OG	–	37.562	0.744	97.282

TABLE VI: Ablation results of major components on MVDoppler-Pose. RME: Radar Motion Encoder; KME: Keypoint Motion Encoder and motion feature alignment; OG: Observation Cross-Attention and prior supervision.

Input configuration			Evaluation metrics
Experiment	Radar Motion Encoder	Radar Observation Encoder	MPJPE (mm) $\downarrow$	$\rho\uparrow$	PCK 150 (%) $\uparrow$
F1	TD/TR	TD/TR	160.664	0.092	63.092
F2	PC	PC	129.311	0.189	70.877
F3	TD/TR	PC	150.882	0.163	65.784
F4	TD/TR	TD/TR–PC	149.910	0.174	67.369
F5	TD/TR–PC	TD/TR–PC	129.669	0.199	71.018

TABLE VII: Ablation study of motion and observation input configurations on RaMoS-Pose.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Feature	Mathematical definition	Dim.	Description
Body Geometry (6D): Body geometric structure
Root-relative position	$\mathbf{p}_{t,j}^{\mathrm{rel}}=\mathbf{p}_{t,j}-\mathbf{p}_{t,r}$	3	Position relative to the root joint
Bone vector	$\mathbf{b}_{t,j}=\mathbf{p}_{t,j}-\mathbf{p}_{t,\pi(j)}$	3	Position relative to the parent joint
Temporal Changes (18D): Temporal changes
Backward difference	$\mathbf{d}^{-}_{t,j}=\mathbf{p}_{t,j}-\mathbf{p}_{t-1,j}$	3	Position change from the previous time step
Forward difference	$\mathbf{d}^{+}_{t,j}=\mathbf{p}_{t+1,j}-\mathbf{p}_{t,j}$	3	Position change to the next time step

Appendix

TABLE VIII: Detailed feature composition and mathematical definitions of the 52-dimensional Keypoint Motion Descriptor.

3D Human Pose Estimation

3D human pose estimation recovers people’s body-joint positions or full-body shape and motion from images, video, and other sensors.

See all 113 papers

MAEPose: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on mmWave Video
Apr 30, 2026Xijia Wei, Yuan Fang, Kevin Chetty +2Masked Autoencoders Self-Supervised Learning
University College London London, UK
mmGAT: Pose Estimation by Graph Attention with Mutual Features from mmWave Radar Point Cloud
Mar 9, 2026Abdullah Al Masud, Shi Xintong, Mondher Bouazizi +1Human Pose Estimation 3D Human Pose Estimation
Graduate School of Science and Technology, Keio University, Kanagawa, 223-8522, Japan · Faculty of Science and Technology, Keio University, Kanagawa, 223-8522, Japan
Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation
Jul 21, 2026Bo Liang, Chen Gong, Wei Gao +1Millimeter-Wave Radar mmWave Radar Sensing
School of Computer Science, Peking University · University of Pittsburgh · Key Laboratory of High Confidence Software Technologies, Ministry of Education (PKU)

Apr 30, 2026J/K move · Enter open · S save

MAEPose: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on mmWave Video

Xijia Wei, Yuan Fang, Kevin Chetty, Youngjun Cho+1

University College London London, UK

Masked Autoencoders Self-Supervised Learning Millimeter-Wave Radar mmWave Radar Sensing 3D Human Pose Estimation Human Pose Estimation

Open paper AskPDF arXiv

Millimetre-wave (mmWave) radar offers a more privacy-preserving alternative to RGB-based human pose estimation. However, existing methods typically rely on pre-extracted intermediate representations such as sparse point clouds or spectrogram images, where the rich spatiotemporal information naturally present in radar video streams is discarded for model learning, while such signal processing adds system complexity. In addition, existing solutions are mainly conducted in an end-to-end supervised manner without leveraging unlabelled raw video streams to learn generalized representations. In this study, we present MAEPose, a masked autoencoding-based human pose estimation approach that operates directly on mmWave spectrogram videos. MAEPose learns spatiotemporal motion-aware generalized representations from unlabelled radar video, and leverages its heatmap decoder for multi-frame pose estimation predictions. We evaluate it across three datasets based on leave-one-person-out cross-validation with rigorous statistical testing. MAEPose consistently outperforms state-of-the-art baselines by up to 22.1% in MPJPE p<0.05, and maintains robust accuracy under zero-shot bystander interference with only a 6.5% error increase. Ablation studies confirm that both the pre-training and the heatmap decoder contribute substantially, while modality analysis indicates that leveraging Range-Doppler video as input achieves better pose estimation performance than Range-Azimuth or their fusion, with lower computational cost.

Abstract

3D Human Pose Estimation

Explore similar work

MAEPose: Self-Supervised Spatiotemporal Learning for Human Pose Estimation on mmWave Video

mmGAT: Pose Estimation by Graph Attention with Mutual Features from mmWave Radar Point Cloud

Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation