Radar-Based 3D Human Pose Estimation via Motion-Aware Pose Initialization and Observation-Guided Refinement
Authors: Woo-Jin Jung, Chan-Bin Lim, Seung-Hyun Kong
Organizations: CCS Graduate School of Mobility, Korea Advanced Institute of Science and Technology, Daejeon 34051, South Korea · Robotics Program, Korea Advanced Institute of Science and Technology, Daejeon 34141, South Korea
3D Human Pose Estimation (3D HPE) is used in indoor environments, such as smart homes, to analyze human movement and monitor human states. Millimeter-wave (mmWave) radar is suitable for this task because it is robust to lighting variations and does not capture images of human appearance. However, radar observations often provide insufficient information about certain body parts or contain overlapping reflections from multiple body parts, limiting accurate 3D joint localization. To address these limitations, we propose the Initialization-to-Refinement Pose Framework (I2R-Pose), comprising two stages: Motion-Aware Pose Initialization and Observation-Guided Refinement. In the first stage, joint-wise features are learned by incorporating temporal relationships across consecutive radar observations and human skeletal structure to compensate for insufficient joint information and estimate an initial 3D pose sequence. In the second stage, separately extracted radar observation features predict joint-wise positional corrections that refine the initial pose and generate the final pose sequence. The framework supports individual and fused inputs of time-Doppler/time-range (TD/TR) maps and radar point clouds (PCs). To evaluate performance across indoor activities, we construct the Radar Motion and Spatial Information for 3D Human Pose Estimation (RaMoS-Pose) dataset, providing synchronized TD/TR maps, radar PCs, and 3D joint ground truth (GT). Evaluations on RaMoS-Pose and the publicly available MVDoppler-Pose dataset show that I2R-Pose reduces mean per-joint position error (MPJPE) by 39.50% on MVDoppler-Pose and by 15.13% on RaMoS-Pose with fused inputs, compared with the best-performing existing methods evaluated on each dataset.
Figures & tables
Fig. 1: Conceptual overview of I2R-Pose. In the Motion-Aware Pose Initialization stage, joint-wise features are learned by incorporating temporal relationships across consecutive radar observations (TD/TR maps, PCs, or both) and human skeletal structure to estimate an initial 3D pose sequence. In the subsequent Observation-Guided Refinement stage, radar observation features extracted separately from the same radar input are combined with the joint-wise features used for initial pose estimation to predict joint-wise positional corrections. These corrections are applied to the initial pose to generate the final pose sequence. The refinement process and changes in MPJPE are illustrated using a representative frame, while RGB images are provided solely for visual reference.
Fig. 2: Overview of the I2R-Pose framework. The TD/TR-only configuration with two observation views is illustrated as an example. The Motion Branch learns joint-wise features from radar observations, while the Pose Branch sequentially performs initial pose estimation using these features and pose refinement using separately extracted observation features. The GT-based Keypoint Motion Encoder and Keypoint HPE Head are used only during training, and Cross-Modal Motion Feature Alignment guides the learning of joint-wise radar features. For visual clarity, the feature tensors are illustrated with the temporal dimension ( T ) along the X-axis, the joint or token dimension ( J or NO ) along the Y-axis, and the channel dimension ( C ) along the Z-axis.
Fig. 3: Main feature extraction and representation processes in I2R-Pose. (a) Generation of joint-wise radar motion features through TD/TR feature extraction and Time–Joint Cross-Attention. (b) Generation of PC spatial tokens through voxel-based feature extraction and global and local tokenization. The illustrated motion features are generated in the Radar Motion Encoder, while the same tokenization structure is independently used in the Radar Observation Encoder with separate weights. (c) Construction of joint-wise motion descriptors from GT joint sequences and their transformation into keypoint motion features.
Feature group
Components
Dim.
Body geometry
Root-relative positions and bone vectors
6
Temporal changes
Joint, bone, and parent-relative differences
18
Motion statistics
Motion magnitudes and reference-axis differences
4
Multiscale relations
Root and bone temporal cross-products
24
Total
52
TABLE I: Composition of the 52-dimensional keypoint motion descriptor.
Dataset
Year
Data
Pose and activity coverage
Size
Radar
Main characteristics
MARS [ 38 ]
2021
PC
Arm and leg exercises, squats, lunges, and other rehabilitation activities (10 types)
40,083 frames, approx. 70 min
TI IWR1443BOOST
Rehabilitation-focused; PC representation
mRI [ 1 ]
2022
PC
Squats, lunges, stretching, walking, and other rehabilitation activities (12 types)
160k+ frames
TI IWR1443BOOST
Rehabilitation-focused; PC representation
mmBody [ 31 ]
2022
PC
Static poses, daily activities, exercise, and sports
200k+ frames
Arbe Phoenix
PC-based 3D human body reconstruction
HIBER [ 39 ]
2022
HM
Standing, sitting, walking, and squatting
804,760 frames
TI AWR2243 Cascade ×2
radio-frequency heatmap-based pose and activity observations
MM-Fi [ 32 ]
2023
PC
Squats, lunges, jumping, object picking, and other daily and rehabilitation activities (27 types)
320,760 frames, 1,080 sequences
TI IWR6843AOP
Diverse activities and PC-based HPE
MiliPoint [ 33 ]
2023
PC
Arm- and leg-focused aerobic exercises (49 types)
545,059 frames
TI IWR1843
PC-based evaluation of diverse movements
TABLE II: Comparison of radar-based HPE datasets providing 3D joint GT.
Fig. 4: Data collection area and sensor configuration of RaMoS-Pose. The left panel shows the radar, camera, and LiDAR arrangement and the data collection area, while the right panel shows the actual sensor setup and relative distances between sensors.
Category
Included activities
Sequences
Ratio (%)
Static poses
Standing, sitting, lying, leaning against a wall
335
19.13
Pose transitions
Sitting down, standing up, lying down, getting up from a lying position
Cleaning, mobile phone use and calls, object manipulation and organization, moving furniture, desk work, drinking, eating
877
50.09
Falls
Simulated falls
27
1.54
TABLE III: Distribution of representative motion categories in RaMoS-Pose. The 1,751 sequences are classified into mutually exclusive categories based on their primary motion, and percentages are calculated based on the number of sequences.
Method
Evaluation metrics
Model
Year
Input
MPJPE (mm) ↓
ρ↑
PCK 150 (%) ↑
MD-Pose [ 21 ]
2023
TD
113.780
0.173
71.780
MDPose [ 22 ]
2024
TD
122.240
0.482
73.670
MVDoppler-Pose [ 23 ]
2025
TD/TR (1 view)
87.604
0.499
82.920
MVDoppler-Pose [ 23 ]
2025
TD/TR (2 views)
60.650
0.596
92.680
I2R-Pose (Ours)
2026
TD/TR (1 view)
72.942
0.514
87.164
TABLE IV: Quantitative comparison on the MVDoppler-Pose dataset.
Method
Evaluation metrics
Model
Year
Input
MPJPE (mm) ↓
ρ↑
PCK 150 (%) ↑
MD-Pose [ 21 ]
2023
TD
201.793
0.044
52.358
MDPose [ 22 ]
2024
TD
227.028
0.091
42.774
MVDoppler-Pose [ 23 ]
2025
TD/TR
170.418
0.109
57.126
VCF-Pose [ 16 ]
2026
PC
151.098
0.143
64.290
MS-STPoseNet [ 7 ]
2026
PC
157.680
0.187
61.100
TABLE V: Quantitative comparison on the RaMoS-Pose dataset. TD/TR uses a single radar view, and PC denotes SRS radar point clouds.
Fig. 5: Qualitative comparison of 3D human pose estimation results on (a) MVDoppler-Pose and (b) RaMoS-Pose. Each row shows the ground-truth (GT) pose and the poses estimated by existing methods and I2R-Pose (Ours) for a representative motion. The numbers below each estimated pose indicate the MPJPE (mm) for the corresponding frame, with bold values indicating the lowest MPJPE in each row. RGB images in (b) are provided solely for visual reference.
Configuration
Evaluation metrics
Applied modules
KME input
MPJPE (mm) ↓
ρ↑
PCK 150 (%) ↑
Baseline
–
60.650
0.596
92.680
+RME
–
39.465
0.734
97.195
+KME
52D
56.804
0.660
93.408
+OG
–
48.901
0.699
95.348
+RME+OG
–
37.562
0.744
97.282
TABLE VI: Ablation results of major components on MVDoppler-Pose. RME: Radar Motion Encoder; KME: Keypoint Motion Encoder and motion feature alignment; OG: Observation Cross-Attention and prior supervision.
Input configuration
Evaluation metrics
Experiment
Radar Motion Encoder
Radar Observation Encoder
MPJPE (mm) ↓
ρ↑
PCK 150 (%) ↑
F1
TD/TR
TD/TR
160.664
0.092
63.092
F2
PC
PC
129.311
0.189
70.877
F3
TD/TR
PC
150.882
0.163
65.784
F4
TD/TR
TD/TR–PC
149.910
0.174
67.369
F5
TD/TR–PC
TD/TR–PC
129.669
0.199
71.018
TABLE VII: Ablation study of motion and observation input configurations on RaMoS-Pose.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Feature
Mathematical definition
Dim.
Description
Body Geometry (6D): Body geometric structure
Root-relative position
pt,jrel=pt,j−pt,r
3
Position relative to the root joint
Bone vector
bt,j=pt,j−pt,π(j)
3
Position relative to the parent joint
Temporal Changes (18D): Temporal changes
Backward difference
dt,j−=pt,j−pt−1,j
3
Position change from the previous time step
Forward difference
dt,j+=pt+1,j−pt,j
3
Position change to the next time step
Appendix
TABLE VIII: Detailed feature composition and mathematical definitions of the 52-dimensional Keypoint Motion Descriptor.
Millimetre-wave (mmWave) radar offers a more privacy-preserving alternative to RGB-based human pose estimation. However, existing methods typically rely on pre-extracted intermediate representations such as sparse point clouds or spectrogram images, where the rich spatiotemporal information naturally present in radar video streams is discarded for model learning, while such signal processing adds system complexity. In addition, existing solutions are mainly conducted in an end-to-end supervised manner without leveraging unlabelled raw video streams to learn generalized representations. In this study, we present MAEPose, a masked autoencoding-based human pose estimation approach that operates directly on mmWave spectrogram videos. MAEPose learns spatiotemporal motion-aware generalized representations from unlabelled radar video, and leverages its heatmap decoder for multi-frame pose estimation predictions. We evaluate it across three datasets based on leave-one-person-out cross-validation with rigorous statistical testing. MAEPose consistently outperforms state-of-the-art baselines by up to 22.1% in MPJPE p<0.05, and maintains robust accuracy under zero-shot bystander interference with only a 6.5% error increase. Ablation studies confirm that both the pre-training and the heatmap decoder contribute substantially, while modality analysis indicates that leveraging Range-Doppler video as input achieves better pose estimation performance than Range-Azimuth or their fusion, with lower computational cost.
Pose estimation and human action recognition (HAR) are pivotal technologies spanning various domains. While the image-based pose estimation and HAR are widely admired for their superior performance, they lack in privacy protection and suboptimal performance in low-light and dark environments. This paper exploits the capabilities of millimeter-wave (mmWave) radar technology for human pose estimation by processing radar data with Graph Neural Network (GNN) architecture, coupled with the attention mechanism. Our goal is to capture the finer details of the radar point cloud to improve the pose estimation performance. To this end, we present a unique feature extraction technique that exploits the full potential of the GNN processing method for pose estimation. Our model mmGAT demonstrates remarkable performance on two publicly available benchmark mmWave datasets and establishes new state of the art results in most scenarios in terms of human pose estimation. Our approach achieves a noteworthy reduction of pose estimation mean per joint position error (MPJPE) by 35.6% and PA-MPJPE by 14.1% from the current state of the art benchmark within this domain.
Abdullah Al Masud, Shi Xintong, Mondher Bouazizi +1
Graduate School of Science and Technology, Keio University, Kanagawa, 223-8522, Japan · Faculty of Science and Technology, Keio University, Kanagawa, 223-8522, Japan
Millimeter-wave (mmWave) radar enables privacy-friendly human sensing, but its sparse point clouds are physical measurements of view-dependent electromagnetic reflections and only indirectly characterize body articulation. Recovering a complete 3D pose from such partial, geometry-dependent observations is therefore under-constrained. Existing methods directly regress joint coordinates from paired radar-pose data, relying on the same limited paired supervision to learn radar perception, human-body structure, and their alignment. This coupling can encourage dataset-specific shortcuts under ambiguous radar observations. We propose Wave2Body, a radar-to-body token translation framework that decouples these learning targets using a self-supervised mmWave tokenizer, a pretrained compositional body tokenizer that defines the output space, and a lightweight translator between them. Experiments on M4Human and mmBody show that Wave2Body achieves stronger cross-domain generalization than previous methods while incurring much lower computational costs for training and inference. All the code and experiment results are publicly available at https://github.com/Galaxywalk/Wave2Body.
Bo Liang, Chen Gong, Wei Gao +1
School of Computer Science, Peking University · University of Pittsburgh · Key Laboratory of High Confidence Software Technologies, Ministry of Education (PKU)