Privacy-Preserving Full-Body Meshing from mmWave Radar via Mesh Foundation Model Supervision

Shuxing Zhang; Yongquan Ni; Zhenyu Ding; Yawen Lin

cs.AISep 28, 2026

Privacy-Preserving Full-Body Meshing from mmWave Radar via Mesh Foundation Model Supervision

Authors: Shuxing Zhang, Yongquan Ni, Zhenyu Ding, Yawen Lin

Organizations: AI Value Center, Incaier (Haier), Qingdao, China

Abstract

Millimeter-wave (mmWave) radar enables privacy-preserving human perception, but the extreme sparsity of point clouds from commercial single-chip sensors (mean ~6.5 points/frame; ~28% empty frames) has confined prior art to body-part keypoints or discrete action classification. We present a cross-modal teacher-student framework that lifts commercial radar to full-body, per-frame, metric 3D mesh reconstruction with per-joint uncertainty. Three innovations: (1) a mesh-foundation-model teacher - SAM 3D Body produces whole-body MHR ground truth (70 joints, 18,439 mesh vertices) from a single RGB frame with zero training, slashing annotation cost by orders of magnitude; (2) StudentPoseFormer - set encoding with masked attention pooling, a temporal Transformer, and a CVAE multi-hypothesis head that outputs both the pose mean and per-joint variance, honestly reporting where the radar cannot see; and (3) a multi-stage ground-truth quality pipeline (confidence gating, depth validation, temporal smoothing, bone-length consistency, bad-frame rejection) plus systematic information-lever ablations. On the public MM-Fi benchmark (same TI IWR6843 sensor, cross-subject), our full configuration reaches 7.45 cm 12-joint MPJPE, with ablations proving the causal value of point accumulation (k = 3, -0.34 cm), Doppler (-0.85 cm; -2 cm at the wrist on fast actions), and velocity loss (-0.27 cm). On our own synchronized radar + RGB-D corpus with block-level held-out splits, the pipeline achieves 21.47 cm end-to-end (per-joint hierarchy from 4.8 cm at the hip to 34.7 cm at the wrist - matching physical information limits), could be improved to 15 cm with ~30k diverse samples, and a scaling law shows sample diversity, not volume, is the binding constraint. Deployment inference is radar-only - no camera, no image.

Figures & tables

Panel of Figure 1 — Figure 1: Overall framework. Left — training: the commercial mmWave radar and an RGB-D camera synchronously capture the same scene; a frozen mesh foundation model (SAM 3D Body) converts each RGB frame into whole-body 3D pseudolabels, which must pass a five-stage quality gate before they supervise the radar-only student. Right — deployment: only the radar and the trained student run. The dashed box lists the modules that are physically absent, so privacy is guaranteed by construction rather than by policy.

Panel of Figure 2 — Figure 2: Point-cloud sequence representation. Top: the radar emits a variable number of detections per frame and some frames are empty. Middle: a sliding window of $T{=}5$ frames centered on the target frame $t$ is zero-padded into a fixed $[T,M,5]$ tensor whose channels are $(x,y,z,\text{doppler},\text{snr})$ . Bottom: the companion binary mask $[T,M]$ flags real detections (white, $1$ ) versus padding (gray, $0$ ); fully masked frames are mapped to a learned empty-frame embedding.

Panel of Figure 3 — Figure 3: StudentPoseFormer architecture. Stage 1: a shared per-point MLP followed by masked attention pooling compresses each frame’s variable-size point set into one 256-d embedding (padded points are masked to $-\infty$ in the attention logits, making empty frames NaN-safe). Stage 2: a 4-layer, 8-head Transformer fuses the $T{=}5$ frame embeddings; the target-frame output is the human condition feature $\mathbf{c}$ . Stage 3: at inference the CVAE head draws $K{=}20$ latent samples from the prior; the per-joint mean is the pose estimate and the per-joint variance is the uncertainty. Stage 4: an MHR head shares the fused latent $(\mathbf{c}\,\|\,z)$ and decodes a 18,439-vertex body mesh.

Panel of Figure 4 — Figure 4: The CVAE multi-hypothesis head in training and inference. (a) Training: a posterior encoder maps the teacher ground truth and the condition feature $\mathbf{c}$ to latent parameters $(\mu,\sigma)$ ; reparameterization samples $z$ , a decoder reconstructs the joints, and a KL term (with $\beta$ annealed from 0 to 1 over the first 20% of the schedule to prevent posterior collapse) regularizes the posterior toward a unit Gaussian. (b) Inference: the posterior encoder is discarded — the radar never sees the GT — and $K{=}20$ latents are drawn from the standard normal prior. The per-joint mean over the $K$ decoded hypotheses is the pose estimate, and the per-joint variance is the uncertainty signal.

Panel of Figure 5 — Figure 5: Five-stage quality control for teacher pseudolabels. A candidate frame is propagated to training only if every stage passes: (1) per-joint teacher 2D/3D confidence above threshold; (2) joints inside the RGB-D range envelope and away from depth holes; (3) One-Euro temporal filtering removes single-frame outliers; (4) per-frame bone lengths within tolerance of the subject’s median (segment kinematics are bounded); (5) frames with too few valid joints are dropped entirely. Rejected joints are excluded from all losses and metrics rather than being imputed.

Panel of Figure 6 — Figure 6: Cross-modal spatiotemporal alignment. (i) Temporal: radar frames are paired to camera frames by nearest-neighbor matching of hardware timestamps (median inter-sensor jitter $\approx 11$ ms in our corpus). (ii) Spatial: monocular mesh estimates carry a depth-scale and cam_t bias, so the mesh root is first anchored to the RGB-D measured depth, making the labels metric; per-session extrinsics $(\mathbf{R},\mathbf{t})$ then map teacher joints from the camera frame into the radar frame via $\mathbf{p}_{\text{radar}}=\mathbf{R}\,\mathbf{p}_{\text{cam}}+\mathbf{t}$ , where $(\mathbf{R},\mathbf{t})$ is estimated by RANSAC-robust Umeyama registration of radar centroid trajectories against depth-anchored camera trajectories recorded during a dedicated walking phase.

Panel of Figure 7 — Figure 7: Deployment runs radar-only. A FIFO buffer maintains the most recent $T$ frames, which are packed into the $[T,M,5]$ input tensor and passed through the student’s forward computation to produce the skeleton, mesh, and per-joint confidence at radar frame rate. The dashed box isolates everything that is not present — no camera, no teacher, no pseudolabel generation — making the privacy boundary a physical property of the deployment rather than an access-control policy.

Panel of Figure 8 — Figure 8: Multi-person handling by track-ID demultiplexing. The radar’s own tracker tags each detection with a target ID ( trackData.tid ); points are split per ID into independent temporal windows and state machines, with a one-frame track-index displacement correction to absorb ID flicker. The same student weights are applied to each target independently, yielding one pose, mesh, and uncertainty vector per person.

Panel of Figure 9 — Figure 9: Per-joint uncertainty produced by the CVAE head, shown as confidence circles on the estimated skeleton (radius $\propto$ per-joint standard deviation over $K$ prior samples). Trunk joints that reflect strongly off large, slow surfaces (hips, shoulders, neck) are tightly concentrated; distal joints with small radar cross-section and fast motion (wrists, ankles) carry large, honest uncertainty. The system therefore annotates its own perception boundary: downstream applications can trust the trunk while treating the extremities as low-confidence.

Configuration (single variable)	MPJPE	PA	track_r	A10	Wrist L/R
Baseline ceiling_tune	8.06	67.1	0.616	18.2/17.6	15.0/15.4
$+$ G1: actions $10\to 27$	8.07	65.1	0.624	(mixed)	15.4/15.8
$+$ G2: accumulation $k{=}3$	7.64	61.1	0.642	14.65	14.1/14.8
$+$ G3: Doppler present	7.64 †	—	0.642	14.65	14.8
$+$ G4: velocity loss $v_{w}{=}1.0$	7.45	—	0.651	14.70	14.5

Table 1: MM-Fi cross-subject results (S01–S07 train / S08–S10 test). Each row changes exactly one variable (G-series protocol). MPJPE is the 12 shared-limb mean per-joint position error in cm; PA-MPJPE in mm; track_r is the moving-track reliability score; A10 is the fastest action class; wrist L/R are per-wrist errors in cm. Best values in bold .

Joint	MPJPE (cm)
Hip	1.7
Shoulder	6.5–6.6
Knee	6.1–6.2
Head	7.7
Ankle	8.1–8.2
Elbow	10.1–10.4

Table 2: Per-joint MPJPE (cm) on MM-Fi for configuration G1 v1a, averaged over left/right. Errors increase monotonically from proximal to distal joints, matching the physical illumination law of RF-based pose estimation [ 2 ] .

Model	Metric	Value
v1a deterministic	MPJPE (cm)	21.47
	PA-MPJPE (mm)	127.0
v1b CVAE	MPJPE (cm)	21.44
	mean per-joint std (cm)	7.45
v2 MHR mesh	body_pose MSE	0.0268
	global_rot MSE	0.908

Table 3: End-to-end results on our own synchronous radar + RGB-D corpus (six sessions, one subject, block-level held-out splits with a 4-frame leakage buffer; 1,576 train / 600 validation samples).

Panel of Figure 10 — Figure 10: Scaling behavior on log–log axes. Circles: measured points — our corpus at $n{=}2{,}176$ (21.47 cm) and the MM-Fi anchor at $n{=}6{,}860$ diverse cross-subject samples (8.3 cm). The blue solid line is the within-distribution fit $\mathrm{err}=53.8\,n^{-0.124}$ : reaching 15 cm would require ${\sim}30$ k samples, 12 cm 185k, and 10 cm 810k (squares). The red dashed line shows the slope $b\approx 0.65$ implied by diverse samples — over $100\times$ steeper. Sample diversity , not volume, is the binding constraint.

Explore similar work

Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation
Jul 21, 2026Bo Liang, Chen Gong, Wei Gao +1Wifi-Csi 3D Human Pose Estimation Radar
School of Computer Science, Peking University · University of Pittsburgh · Key Laboratory of High Confidence Software Technologies, Ministry of Education (PKU)
A Two-Stage Motion-Aware Framework for mmWave-based Human Mesh Recovery
May 8, 2026Hoang Hai Pham, Shuntian Zheng, Jiaqi Li +1Human Mesh Recovery Avatar Reconstruction
Department of Computer Science, University of Warwick, Coventry, UK
DGHMesh: A Large-scale Dual-radar mmWave Dataset and Generalization-focused Benchmark for Human Mesh Reconstruction
Apr 19, 2026Rongxiao Guo, Qingchao ChenMillimeter Wave Human Mesh Recovery
Institute of Medical Technology, Peking University Health Science Center · National Institute of Health Data Science at Peking University · State Key Laboratory of General Artificial Intelligence, Peking University

Jul 21, 2026, cs.CVJ/K move · Enter open · S save

Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation

Bo Liang, Chen Gong, Wei Gao, Chenren Xu

School of Computer Science, Peking University · University of Pittsburgh · Key Laboratory of High Confidence Software Technologies, Ministry of Education (PKU)

Wifi-Csi 3D Human Pose Estimation Radar

Open paper AskPDF arXiv

Millimeter-wave (mmWave) radar enables privacy-friendly human sensing, but its sparse point clouds are physical measurements of view-dependent electromagnetic reflections and only indirectly characterize body articulation. Recovering a complete 3D pose from such partial, geometry-dependent observations is therefore under-constrained. Existing methods directly regress joint coordinates from paired radar-pose data, relying on the same limited paired supervision to learn radar perception, human-body structure, and their alignment. This coupling can encourage dataset-specific shortcuts under ambiguous radar observations. We propose Wave2Body, a radar-to-body token translation framework that decouples these learning targets using a self-supervised mmWave tokenizer, a pretrained compositional body tokenizer that defines the output space, and a lightweight translator between them. Experiments on M4Human and mmBody show that Wave2Body achieves stronger cross-domain generalization than previous methods while incurring much lower computational costs for training and inference. All the code and experiment results are publicly available at https://github.com/Galaxywalk/Wave2Body.

Abstract

Explore similar work

Wave2Body: Rethinking mmWave Human Pose Estimation as Radar-to-Body Token Translation

A Two-Stage Motion-Aware Framework for mmWave-based Human Mesh Recovery

DGHMesh: A Large-scale Dual-radar mmWave Dataset and Generalization-focused Benchmark for Human Mesh Reconstruction