PCHMR: Empowering and Benchmarking Human Mesh Recovery in Privacy-Constrained Real-World Settings

Zeyu Cao; Qingxuan Wu; Zhiyang Dou; Rui Xu; Yuan Liu; Javier Fernandez-Marques; Nicholas D. Lane; Taku Komura

cs.CVOct 17, 2026

PCHMR: Empowering and Benchmarking Human Mesh Recovery in Privacy-Constrained Real-World Settings

Authors: Zeyu Cao, Qingxuan Wu, Zhiyang Dou, Rui Xu, Yuan Liu, Javier Fernandez-Marques, Nicholas D. Lane, Taku Komura,

Organizations: University of Cambridge · University of Pennsylvania · The University of Hong Kong · Hong Kong University of Science and Technology ·

Abstract

Fine-tuning directly on user-side data is an effective way to improve the performance of widely adopted data-driven human mesh recovery (HMR) models. However, doing so for existing HMR methods often assumes aggregating user-side data from real-world HMR-deployed devices to a central training server, posing a significant privacy risk as sensitive human images are transmitted. How can human mesh recovery be evaluated and improved in privacy-constrained real-world settings? We study this question through a benchmark in which state-of-the-art HMR models are trained with federated learning and a secure-aggregation variant, neither of which centralizes raw images, and evaluated across a wide range of client-scale and data-heterogeneity settings. We document that common HMR training pipelines are built around centralized data access and quantify how representative HMR backbones behave when that assumption is removed. Furthermore, to study the data bottleneck of privacy-constrained HMR training, we propose DePoser, a local annotation and fine-tuning pipeline enhanced with depth foundation models, with which collaboratively trained HMR models can be locally tailored to the end user's distribution. On in-the-wild data, fine-tuning on DePoser annotations improves personalization accuracy, measured against DePoser-generated pseudo-ground truth. This benchmark aims to support future work on privacy-constrained HMR models and their real-world deployment and evaluation.

Figures & tables

Figure 1: Comparison between centralized and privacy-constrained HMR frameworks. Existing HMR frameworks commonly assume centralized training (a), which requires uploading sensitive user data and therefore raises privacy concerns. In comparison, PCHMR benchmarks privacy-constrained settings for HMR models (b), where raw data remain on the device and only model updates are exchanged. Locally generated annotations then provide additional supervision, so HMR models can be improved with privacy-constrained real-world human data .

Figure 2: Overview of the Pipeline for Privacy-Constrained HMR. Data labeling can be done offline via a local data annotator (1, 2). After that, the labels of user-generated data are used to collaboratively train a new global model with privacy constraints (3, 4). User data never leave the device. The global model is then deployed for real-time on-device inference (5, 6). PCHMR benchmarks implementations of such a privacy-constrained pipeline under various real-world settings.

#Clients (Total / Per Round )	Model	MPJPE	PA-MPJPE	Raw Data Local?
Centralized	FM-S	57.98	40.62	✗
10 / 10	FM-S	56.85	40.52	✓
100 / 10	FM-S	59.31	41.48	✓
1000 / 10	FM-S	60.82	44.53	✓
Centralized	TORE-S	63.88	41.99	✗
10 / 10	TORE-S	61.27	41.60	✓

Table 1: Effect of Number of Clients. HMR results (MPJPE and PA-MPJPE) for the PCHMR baseline experiments ( 10/10 ), the Independent and Identically Distributed (IID) data 100-client experiments ( 100/10 ), and IID data 1000-client experiments ( 1000/10 ) with their best evaluation results. All experiments select 10 clients per round uniformly at random. For comparison, we also add the centralized baseline trained with the same model and the same dataset.

IID Distribution
Optimizer	FM-S Model		TORE-S Model
	MPJPE	PA-MPJPE	MPJPE	PA-MPJPE
FedAvg	59.31	41.48	62.92	43.11
FedOpt	58.49	41.43	62.82	42.34
FedProx	57.71	41.26	63.41	43.17
LDA $\alpha=0.5$ Distribution

Table 2: Effect of Federated Optimizer. Performance of HMR models with different federated optimizers (FedAvg, FedOpt, FedProx) on IID data and LDA skewed data. The results are measured with 100-client experiments ( 100/10 ) with 10 clients sampled per round.

Figure 3: Data Distribution Evaluation. Performance metrics for heterogeneous data distribution settings with different LDA

\alpha

, evaluated with the Human3.6M ( Ionescu et al., 2013 ) dataset.

Setting	Model	MPJPE	PA-MPJPE	Raw Data Local?
Centralized	FM-S	61.38	41.63	✗
PCHMR	FM-S	58.28	40.42	✓
Centralized	TORE-S	63.89	41.40	✗
PCHMR	TORE-S	61.59	41.94	✓

Table 3: Mixed Dataset Training. Comparison between FastMETRO-S ( Cho et al., 2022 ) and TORE-S ( Dou et al., 2023 ) trained under centralized or privacy-constrained settings on a mixed dataset composed of Human3.6M, UP-3D, COCO, MuCo-3DHP, and MPII ( Ionescu et al., 2013 ; Lassner et al., 2017 ; Lin et al., 2014 ; Mehta et al., 2018 ; Andriluka et al., 2014 ) , evaluated on Human3.6M.

Setting	Model	MPJPE	PA-MPJPE	MPVPE	Raw Data Local?
Centralized	FM-S	84.92	54.69	97.60	✗
PCHMR	FM-S	84.66	54.78	97.42	✓
PCHMR (Natural Part.)	FM-S	86.83	55.61	99.66	✓
Centralized	TORE-S	87.97	55.35	101.88	✗
PCHMR	TORE-S	87.55	54.08	101.86	✓
PCHMR (Natural Part.)	TORE-S	88.00	55.07	102.50	✓

Table 4: Performance of PCHMR on 3DPW. Reconstruction accuracy when PCHMR is fine-tuned on the 3DPW dataset. We further evaluate the model’s performance on the natural scene partition (Natural Part.), where each scene is assigned to a device. PCHMR reaches comparable performance on 3DPW fine-tuning compared with the centralized setting. PCHMR is trained with 100 devices and 10 randomly selected each round, while the natural partition is 20 devices with 5 randomly selected each round.

Dataset	Before/After	Model	MPVPE	MPJPE	PA-MPJPE
VR-runner	Before PCHMR	FM-S	109.50	114.51	60.07
VR-runner	After PCHMR	FM-S	93.23	58.24	43.16
VR-game-1	Before PCHMR	FM-S	176.24	172.57	78.21
VR-game-1	After PCHMR	FM-S	65.57	62.30	43.82
VR-game-2	Before PCHMR	FM-S	150.64	161.82	84.36
VR-game-2	After PCHMR	FM-S	82.53	69.49	53.57

Table 5: Local Fine-Tuning Results. Reconstruction error before and after local fine-tuning with PCHMR, computed on the held-out validation split of the locally annotated data against DePoser pseudo-labels (Appendix D ).

Figure 4: Qualitative results of PCHMR on Human3.6M ( Ionescu et al., 2013 ) (a,b) and 3DPW ( Von Marcard et al., 2018 ) (c,d), trained with 100 clients and 10 randomly sampled per round. The backbones are FastMETRO ( Cho et al., 2022 ) (a,c) and TORE ( Dou et al., 2023 ) (b,d).

Model	MPVPE	MPJPE	PA-MPJPE
SMPLify-X ( Pavlakos et al., 2019a )	198.66	181.07	87.17
DePoser (Ours)	162.60	151.19	71.14

Table 6: Comparison between DePoser and SMPLify-X ( Pavlakos et al., 2019a ) on a sub-sampled 3DPW ( Von Marcard et al., 2018 ) dataset. DePoser demonstrates improved reconstruction accuracy compared to SMPLify-X under the combined Sapiens pose and depth prior.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Model	Clients	Learning Rate	Epochs	Local Epochs ( $E$ )	Global Rounds ( $T$ )
Centralized Baseline	FM-S	-	0.001	60	–	–
Human3.6M naïve on-device training	FM-S	-	0.001	60	–	–
3DPW Centralized Fine-tuning	FM-S	-	0.0001	20	–	–
PCHMR + IID Data 10 / 10	FM-S	10 / 10	0.001	–	1	200
PCHMR + IID Data 100 / 10	FM-S	100 / 10	0.001	–	1	2000
PCHMR + IID Data 1000 / 10	FM-S	1000 / 10	0.001	–	1	6000

Appendix

Table 7: The hyperparameters used by all PCHMR experiments. The two numbers in the Clients column denote the total number of clients and the number of clients randomly selected in each round. Epochs counts passes over the full training set and applies only to the centralized runs; the federated runs are specified instead by the number of local epochs

E

each sampled client performs per round and the number of global rounds

T

. The two units are not interchangeable, and we quote a budget in only one of them per run.

Regime	Budget	Eff. batch	Optimizer steps	Sample-visits	Epoch-equiv.	Critical path	FM-S	TORE-S	Val. evals
							Wall-clock (h)
Centralized (as published)	60 epochs	512	$55{,}800$	$28{,}563{,}720$	$60.0$	–	$18.9$	$18.7$	$60$
Centralized, budget-matched	200 epochs	512	$186{,}000$	$95{,}212{,}400$	$200.0$	–	$63.0$	$62.3$	$200$
Centralized, step-matched	200 epochs	64	$1{,}487{,}800$	$95{,}212{,}400$	$200.0$	–	$\approx 500^{\dagger}$	$\approx 500^{\dagger}$	$200$
PCHMR, IID 10/10	200 rounds	64	$1{,}488{,}000$	$95{,}212{,}400$	$200.0$	$148{,}800$	$44.3$	$47.1$	$200$
PCHMR, IID 100/10	2000 rounds	64	$1{,}500{,}203$	$95{,}212{,}580$	$200.0$	$150{,}203$	$60.5$	$55.9$	$200$

Appendix

Table 8: Training Budget of Every Regime. Optimizer steps are gradient updates summed over all clients; sample-visits count one image passed through the model once, and epoch-equivalents divide them by the

476{,}062

-image training set; the critical path counts the steps of the slowest client sampled in each round, summed over rounds. All federated rows use one local epoch (

E=1

) and a local batch of

64

; the LDA rows use

100

clients with

10

sampled per round and are averaged over partition and client-sampling seeds. Wall-clock is the only backbone-dependent quantity and is reported separately for each; measured values are the median over the runs of that backbone that completed their configured budget in that setting, and † marks a projected value for the step-matched configuration. Runs were executed on a shared cluster with heterogeneous nodes, so wall-clock should be read as an order-of-magnitude comparison rather than a controlled timing benchmark, and it measures our simulation cost rather than the latency of a real deployment; Val. evals is the number of validation passes performed over a run.

Panel of Figure 7 — Figure 7: Validation Error against Three Measures of Training Budget. Each row shows the same six runs for one backbone — the budget-matched $200$ -epoch centralized re-run, the three IID client scales, and the LDA partitions at $\alpha=0.1$ and $\alpha=10$ — plotted against sample-visits (left), optimizer steps (middle), and the native progress counter each regime was logged on, epochs centrally and communication rounds federated (right). The left two panels place the two regimes on a common axis and are the comparison table 8 quantifies; the right panel is what that comparison looks like when it is not aligned, and is included because epochs and rounds are not interchangeable units. Faint traces are the raw per-epoch and per-round evaluations and bold traces an exponential moving average over them. All axes are linear, and the vertical range is clipped to the region where the curves separate, so the first few evaluations of each run fall above the panel. On the right-hand panel the horizontal extent of a curve is set by the number of rounds its configuration was run for rather than by the training it received, which is why 1000/10 spans six times the axis of the runs it is compared against while receiving less than a third of their gradient updates.

Figure 8: Details of the Transformer encoder-decoder architecture used in FastMETRO ( Cho et al., 2022 ) and TORE ( Dou et al., 2023 ) .

Methods	Model	Round Time	Convergence
FL	TORE	139.8s	Converge
MPC	TORE	265.5s	Converge
DP	TORE	139.8s	Non-Converge

Appendix

Table 9: Comparison of different privacy-preserving machine learning paradigms. The experiment uses the 10/10 TORE setting on an IID Human3.6M partition and reports average per-round runtime.

Settings	Model	MPJPE	PA-MPJPE	Raw Data Local?
Centralized Baseline	FM-S	57.98	40.62	✗
Naïve 5% Data Training	FM-S	83.78	63.65	-
Naïve 10% Data Training	FM-S	73.04	53.36	-
Naïve 20% Data Training	FM-S	65.10	47.01	-
PCHMR 100% Data Training	FM-S	59.31	41.48	✓
Centralized Baseline	TORE-S	63.88	41.99	✗

Appendix

Table 10: Comparison with Naïve Single-Device Training. This table presents the performance metrics (MPJPE, PA-MPJPE) for training using various sizes (5, 10, 20%) of the Human3.6M dataset to assess the on-device training scenario using only local data. Compared with naïve local training, which suffers from poor performance due to data scarcity, our PCHMR could mitigate both issues due to the inter-device parameter sharing, achieving much higher performance while keeping raw data local. The PCHMR in this table is trained in a distributed setting with 100 devices and 10 devices selected at random each time.

Local Epochs ( $E$ )	Model	MPJPE	PA-MPJPE	Raw Data Local?
1	FM-S	59.31	41.48	✓
2	FM-S	59.28	42.42	✓
3	FM-S	60.49	42.43	✓
5	FM-S	59.52	42.07	✓
10	FM-S	60.03	42.96	✓
1	TORE-S	62.92	43.11	✓

Appendix

Table 11: Effect of Multiple Local Epochs. Ablation on training the HMR model with more than one local epoch

E

per global round in the federated setting. Note that raising

E

raises the number of local optimizer steps each sampled client takes per round, and hence the total training budget.

Panel of Figure 9 — Figure 9: Data Distribution of Applying LDA. Distribution of labels from each dataset using LDA with 5 clients under (a) $\alpha=0.5$ and (b) $\alpha=1000$ . Note that the high $\alpha$ LDA retained a similar label distribution to the original category distribution, whereas the lower $\alpha$ introduced a much different category distribution.

Panel of Figure 10 — Figure 10: LDA-Partitioned Dataset Size for Individual Clients. This figure illustrates the size of the dataset assigned to each client, sorted by the size of data for (a) 100 partitions and (b) 1000 partitions, using LDA. Note that the lower $\alpha$ creates a much more skewed long-tail distribution following the Dirichlet distribution that it is sampled from.

Figure 11: CNN-Partitioned Data Visualization. For visualization, this diagram clusters the selected datasets into 10 partitions with MiniBatchKMeans (the experiments use 100) and plots them using t-SNE ( Van der Maaten and Hinton, 2008 ) . Each partition is sampled with 100 data points for ease of computation. The data points with the same color come from the same cluster.

Figure 12: 3DPW Dataset Distribution. This diagram shows the natural distribution of different scenes in the 3DPW ( Von Marcard et al., 2018 ) dataset. Note that this is only the train set of the 3DPW dataset following the established data train-val partition.

Settings	Model	MPJPE	PA-MPJPE	Raw Data Local?
Centralized	FM-S	57.98	40.62	✗
$\alpha$ = 0.1	FM-S	51.11	37.24	✓
$\alpha$ = 0.5	FM-S	57.80	39.82	✓
$\alpha$ = 1	FM-S	59.63	39.80	✓
$\alpha$ = 10	FM-S	56.09	39.84	✓
$\alpha$ = 100	FM-S	57.53	40.69	✓

Appendix

Table 12: Data Distribution Evaluation. MPJPE and PA-MPJPE under different data distributions (LDA with varying

\alpha

and CNN clustering), evaluated on Human3.6M ( Ionescu et al., 2013 ) . PCHMR’s error is competitive with the centralized baseline in every setting and lower in some skewed partitions; we do not isolate the cause of these gains. All models (except centralized) in this table are trained in a distributed setting with 100 devices, with 10 devices selected at random each round.

Setting	Model	MPJPE	PA-MPJPE	MPVPE	Raw Data Local?
No Fine-tuning	FM-S	116.35	70.20	132.96	✗
Centralized Fine-tuning	FM-S	84.92	54.69	97.6	✗
PCHMR Fine-tuning	FM-S	84.66	54.78	97.42	✓
No Fine-tuning	TORE-S	119.64	68.35	138.93	✗
Centralized Fine-tuning	TORE-S	87.97	55.35	101.88	✗
PCHMR Fine-tuning	TORE-S	87.55	54.08	101.86	✓

Appendix

Table 13: Privacy-Constrained Fine-Tuning Table. Results of privacy-constrained fine-tuning of models ( Dou et al., 2023 ; Cho et al., 2022 ) pre-trained on Human3.6M, UP-3D, COCO, MPII and MuCo-3DHP ( Ionescu et al., 2013 ; Lassner et al., 2017 ; Lin et al., 2014 ; Andriluka et al., 2014 ; Mehta et al., 2018 ) using PCHMR on the 3DPW dataset ( Von Marcard et al., 2018 ) , compared with centralized fine-tuning with the same datasets and the model without fine-tuning. The privacy-constrained fine-tuning is conducted with 100 clients and 10 randomly sampled per round.

Figure 13: Qualitative results of Cho et al. (2022) before and after privacy-constrained fine-tuning on the 3DPW ( Von Marcard et al., 2018 ) dataset using the PCHMR framework.

Model	Device	100 Partitions		1000 Partitions
Model	Device	Throughput	Time	Throughput	Time
FM-S	Jetson AGX - 15W	2.46	1629.33	2.42	165.33
FM-S	Jetson AGX - 50W	8.01	499.67	7.81	51.23
FM-S	RTX A6000 Ada	56.59	70.7	48.0	8.33
TORE-S	Jetson AGX - 15W	3.43	1166.06	3.35	119.29
TORE-S	Jetson AGX - 50W	8.65	461.5	8.42	47.49

Appendix

Table 14: On-Device Performance Table. This table shows the image throughput and total time used for training with 100 (4000 images) and 1000 (400 images) partitions of client data on different power profiles for Nvidia AGX Orin. We also compare the image throughput and total time with a desktop-level graphics card (RTX A6000 Ada). The unit for Throughput is images per second (img/s), and the unit for Time is seconds (s).

Human Mesh Recovery

Human mesh recovery estimates articulated 3D body shape and pose from images, video, or other sensors for animation, tracking, and physical analysis.

See all 51 papers

Monocular Models are Strong Learners for Multi-View Human Mesh Recovery
Mar 20, 2026Haoyu Xie, Shengkai Xu, Cheng Guo +6Human Mesh Recovery Multi-View 3D Reconstruction
University of North Carolina at Charlotte · University of Central Florida
Multiview Multi-Person Human Mesh Recovery Under Large Scenes with Occlusions
Jul 27, 2026Qi Zhang, Tao Yu, Jiechao He +2Human Mesh Recovery 3D Human Pose Estimation
Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, College of Computer Science and Software Engineering, Shenzhen University, China · City University of Hong Kong, China
DGHMesh: A Large-scale Dual-radar mmWave Dataset and Generalization-focused Benchmark for Human Mesh Reconstruction
Apr 19, 2026Rongxiao Guo, Qingchao Chen3D Reconstruction Human Mesh Recovery
Institute of Medical Technology, Peking University Health Science Center · National Institute of Health Data Science at Peking University · State Key Laboratory of General Artificial Intelligence, Peking University

Mar 20, 2026J/K move · Enter open · S save

Monocular Models are Strong Learners for Multi-View Human Mesh Recovery

Haoyu Xie, Shengkai Xu, Cheng Guo, Muhammad Usama Saleem+5

University of North Carolina at Charlotte · University of Central Florida

Human Mesh Recovery Multi-View 3D Reconstruction

Open paper AskPDF arXiv

Multi-view human mesh recovery (HMR) is broadly deployed in diverse domains where high accuracy and strong generalization are essential. Existing approaches can be broadly grouped into geometry-based and learning-based methods. However, geometry-based methods (e.g., triangulation) rely on cumbersome camera calibration, while learning-based approaches often generalize poorly to unseen camera configurations due to the lack of multi-view training data, limiting their performance in real-world scenarios. To enable calibration-free reconstruction that generalizes to arbitrary camera setups, we propose a training-free framework that leverages pretrained single-view HMR models as strong priors, eliminating the need for multi-view training data. Our method first constructs a robust and consistent multi-view initialization from single-view predictions, and then refines it via test-time optimization guided by multi-view consistency and anatomical constraints. Extensive experiments demonstrate state-of-the-art performance on standard benchmarks, surpassing multi-view models trained with explicit multi-view supervision.

Abstract

Human Mesh Recovery

Explore similar work

Monocular Models are Strong Learners for Multi-View Human Mesh Recovery

Multiview Multi-Person Human Mesh Recovery Under Large Scenes with Occlusions

DGHMesh: A Large-scale Dual-radar mmWave Dataset and Generalization-focused Benchmark for Human Mesh Reconstruction