Monocular Models are Strong Learners for Multi-View Human Mesh Recovery

Haoyu Xie; Shengkai Xu; Cheng Guo; Muhammad Usama Saleem; Wenhan Wu; Chen Chen; Ahmed Helmy; Pu Wang

cs.CVMar 20, 2026

Monocular Models are Strong Learners for Multi-View Human Mesh Recovery

Authors: Haoyu Xie, Shengkai Xu, Cheng Guo, Muhammad Usama Saleem, Wenhan Wu, Chen Chen, Ahmed Helmy, Pu Wang,

Organizations: University of North Carolina at Charlotte · University of Central Florida

Abstract

Multi-view human mesh recovery (HMR) is broadly deployed in diverse domains where high accuracy and strong generalization are essential. Existing approaches can be broadly grouped into geometry-based and learning-based methods. However, geometry-based methods (e.g., triangulation) rely on cumbersome camera calibration, while learning-based approaches often generalize poorly to unseen camera configurations due to the lack of multi-view training data, limiting their performance in real-world scenarios. To enable calibration-free reconstruction that generalizes to arbitrary camera setups, we propose a training-free framework that leverages pretrained single-view HMR models as strong priors, eliminating the need for multi-view training data. Our method first constructs a robust and consistent multi-view initialization from single-view predictions, and then refines it via test-time optimization guided by multi-view consistency and anatomical constraints. Extensive experiments demonstrate state-of-the-art performance on standard benchmarks, surpassing multi-view models trained with explicit multi-view supervision.

Figures & tables

Figure 1: The model architectures of (a) HMR2.0, (b) CameraHMR, (c) TokenHMR, and (d) a summary of these generic single-view transformer-based HMR models. The optimizable components can be selected from three representative groups: tokens (pink/orange), model parameters (blue), or explicit SMPL parameters (green).

Figure 2: Test-time adaptation (TTA) framework for multi-view human mesh recovery. The pipeline is composed of two stages: (a) aggregating single-view SMPL estimates into a set of SMPL parameters initialization termed virtual view ; (b) Test-time adaptation refines single-view optimizable token and virtual view SMPL parameters using 2D Loss, multi-view consistency Loss, and regularization loss.

	Method	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
	HMR2.0(a) [ 9 ]	44.8	33.6
	HMR2.0(b) [ 9 ]	50.0	32.4
Single Camera	HMR2.0+scoreHMR-a [ 47 ]	47.9	28.4
	HMR2.0+scoreHMR-b [ 47 ]	44.7	29.0
	PostureHMR [ 46 ]	44.5	31.0
	Shape Aware [ 25 ]	79.9	45.1

Table 1: Comparison on the Human3.6M dataset. Methods are grouped into single-view, multi-view calibration-free, and multi-view calibration-requiring approaches. Bold and underline indicate the best and second-best results respectively.

Method	MPJPE $\downarrow$	PCK $\uparrow$	AUC $\uparrow$
PaFF [ 16 ]	48.4	98.6	67.3
U-HMR [ 23 ]	39.7	74.0	99.4
HeatFormer [ 30 ]	39.8	99.5	72.8
Ours (TokenHMR)	45.2	99.9	79.6
Ours (HMR2.0)	40.3	99.6	80.7
Ours (CameraHMR)	39.0	99.9	83.8

Table 2: Comparison results on MPI-INF-3DHP dataset with multi-view methods. Bold indicates the best result, Underline indicates the second-best result.

Method	MPJPE $\downarrow$	PCK $\uparrow$	AUC $\uparrow$
HeatFormer	45.74	99.15	68.93
Ours(CameraHMR)	43.71	99.44	80.62

Table 3: Comparison results on MPI-INF-3DHP dataset with cross camera settings.

Components		Camerahmr		HMR2.0		TokenHMR
Virtual View	TTA	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
✗	✓	27.8	22.7	35.1	24.1	43.9	27.5
✓	✗	44.2	29.9	44.4	24.0	37.1	25.7
✓	✓	26.9	20.6	31.6	21.5	32.6	23.2

Table 4: Ablation study on the effects of Virtual View design and TTA.

	Camerahmr		HMR2.0		TokenHMR
Strategy	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
No Virtual View	27.8	22.7	35.1	24.1	43.9	27.5
T-pose	27.2	21.2	32.5	22.4	33.8	23.6
Averaged	27.1	20.7	32.1	21.7	33.5	23.3
Weighted	26.9	20.6	31.6	21.5	32.6	23.2

Table 5: Ablation study on virtual view initialization strategy.

	Camerahmr		HMR2.0		TokenHMR
Guidance	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
anatomical (ana)	39.9	29.4	35.2	24.6	33.5	24.5
keypoint (kp)	39.6	28.5	35.0	24.0	33.2	23.7
kp+ana	36.8	27.2	32.3	22.5	33.0	23.6
kp+ana+consistency (con)	27.3	20.9	32.2	21.6	32.9	23.4
kp+ana+con+regularizer	26.9	20.6	31.6	21.5	32.6	23.2

Table 6: Ablation study on TTA loss components.

	Camerahmr		HMR2.0		TokenHMR
Component	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
SMPL Parameters	58.2	32.2	53.0	31.9	48.4	31.1
Decoder	30.8	22.8	37.2	23.3	36.9	27.5
Learnable Tokens	29.7	20.9	36.0	23.0	35.2	25.7
Learned Tokens	26.9	20.6	31.6	21.5	32.6	23.2

Table 7: Ablation study on single-view backbone optimizable component.

	Camerahmr		HMR2.0		TokenHMR
Component	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
Pose	31.6	21.9	35.2	24.6	34.6	23.3
Pose+Orientation	29.0	21.4	33.4	22.0	34.4	23.3
Pose+Orientation+Shape	26.9	20.6	31.6	21.5	32.6	23.2

Table 8: Ablation study on Virtual-view Optimizable Component.

Figure 3: Mesh alignment on the Human3.6M [ 14 ] (left) and MPI-INF-3DHP [ 31 ] (right). From left to right: input image, mesh at Step 0, and mesh at step 200.

Figure 4: TTA 2D clues alignment. Green markers denote the pseudo ground-truth from 2D detector, while the red markers represent the model’s predictions.

Hyper-parameter	Camerahmr	HMR2.0	TokenHMR
$\eta$	$6e^{-2}$	$1e^{-1}$	$6e^{-2}$
$\lambda_{kp}$	$3e^{-1}$	$3e^{-1}$	$3e^{-1}$
$\lambda_{ana}$	$3e^{-1}$	$3e^{-1}$	$3e^{-1}$
$\lambda_{reg\_orient}$	$3e^{1}$	$1e^{1}$	$1e^{1}$
$\lambda_{reg\_pose}$	$1e^{-1}$	$1e^{-1}$	$1e^{-1}$
$\lambda_{reg\_betas}$	$2e^{-2}$	$2e^{-2}$	$2e^{-2}$

Table 9: Hyper-parameters for each model.

Steps	MPJPE ↓	PA-MPJPE ↓	AITI(s) ↓
0	44.23	29.86	0.03
50	36.54	26.64	0.63
100	30.36	23.26	1.26
150	28.52	22.17	1.88
200	27.56	21.61	2.50

Table 10: Inference time with star-structured consisntecy graph.

# of view	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
2	31.2	23.0
3	29.5	22.0
4	26.9	20.6

Table 11: Ablation study on the number of views.

learning rate	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
$1e^{-2}$	29.8	21.1
$3e^{-2}$	27.4	21.0
$6e^{-2}$	26.9	20.6
$1e^{-1}$	29.5	22.0

Table 12: Ablation study on learning rate.

	Camerahmr		HMR2.0		TokenHMR
Steps	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
50	35.6	25.9	35.4	22.0	35.1	24.3
100	29.9	22.3	32.4	21.5	33.5	23.6
150	28.0	21.2	31.6	21.5	32.8	23.3
200	26.9	20.6	31.6	21.5	32.6	23.2
250	27.4	21.0	31.7	21.5	32.7	23.3

Table 13: Ablation study on TTA steps.

		CameraHmr		HMR2.0		TokenHMR
2D detector	EPE	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
w/o evaluation data(ours)	5.94	26.9	20.6	31.6	21.5	32.6	23.2
w/ 1k evaluation data	5.05	23.8	18.0	30.1	20.3	31.7	22.2
w/ 2k evaluation data	4.74	21.8	16.1	29.9	20.1	31.5	21.6
w/ whole evaluation data	2.41	19.0	13.2	27.6	18.3	31.0	21.4
ground truth	0	15.8	10.4	26.7	17.5	30.0	20.7

Table 14: Ablation study on quality of 2D detector.

	Camerahmr		HMR2.0		TokenHMR
TTA Strategy	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$	MPJPE $\downarrow$	PA-MPJPE $\downarrow$
w/o gradient clip norm	30.5	22.5	37.0	24.6	35.2	24.1
w/ gradient clip norm	26.9	20.6	31.6	21.5	32.6	23.2

Table 15: Ablation study on gradient clip norm during TTA

Figure 5: TTA 2D clues alignment. Green markers denote the pseudo ground-truth from 2D detector, while the red markers represent the model’s predictions.

Figure 6: Qualitative comparison between HeatFormer and our method on the Human3.6M dataset. Pink meshes denote our predictions, while white meshes correspond to HeatFormer (iteration 4).

Figure 7: Mesh recovery and TTA alignment on real-world scenarios using the CVLab–EPFL Basketball Sequence. Red boxes highlight the misalignment in the initial predictions (Step 0).

Human Mesh Recovery

Human mesh recovery estimates articulated 3D body shape and pose from images, video, or other sensors for animation, tracking, and physical analysis.

See all 51 papers

Multiview Multi-Person Human Mesh Recovery Under Large Scenes with Occlusions
Jul 27, 2026Qi Zhang, Tao Yu, Jiechao He +2Human Mesh Recovery 3D Human Pose Estimation
Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, College of Computer Science and Software Engineering, Shenzhen University, China · City University of Hong Kong, China
FactorizedHMR: A Hybrid Framework for Video Human Mesh Recovery
May 14, 2026Patrick Kwon, Chen Chen3D Reconstruction Human Mesh Recovery
Institute of Artificial Intelligence University of Central Florida Orlando, FL 32817
MetricHMSR:Metric Human Mesh and Scene Recovery from Monocular Images
Jun 11, 2025Chentao Song, He Zhang, Haolei Yuan +4Metric Depth Estimation 3D Reconstruction
Tsinghua University · Beihang University · Beijing Normal University

Jul 27, 2026J/K move · Enter open · S save

Multiview Multi-Person Human Mesh Recovery Under Large Scenes with Occlusions

Qi Zhang, Tao Yu, Jiechao He, Antoni B. Chan+1

Guangdong Provincial Key Laboratory of Visual Media and Multidimensional Intelligence, College of Computer Science and Software Engineering, Shenzhen University, China · City University of Hong Kong, China

Human Mesh Recovery 3D Human Pose Estimation 3D Mesh Reconstruction 3D Human Reconstruction

Open paper AskPDF arXiv

Human mesh recovery (HMR) aims to recover 3D human meshes from images. Most existing HMR benchmarks and methods focus on either multi-person reconstruction from a single view or single-person reconstruction from multiple views, where the number of subjects and the scene scale are relatively limited. Such settings are insufficient for real-world applications with large scenes and severe inter-person occlusions. To address this limitation, we introduce a large-scale synthetic benchmark for multiview multi-person HMR, termed MVMP-HMR. The proposed dataset contains 15 complex scenes with up to 50 camera views and 30 interacting persons, featuring large spatial coverage and severe occlusions, which significantly increases the difficulty of human mesh recovery. Based on this benchmark, we further propose a multiview multi-person whole-body human mesh recovery model, referred to as MVMP-HMR model. The model first fuses multiview features into a scene-level 3D feature volume, and then leverages pelvis joints predicted by a 3D pose estimation network to extract person-specific queries from the 3D feature volume. These human queries are cross-attended with the 3D feature volume and integrated to decode each person's 3D mesh. Moreover, we introduce two novel losses--the orientation loss and the 3D joint density loss--to alleviate orientation and pose ambiguities under severe occlusions. Experiments demonstrate that existing state-of-the-art HMR methods struggle on the proposed MVMP-HMR benchmark, while our method consistently outperforms prior SOTAs in large-scale scenes with severe occlusions.

Abstract

Human Mesh Recovery

Explore similar work

Multiview Multi-Person Human Mesh Recovery Under Large Scenes with Occlusions

FactorizedHMR: A Hybrid Framework for Video Human Mesh Recovery

MetricHMSR:Metric Human Mesh and Scene Recovery from Monocular Images