WildHSR: Metric Feed-Forward 4D People-Scene Reconstruction from a 3D Foundation Model

Jerrin Bright; John Zelek

cs.CVSep 24, 2026

WildHSR: Metric Feed-Forward 4D People-Scene Reconstruction from a 3D Foundation Model

Authors: Jerrin Bright, John Zelek

Organizations: Vision and Image Processing Lab, University of Waterloo, Canada

Abstract

3D foundation models recover video cameras and geometry in one forward pass, but some of the strongest are up to scale. Joint people-scene reconstruction then requires two missing outputs: metric scale and persistent person identity. We ask whether one up-to-scale foundation representation can support both through lightweight adaptation. Exact metric labels are scarce, but unlabeled in-the-wild video is abundant. We use people in curated web video to initialise the solution: a posed metric body and 2D keypoints give an approximate, closed-form scale pseudo-label. These pseudo-labels pretrain a Scale Readout, which is then fine-tuned together with a lightweight adapter using exact metric supervision from standard real-video training splits. At inference the head predicts metric scale from foundation-model tokens, without the ruler or its teachers. For person identity, we probe the pretrained foundation model alone and find evidence that its intermediate query-key features encode person correspondence across frames. In most evaluated moving-person clips, a mid-layer token prefers that person over the vacated location and other people. A tiny projection reads this correspondence; together with metric pelvis motion and proposal confidence, it drives dustbin-aware Sinkhorn association of per-frame bodies. WildHSR combines both readouts to reconstruct metric cameras, scene and people from monocular video. Each window is predicted feed-forward; analytic association and Sim(3) composition connect windows. On EMDB-2, WildHSR is the first feed-forward method in the published comparison to beat the best optimization-based WA-MPJPE and RTE while leading feed-forward methods on all three world-frame metrics. On RICH, it leads feed-forward people-and-scene methods on WA-MPJPE and W-MPJPE. The complete pipeline runs at 10.1 fps on one GPU.

Figures & tables

Figure 1: Metric people, scene and camera from one moving camera. Each panel overlays persistent people, the person-free scene and the camera path on a 1 m grid. The large EMDB example spans 33 s; its insets show pelvis-token attention and a top view against ground truth after rigid rotation-and-translation alignment, preserving the predicted scale. The remaining clips cover cyclic motion, a 14-person street, stairs and an indoor walk. Inference receives a single monocular video.

Figure 2: Person-patch matching at layer 13. Rows are patches on three people at $t$ ; columns are patches on the same people at $t^{\prime}$ . Color shows pre-rotation query-key probe scores, normalized only over these candidates per head and then averaged, not scene-wide attention. Dashed blocks mark same-person pairs; $82\%$ of top matches have the right identity here.

Figure 3: Training and deploying the Scale Readout. (1) A body ruler combines observed 2D torso extent $e_{2}$ , the posed body’s metric in-plane extent $e_{3}$ , backbone depth $\tilde{z}_{k}$ and focal length $f$ to estimate $\sigma_{k}=(fe_{3}/e_{2})/\tilde{z}_{k}$ ; an agreement gate yields window pseudo-labels. (2) These labels pretrain the readout on web clips. (3) Exact RICH and 3DPW targets fine-tune the readout and backbone LoRA. (4) Deployment predicts $\sigma$ from backbone tokens alone, without the ruler or teachers.

Figure 4: Overview of WildHSR . A 3D foundation model produces up-to-scale cameras, geometry and scene tokens. A mesh branch proposes people; a Pelvis Readout forms a body-localized query, and cross-attention decodes a metric SMPL-X body. The Scale Readout supplies $\sigma$ ; analytic association and Sim(3) composition link people and windows in one metric world.

Figure 5: Pretrained person correspondence. At layer 13, a VGGT- $\Omega$ pelvis token attends to its own person within a frame ( $3.8\times$ uniform over four clips) and across the window ( $5.8\times$ ). When the person moves, attention follows the person ( $5.0\times$ ), not the vacated location ( $1.2\times$ ). This probe runs the backbone alone; Appendix I gives controls and the complete sample accounting.

Figure 6: Scene and people in one metric world (three EMDB-2 clips), each method at its own metric scale; labels give placement error and, for UniSH and JOSH3R, view turn. WildHSR is $0.2$ to $0.3$ m off, against $0.3$ to $1.6$ m (Human3R), $0.9$ to $3.9$ m (UniSH) and $1.3$ to $5.8$ m (JOSH3R).

Table 3: Training-data ablation on EMDB-2. End-to-end joint-position (WA, W) and trajectory (RTE) errors for checkpoints trained with the indicated data sources.

method	IDF1 $\uparrow$	HOTA $\uparrow$	ID switches $\downarrow$	fragments $\downarrow$	MOTA $\uparrow$
Human3R (native)	87.4	72.3	1682	87	87.1
motion + confidence DP	66.9	56.9	97	77	87.5
raw VGGT query/key	51.8	53.4	8065	861	51.7
projected identity	60.2	54.5	664	284	73.7
projection + motion + confidence	82.1	74.5	39	73	96.5

Table 5: Association on 13 two-person 3DPW test sequences. WildHSR variants share fixed Multi-HMR outputs; Human3R uses its native pipeline. Assignment thresholds were tuned on four validation sequences.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

method	params run $\downarrow$	TFLOPs/frame $\downarrow$	fps $\uparrow$	s / 60 frames $\downarrow$	peak GPU $\downarrow$
Human3R	1.17B	n/a	8.8	6.8	5.95 GB
UniSH	1.86B	7.29	4.9	12.2	7.06 GB
JOSH3R	1.50B	3.77	6.1	9.8	7.55 GB
WildHSR ( $512\times 288$ )	1.52B	6.94	12.6	4.8	8.85 GB
WildHSR ( $688\times 384$ , ours)	1.52B	11.43	10.1	6.0	11.22 GB

Appendix

Table 6: Compute on one GPU, same 60-frame clip. End-to-end throughput and peak memory; FLOPs count traced model stages.

scale source	error $\downarrow$
body-ruler pseudo-label	14.1
Scale Readout, ruler-pretrained	8.4
Scale Readout, metric fine-tuned	7.2

Appendix

Table 7: Ruler calibration on held-out RICH. Median multiplicative scale error (%).

method	Acc. (cm) $\downarrow$
Spann3R	4.81
MapAnything	3.48
CUT3R	2.88
VGGT	2.32
AMB3R	1.75
WildHSR	1.66

Appendix

Table 8: 7Scenes scene accuracy. Mean point-map accuracy in cm (

\downarrow

);

Figure 7: Metric people-scene placement on nine EMDB-2 clips. Each method’s body history and scene retain predicted scale after rigid camera-path alignment. Labels report pelvis offset in metres; UniSH and JOSH3R also show the view rotation required for alignment.

Figure 9: Metric scene, camera and ground. Top three rows: person-free scene and camera path after rigid alignment, preserving predicted scale; labels give world-size ratio and mean camera error. Bottom: reconstructed floor height along the walk against the ground-truth feet; values are median foot-floor gap for Human3R / UniSH / WildHSR .

Figure 10: Accuracy against video length , averaged over six EMDB-2 clips of at least 1000 frames. (a) Camera trajectory error after Sim(3) alignment. (b) Metric camera-to-person distance error without alignment. (c) World-frame person-path error after rigid alignment, retaining scale error.

Figure 11: Crowded in-the-wild reconstruction on 3DPW downtown_runForBus . Top: input and persistent tracks. Bottom: metric scene, camera trajectory and bodies colored by identity. The sequence contains 14 tracks over 10 s, with 10 to 11 people reconstructed per frame.

	EMDB-2
configuration	WA $\downarrow$	W $\downarrow$	RTE $\downarrow$
WildHSR (full)	66.3	193.6	0.90
synthetic-only scale	182.2	1009.2	11.3
$-$ scene contact	72.2	203.4	0.91

Appendix

Table 9: Ablations on EMDB-2. Each row substitutes one component of the full system.

method	foot gap (cm) $\downarrow$	sliding (cm/s) $\downarrow$	world size ( $\times$ GT)	size err. $\downarrow$	camera (m) $\downarrow$	WA (mm) $\downarrow$
Human3R	12.7	22.6	1.03	15%	0.20	112.2
UniSH	15.5	28.1	0.47	73%	0.98	118.5
WildHSR	3.6	17.4	0.99	6%	0.13	66.3

Appendix

Table 10: People-scene consistency on EMDB-2. Five diagnostics use the first 300 frames of each sequence; WA-MPJPE uses full sequences. Shades rank methods per column, with world size ranked by distance from one.

Figure 12: Feet on the reconstructed ground (EMDB 79), each method on its own scene, at four moments with the same viewing direction and zoom.

Figure 13: Position or person? Pretrained VGGT- $\Omega$ by itself. Top: a query token on a walking person (white box) and its global attention on a later frame, at layers 4, 8, 13 and 21; dashed box, where the person was; green box, where they are. The example is the clip’s median, not its best. Bottom, means over clips, layers 10 to 16 shaded: (a) same-person retrieval on pairs where the nearest candidate in the image is the wrong person, in the attention’s query and key space and in token content; (b) attention lift after the person has moved.

Figure 14: Truncation and occlusion on a 300-frame EMDB-2 clip. Top: masked input and ground-truth outline. Middle: reconstructed bodies after WA alignment, with ground truth as dots. Bottom: per-frame WA-MPJPE on a log scale; shading marks masked frames.

condition	WildHSR $\downarrow$	Human3R $\downarrow$	coverage
clean	63.6	89.1	$100/100\%$
legs cut	119.5	238.1	$100/100\%$
head cut	80.9	131.8	$48/52\%$
moving occluder	71.3	181.6	$82/82\%$
frame dropout	183.6	254.4	$80/80\%$

Appendix

Table 11: Six-clip truncation and occlusion. Median WA-MPJPE (mm); body coverage is WildHSR / Human3R. Shading compares methods within each condition.

Figure 15: Measured failure modes. (a) Stair contact, (b) knee articulation, and (c) identity continuity during a burpee. Distances use each method’s own metric scale.

Explore similar work

Scene and Human in One World: Reconstruction in a Feedforward Pass
Jun 26, 2026Boao Shi, Qiao Feng, Yiming Huang +1Human Mesh Recovery Monocular Video
University of Pennsylvania, Philadelphia, Pennsylvania, USA
HARMONI: Aligning Human and Scene Priors for Multi-View 4D Reconstruction
Mar 13, 2026Sangmin Kim, Minhyuk Hwang, Geonho Cha +2Multi-View 4D Reconstruction
Seoul National University · NAVER Cloud
MetricHMSR:Metric Human Mesh and Scene Recovery from Monocular Images
Jun 11, 2025Chentao Song, He Zhang, Haolei Yuan +4Human Mesh Recovery Scene Reconstruction
Tsinghua University · Beihang University · Beijing Normal University

Jun 26, 2026, cs.CVJ/K move · Enter open · S save

Scene and Human in One World: Reconstruction in a Feedforward Pass

Boao Shi, Qiao Feng, Yiming Huang, Lingjie Liu

University of Pennsylvania, Philadelphia, Pennsylvania, USA

Human Mesh Recovery Monocular Video Scene Understanding Articulated Objects Body World

Open paper AskPDF arXiv

Reconstructing humans in dynamic scenes from moving monocular cameras remains challenging due to scale ambiguity, human-scene misalignment, and occlusion interference. Rather than treating human mesh recovery and scene reconstruction as separate tasks, we believe that accurate human-scene reconstruction requires the two tasks to mutually inform each other: parametric human models offer semantic structure and metric-scale priors, while scene geometry provides spatial context for human localization and alignment. Built on this insight, we introduce SHOW, a mask-promptable human mesh recovery framework that couples feed-forward 3D scene reconstruction with Human Mesh Recovery in a unified metric space. SHOW injects human semantics and scale priors from parametric human models into normalized point-map prediction, enabling metric-scale scene reconstruction from inherently scale-ambiguous monocular input. In turn, the recovered scene geometry constrains human mesh estimation, encouraging spatially consistent human placement and improved human-scene alignment. To handle complex multi-person and cluttered scenes, SHOW further incorporates a promptable masking mechanism that enables flexible target-human selection while suppressing background distractions and occlusion interference. Through joint training, the model learns both human-aware geometric features and geometry-constrained human features, producing aligned metric-scale reconstructions from monocular human-centric videos. Extensive experiments demonstrate that SHOW improves metric-scale consistency, human-scene alignment, and reconstruction accuracy under challenging camera motion, occlusion, and cluttered backgrounds.

Abstract

Explore similar work

Scene and Human in One World: Reconstruction in a Feedforward Pass

HARMONI: Aligning Human and Scene Priors for Multi-View 4D Reconstruction

MetricHMSR:Metric Human Mesh and Scene Recovery from Monocular Images