HARMONI: Aligning Human and Scene Priors for Multi-View 4D Reconstruction

Sangmin Kim; Minhyuk Hwang; Geonho Cha; Dongyoon Wee; Jaesik Park

cs.CVMar 13, 2026

HARMONI: Aligning Human and Scene Priors for Multi-View 4D Reconstruction

Authors: Sangmin Kim, Minhyuk Hwang, Geonho Cha, Dongyoon Wee, Jaesik Park

Organizations: Seoul National University · NAVER Cloud

Abstract

Recent advances in 3D foundation models have enabled joint reconstruction of humans and their surrounding environments. However, combining independently trained human and scene priors often produces misalignment in scale and depth. We observe that the two priors have complementary strengths. The scene prior provides consistent depth but approximate scale, while the human prior provides fixed body scale but less reliable depth. Motivated by this observation, we present HARMONI, a feed-forward framework that reconstructs cameras, scene, and humans with identities from monocular or multi-view video without test-time optimization. We introduce bidirectional anchoring, in which scene depth guides human placement, while human keypoints calibrate the scene scale to match the human body. While most existing approaches target monocular inputs and multi-view methods rely on optimization or re-identification, our approach naturally extends to multiple views. Building on bidirectional anchoring, we introduce a multi-view fusion module that merges per-view estimates, while reducing the influence of unreliable views and visually similar individuals. Experiments show that our framework outperforms previous human-scene methods in global motion and multi-view pose estimation by up to 28% and 64%, while running 28 times faster than optimization-based approaches. Project page: https://nstar1125.github.io/harmoni.

Figures & tables

Figure 1: (a) We present HARMONI , which reconstructs cameras, scene, and humans with identities from multi-view or monocular videos without test-time optimization. (b) With bidirectional anchoring , our model outperforms previous human-scene reconstruction methods in both settings.

Figure 2: Overview of our framework. The scene encoder and decoder reconstruct cameras $\pi$ , point maps $P$ , and an approximate metric scale $s$ . The human decoder and SMPL head predict per-view human estimates $h$ . Bidirectional anchoring (B.A.) couples the two branches. Scene depth $d^{\text{anchor}}$ guides human placement, and the keypoint-based ratio $r$ calibrates the scene scale. For multi-view inputs, the multi-view fusion module merges per-view estimates into a global estimate $\bar{h}$ .

Figure 3: Bidirectional Anchoring. (a) Human depth is anchored on the head surface depth with a predicted residual to the head joint. (b) Keypoints for calibration are localized using heatmaps. (c) Scene scale is calibrated by comparing SMPL and image body proportions.

	Settings			RICH			EMDB-2
	Multi-View	Multi-Person	Scene	WA-MPJPE $\downarrow$	W-MPJPE $\downarrow$	RTE(%) $\downarrow$	WA-MPJPE $\downarrow$	W-MPJPE $\downarrow$	RTE(%) $\downarrow$
JOSH3R	✗	✓	✓	-	-	-	220.0	661.7	13.1
UniSH	✗	✗	✓	118.1	183.2	4.8	118.5	270.1	5.8
Human3R	✗	✓	✓	110.0	184.9	3.3	112.2	267.9	2.2
Ours-mono	✗	✓	✓	84.9	132.4	2.6	83.6	214.4	1.4
Ours-multi	✓	✓	✓	45.4	61.2	0.9	-	-	-

Table 1: Quantitative comparison on RICH and EMDB-2 datasets. Ours-multi uses all available views and is reported for reference. red : the best, yellow : the second best.

	Settings			EgoExo4D		EgoHumans
	Video	No ReID	No Optim.	W-MPJPE ${}^{\dagger}\downarrow$	PA-MPJPE $\downarrow$	W-MPJPE ${}^{\dagger}\downarrow$	GA-MPJPE $\downarrow$	PA-MPJPE $\downarrow$	Runtime $\downarrow$
HSfM (init.)	✗	✗	✓	5.29	0.07	4.28	0.51	0.06	$\sim$ 21s
HSfM	✗	✗	✗	0.56	0.06	1.04	0.21	0.05	$\sim$ 118s
HAMSt3R	✗	✓	$\triangle$	0.51	0.09	3.80	0.42	0.14	$\sim$ 32s
Ours	✓	✓	✓	0.23	0.06	0.37	0.20	0.05	$\sim$ 1.1s

Table 2: Quantitative comparison on EgoExo4D and EgoHumans datasets. For fair comparison, all methods are evaluated with ground-truth identities. All metrics are in meters.

	EgoExo4D		EgoHumans
	W-M. ${}^{\dagger}\downarrow$	TE. $\downarrow$	W-M. ${}^{\dagger}\downarrow$	TE. $\downarrow$
Ours- $s$	0.31	0.54	1.56	3.01
Ours- $s^{*}$	0.23	0.19	0.37	0.31

Table 4: Ablation on scale calibration. Both metrics are computed in meters.

	Camera (EMDB-2)		Depth (Bonn)
	ATE $\downarrow$	RPE ${}_{\text{t}}\downarrow$	AbsRel $\downarrow$	$\delta{<}1.25\uparrow$
CUT3R	.059	.013	.048	.970
Human3R	.037	.012	.050	.961
Pi3	.025	.010	.026	.985
UniSH	.121	.021	.034	.985
Pi3X	.026	.008	.026	.986

Table 5: Camera pose and depth accuracy.

EMDB-2	WA-M. $\downarrow$	W-M. $\downarrow$	RTE(%) $\downarrow$
w/o scale calib.	157.8	422.3	4.2
Direct transl.	100.9	251.4	1.9
Direct depth	111.8	276.4	2.2
Ours-Stage 1	84.7	216.4	1.4

Table 6: Ablation on alignment strategies. All variants are trained up to Stage 1.

$\lambda_{c}$	$\lambda_{p}$	Acc.(%)	F1(%)	Prec.(%)	Recall(%)
0.0	1.0	89.5	78.9	94.1	67.9
0.2	0.8	89.9	79.9	94.0	69.6
0.5	0.5	90.7	82.1	92.8	73.6
0.8	0.2	90.9	83.3	88.5	78.7
1.0	0.0	68.3	59.6	47.2	80.8

Table 7: Ablation on association weights.

Figure 6: Qualitative results on the 3DPW dataset. We compare our method with monocular human-scene reconstruction baselines, including UniSH and Human3R.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

RICH	Penet. $\downarrow$	Float $\downarrow$
UniSH	13.7	11.5
Human3R	5.3	18.1
Ours	5.0	7.6

Appendix

Table 8: Physical plausibility evaluation.

Figure 8: Heatmap visualization of truncated scene and the reconstructed output.

Figure 9: Heatmap visualization of occluded scene and the reconstructed output.

Figure 10: Challenging scenario with truncation, occlusion, and object in front of the head.

Figure 11: Challenging scenario with crowded scene.

Figure 12: Multi-view reconstruction process.

Module	Runtime $\downarrow$
Scene encoder	0.09s
Scene decoder	0.25s
Scene heads	0.23s
Human encoder	0.40s
Human decoder	0.01s
SMPL head	0.02s

Appendix

Table 9: Runtime analysis on multi-view scene.

	Settings		EMDB-2
	Scene	Cam	WA-MPJPE $\downarrow$	W-MPJPE $\downarrow$	RTE(%) $\downarrow$
SLAHMR	✗	DROID-SLAM	326.9	776.1	10.2
WHAM	✗	DPVO	135.6	354.8	6.0
GVHMR	✗	DPVO	111.0	276.5	2.0
TRAM	✗	DROID-SLAM	76.4	222.4	1.4
Ours	✓	Feed-Forward	83.6	214.4	1.4

Appendix

Table 10: Comparison with world-grounded human motion recovery methods on EMDB-2.

	RICH			EMDB-2
	WA-MPJPE $\downarrow$	W-MPJPE $\downarrow$	RTE(%) $\downarrow$	WA-MPJPE $\downarrow$	W-MPJPE $\downarrow$	RTE(%) $\downarrow$
Human3R	110.0	184.9	3.3	112.2	267.9	2.2
Human3R + Calib.	111.2	190.1	3.5	106.4	250.4	2.1

Appendix

Table 11: Scale calibration applied to Human3R without depth residual prediction.

Figure 14: Qualitative results in multi-view settings.

Figure 15: Qualitative results in monocular settings.

Figure 16: Qualitative results on single images.

Explore similar work

TROPHIES: Temporal Reconstruction of Places, Humans, and Cameras from Multi-view Videos
Jun 1, 2026Jinpeng Liu, Yukang Xu, Yutong Li +1Scene Reconstruction Camera
National University of Singapore
MetricHMSR:Metric Human Mesh and Scene Recovery from Monocular Images
Jun 11, 2025Chentao Song, He Zhang, Haolei Yuan +4Human Mesh Recovery Scene Reconstruction
Tsinghua University · Beihang University · Beijing Normal University
Scene and Human in One World: Reconstruction in a Feedforward Pass
Jun 26, 2026Boao Shi, Qiao Feng, Yiming Huang +1Human Mesh Recovery Monocular Video
University of Pennsylvania, Philadelphia, Pennsylvania, USA

Jun 1, 2026, cs.CVJ/K move · Enter open · S save

TROPHIES: Temporal Reconstruction of Places, Humans, and Cameras from Multi-view Videos

Jinpeng Liu, Yukang Xu, Yutong Li, Xingyu Liu

National University of Singapore

Scene Reconstruction Camera Spatial Coherence Humanoid

Open paper AskPDF arXiv

Reconstructing humans and their surrounding environments in a globally consistent 4D space is essential for comprehensive perception. However, prior works typically assume single-view inputs or decouple humans, scenes, and cameras, making them unable to recover coherent geometry, stable motion, and physically aligned trajectories. These limitations motivate us to introduce a new task: unified human-scene-camera reconstruction from multi-view videos, which aims to jointly estimate dynamic humans, static scenes, and camera poses in one global coordinate frame. We propose TROPHIES--Temporal Reconstruction of Places, Humans, and Cameras from Multi-view Videos-a unified framework tailored for this task. TROPHIES features a Human Branch that models humans through temporal and spatial reasoning, and a Scene Branch that reconstructs static geometry with human-aware attention. A global alignment and optimization module couples both branches by enforcing scale consistency, contact priors, and cross-view temporal coherence. Experiments on EgoHuman and EgoExo4D demonstrate that TROPHIES achieves globally aligned, physically plausible 4D reconstructions and consistently outperforms existing paradigms in both global fidelity and human-scene consistency.

Abstract

Explore similar work

TROPHIES: Temporal Reconstruction of Places, Humans, and Cameras from Multi-view Videos

MetricHMSR:Metric Human Mesh and Scene Recovery from Monocular Images

Scene and Human in One World: Reconstruction in a Feedforward Pass