HORIA: 4D Human and Object Reconstruction from a Monocular Video with Interaction-Aware Re-Rendering

Hyeongjin Nam; Taeryung Lee; Daniel Sungho Jung; Kyoung Mu Lee

cs.CVOct 19, 2026

HORIA: 4D Human and Object Reconstruction from a Monocular Video with Interaction-Aware Re-Rendering

Authors: Hyeongjin Nam, Taeryung Lee, Daniel Sungho Jung, Kyoung Mu Lee

Organizations: Dept. of ECE&ASRI, Seoul National University · IPAI, Seoul National University

Abstract

Joint 4D reconstruction of a human and object from a monocular video is essential for understanding how people interact with their environment. While recent methods have made noticeable progress, capturing such human-object interactions remains difficult due to two main challenges. First, a video typically contains composite semantic context spanning multiple interaction phases, making it difficult to track the temporal changes of the interaction in the reconstruction. Second, under human-object occlusion, a single-view observation contains incomplete visual information about the interaction, resulting in geometrically inconsistent reconstructions. To address these challenges, we introduce HORIA, built upon two core designs. First, we decompose the entire input video into temporally grounded and semantically distinct interaction phases with motion-aware captioning, obtaining a specific text description for each phase. Second, conditioned on these text descriptions of the interaction, we propose an interaction-aware re-rendering that compensates for the limited single-view observation. Specifically, we render the initially reconstructed 4D human and object from novel viewpoints and refine the renderings with a text-to-video diffusion prior, which provides auxiliary novel-view supervision beyond single-view cues. As a result, our framework produces temporally coherent and accurate 4D reconstructions under diverse interaction scenarios, achieving state-of-the-art performance.

Figures & tables

Figure 2: Re-rendering for reconstruction. Conditioned on the text description $P_{\text{semantic}}$ , our re-rendering refines the novel-view renderings of the initial reconstruction, and the refined videos serve as auxiliary supervision for the final 4D reconstruction.

Figure 3: Overall pipeline of HORIA. Given an input video, our framework first decomposes it into interaction phases based on initially estimated human and object motion. Then, the text description of each phase guides the re-rendering of the human and object under novel viewpoints and provides auxiliary supervision for 4D HOI optimization.

Figure 4: Detailed process of video decomposition (left) and text-guided video refinement (right).

Caption	Decomposition	CD ${}_{\text{H}}{\downarrow}$	CD ${}_{\text{O}}{\downarrow}$	Contact ${\uparrow}$	Collision ${\downarrow}$	Accel ${}_{\text{H}}{\downarrow}$	Accel ${}_{\text{O}}{\downarrow}$
✗	✗	5.723	54.505	0.349	0.072	0.307	3.468
Standard	✗	5.500	51.155	0.391	0.059	0.291	3.129
Motion-aware	✗	5.511	42.102	0.408	0.060	0.294	3.210
Motion-aware	Uniform	5.404	37.121	0.469	0.056	0.298	2.986
Motion-aware (Ours)	Grounded	5.385	34.894	0.520	0.056	0.273	1.571

Table 1: Effect of video decomposition.

Re-rendering	Regional cond.	CD ${}_{\text{H}}{\downarrow}$	CD ${}_{\text{O}}{\downarrow}$	Contact ${\uparrow}$	Collision ${\downarrow}$	Accel ${}_{\text{H}}{\downarrow}$	Accel ${}_{\text{O}}{\downarrow}$
✗	✗	6.009	66.138	0.300	0.064	0.308	3.115
✓	✗	5.519	36.151	0.499	0.058	0.290	1.614
✓ (Ours)	✓	5.385	34.894	0.520	0.056	0.273	1.571

Table 2: Effect of re-rendering.

Methods	CD ${}_{\text{H}}{\downarrow}$	CD ${}_{\text{O}}{\downarrow}$	Contact ${\uparrow}$	Collision ${\downarrow}$	Accel ${}_{\text{H}}{\downarrow}$	Accel ${}_{\text{O}}{\downarrow}$
Initial reconstruction	5.821	88.508	0.275	0.069	0.334	20.034
w/o $\mathcal{L}_{\text{SDF}}$	5.706	48.121	0.354	0.063	0.291	2.410
w/o $\mathcal{L}_{\text{contact}}$	5.418	40.162	0.459	0.057	0.288	1.709
w/o $\mathcal{L}_{\text{collision}}$	5.390	38.945	0.512	0.061	0.277	1.602
HORIA (Ours)	5.385	34.894	0.520	0.056	0.273	1.571

Table 3: Ablation on 4D HOI optimization.

		Open4DHOI [ 51 ]						BEHAVE [ 3 ]
Methods		$\text{CD}_{\text{H}}{\downarrow}$	$\text{CD}_{\text{O}}{\downarrow}$	$\text{Contact}{\uparrow}$	$\text{Collision}{\downarrow}$	$\text{Accel}_{\text{H}}{\downarrow}$	$\text{Accel}_{\text{O}}{\downarrow}$	$\text{CD}_{\text{H}}{\downarrow}$	$\text{CD}_{\text{O}}{\downarrow}$	$\text{Contact}{\uparrow}$	$\text{Collision}{\downarrow}$	$\text{Accel}_{\text{H}}{\downarrow}$	$\text{Accel}_{\text{O}}{\downarrow}$
Image-based	PHOSA [ 62 ]	6.434	61.138	0.267	0.061	–	–	5.661	40.355	0.326	0.099	–	–
	HOI-Gaussian [ 50 ]	5.330	47.086	0.382	0.072	–	–	3.420	39.722	0.521	0.097	–	–
	TeHOR [ 30 ]	5.501	44.397	0.456	0.070	–	–	3.627	31.168	0.552	0.099	–	–
	HORIA (Ours)	5.380	37.268	0.501	0.056	–	–	3.512	13.013	0.569	0.080	–	–
Video-based	VisTracker [ 55 ]	6.392	51.289	0.383	0.152	0.459	2.197	6.049	16.664	0.464	0.108	1.553	1.336

Table 4: Quantitative comparison with state-of-the-art methods on Open4DHOI [ 51 ] and BEHAVE [ 3 ] . Bold and underlined values indicate the best and second-best scores, respectively.

Figure 5: Effectiveness of video decomposition. A single description of the entire video does not distinguish the interaction phases, resulting in implausible re-rendering and reconstruction. Our decomposition assigns a description to each phase, guiding the re-rendering to follow the interaction at each moment.

Figure 6: Effectiveness of 4D HOI optimization. For each interaction phase described with $P_{\text{semantic}}$ , our 4D HOI optimization corrects the artifacts (red circles) in the initial reconstructions, yielding accurate reconstruction results.

Figure 7: Qualitative comparison with state-of-the-art methods. We render each method’s reconstruction at $-30^{\circ}$ azimuth and highlight representative failure cases with red circles.

Methods		$\text{CD}_{\text{H}}{\downarrow}$	$\text{CD}_{\text{O}}{\downarrow}$	$\text{Contact}{\uparrow}$	$\text{Collision}{\downarrow}$	$\text{Accel}_{\text{H}}{\downarrow}$	$\text{Accel}_{\text{O}}{\downarrow}$
Image-based	PHOSA [ 62 ]	4.622	42.161	0.375	0.079	–	–
	HOI-Gaussian [ 50 ]	2.891	39.542	0.509	0.080	–	–
	TeHOR [ 30 ]	2.528	32.160	0.513	0.073	–	–
	HORIA (Init.)	2.705	63.394	0.406	0.098	–	–
	HORIA (Ours)	2.524	14.951	0.543	0.069	–	–
Video-based	VisTracker [ 55 ]	6.152	23.257	0.405	0.105	1.216	2.781

Table S1 : Quantitative comparison with state-of-the-art methods on InterCap [ 18 ] . Bold and underlined values indicate the best and second-best scores, respectively.

VisTracker [ 55 ]	InterTrack [ 56 ]	CARI4D [ 57 ]	HORIA (Ours)
403	783	453	439

Table S2 : Running time comparison with video-based methods. We report the average runtime (sec) per one second of video.

Methods	$\text{CD}_{\text{H}}{\downarrow}$	$\text{CD}_{\text{O}}{\downarrow}$	$\text{Contact}{\uparrow}$	$\text{Collision}{\downarrow}$	$\text{Accel}_{\text{H}}{\downarrow}$	$\text{Accel}_{\text{O}}{\downarrow}$
TeHOR [ 30 ]	5.750	44.176	0.426	0.080	1.590	24.685
HORIA (Ours)	5.385	34.894	0.520	0.056	0.273	1.571

Table S3 : Comparison with TeHOR [ 30 ] on every frame of Open4DHOI [ 51 ] .

Figure S3 : Failure case of reconstructing local details.

Figure S4 : More qualitative results of 4D HOI optimization. We render all reconstructions at $-30^{\circ}$ azimuth and highlight representative failure cases with red circles.

Figure S5 : More qualitative comparison with VisTracker [ 55 ] and CARI4D [ 57 ] . We render each method’s reconstruction at $-30^{\circ}$ azimuth and highlight representative failure cases with red circles.

Figure S6 : More qualitative comparison with VisTracker [ 55 ] and CARI4D [ 57 ] . We render each method’s reconstruction at $-30^{\circ}$ azimuth and highlight representative failure cases with red circles.

4D Reconstruction

4D reconstruction recovers changing 3D scenes, shapes, and motion over time from images, videos, or other sparse measurements.

See all 126 papers

HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration
Jun 26, 2026Jiaxin Li, Yuxiang Wu, Zhenkai Zhang +114D Reconstruction Monocular 3D Reconstruction
Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China · University of Science and Technology of China, China +1
ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors
Mar 4, 2026Zihao Huang, Tianqi Liu, Zhaoxi Chen +7HOI Generation Dynamic Scene Reconstruction
the School of AIA, Huazhong University of Science and Technology, Wuhan 430074, China · the S-Lab, Nanyang Technological University (NTU), Singapore 639798 · the Beijing Academy of Artificial Intelligence (BAAI), Beijing 100083, China +2
UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video
Apr 21, 2026Tanuj Sur, Shashank Tripathi, Nikos Athanasiou +44D Reconstruction Human Mesh Recovery
1National University of Singapore · 2Max Planck Institute for Intelligent Systems, Tübingen, Germany

Jun 26, 2026J/K move · Enter open · S save

HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration

Jiaxin Li, Yuxiang Wu, Zhenkai Zhang, Xinrui Shi+10

Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China · University of Science and Technology of China, China+1

4D Reconstruction Monocular 3D Reconstruction Dynamic Scene Reconstruction

Open paper AskPDF arXiv

Extracting dynamic 4D object interactions from massive, in-the-wild monocular videos offers a highly efficient data collection pathway for scaling Embodied AI and training VLAs. However, existing monocular 4D reconstruction methods primarily focus on isolated objects, often failing under the severe occlusions and complex dynamics inherent in multi-object interactions. To bridge this gap, we propose HAT-4D, the first agentic framework designed to reconstruct the 3D geometry, temporal dynamics, and physical interactions of multiple objects from a single video. By integrating VLMs with a multi-level human-in-the-loop feedback mechanism, HAT-4D efficiently resolves depth ambiguities and interaction-induced occlusions during 3D generation and 4D propagation, yielding physically plausible assets without relying on expensive multicamera rigs. As a scalable data engine, HAT-4D facilitates the creation of MVOIK-4D, an open-world benchmark for monocular 4D interaction reconstruction, accompanied by a novel multi-dimensional evaluation protocol focused on physical plausibility and temporal consistency. Extensive experiments demonstrate that HAT-4D achieves SOTA performance on most evaluation metrics, while maintaining competitive semantic alignment. Ablation studies show that introducing a small amount of human feedback improves interaction reconstruction. Moreover, the data produced by HAT-4D effectively improves baseline performance when used for fine-tuning. Our data and code are available at https://lijiaxin0111.github.io/HAT4D/

Abstract

4D Reconstruction

Explore similar work

HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration

ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors

UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video