End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction

Haoyu Zhang; Wei Zhai; Yuhang Yang; Yang Cao; Zheng-Jun Zha

cs.CVMar 15, 2026

End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction

Authors: Haoyu Zhang, Wei Zhai, Yuhang Yang, Yang Cao, Zheng-Jun Zha

Organizations: University of Science and Technology of China

Abstract

Monocular 4D human-object interaction (HOI) reconstruction - recovering a moving human and a manipulated object from a single RGB video - remains challenging due to depth ambiguity and frequent occlusions. Existing methods often rely on multi-stage pipelines or iterative optimization, leading to high inference latency, failing to meet real-time requirements, and susceptibility to error accumulation. To address these limitations, we propose THO, an end-to-end Spatial-Temporal Transformer that predicts human motion and coordinated object motion in a forward fashion from the given video and 3D template. THO achieves this by leveraging spatial-temporal HOI tuple priors. Spatial priors exploit contact-region proximity to infer occluded object features from human cues, while temporal priors capture cross-frame kinematic correlations to refine object representations and enforce physical coherence. Extensive experiments demonstrate that THO operates at an inference speed of 31.5 FPS on a single RTX 4090 GPU, achieving a >600x speedup over prior optimization-based methods while simultaneously improving reconstruction accuracy and temporal consistency. The project page is available at: https://nianheng.github.io/THO-project/

Figures & tables

Figure 2: Overall pipeline of THO. From a monocular video, GVHMR extracts human priors and Interaction-Centric Crops ( $\mathcal{I}^{crop}$ ). The 3D Vertex Encoder generates unified embeddings ( $F^{J},F^{H},F^{O}$ ). Spatially, SCAT recovers occluded $F^{O}$ and injects $F^{H}$ via contact-aware cues. Fusing refined $f^{O}$ with joint and global contexts yields an aggregated motion token ( $f^{\text{token}}$ ). Temporally, TIAT processes this token sequence to model temporal dynamics ( $\hat{f}^{\text{token}}$ ), which MLPs decode into 4D HOI reconstruction.

Figure 3: Qualitative comparison on BEHAVE [ 1 ] . THO mitigates inaccurate symmetric object pose predictions (Rows 1-2), yields more reasonable poses under severe occlusion (Row 3), and ensures better human-object contact (Row 4).

Method	BEHAVE						InterCap						FPS $\uparrow$
	$\text{CD}_{h}\downarrow$	$\text{CD}_{o}\downarrow$	$\text{CD}_{c}\downarrow$	$\text{V2V}\downarrow$	$\text{Acc}_{h}\downarrow$	$\text{Acc}_{o}\downarrow$	$\text{CD}_{h}\downarrow$	$\text{CD}_{o}\downarrow$	$\text{CD}_{c}\downarrow$	$\text{V2V}\downarrow$	$\text{Acc}_{h}\downarrow$	$\text{Acc}_{o}\downarrow$
CHORE [ 50 ]	25.82	28.94	26.52	38.16	2.75	4.59	25.14	32.45	27.31	41.72	2.61	4.37	0.07
HOI-TG [ 44 ]	21.19	24.70	22.67	34.69	2.44	3.76	23.52	26.63	24.20	36.94	2.47	3.64	4.81
VisTracker [ 51 ]	11.24	12.58	11.63	22.45	0.83	1.21	12.57	13.80	12.76	24.35	0.86	1.27	0.05
Ours	8.49	13.36	10.06	21.68	0.79	1.02	9.31	14.45	11.51	23.72	0.83	1.12	31.5

Table 1: Quantitative comparison with state-of-the-art methods on BEHAVE [ 1 ] and InterCap [ 19 ] datasets. Reconstruction accuracy metrics ( CD and V2V ) are scaled in cm, temporal smoothness metrics ( Acc ) in cm/s 2 , and inference speed in FPS. The best results are highlighted in bold.

Figure 4: Efficiency vs. Performance on BEHAVE. THO achieves real-time inference (31.5 FPS), outperforming all baselines not only in inference speed but also yielding the best temporal smoothness (

\text{Acc}_{h},\text{Acc}_{o}

) as shown in (a) , and the highest reconstruction accuracy (

\text{CD}_{c}

) as shown in (b) .

	$\text{CD}_{h}\downarrow$	$\text{CD}_{o}\downarrow$	$\text{CD}_{c}\downarrow$	$\text{V2V}\downarrow$
w/o SCAT	8.53	14.59	10.47	24.32
w/ OR	8.52	14.22	10.35	23.08
w/ OR&HI	8.49	13.36	10.06	21.68

Table 2: Ablation study of SCAT. We evaluate the progressive contributions of Occlusion-Aware Internal Refinement (OR) and Human-Guided Contact Injection (HI) to refining object geometry and enhancing contact precision.

Figure 5: Impact of SCAT. OR recovers complete object geometry from occluded views, while HI corrects the relative pose to ensure physically consistent human-object contact, eliminating the misalignment observed in the baseline.

Figure 6: Visualization of Human-Guided Contact Injection. Left: Attention highlights the interacting hand while suppressing the proximal calf (semantic mismatch). Right: It isolates the contacting foot, ignoring the distant non-interacting foot.

	$\text{CD}_{h}\downarrow$	$\text{CD}_{o}\downarrow$	$\text{CD}_{c}\downarrow$	$\text{V2V}\downarrow$	$\text{Acc}_{h}\downarrow$	$\text{Acc}_{o}\downarrow$
w/o TIAT	10.34	16.45	12.38	27.31	1.12	1.97
w/o RoPE	8.65	13.60	10.27	22.74	0.84	1.20
w/o GLMD	8.63	15.26	10.79	25.85	0.81	1.16
w/o $f^{g}$	14.70	21.32	17.12	31.55	1.67	2.75
full model	8.49	13.36	10.06	21.68	0.79	1.02

Table 3: Ablation study of TIAT and GLMD. We assess the impact of removing full modules and their specific sub-components (RoPE in TIAT,

f^{g}

in GLMD) on reconstruction accuracy and temporal coherence.

Figure 7: Failure cases. We show reconstruction failures for two elongated objects (top and bottom rows). Each case compares our predicted meshes against the GT at two timestamps separated by 0.5s.

Spatiotemporal Transformers

See all 20 papers

Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos
May 14, 2026Yubo Zhao, Yujin Chai, Yunao Dong +43D Reconstruction Monocular 3D Reconstruction
The Hong Kong University of Science and Technology · Tencent IEG
HORIA: 4D Human and Object Reconstruction from a Monocular Video with Interaction-Aware Re-Rendering
Oct 19, 2026Hyeongjin Nam, Taeryung Lee, Daniel Sungho Jung +14D Reconstruction 3D Reconstruction
Dept. of ECE&ASRI, Seoul National University · IPAI, Seoul National University
High-Fidelity 4D Hand-Object Capture via Multi-View Spatiotemporal Tracking and Physics-Aware Gaussians
Jun 14, 2026Bo Peng, Xu Chen, Yi Gu +64D Reconstruction 3D Hand Pose Estimation
Google XR · The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China (USTC)

May 14, 2026J/K move · Enter open · S save

Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos

Yubo Zhao, Yujin Chai, Yunao Dong, Chengfeng Zhao+3

The Hong Kong University of Science and Technology · Tencent IEG

3D Reconstruction Monocular 3D Reconstruction Dynamic Scene Reconstruction Human-Object Interaction

Open paper AskPDF arXiv

Recovering 4D human-object interaction (HOI) from monocular video is a key step toward scalable 3D content creation, embodied AI, and simulation-based learning. Recent methods can reconstruct temporally coherent human and object trajectories, but these trajectories often remain visual artifacts while failing to preserve stable contact, functional manipulation, or physical plausibility when used as reference motions for humanoid-object simulation. This reveals a fundamental interaction gap: HOI reconstruction should not stop at tracking a human and an object, but should recover the relation that makes their motion a coherent interaction. We introduce

\textbf{HA-HOI}

, a framework for reconstructing physically plausible 4D HOI animation from in-the-wild monocular videos. Instead of treating the human and object as independent entities in an ambiguous monocular 3D space, we propose a

\textit{human-first, object-follow}

formulation. The human motion is recovered as the interaction anchor, and the object is reconstructed, aligned, and refined relative to the human action. The resulting kinematic trajectory is then projected into a physics-based humanoid-object simulation, where it acts as a teacher trajectory for stable physical rollout. Across benchmark and in-the-wild videos,

\textbf{HA-HOI}

improves human-object alignment, contact consistency, temporal stability, and simulation readiness over prior monocular HOI reconstruction methods. By moving beyond visually plausible trajectory recovery toward physically grounded interaction animation, our work takes a step toward turning general monocular HOI videos into scalable demonstrations for humanoid-object behavior. Project page: https://knoxzhao.github.io/real2sim_in_HOI/

Abstract

Spatiotemporal Transformers

Explore similar work

Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos

HORIA: 4D Human and Object Reconstruction from a Monocular Video with Interaction-Aware Re-Rendering

High-Fidelity 4D Hand-Object Capture via Multi-View Spatiotemporal Tracking and Physics-Aware Gaussians