CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction

Xianghui Xie; Bowen Wen; Yan Chang; Hesam Rabeti; Jiefeng Li; Ye Yuan; Gerard Pons-Moll; Stan Birchfield

cs.CVDec 12, 2025

CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction

Authors: Xianghui Xie, Bowen Wen, Yan Chang, Hesam Rabeti, Jiefeng Li, Ye Yuan, Gerard Pons-Moll, Stan Birchfield

Organizations: NVIDIA · University of Tübingen · Tübingen AI Center · Max Planck Institute for Informatics

Abstract

Accurate capture of human-object interaction from ubiquitous sensors like RGB cameras is important for applications in human understanding, gaming, and robot learning. However, inferring 4D interactions from a single RGB view is highly challenging due to the unknown object and human information, depth ambiguity, occlusion, and complex motion, which hinder consistent 3D and temporal reconstruction. Previous methods simplify the setup by assuming ground truth object template or constraining to a limited set of object categories. We present CARI4D, the first category-agnostic method that reconstructs spatially and temporarily consistent 4D human-object interaction at metric scale from monocular RGB videos. To this end, we propose a pose hypothesis selection algorithm that robustly integrates the individual predictions from foundation models, jointly refine them through a learned render-and-compare paradigm to ensure spatial, temporal and pixel alignment, and finally reasoning about intricate contacts for further refinement satisfying physical constraints. Experiments show that our method outperforms prior art by 38% on in-distribution dataset and 36% on unseen dataset in terms of reconstruction error. Our model generalizes beyond the training categories and thus can be applied zero-shot to in-the-wild internet videos. Our code and pretrained models will be publicly released.

Figures & tables

Figure 2: CARI4D method overview. Given a monocular RGB video, we reconstruct the 4D human and object at metric scale with consistent contacts. We start by estimating the metric-scale object mesh ( Sec. 3.1 ), followed by initialization of human and object poses using dynamic pose hypothesis selection ( Sec. 3.2 ). We then train a category agnostic contact reasoning model (CoCoNet) to refine the interaction poses and estimate hand contacts ( Sec. 3.3 ) which are used to perform contact aware joint optimization ( Sec. 3.4 ).

Figure 3: Qualitative comparison on BEHAVE dataset [ 3 ] . InterTrack [ 59 ] reconstructs human and object as point clouds only, and the shapes are noisy. VisTracker [ 57 ] requires known object templates, hence we augment it with our reconstructed objects (denoted as † ). Our method reconstructs the objects and tracks the poses accurately. (Purple balls indicate contact predictions.)

Figure 4: Zero-shot generalization to unseen InterCap dataset [ 18 ] . † Uses our reconstructed object meshes. Our method reconstructs the metric-scale object accurately and generalizes to unseen objects. (Purple balls indicate contact predictions.)

Method	CD-h $\downarrow$	CD-o $\downarrow$	CD-c $\downarrow$	Acc-h $\downarrow$	Acc-o $\downarrow$
InterTrack	25.71	47.66	30.20	5.23	5.64
VisTracker	13.52	18.29	14.22	0.54	0.77
Ours	7.74	12.05	9.23	1.14	0.75

Table 1: Evaluation results on BEHAVE [ 3 ] dataset (unit: cm). Our method significantly outperforms previous instance-specific VisTracker [ 57 ] and category-specific InterTrack [ 11 ] .

Method	CD-h $\downarrow$	CD-o $\downarrow$	CD-c $\downarrow$	Acc-h $\downarrow$	Acc-o $\downarrow$
PICO ∗	5.15	27.63	87.73	-	-
Ours ∗	3.21	9.04	5.90	-	-
InterTrack	24.83	35.88	27.69	4.73	4.79
VisTracker	16.12	27.41	20.17	0.98	1.25
Ours	11.06	15.69	12.88	1.25	0.82

Table 2: Zero-shot generalization to unseen InterCap [ 18 ] dataset (unit: cm). ∗ Denotes key-frames only, where acceleration metrics do not apply. Our method outperforms both image based method PICO [ 10 ] and video based tracking methods [ 59 , 57 ] .

Figure 5: Generalization to in-the-wild videos . Prior methods predict noisy shape (InterTrack [ 59 ] ), flipped object pose (VisTracker [ 57 ] , † with our object reconstruction) or wrong contacts and object position (PICO [ 10 ] ). Our method generalizes better overall. (Purple balls indicate contact predictions.)

Method	CD-h $\downarrow$	CD-o $\downarrow$	CD-c $\downarrow$	Acc-h $\downarrow$	Acc-o $\downarrow$
a. Raw NLF + FP tracking	$11.53$	$1565.42$	$405.13$	$3.06$	$4.34$
b. Raw NLF + FP pose est.	$11.53$	$40.54$	$13.02$	$3.06$	$9.26$
c. Our init. NLF + FP	$7.81$	$16.85$	$10.79$	$2.21$	$8.36$
d. Our init. + ref. w/o align	$8.01$	$13.55$	$9.95$	$2.08$	$3.74$
e. Our init. + ref. w/ align	7.01	$11.59$	8.62	$1.75$	$3.78$
f. Our full model	$8.41$	11.57	$9.35$	1.06	0.38

Table 3: Ablation studies. Our proposed initialization (c) is better than running vanilla (a, b) NLF [ 37 ] and FoundationPose (FP [ 52 ] ). Our contact reasoning model trained with the proposed alignment (e) further improves the accuracy. Joint optimization (f) improves smoothness and contact consistency (see Fig. 6 ).

Figure 6: Importance of contacts . Without our contact-aware optimization, the model does not properly handle the fine-grained hand-object interaction, leading to floating object or penetration errors. (Purple balls indicate contact predictions.)

Method	CD-h $\downarrow$	CD-o $\downarrow$	CD-c $\downarrow$	Acc-h $\downarrow$	Acc-o $\downarrow$
Ours	8.41	11.57	9.35	1.06	0.38
+GT mesh	8.05	9.26	8.18	1.03	0.32
+GT dmap	7.50	11.25	8.78	0.94	0.35
+GT mesh&dmap	7.23	7.78	7.20	0.92	0.27

Table 4: Oracle study . Trained on estimated depth, our model also allows input with ground truth depth or object mesh and achieves slightly better results.

Figure 7: CoCoNet architecture . Here $b,t,h,w$ denote batch size, temporal window size, image height and width respectively. We follow a render-and-compare paradigm, hence RGB $a$ and RGB $b$ denote the image from input observation and rendering respectively, same for xyz map and mask (human and object stacked together).

Method	PICO	InterTrack	VisTracker	Ours
Runtime (min) $\downarrow$	1560	198	118	45

Table 5: Average runtime (minutes) to process one video of 300 frames. Our method is much faster than baselines while being more accurate.

Figure 8: Sensitivity analysis of UniDepth and FoundationPose errors on the final performance.

Figure 9: Error per seq. (left) and visibility distribution (right).

Interaction type	CD-h $\downarrow$	CD-o $\downarrow$	CD-c $\downarrow$	Acc-h $\downarrow$	Acc-o $\downarrow$	Contact $\uparrow$
Hand mainly	$8.76$	$11.71$	$7.12$	$1.06$	$0.42$	$0.93$
Sitting	$8.16$	$10.10$	$7.23$	$1.01$	$0.32$	$0.95$
Shoulder contact	$10.24$	$12.39$	$9.27$	$1.03$	$0.48$	$0.95$
Leaning	$9.37$	$11.44$	$8.28$	$1.05$	$0.31$	$0.93$

Table 6: Performance of our method per interaction type.

Figure 10: Additional qualitative examples.

Figure 11: Failure case examples. Our method focuses on full body interaction and the detailed hand poses are not handled, which can be important for fine-grained object manipulation task (top row). Our method thus failed to reconstruct realistic finger poses for holding the plate. Under highly dynamic motion and extreme occlusion (bottom row), FoundationPose predicts flipped object pose for initialization. Such large rotation error is not able to be corrected by our refinement process in subsequent steps, leading to inaccurate reconstruction in the end.

Human-Scene Interaction Reconstruction

See all 13 papers

HORIA: 4D Human and Object Reconstruction from a Monocular Video with Interaction-Aware Re-Rendering
Oct 19, 2026Hyeongjin Nam, Taeryung Lee, Daniel Sungho Jung +1Human-Scene Interaction Reconstruction 4D Reconstruction
Dept. of ECE&ASRI, Seoul National University · IPAI, Seoul National University
HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration
Jun 26, 2026Jiaxin Li, Yuxiang Wu, Zhenkai Zhang +114D Reconstruction Monocular 3D Reconstruction
Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China · University of Science and Technology of China, China +1
UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video
Apr 21, 2026Tanuj Sur, Shashank Tripathi, Nikos Athanasiou +44D Reconstruction Human Mesh Recovery
1National University of Singapore · 2Max Planck Institute for Intelligent Systems, Tübingen, Germany

Oct 19, 2026J/K move · Enter open · S save

HORIA: 4D Human and Object Reconstruction from a Monocular Video with Interaction-Aware Re-Rendering

Hyeongjin Nam, Taeryung Lee, Daniel Sungho Jung, Kyoung Mu Lee

Dept. of ECE&ASRI, Seoul National University · IPAI, Seoul National University

Human-Scene Interaction Reconstruction 4D Reconstruction 3D Reconstruction Monocular 3D Reconstruction

Open paper AskPDF arXiv

Joint 4D reconstruction of a human and object from a monocular video is essential for understanding how people interact with their environment. While recent methods have made noticeable progress, capturing such human-object interactions remains difficult due to two main challenges. First, a video typically contains composite semantic context spanning multiple interaction phases, making it difficult to track the temporal changes of the interaction in the reconstruction. Second, under human-object occlusion, a single-view observation contains incomplete visual information about the interaction, resulting in geometrically inconsistent reconstructions. To address these challenges, we introduce HORIA, built upon two core designs. First, we decompose the entire input video into temporally grounded and semantically distinct interaction phases with motion-aware captioning, obtaining a specific text description for each phase. Second, conditioned on these text descriptions of the interaction, we propose an interaction-aware re-rendering that compensates for the limited single-view observation. Specifically, we render the initially reconstructed 4D human and object from novel viewpoints and refine the renderings with a text-to-video diffusion prior, which provides auxiliary novel-view supervision beyond single-view cues. As a result, our framework produces temporally coherent and accurate 4D reconstructions under diverse interaction scenarios, achieving state-of-the-art performance.

Abstract

Human-Scene Interaction Reconstruction

Explore similar work

HORIA: 4D Human and Object Reconstruction from a Monocular Video with Interaction-Aware Re-Rendering

HAT-4D: Lifting Monocular Video for 4D Multi-Object Interactions via Human-Agent Collaboration

UniCon3R: Unified Contact-aware 4D Human-Scene Reconstruction from Monocular Video