Accurate capture of human-object interaction from ubiquitous sensors like RGB cameras is important for applications in human understanding, gaming, and robot learning. However, inferring 4D interactions from a single RGB view is highly challenging due to the unknown object and human information, depth ambiguity, occlusion, and complex motion, which hinder consistent 3D and temporal reconstruction. Previous methods simplify the setup by assuming ground truth object template or constraining to a limited set of object categories. We present CARI4D, the first category-agnostic method that reconstructs spatially and temporarily consistent 4D human-object interaction at metric scale from monocular RGB videos. To this end, we propose a pose hypothesis selection algorithm that robustly integrates the individual predictions from foundation models, jointly refine them through a learned render-and-compare paradigm to ensure spatial, temporal and pixel alignment, and finally reasoning about intricate contacts for further refinement satisfying physical constraints. Experiments show that our method outperforms prior art by 38% on in-distribution dataset and 36% on unseen dataset in terms of reconstruction error. Our model generalizes beyond the training categories and thus can be applied zero-shot to in-the-wild internet videos. Our code and pretrained models will be publicly released.
Figures & tables
Figure 2: CARI4D method overview. Given a monocular RGB video, we reconstruct the 4D human and object at metric scale with consistent contacts. We start by estimating the metric-scale object mesh ( Sec. 3.1 ), followed by initialization of human and object poses using dynamic pose hypothesis selection ( Sec. 3.2 ). We then train a category agnostic contact reasoning model (CoCoNet) to refine the interaction poses and estimate hand contacts ( Sec. 3.3 ) which are used to perform contact aware joint optimization ( Sec. 3.4 ).
Figure 3: Qualitative comparison on BEHAVE dataset [ 3 ] . InterTrack [ 59 ] reconstructs human and object as point clouds only, and the shapes are noisy. VisTracker [ 57 ] requires known object templates, hence we augment it with our reconstructed objects (denoted as † ). Our method reconstructs the objects and tracks the poses accurately. (Purple balls indicate contact predictions.)
Figure 4: Zero-shot generalization to unseen InterCap dataset [ 18 ] . † Uses our reconstructed object meshes. Our method reconstructs the metric-scale object accurately and generalizes to unseen objects. (Purple balls indicate contact predictions.)
Table 2: Zero-shot generalization to unseen InterCap [ 18 ] dataset (unit: cm). ∗ Denotes key-frames only, where acceleration metrics do not apply. Our method outperforms both image based method PICO [ 10 ] and video based tracking methods [ 59 , 57 ] .
Table 3: Ablation studies. Our proposed initialization (c) is better than running vanilla (a, b) NLF [ 37 ] and FoundationPose (FP [ 52 ] ). Our contact reasoning model trained with the proposed alignment (e) further improves the accuracy. Joint optimization (f) improves smoothness and contact consistency (see Fig. 6 ).
Figure 6: Importance of contacts . Without our contact-aware optimization, the model does not properly handle the fine-grained hand-object interaction, leading to floating object or penetration errors. (Purple balls indicate contact predictions.)
Method
CD-h ↓
CD-o ↓
CD-c ↓
Acc-h ↓
Acc-o ↓
Ours
8.41
11.57
9.35
1.06
0.38
+GT mesh
8.05
9.26
8.18
1.03
0.32
+GT dmap
7.50
11.25
8.78
0.94
0.35
+GT mesh&dmap
7.23
7.78
7.20
0.92
0.27
Table 4: Oracle study . Trained on estimated depth, our model also allows input with ground truth depth or object mesh and achieves slightly better results.
Figure 7: CoCoNet architecture . Here b,t,h,w denote batch size, temporal window size, image height and width respectively. We follow a render-and-compare paradigm, hence RGB a and RGB b denote the image from input observation and rendering respectively, same for xyz map and mask (human and object stacked together).
Method
PICO
InterTrack
VisTracker
Ours
Runtime (min) ↓
1560
198
118
45
Table 5: Average runtime (minutes) to process one video of 300 frames. Our method is much faster than baselines while being more accurate.
Figure 8: Sensitivity analysis of UniDepth and FoundationPose errors on the final performance.
Figure 9: Error per seq. (left) and visibility distribution (right).
Interaction type
CD-h ↓
CD-o ↓
CD-c ↓
Acc-h ↓
Acc-o ↓
Contact ↑
Hand mainly
8.76
11.71
7.12
1.06
0.42
0.93
Sitting
8.16
10.10
7.23
1.01
0.32
0.95
Shoulder contact
10.24
12.39
9.27
1.03
0.48
0.95
Leaning
9.37
11.44
8.28
1.05
0.31
0.93
Table 6: Performance of our method per interaction type.
Figure 10: Additional qualitative examples.
Figure 11: Failure case examples. Our method focuses on full body interaction and the detailed hand poses are not handled, which can be important for fine-grained object manipulation task (top row). Our method thus failed to reconstruct realistic finger poses for holding the plate. Under highly dynamic motion and extreme occlusion (bottom row), FoundationPose predicts flipped object pose for initialization. Such large rotation error is not able to be corrected by our refinement process in subsequent steps, leading to inaccurate reconstruction in the end.
Joint 4D reconstruction of a human and object from a monocular video is essential for understanding how people interact with their environment. While recent methods have made noticeable progress, capturing such human-object interactions remains difficult due to two main challenges. First, a video typically contains composite semantic context spanning multiple interaction phases, making it difficult to track the temporal changes of the interaction in the reconstruction. Second, under human-object occlusion, a single-view observation contains incomplete visual information about the interaction, resulting in geometrically inconsistent reconstructions. To address these challenges, we introduce HORIA, built upon two core designs. First, we decompose the entire input video into temporally grounded and semantically distinct interaction phases with motion-aware captioning, obtaining a specific text description for each phase. Second, conditioned on these text descriptions of the interaction, we propose an interaction-aware re-rendering that compensates for the limited single-view observation. Specifically, we render the initially reconstructed 4D human and object from novel viewpoints and refine the renderings with a text-to-video diffusion prior, which provides auxiliary novel-view supervision beyond single-view cues. As a result, our framework produces temporally coherent and accurate 4D reconstructions under diverse interaction scenarios, achieving state-of-the-art performance.
Hyeongjin Nam, Taeryung Lee, Daniel Sungho Jung +1
Dept. of ECE&ASRI, Seoul National University · IPAI, Seoul National University
Extracting dynamic 4D object interactions from massive, in-the-wild monocular videos offers a highly efficient data collection pathway for scaling Embodied AI and training VLAs. However, existing monocular 4D reconstruction methods primarily focus on isolated objects, often failing under the severe occlusions and complex dynamics inherent in multi-object interactions. To bridge this gap, we propose HAT-4D, the first agentic framework designed to reconstruct the 3D geometry, temporal dynamics, and physical interactions of multiple objects from a single video. By integrating VLMs with a multi-level human-in-the-loop feedback mechanism, HAT-4D efficiently resolves depth ambiguities and interaction-induced occlusions during 3D generation and 4D propagation, yielding physically plausible assets without relying on expensive multicamera rigs. As a scalable data engine, HAT-4D facilitates the creation of MVOIK-4D, an open-world benchmark for monocular 4D interaction reconstruction, accompanied by a novel multi-dimensional evaluation protocol focused on physical plausibility and temporal consistency. Extensive experiments demonstrate that HAT-4D achieves SOTA performance on most evaluation metrics, while maintaining competitive semantic alignment. Ablation studies show that introducing a small amount of human feedback improves interaction reconstruction. Moreover, the data produced by HAT-4D effectively improves baseline performance when used for fine-tuning. Our data and code are available at https://lijiaxin0111.github.io/HAT4D/
Jiaxin Li, Yuxiang Wu, Zhenkai Zhang +11
Shanghai Jiao Tong University, China · Shanghai Innovation Institute, China · University of Science and Technology of China, China +1
We introduce UniCon3R, a unified feed-forward framework for online human-scene 4D reconstruction from monocular video. Current feed-forward human-scene reconstruction methods suffer from artifacts, where bodies float above the ground or penetrate parts of the scene. A key reason is the lack of effective interaction modelling between the human and the environment. Our goal is to exploit contact between the human and the scene during inference to actively improve the human mesh reconstruction. To that end, we explicitly model interaction by inferring 4D contact from the human pose and scene geometry and use the contact as a corrective cue for generating the pose. This enables UniCon3R to jointly recover scene geometry and spatially aligned 4D humans within the scene. Experiments on standard human-centric video benchmarks show that UniCon3R outperforms state-of-the-art baselines on physical plausibility and global human motion estimation while preserving fast, feed-forward inference speeds. The results validate our central claim: contact serves as a powerful internal prior, thus establishing a new paradigm for physically grounded joint human-scene reconstruction. Project page is available at https://surtantheta.github.io/UniCon3R .
Tanuj Sur, Shashank Tripathi, Nikos Athanasiou +4
1National University of Singapore · 2Max Planck Institute for Intelligent Systems, Tübingen, Germany