Organizations: the School of AIA, Huazhong University of Science and Technology, Wuhan 430074, China · the S-Lab, Nanyang Technological University (NTU), Singapore 639798 · the Beijing Academy of Artificial Intelligence (BAAI), Beijing 100083, China · Zhejiang University (ZJU), Hangzhou 310058, China · the Institute for AI Industry Research (AIR), Tsinghua University (THU), Beijing 100084, China
Synthesizing physically plausible articulated human-object interactions (HOI) without 3D/4D supervision remains a fundamental challenge. While recent zero-shot approaches leverage video diffusion models to synthesize human-object interactions, they are largely confined to rigid-object manipulation and lack explicit 4D geometric reasoning. To bridge this gap, we formulate articulated HOI synthesis as a 4D reconstruction problem from monocular video priors: given only a video generated by a diffusion model, we reconstruct a full 4D articulated scene without any 3D supervision. This reconstruction-based approach treats the generated 2D video as supervision for an inverse rendering problem, recovering geometrically consistent and physically plausible 4D scenes that naturally respect contact, articulation, and temporal coherence. We introduce ArtHOI, the first zero-shot framework for articulated human-object interaction synthesis via 4D reconstruction from video priors. Our key designs are: 1) Flow-based part segmentation: leveraging optical flow as a geometric cue to disentangle dynamic from static regions in monocular video; 2) Decoupled reconstruction pipeline: joint optimization of human motion and object articulation is unstable under monocular ambiguity, so we first recover object articulation, then synthesize human motion conditioned on the reconstructed object states. ArtHOI bridges video-based generation and geometry-aware reconstruction, producing interactions that are both semantically aligned and physically grounded. Across diverse articulated scenes (e.g., opening fridges, cabinets, microwaves), ArtHOI significantly outperforms prior methods in contact accuracy, penetration reduction, and articulation fidelity, extending zero-shot interaction synthesis beyond rigid manipulation through reconstruction-informed synthesis.
Figures & tables
Fig. 1: ArtHOI recovers zero-shot articulated human-object scene geometry and dynamics from monocular video priors without 3D supervision. Unlike prior works ( e.g. , TRUMANS, ZeroHSI), our method achieves all four capabilities simultaneously: RGB rendering, articulated object modeling, physical constraint modeling, and zero-shot generalization, notably without using 3D supervision.
TABLE I: Comparison of capabilities across different approaches. HOI : Human-object interaction; RGB : RGB rendering; Art : Articulated objects; Phy : Physical constraints; ZS : Zero-shot.
Fig. 2: ArtHOI synthesizes 3D articulated interactions by reconstructing 4D scenes from monocular video priors. Stage I reconstructs object articulation with kinematic constraints. Stage II refines human motion under the reconstructed geometry.
Fig. 3: Key components for articulated interaction under monocular supervision. (a) Back projection maps masks to 3D to identify moving parts. (b) Quasi-static point pairs link dynamic/static regions for kinematic stability. (c) Contact loss projects 2D keypoints into 3D using object depth, guiding human motion without multi-view cues. Ablations in Figure 5 (middle: (b), right: (c)).
Fig. 4: Qualitative comparison of our method with baselines. Our method synthesizes more realistic articulated human-object interactions with proper contact and natural motion coordination. Better inspected in our supplementary video.
TABLE II: Comparison of interaction quality. Smoothness (↓) is best interpreted among zero-shot methods. Non-zero-shot high smoothness stems from minimal contact, not motion instability.
TABLE IV: Rigid object comparison under monocular RGB.
Fig. 5: Comparing our full model with variants. Better inspected in our supplementary video.
Method
Realism ↑
Contact Quality ↑
Motion Smoothness ↑
Overall Preference ↑
Ours vs. TRUMANS [ 4 ]
96.08%
98.00%
92.16%
98.04%
Ours vs. CHOIS [ 1 ]
95.20%
89.08%
94.83%
95.28%
Ours vs. LINGO [ 3 ]
90.20%
87.13%
92.00%
91.51%
Ours vs. ZeroHSI [ 6 ]
91.18%
85.41%
84.21%
89.42%
TABLE V: User study shows the percentage of participants who preferred our method over each baseline across four evaluation dimensions.
Method
Interaction
Articulation
X-CLIP ↑
Foot Sliding ↓
Contact% ↑
Rot (mean) ↓
Rot (max) ↓
Rot (min) ↓
Joint Opt.
0.187
0.67
61.45
12.34
35.89
2.01
w/o La
0.223
0.42
68.75
15.67
42.18
4.56
w/o Lk
0.201
0.58
59.82
6.71
21.41
0.58
w/o Ls
0.218
0.49
65.43
8.23
25.45
0.79
Full Model
0.244
0.31
75.64
6.71
21.41
0.58
TABLE VI: Ablation study results. We remove individual components and evaluate their impact on both interaction and articulation.
Fig. 6: Failure cases. Co-tracker struggles with low-texture or reflective regions, leading to distortions that propagate into articulation prediction.
Synthesizing human--object interaction (HOI) videos has broad practical value in e-commerce, digital advertising, and virtual marketing. However, current diffusion models, despite their photorealistic rendering capability, still frequently fail on (i) the structural stability of sensitive regions such as hands and faces and (ii) physically plausible contact (e.g., avoiding hand--object interpenetration). We present CoInteract, an end-to-end framework for HOI video synthesis conditioned on a person reference image, a product reference image, text prompts, and speech audio. CoInteract introduces two complementary designs embedded into a Diffusion Transformer (DiT) backbone. First, we propose a Human-Aware Mixture-of-Experts (MoE) that routes tokens to lightweight, region-specialized experts via spatially supervised routing, improving fine-grained structural fidelity with minimal parameter overhead. Second, we propose Spatially-Structured Co-Generation, a dual-stream training paradigm that jointly models an RGB appearance stream and an auxiliary HOI structure stream to inject interaction geometry priors. During training, the HOI stream attends to RGB tokens and its supervision regularizes shared backbone weights; at inference, the HOI branch is removed for zero-overhead RGB generation. Experimental results demonstrate that CoInteract significantly outperforms existing methods in structural stability, logical consistency, and interaction realism.
Existing hand-object interactions (HOI) methods are largely limited to rigid objects, while 4D reconstruction methods of articulated objects generally require pre-scanning the object or even multi-view videos. It remains an unexplored but significant challenge to reconstruct 4D human-articulated-object interactions from a single monocular RGB video. Fortunately, recent advancements in foundation models present a new opportunity to address this highly ill-posed problem. To this end, we introduce ArtHOI, an optimization-based framework that integrates and refines priors from multiple foundation models. Our key contribution is a suite of novel methodologies designed to resolve the inherent inaccuracies and physical unreality of these priors. In particular, we introduce an Adaptive Sampling Refinement (ASR) method to optimize object's metric scale and pose for grounding its normalized mesh in world space. Furthermore, we propose a Multimodal Large Language Model (MLLM) guided hand-object alignment method, utilizing contact reasoning information as constraints of hand-object mesh composition optimization. To facilitate a comprehensive evaluation, we also contribute two new datasets, ArtHOI-RGBD and ArtHOI-Wild. Extensive experiments validate the robustness and effectiveness of our ArtHOI across diverse objects and interactions. Project: https://arthoi-reconstruction.github.io.
Zikai Wang, Zhilu Zhang, Yiqing Wang +2
Harbin Institute of Technology · Shanghai Jiao Tong University
Generating realistic and physically plausible 3D Human-Object Interactions (HOI) remains a key challenge in motion generation. One primary reason is that describing these physical constraints with words alone is difficult. To address this limitation, we propose a new paradigm: extracting rich interaction priors from easily accessible 2D images. Specifically, we introduce ViHOI, a novel framework that enables diffusion-based generative models to leverage rich, task-specific priors from 2D images to enhance generation quality. We utilize a large Vision-Language Model (VLM) as a powerful prior-extraction engine and adopt a layer-decoupled strategy to obtain visual and textual priors. Concurrently, we design a Q-Former-based adapter that compresses the VLM's high-dimensional features into compact prior tokens, which significantly facilitates the conditional training of our diffusion model. Our framework is trained on motion-rendered images from the dataset to ensure strict semantic alignment between visual inputs and motion sequences. During inference, it leverages reference images synthesized by a text-to-image generation model to improve generalization to unseen objects and interaction categories. Experimental results demonstrate that ViHOI achieves state-of-the-art performance, outperforming existing methods across multiple benchmarks and demonstrating superior generalization.