ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions

Zikai Wang; Zhilu Zhang; Yiqing Wang; Hui Li; Wangmeng Zuo

cs.CVMar 26, 2026

ArtHOI: Taming Foundation Models for Monocular 4D Reconstruction of Hand-Articulated-Object Interactions

Authors: Zikai Wang, Zhilu Zhang, Yiqing Wang, Hui Li, Wangmeng Zuo

Organizations: Harbin Institute of Technology · Shanghai Jiao Tong University

Abstract

Existing hand-object interactions (HOI) methods are largely limited to rigid objects, while 4D reconstruction methods of articulated objects generally require pre-scanning the object or even multi-view videos. It remains an unexplored but significant challenge to reconstruct 4D human-articulated-object interactions from a single monocular RGB video. Fortunately, recent advancements in foundation models present a new opportunity to address this highly ill-posed problem. To this end, we introduce ArtHOI, an optimization-based framework that integrates and refines priors from multiple foundation models. Our key contribution is a suite of novel methodologies designed to resolve the inherent inaccuracies and physical unreality of these priors. In particular, we introduce an Adaptive Sampling Refinement (ASR) method to optimize object's metric scale and pose for grounding its normalized mesh in world space. Furthermore, we propose a Multimodal Large Language Model (MLLM) guided hand-object alignment method, utilizing contact reasoning information as constraints of hand-object mesh composition optimization. To facilitate a comprehensive evaluation, we also contribute two new datasets, ArtHOI-RGBD and ArtHOI-Wild. Extensive experiments validate the robustness and effectiveness of our ArtHOI across diverse objects and interactions. Project: https://arthoi-reconstruction.github.io.

Figures & tables

Figure 2: Pipeline of our ArtHOI. ArtHOI is an optimization-based framework (see subfigure (a)) that integrates and refines priors from multiple foundation models for monocular 4D reconstruction of human-articulated-object interactions. In particular, the proposed object’s metric scale and pose optimization (see subfigure (b)) recovers 3D mesh in world space from a normalized one, while MLLM-guided hand-object alignment method (see subfigure (c)) promotes physically plausible hand-object mesh composition.

Figure 3: This gallery showcases the results of our hand-articulated-object reconstruction on three data sources: ArtHOI-RGBD, RSRD and ArtHOI-Wild.(more results in the supp.). The first column shows sampled input frames. We present the camera view and a side view to display the reconstructed HOI meshes. Hand reconstructions for RSRD are produced using the same WiLoR model as ours for a fair comparison. Note that RSRD is unable to process the video from ArtHOI-Wild, as it requires an object surrounding scan that is unavailable for internet videos.

Objects	Method	CD (mm) $\downarrow$	MSSD (mm) $\downarrow$	F10 $\uparrow$	F5 $\uparrow$
Headphone	EasyHOI [ 40 ]	209.295 $\pm$ 107.36	291.02 $\pm$ 105.03	1.26	0.59
	RSRD [ 24 ]	14.708 $\pm$ 0.18	41.06 $\pm$ 3.65	41.67	20.91
	Ours	8.124 $\pm$ 0.44	30.43 $\pm$ 2.20	69.68	42.19
Scissor	EasyHOI [ 40 ]	170.946 $\pm$ 110.29	210.08 $\pm$ 114.00	4.14	2.24
	RSRD [ 24 ]	13.841 $\pm$ 5.89	31.53 $\pm$ 5.20	37.98	17.55
	Ours	4.256 $\pm$ 1.02	15.14 $\pm$ 3.14	92.57	65.00

Table 1: 4D reconstruction accuracy of articulated object on monocular RGB videos from ArtHOI-RGBD dataset. Lower CD/MSSD and higher F-scores indicate better performance.

Objects	Method	CD (mm) $\downarrow$	MSSD (mm) $\downarrow$	F10 $\uparrow$	F5 $\uparrow$
Scissor	EasyHOI [ 40 ]	205.567 $\pm$ 214.40	270.99 $\pm$ 232.23	6.04	3.07
	RSRD [ 24 ]	68.564 $\pm$ 20.94	109.31 $\pm$ 19.91	2.09	1.20
	Ours	5.447 $\pm$ 3.34	13.12 $\pm$ 6.38	80.95	61.81
LED Light	EasyHOI [ 40 ]	52.435 $\pm$ 61.43	100.18 $\pm$ 72.43	16.43	7.64
	RSRD [ 24 ]	10.144 $\pm$ 1.31	32.80 $\pm$ 5.72	63.21	46.21
	Ours	10.836 $\pm$ 7.50	35.30 $\pm$ 21.84	60.95	31.81

Table 2: 4D reconstruction accuracy of articulated object on monocular RGB videos from RSRD [ 24 ] dataset. Lower CD/MSSD and higher F-scores indicate better performance.

	Method	CD(mm) $\downarrow$	F10 $\uparrow$	MSSD(mm) $\downarrow$	Cont.Acc% $\uparrow$	Fing.Acc% $\uparrow$
Mixer	EasyHOI [ 40 ]	226.0	0.05	326.1	N/A	N/A
	RSRD [ 24 ]	Failed to reconstruct articulated object
	Ours	12.1	0.55	41.4	82.5	76.2
Box	EasyHOI [ 40 ]	207.9	0.03	356.4	N/A	N/A
	RSRD [ 24 ]	Failed to reconstruct articulated object
	Ours	14.0	0.44	51.2	76.6	62.3

Table 3: Comparison on a subset of ARCTIC [ 12 ] . ‘Cont.Acc’ denotes binary contact accuracy and ‘Fing.Acc’ denotes main contacting finger (thumb, index, middle) accuracy of MLLM reasoning results.

	ArtHOI-RGBD	RSRD [ 24 ]	ArtHOI-Wild
No Alignment	0.972	0.517	0.514
RSRD [ 24 ] + WiLoR [ 52 ]	0.392	0.166	N/A
Ours w/o MLLM	0.046	0.035	0.059
Ours w/ MLLM	0.029	0.022	0.039

Table 4: Comparison of

Co^{2}

scores for unaligned and aligned articulated HOI reconstruction under different contact reasoning strategies. We evaluate four settings: (1) unaligned hand-object reconstruction, (2) RSRD with WiLoR [ 52 ] hands, (3) our alignment using a mask-intersection contact heuristic (w/o MLLM), and (4) our full alignment with MLLM-based contact reasoning (w/ MLLM). Lower is better. RSRD fails on ArtHOI-Wild due to missing object-scanning inputs.

Method	ArtHOI-RGBD		RSRD [ 24 ]		ArtHOI-Wild
Method	IoU $\uparrow$	SR( $\%$ ) $\uparrow$	IoU $\uparrow$	SR( $\%$ ) $\uparrow$	IoU $\uparrow$	SR( $\%$ ) $\uparrow$
FoundationPose [ 62 ]	0.820	60%	0.706	78%	0.749	71%
Any6D [ 28 ]	0.876	60%	0.857	78%	0.683	57%
ASR (ours)	0.905	100%	0.876	100%	0.882	100%

Table 5: Comparison of canonical mesh pose and scale optimization. We compare with FoundationPose and Any6D [ 28 ] . Metrics include the IoU between rendered and ground-truth masks under the optimized pose, and the optimization success rate (SR

\%

). A case is considered failed if subsequent part motion reconstruction or HOI alignment cannot proceed.

Prompting Strategy				RSRD [ 24 ]		ArtHOI-Wild
Temp.	Persp.	MinFP	Depth	Acc. $\uparrow$	FP $\downarrow$	Acc. $\uparrow$	FP $\downarrow$
				81.53	18.24	83.50	16.59
✓				82.75	17.08	82.62	17.02
✓		✓	✓	86.42	13.49	85.92	13.27
✓	✓		✓	86.27	13.66	86.21	13.79
✓	✓	✓		87.65	12.13	87.52	11.35

Table 6: Ablation study on prompting strategies for MLLM contact reasoning, evaluated by accuracy and false positive rate (FP,

\%

). “Temp.” incorporates temporal context from neighboring frames. “Persp.” indicates introducing camera-perspective cues; “MinFP” uses prompts designed to suppress false positives; and “Depth” augments image prompts with colorized depth. Results of ArtHOI-RGBD is excluded due to its near

100\%

accuracy.

Figure 4: Qualitative comparison of metric scale and pose estimation on in-the-wild videos without ground-truth depth. Images are cropped and zoomed-in for better visualization.

Figure A : Demonstration of our MLLM contact reasoning pipeline. For clarity, we merge 2 neighbouring frames, but in practice, it’s typically set to 3. The top row shows RGB frames, the bottom row shows colorized depth maps. The MLLM analyzes visual and depth cues across frames to determine contact status and engaged fingers for each hand.

	RSRD [ 24 ]		ArtHOI-Wild
Contact Judge	Acc. $\uparrow$	FP $\downarrow$	Acc. $\uparrow$	FP $\downarrow$
Mask Intersection	0.86	0.14	0.76	0.23
MLLM	0.89	0.11	0.87	0.10

Table A : Comparison of contact accuracy (Acc.) and false positive rate (FP) between our MLLM-based contact reasoning and a rule-based mask-intersection heuristic. While both methods perform similarly on the controlled RSRD dataset, the heuristic degrades notably on in-the-wild videos, whereas the MLLM remains robust. ArtHOI-RGBD is excluded due to its near-perfect accuracy.

Figure B : Qualitative comparison between our method and EasyHOI [ 40 ] on ArtHOI-RGBD. EasyHOI often fails to recover articulated object scale and pose, and exhibits inconsistent hand-object alignment across frames.

Panel of Figure 13 — Figure C : Stage 1: Perspective Detection Prompt. This prompt determines whether the input video is from a first-person or third-person viewpoint, which is essential for correctly identifying hand laterality in subsequent stages.

Panel of Figure 14 — Figure D : Stage 2: Hand Mapping Prompt. This stage identifies and maps visible hands to left/right labels. Stage 2a handles first-person perspective videos using spatial positioning and thumb direction cues. Stage 2b handles third-person perspective videos by analyzing camera angle relative to the operator’s body and arm connectivity patterns.

Panel of Figure 15 — Figure E : Stage 3: Frame-wise Contact Reasoning Prompt. This stage performs detailed analysis of each frame to determine contact state and identify engaged fingers. The critical depth map verification step (Phase C) distinguishes true physical contact from mere proximity using depth discontinuity analysis.

Hand-Object Interaction Reconstruction

See all 18 papers

Open-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstruction
May 20, 2026Hao Xu, Yilin Liu, Yinqiao Wang +2Human-Object Interaction 3D Reconstruction
The Chinese University of Hong Kong, China · University College London, United Kingdom · Adobe Research, United Kingdom
ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors
Mar 4, 2026Zihao Huang, Tianqi Liu, Zhaoxi Chen +7HOI Generation Dynamic Scene Reconstruction
the School of AIA, Huazhong University of Science and Technology, Wuhan 430074, China · the S-Lab, Nanyang Technological University (NTU), Singapore 639798 · the Beijing Academy of Artificial Intelligence (BAAI), Beijing 100083, China +2
4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction
Oct 6, 2026Shiqi Li, Sean Cho, Yijie Li +3Physics-Informed Generative Modeling 4D Reconstruction
Texas A&M University · NVIDIA

May 20, 2026J/K move · Enter open · S save

Open-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstruction

Hao Xu, Yilin Liu, Yinqiao Wang, Chi-Wing Fu+1

The Chinese University of Hong Kong, China · University College London, United Kingdom · Adobe Research, United Kingdom

Human-Object Interaction 3D Reconstruction Monocular 3D Reconstruction Object Pose Estimation Hand-Object Interaction Reconstruction 3D Hand Pose Estimation+2

Open paper AskPDF arXiv

We ask whether everyday open-world monocular videos can be turned into reusable 4D interaction primitives: articulated hand motion, object shape with 6D pose over time, and the when/where of contact. Such a capability would enable scalable mining of real interactions and, beyond reconstruction, support scene-aware synthesis and planning. However, reconstructing hand-object interaction (HOI) from challenging monocular videos remains difficult: methods often assume known objects or curated scenes, and separately estimated hands and objects easily become misaligned under clutter, occlusion, and unseen object geometries. Targeting this setting, we present Open-CHOIR, an Open-world Contact-aware HOI Reconstruction framework for a monocular camera, using contact as an explicit coupling signal between hands and objects. Open-CHOIR first initializes a coarse, contact-agnostic 4D HOI sequence from open-world visual priors. It then introduces a generative HOI spatial rectification module to predict ray-depth corrections and rectify hand-object relative placement, then derive initial per-frame contact correspondences on the rectified geometry. Last, a contact-aware joint optimization with dynamically updated contact constraints enforces geometric, temporal, and contact consistency. Experiments on controlled and challenging videos show that Open-CHOIR improves object reconstruction, physical plausibility, and temporal consistency over state-of-the-art methods. Code, data, and pretrained weights are available at https://github.com/hxwork/CHOIR.

Abstract

Hand-Object Interaction Reconstruction

Explore similar work

Open-CHOIR: Open-World Contact-Aware 4D Hand-Object Interaction Reconstruction

ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors

4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction