DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction

Wenliang Guo; Zhanbo Huang; Yu Kong

cs.CVSep 29, 2026

DynamicHOI: Coupled Dynamics for Physics-aware HOI Reconstruction

Authors: Wenliang Guo, Zhanbo Huang, Yu Kong

Organizations: Michigan State University

Abstract

We study hand-object interaction (HOI) reconstruction from monocular RGB videos, where partial observations can produce visually plausible yet mechanically inconsistent trajectories. Existing methods mainly enforce visual and geometric agreement, leaving the underlying interaction dynamics insufficiently constrained. We propose DynamicHOI, a physics-aware HOI reconstruction framework combining geometry-grounded diffusion refinement with coupled hand-object dynamics. Geometry spatially grounds visual evidence for trajectory refinement, while articulated inverse dynamics and Newton-Euler dynamics derive hand generalized forces and object wrenches for dynamics-level supervision. We further couple hand and object dynamics through contact-force transfer and recover active hand actuation as an interaction-level physical quantity. We formulate its empirical magnitude distribution into a probabilistic prior that penalizes unlikely actuation and suppresses mechanically implausible reconstructed motion. Experiments on three HOI datasets show consistent improvements in both hand and object reconstruction. The reconstructed trajectories further benefit downstream applications including hand world-model generation and robotic manipulation learning, demonstrating the value of physics-aware HOI modeling beyond reconstruction.

Figures & tables

Figure 1: Framework overview. Framewise reconstruction models first estimate the initial trajectory in parametric states, which is then refined progressively by a diffusion-based refinement network, and finally decoded to HOI mesh trajectory.

Figure 2: Coupled HOI dynamics. The bottom-right plot shows the empirical distribution of standardized active actuation $z=(a_{t,d}-\mu_{d})/\sigma_{d}$ for an example DoF. The logarithmic density axis highlights the heavy-tailed behavior, which is empirically better captured by the Student- $t$ distribution than the Gaussian distribution.

Table 1: Comparison on HOT3D, HO3D-v2, and DexYCB datasets. Bold / underline denote the best / second-best reconstruction methods.

\dagger

denotes our reproduction, and unmarked values are their paper-reported. “–” denotes metrics that are either inapplicable to the method or unavailable due to missing reports and unreleased code.

Table 2: Module ablation. Bold / underline denote the best / second-best configurations.

Table 3: Loss ablation. Bold / underline denote the best / second-best configurations.

Figure 3: Visualization for reconstruction.

Student- $t$ distributions
Distribution	NLL $\downarrow$	PA $\downarrow$	ADD@.1 $\uparrow$
w/o $\mathcal{L}_{\text{act}}$	–	4.22	24.9
Gaussian	−0.39	4.54	24.2
Laplace	−0.11	4.17	24.9
Cauchy ( $\nu{=}1$ )	−1.45	4.20	24.7
Fixed $\nu{=}10$	−0.70	4.17	25.6

Table 4: Ablation of different distribution choices for modeling active hand actuation on HOT3D dataset.

Panel of Figure 4 — Figure 4: Downstream applications with different hand-trajectory sources.

Explore similar work

CHOIR: Contact-aware 4D Hand-Object Interaction Reconstruction
May 20, 2026Hao Xu, Yilin Liu, Yinqiao Wang +2Hand-Object Interaction Detection 3D Hand
The Chinese University of Hong Kong, China · University College London, United Kingdom · University College London, Adobe Research, United Kingdom
Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos
May 14, 2026Yubo Zhao, Yujin Chai, Yunao Dong +4Articulated Objects Monocular Video
The Hong Kong University of Science and Technology · Tencent IEG
Recovering Physically Plausible Human-Object Interactions from Monocular Videos
Jun 3, 2026Dingbang Huang, Etienne Vouga, Qixing Huang +1Monocular Video Motion
University of Texas at Austin

May 20, 2026, cs.CVJ/K move · Enter open · S save

CHOIR: Contact-aware 4D Hand-Object Interaction Reconstruction

Hao Xu, Yilin Liu, Yinqiao Wang, Chi-Wing Fu+1

The Chinese University of Hong Kong, China · University College London, United Kingdom · University College London, Adobe Research, United Kingdom

Hand-Object Interaction Detection 3D Hand Monocular Video Motion Capture 4D

Open paper AskPDF arXiv

We ask whether everyday open-world monocular videos can be turned into reusable 4D interaction primitives: articulated hand motion, object shape with 6D pose over time, and the when/where of contact. Such a capability would enable scalable mining of real interactions and, beyond reconstruction, support scene-aware synthesis and planning. However, reconstructing hand-object interaction (HOI) from challenging monocular videos remains difficult: methods often assume known objects or curated scenes, and separately estimated hands and objects easily become misaligned under clutter, occlusion, and unseen object geometries. Targeting this setting, we present CHOIR, a Contact-aware HOI Reconstruction framework for a monocular camera, using contact as an explicit coupling signal between hands and objects. CHOIR first initializes a coarse, contact-agnostic 4D HOI sequence from open-world visual priors. It then introduces a generative HOI spatial rectification module to predict ray-depth corrections and rectify hand-object relative placement, then derive initial per-frame contact correspondences on the rectified geometry. Last, a contact-aware joint optimization with dynamically updated contact constraints enforces geometric, temporal, and contact consistency. Experiments on controlled and challenging videos show that CHOIR improves object reconstruction, physical plausibility, and temporal consistency over state-of-the-art methods.

Abstract

Explore similar work

CHOIR: Contact-aware 4D Hand-Object Interaction Reconstruction

Real2Sim in HOI: Toward Physically Plausible HOI Reconstruction from Monocular Videos

Recovering Physically Plausible Human-Object Interactions from Monocular Videos