4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction

Shiqi Li; Sean Cho; Yijie Li; Fengzhi Guo; Bowen Wen; Cheng Zhang

cs.CVOct 6, 2026

4D-HOF: Hand-Object Flow Matching for Feed-Forward 4D Interaction Reconstruction

Authors: Shiqi Li, Sean Cho, Yijie Li, Fengzhi Guo, Bowen Wen, Cheng Zhang

Organizations: Texas A&M University · NVIDIA

Abstract

Existing methods for 4D hand-object reconstruction often rely on costly per-sequence optimization, while generative approaches typically synthesize interactions from random noise, which can lead to unstable interaction prediction. We introduce 4D-HOF, a feed-forward framework that reconstructs 4D hand-object interactions from coarse but informative estimates produced by vision foundation models. Concretely, we learn a conditional flow matching model that transports foundation-model-derived hand-object states toward an interaction manifold, allowing the model to correct errors in translation, rotation, and alignment in a feed-forward manner. A key advantage of our generative formulation is that it naturally enables test-time guidance within the transport process. Rather than applying a separate post-hoc optimization after reconstruction, we directly steer the evolving generative states using physical interaction constraints and observed 2D evidence, allowing the reconstruction to be refined as part of the generative process itself. By training the generative model on diverse datasets, 4D-HOF generalizes robustly to challenging in-the-wild scenarios. Experiments on out-of-domain benchmarks show that 4D-HOF achieves state-of-the-art performance, producing more stable and accurate 4D hand-object reconstructions.

Figures & tables

Figure 1: 4D Hand-Object Flow Matching (4D-HOF) . Given a monocular video of a hand interacting with an unseen object, we learn a flow matching model for accurate and efficient 4D hand-object interaction reconstruction. Our method generalizes across diverse and challenging scenarios, including ambiguous depth (headset), occlusion (kettle), complex articulated hand poses (bar), and large object rotation (camera). Moreover, 4D-HOF is substantially faster than prior optimization-based methods due to our feed-forward design.

Figure 2: 4D-HOF overview. Given a monocular video, we construct initial hand-object states by leveraging foundation models, including (a) contextual scene parsing, (b) object reconstruction, (d) hand reconstruction, and (c) depth alignment for recovering metric geometry, producing coarse HOI initialization. We then formulate HOI refinement as a conditional generative bridge matching problem that takes the HOI initialization together with RGB and 3D cues as input and outputs the refined HOI reconstruction. At inference time, test-time guidance adjusts the evolving states to better satisfy physical and image-space constraints.

Method	Interaction				Hand	Object			Robustness
Method	CD h $\downarrow$	IV $\downarrow$	CA $\uparrow$	JM $\downarrow$	MPJPE $\downarrow$	CD $\downarrow$	F5 $\uparrow$	F10 $\uparrow$	SR $\uparrow$
HOLD	18.66	18.76	89.50	10.43	22.09	1.11	81.75	92.42	100.0
MagicHOI	21.81	17.41	83.00	21.66	7.38	0.90	76.74	91.59	83.3
AGILE	15.81	–	–	–	3.92	0.27	86.63	97.77	100.0
CHOIR	42.37	22.90	90.99	0.92	21.12	0.84	63.87	90.29	94.4
4D-HOF (ours)	5.98	13.44	97.27	1.61	4.40	0.28	89.11	97.13	100.0

Table 1: Results on HO3D-v3. Our method shows strong improvements on interaction metrics, which are often underreported in previous methods ( Wang et al., 2025b ; Shi et al., 2026 ) . Hand and object accuracy are determined by foundation models and our method performs on par with AGILE (code not available).

Table 4

Figure 3: Qualitative results on HO3D-v3 ( Hampali et al., 2020 ) and DexYCB ( Chao et al., 2021 ) . 4D-HOF consistently outperforms HOLD ( Fan et al., 2024 ) and MagicHOI ( Wang et al., 2025b ) in preserving accurate geometry and hand-object interactions. Please see our supplementary video for detailed comparisons.

Table 6

Table 7

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Model architecture. Our model adopts a DiT-style architecture for conditional flow matching. The latent hand-object state is processed through temporal self-attention, point-stream cross-attention using Point Transformer V3 features, and visual cross-attention using DINO features. Adaptive layer normalization conditioned on the timestep is applied throughout the network to modulate the generative transport process.

Figure S2: Qualitative results on HO3D-v3 ( Hampali et al., 2020 ) following MagicHOI’s setting.

Method	Hand	Object				Interaction
Method	MPJPE $\downarrow$	CD $\downarrow$	F5 $\uparrow$	F10 $\uparrow$	RS $\downarrow$	CD h $\downarrow$
iHOI	27.75	2.37	35.78	62.11	0.17	25.45
DiffHOI	16.02	2.30	39.59	64.49	0.13	33.33
EasyHOI	16.69	1.86	46.10	70.92	0.28	19.55
HOLD	30.79	1.31	57.20	80.23	0.62	21.28
MagicHOI	4.62	0.87	69.72	92.15	0.11	2.39

Appendix

Table S1: Results on HO3D-v3 ( Hampali et al., 2020 ) following MagicHOI’s setting.

Methods	CD $\downarrow$	F5 $\uparrow$	F10 $\uparrow$	RS $\downarrow$
SAM3D	0.47	77.54	96.20	0.17
4D-HOF (Ours)	0.28	89.11	97.13	0.12

Appendix

Table S2: Object generation comparison. Our multiview-based pipeline outperforms SAM3D across all metrics, demonstrating the benefit of complementary visual observations for object reconstruction.

Figure S3: Qualitative comparison of object generation. We show two views of each object reconstructed by 4D-HOF and SAM3D ( Chen et al., 2026a ) alongside the ground truth. Our pipeline better preserves object geometry and texture details, including labels and fine structures. Ground-truth object widths are provided for scale reference.

Figure S4: Dataset scaling analysis for bridge matching model training.

Figure S5: Qualitative comparison between GPT6-Astra and our 4D-HOF.

Figure S6: Failure analysis. When the estimated point map contains severe artifacts or out-of-distribution predictions, the resulting initialization may become unreliable, which can affect the flow-matching refinement process and lead to reconstruction failures.

Explore similar work

High-Fidelity 4D Hand-Object Capture via Multi-View Spatiotemporal Tracking and Physics-Aware Gaussians
Jun 14, 2026Bo Peng, Xu Chen, Yi Gu +63D Hand 4D Reconstruction
Google XR · The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China (USTC)
HandFlow: Fully Generative 4D Hand Recovery with Flow Matching
Jul 13, 2026Mingxi Xu, Bowen Duan, Yi Gu +33D Hand 4D Reconstruction
The Hong Kong University of Science and Technology (Guangzhou), China · Google, USA
CHOIR: Contact-aware 4D Hand-Object Interaction Reconstruction
May 20, 2026Hao Xu, Yilin Liu, Yinqiao Wang +2Hand-Object Interaction Detection 3D Hand
The Chinese University of Hong Kong, China · University College London, United Kingdom · University College London, Adobe Research, United Kingdom

Jun 14, 2026, cs.CVJ/K move · Enter open · S save

High-Fidelity 4D Hand-Object Capture via Multi-View Spatiotemporal Tracking and Physics-Aware Gaussians

Bo Peng, Xu Chen, Yi Gu, Hidenobu Matsuki+5

Google XR · The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China (USTC)

3D Hand 4D Reconstruction 4D Motion Capture Multi-Object Tracking High-Fidelity+2

Open paper AskPDF arXiv

The growing demand for high-fidelity 4D hand-object interaction (HOI) data in embodied AI and spatial computing is currently bottlenecked by the reliance on pre-scanned object templates and physical markers. While recent methods have demonstrated promising results in reconstructing 4D hand-object interaction from videos, they are highly sensitive to initial estimates of hand and object poses. Yet, estimating these poses from images is challenging, in particular under severe occlusion which is inherent in hand-object interaction scenarios. We propose a novel system for the robust and accurate reconstruction of hands and objects from synchronized and calibrated multi-view videos without requiring any templates or markers. Our system consists of two main components with key innovations: (1) a multi-view feed-forward transformer model that aggregates cross-view geometry and temporal cues to provide a reliable, metric-consistent initialization for both poses and dense object geometry, and (2) a hand-object physics-aware Gaussian-based optimization framework to refine the initial estimates, integrating tetrahedral constraints, collision refinement, and appearance decomposition to produce physically plausible and visually accurate reconstruction. Validated on public benchmarks and an extensive internal dataset, our pipeline achieves highly robust, artifact-free reconstruction, providing an efficient foundation for automated 4D asset generation. Our project page are available at https://zyshen021.github.io/HOSTPG/.

Abstract

Explore similar work

High-Fidelity 4D Hand-Object Capture via Multi-View Spatiotemporal Tracking and Physics-Aware Gaussians

HandFlow: Fully Generative 4D Hand Recovery with Flow Matching

CHOIR: Contact-aware 4D Hand-Object Interaction Reconstruction