HOIMask: Towards Generative Masked Modeling for Human Object Interaction Generation

Yihong Ji; Jinsong Zhang; He Hu; Hongbo Xu

cs.CVAug 15, 2026

HOIMask: Towards Generative Masked Modeling for Human Object Interaction Generation

Authors: Yihong Ji, Jinsong Zhang, He Hu, Hongbo Xu

Organizations: College of Computer Science and Software Engineering, Shenzhen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Fuzhou University

Abstract

Diffusion-based methods have dominated the HOI generation, as they enable critical contact fusions or signals to guide the diffusion process. However, they often result in high artifacts and unstable interaction quality due to error accumulation during iterative denoising. In this work, we propose HOIMask, the first generative masked framework for modeling HOI motion in discrete space. HOIMask first encodes both motion sequences and contact-aware signals into discrete 2D human and object token maps via HOI Vector Quantization (VQ), preserving fine-grained spatial-temporal structure beyond conventional 1D representations. On this basis, a generative masked modeling framework is employed to jointly capture human-object interaction dynamics, leveraging a transformer architecture designed to model complex spatial-temporal and interaction dependencies. To generate more coherent and physically plausible motions, we further introduce a novel contact-aware reconstruction guidance in discrete space during inference, which fuses contact signals to optimize HOI tokens that forces the generated motion with higher spatio-temporal consistency. With craftily designed motion interaction tokens, dedicated architecture and guidance strategy, HOIMask outperforms state-of-the-art diffusion-based methods, generating more realistic and semantically aligned HOI motions. Please refer to https://jyhflash.github.io/HOIMask/ for more results.

Figures & tables

Figure 2: Overview of HOIMask . (a) During training, HOI motions first are quantized via HVQ and OVQ into 2D HOI tokens, which are partially masked and predicted by the HOI-Mask Transformer. During inference, the HOI tokens are optimized with contact-aware reconstruction guidance to generate more physically plausible interactions through HOI-Decoder. (b) The HOI-Mask Transformer incorporates spatial-temporal attention and human/object-centric attention to model fine-grained spatio-temporal dependencies.

Figure 3: At Inference stage, Contact-aware Reconstruction Guidance is applied to optimize HOI tokens. The predicted logits are updated and pass through Soft Vector Quantization (Soft-VQ) to obtain optimized HOI tokens in differentiable paradigm during every iteration.

FullBodyManipulation Dataset
Method	FID ↓	Diversity →	R-Pre (Top3) ↑	PS ↓	$C_{prec}$ ↑	$C_{\%}$
GT	$0.01^{\pm 0.000}$	$1.756^{\pm 0.082}$	$0.750^{\pm 0.000}$	$0.060^{\pm 0.000}$	$-$	$70.68^{\pm 0.000}$
Diffusion Models
MDM ∗ [ 49 ]	$4.273^{\pm 0.001}$	$2.391^{\pm 0.082}$	$0.405^{\pm 0.001}$	$0.088^{\pm 0.001}$	$0.39^{\pm 0.007}$	$22.56^{\pm 0.095}$
InterDiff ∗ [ 58 ]	$6.890^{\pm 0.002}$	$2.290^{\pm 0.053}$	$0.605^{\pm 0.000}$	$0.079^{\pm 0.001}$	$0.42^{\pm 0.004}$	$25.63^{\pm 0.100}$
HOI-Diff [ 38 ]	$1.875^{\pm 0.000}$	$2.463^{\pm 0.079}$	$0.792^{\pm 0.000}$	$0.068^{\pm 0.001}$	$0.57^{\pm 0.005}$	$29.20^{\pm 0.090}$

Table 1: Quantitative comparisons on the FullBodyManipulation and BEHAVE datasets. Bold indicates the best result.

Figure 4: Qualitative comparison on the FullBodyManipulation dataset. We use red boxes to highlight the incorrect interaction. The rotating arrow demonstrates the successful action “rotate”. In contrast, our method generates more realistic and physically consistent human-object interactions than the compared approaches.

Figure 5: User Study. The color bar and numbers represent the preference rate.

Method	FID $\downarrow$	Diversity $\rightarrow$	R-Prec $\uparrow$	$C_{prec}$ $\uparrow$	$C_{\%}$
w/o STA	$0.505^{\pm 0.001}$	$1.126^{\pm 0.044}$	$0.781^{\pm 0.001}$	$0.69^{\pm 0.008}$	$43.92^{\pm 0.056}$
w/o HOCA	$0.242^{\pm 0.000}$	$1.496^{\pm 0.098}$	$0.794^{\pm 0.000}$	$0.73^{\pm 0.005}$	$52.50^{\pm 0.055}$
w/o Guidance	$0.134^{\pm 0.000}$	$1.864^{\pm 0.084}$	$0.800^{\pm 0.001}$	$0.64^{\pm 0.006}$	$37.54^{\pm 0.067}$
HOIMask	$\mathbf{0.109}^{\pm 0.000}$	$\mathbf{1.845}^{\pm 0.083}$	$\mathbf{0.826}^{\pm 0.000}$	$\mathbf{0.74}^{\pm 0.005}$	$\mathbf{52.96}^{\pm 0.080}$

Table 2: Ablation Study results on the FullBodyManipulation test set demonstrating the effectiveness of each key component in the proposed HOI-Mask Transformer . Bold face indicates the best result.

Method	MPJPE $\downarrow$	$R_{o}$ $\downarrow$	$C_{\%}$	PS $\downarrow$
GT	$0.00$	$0.00$	$70.68$	$0.060$
1D-VQVAE	$65.43$	$0.26$	$43.15$	$0.063$
w/o Interaction Loss	$\mathbf{27.56}$	$0.10$	$60.73$	$0.062$
w/o Motion Loss	$35.87$	$0.12$	$55.41$	$0.061$
2D-VQVAE	$28.95$	$\mathbf{0.08}$	$\mathbf{65.33}$	$\mathbf{0.061}$

Table 3: Ablation study results on the FullBodyManipulation test set demonstrating the effectiveness of each key component in the proposed HOI VQ-VAE . Bold face indicates the best result.

Method	FID $\downarrow$	Diversity $\rightarrow$	R-Prec $\uparrow$	$C_{\%}$	PS $\downarrow$
w/o Contact Loss	$0.121^{\pm 0.001}$	$1.880^{\pm 0.041}$	$0.779^{\pm 0.002}$	$39.01^{\pm 0.082}$	$\mathbf{0.061}^{\pm 0.000}$
w/o Penetration Loss	$0.114^{\pm 0.000}$	$1.866^{\pm 0.075}$	$0.814^{\pm 0.002}$	$50.88^{\pm 0.055}$	$0.068^{\pm 0.001}$
w GT Contact	${0.112}^{\pm 0.001}$	$\mathbf{1.823}^{\pm 0.088}$	$0.821^{\pm 0.001}$	$\mathbf{54.60}^{\pm 0.049}$	$0.067^{\pm 0.001}$
Ours	$\mathbf{0.109}^{\pm 0.000}$	${1.845}^{\pm 0.083}$	$\mathbf{0.826}^{\pm 0.000}$	${52.96}^{\pm 0.080}$	$0.065^{\pm 0.000}$

Table 4: Ablation study results on Contact-aware Reconstruction Guidance .

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 1: 2D HOI token map construction. HVQ and OVQ are composed of several 2D convolutional layers, downsamples the input motion sequence from $(T,J,D)$ to $(t,j,d)$ and $(T,K,D)$ to $(t,k,d)$ . The resulting latent representation is then quantized by substituting each feature vector with the index of its nearest entry in learned codebook.

$p_{inter}$	FID $\downarrow$	R-Prec (Top3) $\uparrow$	PS $\downarrow$	$C_{\%}$
0.6	$0.1278$	$0.8570$	$0.070$	$59.40$
0.5	$0.1106$	$0.8433$	$0.067$	$56.36$
0.4	$0.1020$	$0.8125$	$0.065$	$53.43$
0.3	$0.1260$	$0.8196$	$0.066$	$59.36$
0.2	$0.1091$	$0.8264$	$0.065$	$52.96$
0.1	$0.1278$	$0.8188$	$0.070$	$52.37$

Appendix

Table 1: Ablation study results on probability of human-object masking. The gray line represents the configuration used in HOIMask.

Figure 2: Qualitative results for the ablation study on HOIMask. Removing specific modules leads to a noticeable degradation in the quality of the generated results.

Figure 3: Qualitative comparisons on more baseline methods. We use red boxes to highlight the incorrect interaction. The Rotating arrow demonstrates the successful action "rotate". In contrast, our method generates more realistic and physically consistent human–object interactions than the compared approaches.

Figure 4: Visualization of Failure Cases. We use red box to highlight the incorrect interaction, the blue box indicates the correct interaction.

Model	FID $\downarrow$	Diversity	R-Prec (Top3) $\uparrow$	PS $\downarrow$	$C_{prec}$ $\uparrow$	$C_{\%}$
HOI-Diff [ 38 ]	$0.689$	$7.620$	$0.740$	$0.103$	$0.722$	$0.084$
CHOIS [ 28 ]	$0.572$	$7.717$	$0.766$	$0.131$	$0.710$	$0.115$
LIGHT [ 55 ]	$0.148$	$7.712$	$0.754$	$0.132$	$0.731$	$0.132$
HOIMask	$0.139$	$7.790$	$0.769$	$0.132$	$0.750$	$0.136$

Appendix

Table 2: Performance on InterAct [ 59 ] dataset.

Generation Steps	FID $\downarrow$	Diversity $\rightarrow$	R-Prec(Top3) $\uparrow$	$C_{prec}$ $\uparrow$	$C_{\%}$
GT	$0.01^{\pm 0.000}$	$1.7557^{\pm 0.082}$	$0.7504^{\pm 0.000}$	$-$	$70.68^{\pm 0.000}$
5	$0.3409^{\pm 0.001}$	$2.0731^{\pm 0.097}$	$0.8212^{\pm 0.000}$	${0.70}^{\pm 0.008}$	${33.15}^{\pm 0.124}$
10	$0.2349^{\pm 0.001}$	$2.0135^{\pm 0.087}$	$0.8037^{\pm 0.001}$	$0.74^{\pm 0.003}$	$47.55^{\pm 0.092}$
20	$\mathbf{0.1091}^{\pm 0.000}$	$\mathbf{1.8450}^{\pm 0.083}$	$\mathbf{0.8264}^{\pm 0.000}$	$\mathbf{0.74}^{\pm 0.005}$	${52.96}^{\pm 0.080}$
40	$0.1469^{\pm 0.000}$	$1.9128^{\pm 0.092}$	$0.8152^{\pm 0.001}$	$0.73^{\pm 0.004}$	$\mathbf{54.98}^{\pm 0.110}$

Appendix

Table 3: Quantitative results on generation steps. Bold face indicates the best result. The gray line represents the configuration used in HOIMask.

Update Iterations	FID $\downarrow$	Diversity $\rightarrow$	R-Prec(Top3) $\uparrow$	$C_{prec}$ $\uparrow$	$C_{\%}$
GT	$0.01^{\pm 0.000}$	$1.7557^{\pm 0.082}$	$0.7504^{\pm 0.000}$	$-$	$70.68^{\pm 0.000}$
10	$0.1117^{\pm 0.001}$	$\mathbf{1.8231}^{\pm 0.1255}$	$0.7908^{\pm 0.0000}$	$0.72^{\pm 0.007}$	$49.80^{\pm 0.120}$
20	$\mathbf{0.1091}^{\pm 0.000}$	${1.8450}^{\pm 0.083}$	$\mathbf{0.8264}^{\pm 0.000}$	${0.74}^{\pm 0.005}$	${52.96}^{\pm 0.080}$
40	$0.1613^{\pm 0.0000}$	$1.9414^{\pm 0.0885}$	$0.7908^{\pm 0.0001}$	$\mathbf{0.75}^{\pm 0.004}$	$\mathbf{53.93}^{\pm 0.098}$
80	$0.2093^{\pm 0.0001}$	$1.2006^{\pm 0.1090}$	$0.7714^{\pm 0.0001}$	$0.74^{\pm 0.008}$	$53.79^{\pm 0.105}$

Appendix

Table 4: Quantitative results on logits update iterations. Bold face indicates the best result. The gray line represents the configuration used in HOIMask.

Model	Speed $\downarrow$
MDM ∗ [ 49 ]	$4.91s$
CHOIS ∗ [ 28 ]	$7.89s$
ROG [ 60 ]	$42.04s$
SemGeoMo ∗ [ 6 ]	$221.48s$
HOIMask (w/o Guidance)	$\mathbf{4.57}s$
HOIMask	$55.80s$

Appendix

Table 5: Results on inference speed.

HOI Generation

Human-object interaction generation creates 3D motion, grasps, and videos of people manipulating or coordinating with objects.

See all 84 papers

MAD-HOI: Masked Autoregressive Diffusion for Generating Articulated Hand Object Interactions from Text
Aug 10, 2026Ananya Bal, Kartik Sharma, Ethan Lai +4HOI Generation Autoregressive Diffusion
Carnegie Mellon University
MFlowHOI: Whole-Body Human-Object Interaction Generation via Part-Aware MeanFlow
Oct 12, 2026Qiang He, Yonghao Zhang, Yinda Zhang +7HOI Generation One-Step Generative Modeling
Institute of Software, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Google
ViHOI: Human-Object Interaction Synthesis with Visual Priors
Mar 25, 2026Songjin Cai, Linjie Zhong, Ling Guo +1HOI Generation Human Motion Generation
South China University of Technology

Aug 10, 2026J/K move · Enter open · S save

MAD-HOI: Masked Autoregressive Diffusion for Generating Articulated Hand Object Interactions from Text

Ananya Bal, Kartik Sharma, Ethan Lai, Samyak Tiwari+3

Carnegie Mellon University

HOI Generation Autoregressive Diffusion Diffusion Models

Open paper AskPDF arXiv

Methods for text-based generation of hand-object interaction (HOI) sequences primarily focus on producing smooth, physically plausible trajectories. A truly utilitarian method should additionally support variable-length generation, composite motion sequences, motion completion and infilling, and reliable termination without compromising physical plausibility. Standard diffusion models for HOI generation are typically trained only for text-to-motion generation on atomic motions and require the motion length to be specified a priori. Autoregressive (AR) methods provide greater sequence-level flexibility, but commonly depend on discrete motion codes, which can lose contact-sensitive motion detail. To address these key limitations, we present a model performing Masked Autoregression with Diffusion for HOI generation (MAD-HOI). Our method starts by encoding hand and object motions in a continuous latent space while keeping them disentangled to maintain stream-wise control. This is followed by a masked autoregressive transformer to predict context features that condition a flow-matching head. MAD-HOI is capable of motion generation for atomic and composite articulated sequences, conditioned motion completion and infilling, as well as EOM (End of Motion) prediction from a single training objective. We provide comprehensive evaluations for these capabilities and benchmark our method on the ARCTIC and GRAB datasets. Our experiments show that MAD-HOI achieves the strongest text-motion alignment and distributional generation quality against baselines on both ARCTIC and GRAB, while maintaining competitive geometric accuracy and physical plausibility.

Abstract

HOI Generation

Explore similar work

MAD-HOI: Masked Autoregressive Diffusion for Generating Articulated Hand Object Interactions from Text

MFlowHOI: Whole-Body Human-Object Interaction Generation via Part-Aware MeanFlow

ViHOI: Human-Object Interaction Synthesis with Visual Priors