DISPLAY: Directable Human-Object Interaction Video Generation via Sparse Motion Guidance and Multi-Task Auxiliary

Jiazhi Guan; Quanwei Yang; Luying Huang; Junhao Liang; Borong Liang; Haocheng Feng; Wei He; Kaisiyuan Wang

cs.CVMar 10, 2026

DISPLAY: Directable Human-Object Interaction Video Generation via Sparse Motion Guidance and Multi-Task Auxiliary

Authors: Jiazhi Guan, Quanwei Yang, Luying Huang, Junhao Liang, Borong Liang, Haocheng Feng, Wei He, Kaisiyuan Wang, +2 more

Organizations: Baidu Inc.

Abstract

Human-centric video generation has advanced rapidly, yet existing methods struggle to produce controllable and physically consistent Human-Object Interaction (HOI) videos. Existing works rely on dense control signals, template videos, or carefully crafted text prompts, which limit flexibility and generalization to novel objects. We introduce a framework, namely DISPLAY, guided by Sparse Motion Guidance, composed only of wrist joint coordinates and a shape-agnostic object bounding box. This lightweight guidance alleviates the imbalance between human and object representations and enables intuitive user control. To enhance fidelity under such sparse conditions, we propose an Object-Stressed Attention mechanism that improves object robustness. To address the scarcity of high-quality HOI data, we further develop a Multi-Task Auxiliary Training strategy with a dedicated data curation pipeline, allowing the model to benefit from both reliable HOI samples and auxiliary tasks. Comprehensive experiments show that our method achieves high-fidelity, controllable HOI generation across diverse tasks. The project page can be found at \href{https://mumuwei.github.io/DISPLAY/}.

Figures & tables

Figure 1: Human-Object Interactions Synthesis . Our method generates user-intended human-object interactions from zero-shot references and user-specified sparse motion guidance. Blue: a generated video performing human-object interactions with novel visual and object references. Gray: a generated video demonstrating environmental interactions conditioned solely on the visual reference.

Figure 2: The DISPLAY Framework . We organize the proposed framework into three parts: Input Pipeline , Pretrained Video Model , and Condition Branch . 1) In the Input Pipeline , the input video is processed through a predefined HOI data procedure to extract the required multi-modal conditioning signals. 2) The Pretrained Video Model preserves the original T2V denoising formulation. 3) The Condition Branch encodes the multi-modal conditions and modulates the generation process via residual injection to guide the final video synthesis.

Method	Appearance Quality			Temporal Consistency			Hand Fidelity		Object Fidelity
Method	FID $\downarrow$	LPIPS $\downarrow$	AES $\uparrow$	FVD $\downarrow$	MS $\uparrow$	SC $\uparrow$	HF $\uparrow$	CA $\uparrow$	O-CLIP $\uparrow$	O-DINO $\uparrow$
VACE	105.708	0.134	0.524	862.65	0.995	0.953	0.972	0.871	0.802	0.677
HunyuanCustom	72.185	0.060	0.541	742.34	0.994	0.956	0.975	0.832	0.770	0.603
HuMo	254.620	0.552	0.464	1265.26	0.994	0.977	0.982	-	0.810	0.709
Nano+WanAnimate	119.890	0.295	0.537	948.61	0.995	0.953	0.989	0.771	0.806	0.643
Ours	67.501	0.072	0.547	560.29	0.995	0.958	0.987	0.891	0.890	0.832

Table 1: Quantitative Comparison. Videos generated by our method and all baselines are evaluated from four perspectives, with appearance quality, temporal consistency, hand fidelity, and object fidelity assessed accordingly.

Figure 3: Qualitative Comparisons . We present an object-replacement comparison on the left, where the original object is substituted according to the provided object reference. The visual reference and background condition used by our method are displayed in the top row. On the right, we present an object-insertion comparison, where the template video contains no original object.

Figure 4: Qualitative Comparisons. Comparisons with AnchorCraft and Re-HOLD on their official results.

Figure 5: Beyond Object Replacement . Guided by sparse motions authored via the proposed user interface, our model supports environmental interactions. Red arrows indicate motion patterns we provide.

Figure 6: Long-Video Manipulation . Our model effectively generates long-video manipulations without noticeable error accumulation.

Variation	FID $\downarrow$	SC $\uparrow$	HF $\uparrow$	O-CLIP $\uparrow$
HSC	68.210	0.953	0.980	0.885
w/o OSA	78.20	0.943	0.981	0.787
w/o MTT	89.76	0.941	0.968	0.862
w/o VR	107.64	0.947	0.985	0.865
Ours	67.501	0.958	0.987	0.890

Table 2: Ablation study. Quantitative results of different variants.

Figure 7: Ablations. Qualitative results.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Motion-Authoring Interface . We present the proposed user interface, which facilitates the creation of Sparse Motion Guidance for high-freedom HOI generation. In general, the red boxes display the input components, while the blue boxes show the input references and the authored motions.

Figure 9: HOI data construction pipeline. Through HOI Data Curation and Annotation, high-quality HOI data is obtained for model training.

Explore similar work

SparseCtrl-HOI: Sparse Temporal Control for Human-Object Interaction Video Generation
Jul 7, 2026Shenbo Xie, Mingrui Cai, Xu Yang +2HOI Generation Controllable Video Generation
South China University of Technology
AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment
Jul 24, 2026Ziyao Huang, Shunkai Li, Juan Cao +7HOI Generation Video Diffusion Models
University of Chinese Academy of Sciences · Tencent HunYuan
GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training
Jun 11, 2026Zhihai Bi, Qiang Zhang, Guoyang Zhao +8Contact-Rich Robotic Manipulation Robot Motion Generation
The Hong Kong University of Science and Technology (Guangzhou) · Artificial General Intelligence Institute, University of Science and Technology of China · The University of Hong Kong +1

Jul 7, 2026J/K move · Enter open · S save

SparseCtrl-HOI: Sparse Temporal Control for Human-Object Interaction Video Generation

Shenbo Xie, Mingrui Cai, Xu Yang, Yifei Liu+1

South China University of Technology

HOI Generation Controllable Video Generation

Open paper AskPDF arXiv

Human-Object Interaction (HOI) video generation aims to synthesize realistic videos of humans manipulating diverse objects, serving as a promising avenue for AI-driven live streaming e-commerce. A primary obstacle in this domain lies in the complexity of modeling fine-grained physical dynamics and the intricate spatial-temporal coordination between human hands and objects. Existing approaches to this problem typically rely on dense temporal guidance, e.g., frame-wise hand-object pose sequences, to strictly control the interaction process. However, such dense guidance incurs high annotation costs and affects motion synthesis diversity. To overcome these limitations, we introduce SparseCtrl-HOI, a novel sparse temporal control framework for HOI video generation. It requires only a few keyframes that capture interaction states at designated timestamps. Specifically, we employ a Time-Controlled Rotary Positional Embedding (TiRoPE) mechanism to temporally anchor these keyframes while preserving their spatial integrity. Subsequently, to govern the dynamics across intermediate frames, we propose a Motion Prior Injection Module that leverages Multimodal Large Language Models (MLLMs) to extract high-level motion priors. This empowers the model to hallucinate logically and physically plausible transitions. Furthermore, we build SparseHOI-5K, a high-quality and richly annotated dataset for HOI video generation with sparse temporal control. Comprehensive evaluations confirm that our method substantially reduces annotation overhead while synthesizing superior live-streaming e-commerce videos. Both our code and dataset are publicly available at https://mpi-lab.github.io/SparseCtrl-HOI.

Abstract

Explore similar work

SparseCtrl-HOI: Sparse Temporal Control for Human-Object Interaction Video Generation

AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment

GenHOI: Contact-Aware Humanoid-Object Interaction by Imitating Generated Videos without Task-Specific Training