cs.CVApr 13, 2026
SaveOmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation
Organizations: The Chinese University of Hong Kong · ByteDance · Monash University · The University of Hong Kong
Abstract
In this work, we study Human-Object Interaction Video Generation (HOIVG), which aims to synthesize high-quality human-object interaction videos conditioned on text, reference images, audio, and pose. This task holds significant practical value for automating content creation in real-world applications, such as e-commerce demonstrations, short video production, and interactive entertainment. However, existing approaches fail to accommodate all these requisite conditions. We present OmniShow, an end-to-end framework tailored for this practical yet challenging task, capable of harmonizing multimodal conditions and delivering industry-grade performance. To overcome the trade-off between controllability and quality, we introduce Unified Channel-wise Conditioning for efficient image and pose injection, and Gated Local-Context Attention to ensure precise audio-visual synchronization. To effectively address data scarcity, we develop a Decoupled-Then-Joint Training strategy that leverages a multi-stage training process with model merging to efficiently harness heterogeneous sub-task datasets. Furthermore, to fill the evaluation gap in this field, we establish HOIVG-Bench, a dedicated and comprehensive benchmark for HOIVG. Extensive experiments demonstrate that OmniShow achieves overall state-of-the-art performance across various multimodal conditioning settings, setting a solid standard for the emerging HOIVG task.
Figures & tables
Figure 1: Overview of OmniShow . Our model unifies text, reference image, audio, and pose conditions to synthesize high-quality videos. (a) Unified Tasks: OmniShow exhibits exceptional flexibility across diverse conditioning settings, from R2V, RA2V, RP2V to the most challenging RAP2V, consistently delivering industry-grade visual quality and precise multimodal alignment. (b) Broader Applications: OmniShow unlocks creative potential such as audio-driven avatars, object swapping, and video remixing, demonstrating robust compositional control and extensible capabilities. Please visit our project page for more immersive and diverse video demonstrations.
Figure 2: Pipeline of OmniShow . Our framework consists of: (a) Unified Channel-wise Conditioning ( section 3.2 ) effectively injects reference images and pose cues via unified channel concatenation. It augments noisy video tokens with pseudo-frames, which are supervised by a reference reconstruction loss to preserve semantic details. (b) Gated Local-Context Attention ( section 3.3 ) ensures precise audio-visual synchronization. It packs raw audio features with sufficient contextual information and injects them via masked attention to align video frames with corresponding audio segments, followed by adaptive gating to stabilize early training. (c) Decoupled-Then-Joint Training ( section 3.4 ) makes the efficient utilization of heterogeneous datasets possible. We first train specialized R2V and A2V models on separate sub-task datasets, then fuse them via weight interpolation, followed by joint fine-tuning to unify multimodal capabilities. Note that we omit some components, such as normalization, within the MMDiT blocks for brevity.
Figure 5: Variation of the average norm of , which reflects audio impact in different blocks.
Figure 6: Statistics and example of HOIVG-Bench. (a) Word cloud of text prompts illustrating the wide variety of human-object interaction scenarios. (b) Motion intensity distribution of original videos calculated via dense optical flow [ 15 ] , reflecting a diverse dynamic range of pose conditions. (c) A representative sample from the benchmark, featuring text, reference images, pose, and audio. Constructed through a rigorous pipeline, the benchmark provides high-quality data to evaluate the synergy of diverse multimodal inputs.
| Method | Text Align. | Reference Consistency | Audio-Visual Sync. | Pose Accuracy | Video Quality | ||||||
| TA | FaceSim | NexusScore | Sync-C | Sync-D | AKD | PCK | AES | IQA | VQ | MQ | |
| Text+Reference-to-Video (R2V) | |||||||||||
| HunyuanCustom [ 21 ] | 7.523 | 0.440 | 0.359 | - | - | - | - | 0.452 | 0.697 | 10.11 | 5.286 |
| HuMo-1.7B [ 4 ] | 7.087 | 0.647 | 0.333 | - | - | - | - | 0.441 | 0.723 | 9.76 | 3.406 |
| HuMo-17B [ 4 ] | 7.949 | 0.843 | 0.346 | - | - | - | - | 0.448 | 0.726 | 9.97 | 3.685 |
| VACE [ 31 ] | 8.413 | 0.759 | 0.368 | - | - | - | - | 0.457 | 0.722 | 10.72 | 5.442 |
Table 1: Quantitative comparison. We compare our OmniShow with existing state-of-the-art methods, showing that OmniShow can achieve superior/competitive performance across diverse multimodal conditioning settings. The best and second-best results in each setting are marked in bold and underlined . The symbol “-” indicates that the metric is not applicable in the corresponding setting.
Figure 7: Qualitative comparison. We present generated results from our OmniShow and other methods across various multimodal condition settings. Note that text prompts are omitted for brevity.
Figure 8: Side-by-side human evaluation. Judged by human evaluators, our OmniShow achieves superior performance in overall quality, showing great alignment with human preference.
Table 2: Quantitative results of ablation studies. We evaluate three key components of OmniShow , including (a) Unified Channel-wise Conditioning, (b) Gated Local-Context Attention, and (c) Decoupled-Then-Joint Training.
| Method | *IQA | *AES | Sync-C | Sync-D |
| FantasyTalking [ 58 ] | 2.11 | 1.12 | 1.11 | 12.88 |
| HunyuanVideo-Avatar [ 7 ] | 1.76 | 1.18 | 4.89 | 9.37 |
| Hallo3 [ 10 ] | 2.31 | 1.48 | 4.26 | 10.22 |
| MultiTalk [ 34 ] | 2.07 | 1.30 | 6.34 | 8.47 |
| OmniAvatar [ 18 ] | 2.16 | 1.31 | 5.40 | 9.13 |
| OmniShow-A2V (Ours) | 2.26 | 1.51 | 6.49 | 8.97 |
Table 3: Performance on the EMTD benchmark. Note that metrics marked with “*” adopt the definitions from this benchmark, thus leading to different numerical ranges.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Data collection pipeline. We first segment videos into isolated shots and then filter them by diverse metric scores. Subsequently, specialized processing workflows are applied to construct multiple heterogeneous datasets.
| Method | Text Align. | Reference Consistency | Audio-Visual Sync. | Pose Accuracy | Video Quality | ||||||
| TA | FaceSim | NexusScore | Sync-C | Sync-D | AKD | PCK | AES | IQA | VQ | MQ | |
| Cascaded Baseline | 6.885 | 0.591 | 0.341 | 7.016 | 7.823 | 0.198 | 0.340 | 0.417 | 0.709 | 10.05 | 3.911 |
| OmniShow (Ours) | 7.134 | 0.645 | 0.353 | 7.699 | 7.674 | 0.172 | 0.478 | 0.424 | 0.725 | 11.06 | 5.880 |
Table 4: Quantitative comparison with the cascaded baseline on the RAP2V task. We compare our end-to-end OmniShow with a representative cascaded baseline (VACE [ 31 ] +LatentSync [ 35 ] ). Our approach outperforms the cascaded baseline across all evaluation metrics, highlighting the superiority of multi-condition unification.
Figure 10: Qualitative comparison with the cascaded baseline on the RAP2V task. Compared to the cascaded baseline (VACE [ 31 ] +LatentSync [ 35 ] ), our OmniShow generates visually coherent videos with precise lip synchronization and superior visual quality. Notably, it avoids the artifacts and blur often introduced by isolated lip-synchronization generation, particularly under complex occlusions. Zoom in for better view.
Table 5: Quantitative results of additional ablation studies. We further evaluate two design choices within the proposed techniques, including (a) different RoPE strategies for positional encoding of the pseudo-frames in Decoupled-Then-Joint Training and (b) The impact of context window size in Gated Local-Context Attention.
Figure 11: More qualitative results. We present additional generated results from our OmniShow and other methods across various multimodal condition settings, showing the state-of-the-art performance of OmniShow . Note that OmniShow is the first-of-its-kind framework supporting the full spectrum of four multimodal inputs required by HOIVG. We recommend visiting our project page for video demonstrations.
Figure 12: More qualitative results. We present additional generated results from our OmniShow and other methods across various multimodal condition settings, showing the state-of-the-art performance of OmniShow . Note that OmniShow is the first-of-its-kind framework supporting the full spectrum of four multimodal inputs required by HOIVG. We recommend visiting our project page for video demonstrations.
Figure 13: More qualitative results. We present additional generated results from our OmniShow and other methods across various multimodal condition settings, showing the state-of-the-art performance of OmniShow . Note that OmniShow is the first-of-its-kind framework supporting the full spectrum of four multimodal inputs required by HOIVG. We recommend visiting our project page for video demonstrations.