UniVid: Pyramid Diffusion Model for High Quality Video Generation

Xinyu Xiao; Binbin Yang; Tingtian Li; Yipeng Yu; Sen Lei

cs.CVMar 14, 2026

UniVid: Pyramid Diffusion Model for High Quality Video Generation

Authors: Xinyu Xiao, Binbin Yang, Tingtian Li, Yipeng Yu, Sen Lei

Abstract

Diffusion-based text-to-video generation (T2V) or image-to-video (I2V) generation have emerged as a prominent research focus. However, there exists a challenge in integrating the two generative paradigms into a unified model. In this paper, we present a unified video generation model (UniVid) with hybrid conditions of the text prompt and reference image. Given these two available controls, our model can extract objects' appearance and their motion descriptions from textual prompts, while obtaining texture details and structural information from image clues to guide the video generation process. Specifically, we scale up the pre-trained text-to-image diffusion model for generating temporally coherent frames via introducing our temporal-pyramid cross-frame spatial-temporal attention modules and convolutions. To support bimodal control, we introduce a dual-stream cross-attention mechanism, whose attention scores can be freely re-weighted for interpolation of between single and two modalities controls during inference. Extensive experiments showcase that our UniVid achieves superior temporal coherence on T2V, I2V and (T+I)2V tasks.

Figures & tables

Fig. 1: An illustration of our proposed pyramid spatial-temporal attention ( $\mathtt{PSTAttn}\emph{-f}$ ), pyramid spatial-temporal convolution ( $\mathtt{PSTConv}\emph{-f}$ ) and temporal attention, where $\emph{f}\in\{1,2,4,8\}$ indicates the downsampling factors in the U-Net layers. The red patches in frames denotes the queries of attention, and the blue ones denotes the keys / values of attention. (We also visualize the receptive field of $\mathtt{PSTConv}\emph{-f}$ in a similar manner.)

Fig. 2: An overview of UniVid. For the latent $z_{t}$ , a text prompt $\mathcal{C}_{t}$ and an image prompt $\mathcal{C}_{v}$ are used as hybrid condition to guide the denoising (generation) process. Specifically, we use a CLIP text encoder $\tau_{t}$ and an CLIP image encoder $\tau_{v}$ to obtain the linguistic token embedding $E_{t}$ and visual patch embedding $E_{v}$ , respectively. The U-Net architecture of UniVid is built by stacking multiple basic STBlocks, with each block consisting of pyramid spatial-temporal convolution ( $\mathtt{PSTConv}$ ), self-attention ( $\mathtt{SelfAttn}$ ), pyramid spatial-temporal attention ( $\mathtt{PSTAttn}$ ), dual-stream cross-attention ( $\mathtt{CrossAttn}$ ) and temporal attention ( $\mathtt{TemAttn}$ ). Best viewed in color.

Fig. 3: An illustration of our proposed dual-stream cross-attention ( $\mathtt{CrossAttn_{Dual}}$ ). We use the CLIP text encoder $\tau_{t}$ to obtain linguistic token embeddings $E_{t}$ for text prompt $\mathcal{C}_{t}$ . For the image prompt $\mathcal{C}_{v}$ , we use the counterpart CLIP image encoder $\tau_{v}$ to obtain the visual patch embeddings $E_{v}$ . Then we perform a two-path cross-attention between the input feature and these bimodal embeddings. The output feature is obtained by fusing the dual-stream results with coefficient $\lambda_{t}$ and $\lambda_{v}$ .

Fig. 4: The illstration displays examples generated from three tasks: T2V, I2V, and (T+I)2V. The top two rows present the T2V results, the middle two rows show the I2V results, and the final two rows illustrate the (T+I)2V results. The highlighted images in red correspond to the reference images.

Method	UCF-101		MSR-VTT
Method	IS $(\uparrow)$	FVD $(\downarrow)$	CLIP-SIM $(\downarrow)$	FID $(\downarrow)$
CogVideo [ 27 ]	25.27	701.59	0.2631	-
Make-A-Video [ 2 ]	33.00	367.23	0.3049	-
MagicVideo [ 28 ]	-	655	-	36.50
Latent-Shift [ 29 ]	-	360.04	0.2773	-
Video LDM [ 30 ]	33.45	550.61	0.2929	-

TABLE I: Zero-shot text to video generation on UCF-101 and MSR-VTT.

Method	Task	UCF-101		MSR-VTT
Method	Task	IS $(\uparrow)$	FVD $(\downarrow)$	CLIP-FID $(\downarrow)$	FID $(\downarrow)$
PYoCo [ 3 ]	T2V	47.76	355.19	9.73	22.14
EMU [ 34 ]	I2V	42.7	606.2	-	-
SVD [ 35 ]	I2V	-	242	-	-
DynamiCrafter [ 36 ]	(T+I)2V	-	429.23	-	-
Univid	I2V	44.51	255.33	9.91	21.54

TABLE II: Zero-shot text with image and and image-only to video generation on UCF-101 and MSR-VTT.

Explore similar work

UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors
May 1, 2026Houyuan Chen, Hong Li, Xianghao Kong +8Video Diffusion Models Unified Multimodal Models
Beihang University, Beijing, China · Nanjing University, Nanjing, China · BAAI, Beijing, China +3
Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator
Apr 9, 2026Luozheng Qin, Jia Gong, Qian Qiao +6Unified Multimodal Models Video Generation
Shanghai Academy of AI for Science · Independent Researcher · Singapore University of Technology and Design +1
Image-to-Video Diffusion: From Foundations to Open Frontiers
May 17, 2026Xianlong Wang, Wenbo Pan, Shijia Zhou +6Image-to-Video Generation Video Diffusion Models
Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China · School of Cyber Science and Engineering, Huazhong University of Science and Technology, Wuhan 430074, China · School of Cyber Science and Technology, University of Science and Technology of China, Hefei 230026, China +3

May 1, 2026J/K move · Enter open · S save

UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors

Houyuan Chen, Hong Li, Xianghao Kong, Tianrui Zhu+7

Beihang University, Beijing, China · Nanjing University, Nanjing, China · BAAI, Beijing, China+3

Video Diffusion Models Unified Multimodal Models Multimodal Diffusion Models

Open paper AskPDF arXiv

Recent progress has shown that video diffusion models (VDMs) can be repurposed for diverse multimodal graphics tasks. However, existing methods often train separate models for each problem setting, which fixes the input-output mapping and limits the modeling of correlations across modalities. We present UniVidX, a unified multimodal framework that leverages VDM priors for versatile video generation. UniVidX formulates pixel-aligned tasks as conditional generation in a shared multimodal space, adapts to modality-specific distributions while preserving the backbone's native priors, and promotes cross-modal consistency during synthesis. It is built on three key designs. Stochastic Condition Masking (SCM) randomly partitions modalities into clean conditions and noisy targets during training, enabling omni-directional conditional generation instead of fixed mappings. Decoupled Gated LoRA (DGL) introduces per-modality LoRAs that are activated when a modality serves as the generation target, preserving the strong priors of the VDM. Cross-Modal Self-Attention (CMSA) shares keys and values across modalities while keeping modality-specific queries, facilitating information exchange and inter-modal alignment. We instantiate UniVidX in two domains: UniVid-Intrinsic, for RGB videos and intrinsic maps including albedo, irradiance, and normal; and UniVid-Alpha, for blended RGB videos and their constituent RGBA layers. Experiments show that both models achieve performance competitive with state-of-the-art methods across distinct tasks and generalize robustly to in-the-wild scenarios, even when trained on fewer than 1,000 videos. Project page: https://houyuanchen111.github.io/UniVidX.github.io/

Abstract

Explore similar work

UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors

Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator

Image-to-Video Diffusion: From Foundations to Open Frontiers