Evolution of Video Generative Foundations

Teng Hu; Jiangning Zhang; Hongrui Huang; Ran Yi; Zihan Su; Jieyu Weng; Zhucun Xue; Lizhuang Ma

cs.CVApr 7, 2026

Evolution of Video Generative Foundations

Authors: Teng Hu, Jiangning Zhang, Hongrui Huang, Ran Yi, Zihan Su, Jieyu Weng, Zhucun Xue, Lizhuang Ma, +2 more

Abstract

The rapid advancement of Artificial Intelligence Generated Content (AIGC) has revolutionized video generation, enabling systems ranging from proprietary pioneers like OpenAI's Sora, Google's Veo3, and Bytedance's Seedance to powerful open-source contenders like Wan and HunyuanVideo to synthesize temporally coherent and semantically rich videos. These advancements pave the way for building "world models" that simulate real-world dynamics, with applications spanning entertainment, education, and virtual reality. However, existing reviews on video generation often focus on narrow technical fields, e.g., Generative Adversarial Networks (GAN) and diffusion models, or specific tasks (e. g., video editing), lacking a comprehensive perspective on the field's evolution, especially regarding Auto-Regressive (AR) models and integration of multimodal information. To address these gaps, this survey firstly provides a systematic review of the development of video generation technology, tracing its evolution from early GANs to dominant diffusion models, and further to emerging AR-based and multimodal techniques. We conduct an in-depth analysis of the foundational principles, key advancements, and comparative strengths/limitations. Then, we explore emerging trends in multimodal video generation, emphasizing the integration of diverse data types to enhance contextual awareness. Finally, by bridging historical developments and contemporary innovations, this survey offers insights to guide future research in video generation and its applications, including virtual/augmented reality, personalized education, autonomous driving simulations, digital entertainment, and advanced world models, in this rapidly evolving field. For more details, please refer to the project at https://github.com/sjtuplayer/Awesome-Video-Foundations.

Figures & tables

Fig. 1: The evolution of generative paradigms in video generation, illustrating the shift from implicit adversarial learning (GANs) to explicit density estimation via denoising (Diffusion) and next-token prediction (Auto-Regressive)

Fig. 2: The trending of video generation models in the top-tire conferences.

Paradigm	Image Foundation	Video Evolution
VAE	VQ-VAE / VQ-GAN [ 29 ]	VideoGPT [ 27 ] , MAGVIT [ 30 ]
GAN	StyleGAN [ 16 ]	MoCoGAN [ 21 ] , StyleGAN-V [ 31 ]
Diffusion	DDPM [ 1 ] / LDM [ 2 ]	SVD [ 3 ] , VideoCrafter2 [ 32 ]
Diff-Transformer	DiT [ 33 ]	Wan [ 7 ] , HunyuanVideo [ 8 ]

TABLE I: Technological Correspondence between Image and Video Generation. Video models typically inherit the architectural core of image models while introducing specialized temporal adaptation mechanisms.

Fig. 3: A comprehensive taxonomy of video generation methodologies. The framework categorizes existing works based on three dominant generative paradigms—GANs, Diffusion Models, and Auto-Regressive Models—and further structures them according to specific architectural designs and functional objectives.

Fig. 4: Comparison between different model structures for diffusion models.

Fig. 5: Timeline of the leading video generation models.

Pathway	Goal & Capability	Required Compute	Data Strategy	Est. Cost	Representative Models
1. World Foundation	Physical Simulation : Perfect physics,	Exascale	Brute Force :	$60M–$ 80M	NVIDIA Cosmos
(Industrial Scale)	robotics, fluid dynamics, zero-shot generalization.	( $\sim$ 20M H100 Hrs)	$>$ 20M hrs raw video		(14B params)
2. Balanced Research	Visual Fidelity : High aesthetic quality,	High-Performance	Optimization :	$2M–$ 3M	Seaweed-7B
(Lab Scale)	consistent motion, competitive with SOTA.	( $\sim$ 600k H100 Hrs)	Mixed-resolution training		(7B params)
3. Efficient Commercial	Media Generation : Commercial-grade	Accessible	High Curation :	$200k–$ 400k	CogVideoX-5B,
(Startup Scale)	720p/1080p video, short-form content.	( $\sim$ 100k H100 Hrs)	10M–35M HQ clips		Open-Sora 2.0

TABLE II: Resource Requirements and Strategic Pathways for Training Video Generation Models. Data summarized from technical reports of models released in 2024–2025.

Method	Consist.	Visual	Semantic	Total
Veo 3 [ 5 ]	98.18%	74.82%	82.49%	85.06%
Sora [ 4 ]	97.55%	70.55%	79.35%	84.28%
Wan2.2-14B [ 7 ]	98.02%	75.61%	79.50%	84.23%
Wan2.1-14B [ 7 ]	98.34%	73.59%	76.11%	83.69%
Luma [ 164 ]	98.19%	75.38%	84.17%	83.61%
Hunyuan Video [ 8 ]	98.32%	70.22%	76.88%	83.43%

TABLE III: Foundational quality evaluation on VBench [ 161 ] . Consist. : Temporal and background consistency; Visual : Aesthetic and imaging quality; Semantic : Semantic alignment and spatial relationships.

Method	Hard	P.A.	O.I.	All
Wan2.1-14B [ 7 ]	21.9%	31.5%	36.2%	32.6%
CogVideoX-5B [ 88 ]	0.0%	24.6%	26.1%	25.0%
Cosmos-1.0-7B [ 171 ]	10.9%	22.6%	27.4%	24.1%
Sora [ 4 ]	5.3%	22.2%	26.7%	23.3%
Ray2 [ 164 ]	8.3%	21.0%	18.5%	20.3%
Hunyuan-Video [ 8 ]	6.2%	17.6%	15.9%	17.2%

TABLE IV: Physical consistency evaluation (Success Rate %) on VideoPhy-2 [ 170 ] . Hard : Scenarios with complex causal chains; P.A. : Physical Attributes (e.g., fluids); O.I. : Object Interactions (e.g., collisions).

Method	Sexual	Violent	Social	Legal	Avg.
Pika [ 174 ]	40.85%	84.83%	53.05%	53.07%	59.0%
Gen2 [ 175 ]	16.00%	31.70%	59.45%	41.93%	40.9%
SVD [ 3 ]	3.65%	47.83%	63.70%	51.83%	46.8%
Open-Sora [ 176 ]	36.95%	80.07%	54.75%	30.83%	52.1%

TABLE V: Representative safety performance (NSFW rates) on T2VSafetyBench [ 172 ] .

Fig. 6: Benchmarks of video generation model.

Fig. 7: The downstream video generation tasks can be primarily classified into 4 groups according to the modality and granularity of user control, which are video editing (a), video customization (b), motion control (c), and world model (d).

Explore similar work

Apr 3, 2026

Generative AI for Video Trailer Synthesis: From Extractive Heuristics to Autoregressive Creativity

The domain of automatic video trailer generation is currently undergoing a profound paradigm shift, transitioning from heuristic-based extraction methods to deep generative synthesis. While early methodologies relied heavily on low-level feature engineering, visual saliency, and rule-based heuristics to select representative shots, recent advancements in Large Language Models (LLMs), Multimodal Large Language Models (MLLMs), and diffusion-based video synthesis have enabled systems that not only identify key moments but also construct coherent, emotionally resonant narratives. This survey provides a comprehensive technical review of this evolution, with a specific focus on generative techniques including autoregressive Transformers, LLM-orchestrated pipelines, and text-to-video foundation models like OpenAI's Sora and Google's Veo. We analyze the architectural progression from Graph Convolutional Networks (GCNs) to Trailer Generation Transformers (TGT), evaluate the economic implications of automated content velocity on User-Generated Content (UGC) platforms, and discuss the ethical challenges posed by high-fidelity neural synthesis. By synthesizing insights from recent literature, this report establishes a new taxonomy for AI-driven trailer generation in the era of foundation models, suggesting that future promotional video systems will move beyond extractive selection toward controllable generative editing and semantic reconstruction of trailers.

Abhishek Dharmaratnakar, Srivaths Ranganathan, Debanshu Das +1

Video Generation Text-to-Video Generation Controllable Video Generation Autoregressive Video Generation Video Diffusion Models

Ask PDF

Mar 8, 2026

Image Generation Models: A Technical History

Image generation has advanced rapidly over the past decade, yet the literature seems fragmented across different models and application domains. This paper aims to offer a comprehensive survey of breakthrough image generation models, including variational autoencoders (VAEs), generative adversarial networks (GANs), normalizing flows, autoregressive and transformer-based generators, and diffusion-based methods. We provide a detailed technical walkthrough of each model type, including their underlying objectives, architectural building blocks, and algorithmic training steps. For each model type, we present the optimization techniques as well as common failure modes and limitations. We also go over recent developments in video generation and present the research works that made it possible to go from still frames to high quality videos. Lastly, we cover the growing importance of robustness and responsible deployment of these models, including deepfake risks, detection, artifacts, and watermarking.

Rouzbeh Shirvani

Image Generation Diffusion-Based Image Generation Generative Adversarial Networks Variational Autoencoders Normalizing Flows Autoregressive Image Generation

Ask PDF

Apr 17, 2026

Efficient Video Diffusion Models: Advancements and Challenges

Video diffusion models have rapidly become the dominant paradigm for high-fidelity generative video synthesis, but their practical deployment remains constrained by severe inference costs. Compared with image generation, video synthesis compounds computation across spatial-temporal token growth and iterative denoising, making attention and memory traffic major bottlenecks in real-world settings. This survey provides a systematic and deployment-oriented review of efficient video diffusion models. We propose a unified categorization that organizes existing methods into four classes of main paradigms, including step distillation, efficient attention, model compression, and cache/trajectory optimization. Building on this categorization, we respectively analyze algorithmic trends of these four paradigms and examine how different design choices target two core objectives: reducing the number of function evaluations and minimizing per-step overhead. Finally, we discuss open challenges and future directions, including quality preservation under composite acceleration, hardware-software co-design, robust real-time long-horizon generation, and open infrastructure for standardized evaluation. To the best of our knowledge, our work is the first comprehensive survey on efficient video diffusion models, offering researchers and engineers a structured overview of the field and its emerging research directions.

Shitong Shao, Lichen Bai, Pengfei Wan +2

The Hong Kong University of Science and Technology (Guangzhou), China · Kling Team, Kuaishou Technology, China · The Hong Kong University of Science and Technology, Hong Kong

Model Compression Video Diffusion Models Diffusion Model Distillation Efficient Attention Diffusion Model Caching Diffusion Model Inference Acceleration

Ask PDF