T2V Generation

T2V: Text-to-Video

Ask about this topic

Overview

Text-to-video generation studies how to turn language into coherent video, enabling uses such as storytelling, instruction, and creative production. Most systems build on diffusion transformers or flow-matching models, with prompt refinement, inference-time guidance, reference conditioning, and preference-based post-training used to improve control. Evaluation combines visual-quality and prompt-alignment measures such as VBench and T2V-CompBench with targeted tests of physics, temporal events, identity, and spatial relations. Models can produce convincing appearances, but reliable motion, physical interactions, and fine-grained prompt compliance remain difficult, while inference cost and safety add deployment challenges.

Latest papers

CardsList

110 papers14 papers in the last four weeks, up 75% on the four weeks before. 0.1% of all new papers.

Mar 14, 2026

MotionCFG: Boosting Motion Dynamics via Stochastic Concept Perturbation

Despite recent advances in Text-to-Video (T2V) synthesis, generating high-fidelity and dynamic motion remains a significant challenge. Existing methods primarily rely on Classifier-Free Guidance (CFG), often with explicit negative prompts (e.g. "static", "blurry"), to suppress undesired artifacts. However, such explicit negations frequently introduce unintended semantic bias and distort object integrity; a phenomenon we define as Content-Motion Drift. To address this, we propose MotionCFG, a framework that enhances motion dynamics by contrasting a target concept with its noise-perturbed counterparts. Specifically, by injecting Gaussian noise into the concept embeddings, MotionCFG creates localized negative anchors that encapsulate a broad complementary space of sub-optimal motion variations. Unlike explicit negations, this approach facilitates implicit hard negative mining without shifting the global semantic identity, allowing for a focused refinement of temporal details. Combined with a piecewise guidance schedule that confines intervention to the early denoising steps, MotionCFG consistently improves motion dynamics across state-of-the-art T2V frameworks with negligible computational overhead and minimal compromise in visual quality. Additionally, we demonstrate that this noise-induced contrastive mechanism is effective not only for sharpening motion trajectories but also for steering complex, non-linear concepts such as precise object numerosity, which are typically difficult to modulate via standard text-based guidance.

Byungjun Kim, Soobin Um, Jong Chul Ye

Graduate School of AI, KAIST, Daejeon 34141, Republic of Korea · Department of AI, Kookmin University, Seoul 02707, Republic of Korea

T2V GenerationVideo Diffusion ModelsControllable Video GenerationDiffusion Guidance
AskPDF
Mar 14, 2026

UniVid: Pyramid Diffusion Model for High Quality Video Generation

Diffusion-based text-to-video generation (T2V) or image-to-video (I2V) generation have emerged as a prominent research focus. However, there exists a challenge in integrating the two generative paradigms into a unified model. In this paper, we present a unified video generation model (UniVid) with hybrid conditions of the text prompt and reference image. Given these two available controls, our model can extract objects' appearance and their motion descriptions from textual prompts, while obtaining texture details and structural information from image clues to guide the video generation process. Specifically, we scale up the pre-trained text-to-image diffusion model for generating temporally coherent frames via introducing our temporal-pyramid cross-frame spatial-temporal attention modules and convolutions. To support bimodal control, we introduce a dual-stream cross-attention mechanism, whose attention scores can be freely re-weighted for interpolation of between single and two modalities controls during inference. Extensive experiments showcase that our UniVid achieves superior temporal coherence on T2V, I2V and (T+I)2V tasks.

Xinyu Xiao, Binbin Yang, Tingtian Li +2

Video Diffusion ModelsT2V GenerationControllable Video GenerationImage-to-Video GenerationTemporal Consistency in Video Generation
AskPDF
Mar 12, 2026

OSCBench: Benchmarking Object State Change in Text-to-Video Generation

Text-to-video (T2V) generation models have made rapid progress in producing visually high-quality and temporally coherent videos. However, existing benchmarks primarily focus on perceptual quality, text-video alignment, or physical plausibility, leaving a critical aspect of action understanding largely unexplored: object state change (OSC) explicitly specified in the text prompt. OSC refers to the transformation of an object's state induced by an action, such as peeling a potato or slicing a lemon. In this paper, we introduce OSCBench, a benchmark specifically designed to assess OSC performance in T2V models. OSCBench is constructed from instructional cooking data and systematically organizes action-object interactions into regular, novel, and compositional scenarios to probe both in-distribution performance and generalization. We evaluate six representative open-source and proprietary T2V models using both human user study and multimodal large language model (MLLM)-based automatic evaluation. Our results show that, despite strong performance on semantic and scene alignment, current T2V models consistently struggle with accurate and temporally consistent object state changes, especially in novel and compositional settings. These findings position OSC as a key bottleneck in text-to-video generation and establish OSCBench as a diagnostic benchmark for advancing state-aware video generation models.

Xianjing Han, Bin Zhu, Shiqi Hu +4

National University of Singapore · Singapore Management University · Carnegie Mellon University +1

T2V GenerationVideo Generation EvaluationPhysical Consistency in Video Generation
AskPDF
Mar 12, 2026

Event-Driven Video Generation

Current text-to-video models can make individual frames look convincing while still getting simple interactions wrong: objects move before contact, an intended action is skipped, a placed object keeps drifting, or a support relation breaks. Our starting point is that standard frame-first denoising updates every latent region at every step, even when the prompt implies that only a local interaction should be active. We introduce Event-Driven Video Generation (EVD), a small DiT-compatible intervention that gives the sampler an explicit event signal. A lightweight head predicts token-level event activity; training losses tie that activity to latent state change; and event-gated sampling, with hysteresis and an early-step schedule, applies the update field mainly where an interaction is forming. On EVD-Bench, EVD improves human preference and VBench dynamics for state persistence, spatial accuracy, support relations, and contact stability, while keeping appearance quality comparable to the base model. The results suggest that a modest amount of event structure can correct several interaction failures that otherwise remain hidden behind good frame-level appearance.

Chika Maduabuchi, Jindong Wang

William & Mary, Williamsburg, VA, USA

Physical Consistency in Video GenerationVideo Diffusion ModelsT2V GenerationControllable Video Generation
AskPDF
Mar 7, 2026

Two Frames Matter: A Temporal Attack for Text-to-Video Model Jailbreaking

Recent text-to-video (T2V) models can synthesize complex videos from lightweight natural language prompts, raising urgent concerns about safety alignment in the event of misuse in the real world. Prior jailbreak attacks typically rewrite unsafe prompts into paraphrases that evade content filters while preserving meaning. Yet, these approaches often still retain explicit sensitive cues in the input text and therefore overlook a more profound, video-specific weakness. In this paper, we identify a temporal trajectory infilling vulnerability of T2V systems under fragmented prompts: when the prompt specifies only sparse boundary conditions (e.g., start and end frames) and leaves the intermediate evolution underspecified, the model may autonomously reconstruct a plausible trajectory that includes harmful intermediate frames, despite the prompt appearing benign to input or output side filtering. Building on this observation, we propose TFM. This fragmented prompting framework converts an originally unsafe request into a temporally sparse two-frame extraction and further reduces overtly sensitive cues via implicit substitution. Extensive evaluations across multiple open-source and commercial T2V models demonstrate that TFM consistently enhances jailbreak effectiveness, achieving up to a 12% increase in attack success rate on commercial systems. Our findings highlight the need for temporally aware safety mechanisms that account for model-driven completion beyond prompt surface form.

Moyang Chen, Zonghao Ying, Wenzhuo Xu +4

College of Science, Mathematics and Technology, Wenzhou-Kean University · AI Security Lab · State Key Laboratory of Complex & Critical Software Environment, Beihang University

T2V GenerationAdversarial Text AttacksAdversarial Examples
AskPDF
Mar 6, 2026

NEGATE: Constrained Semantic Guidance for Linguistic Negation in Text-to-Video Diffusion

Negation is a fundamental linguistic operator, yet it remains inadequately modeled in diffusion-based generative systems. In this work, we present a formal treatment of linguistic negation in diffusion-based generative models by modeling it as a structured feasibility constraint on semantic guidance within diffusion dynamics. Rather than introducing heuristics or retraining model parameters, we reinterpret classifier-free guidance as defining a semantic update direction and enforce negation by projecting the update onto a convex constraint set derived from linguistic structure. This novel formulation provides a unified framework for handling diverse negation phenomena, including object absence, graded non-inversion semantics, multi-negation composition, and scope-sensitive disambiguation. Our approach is training-free, compatible with pretrained diffusion backbones, and naturally extends from image generation to temporally evolving video trajectories. In addition, we introduce a structured negation-centric benchmark suite that isolates distinct linguistic failure modes in generative systems, to further research in this area. Experiments demonstrate that our method achieves robust negation compliance while preserving visual fidelity and structural coherence, establishing the first unified formulation of linguistic negation in diffusion-based generative models beyond representation-level evaluation.

Taewon Kang, Ming C. Lin

University of Maryland at College Park, United States

Diffusion Model GuidanceT2V GenerationVideo Diffusion ModelsControllable Video Generation
AskPDF
Mar 5, 2026

Accelerating Text-to-Video Generation with Calibrated Sparse Attention

Recent diffusion models enable high-quality video generation, but suffer from slow runtimes. The large transformer-based backbones used in these models are bottlenecked by spatiotemporal attention. In this paper, we identify that a significant fraction of token-to-token connections consistently yield negligible scores across various inputs, and their patterns often repeat across queries. Thus, the attention computation in these cases can be skipped with little to no effect on the result. This observation continues to hold for connections among local token blocks. Motivated by this, we introduce CalibAtt, a training-free method that accelerates video generation via calibrated sparse attention. CalibAtt performs an offline calibration pass that identifies block-level sparsity and repetition patterns that are stable across inputs, and compiles these patterns into optimized attention operations for each layer, head, and diffusion timestep. At inference time, we compute the selected input-dependent connections densely, and skip the unselected ones in a hardware-efficient manner. Extensive experiments on Wan 2.1 14B, Mochi 1, and few-step distilled models at various resolutions show that CalibAtt achieves up to 1.58x end-to-end speedup, outperforming existing training-free methods while maintaining video generation quality and text-video alignment.

Shai Yehezkel, Shahar Yadin, Noam Elata +2

Apple · Tel Aviv University

Diffusion Model Inference AccelerationT2V GenerationVideo Diffusion ModelsSparse AttentionDiffusion Transformer
AskPDF
Mar 3, 2026

PhyPrompt: RL-based Prompt Refinement for Physically Plausible Text-to-Video Generation

State-of-the-art text-to-video (T2V) generators frequently violate physical laws despite high visual quality. We show this stems from insufficient physical constraints in prompts rather than model limitations: manually adding physics details reliably produces physically plausible videos, but requires expertise and does not scale. We present PhyPrompt, a two-stage reinforcement learning framework that automatically refines prompts for physically realistic generation. First, we fine-tune a large language model on a physics-focused Chain-of-Thought dataset to integrate principles like object motion and force interactions while preserving user intent. Second, we apply Group Relative Policy Optimization with a dynamic reward curriculum that initially prioritizes semantic fidelity, then progressively shifts toward physical commonsense. This curriculum achieves synergistic optimization: PhyPrompt-7B reaches 40.8% joint success on VideoPhy2 (8.6pp gain), improving physical commonsense by 11pp (55.8% to 66.8%) while simultaneously increasing semantic adherence by 4.4pp (43.4% to 47.8%). Remarkably, our curriculum exceeds single-objective training on both metrics, demonstrating compositional prompt discovery beyond conventional multi-objective trade-offs. PhyPrompt outperforms GPT-4o (+3.8% joint) and DeepSeek-V3 (+2.2%, 100×\times larger) using only 7B parameters. The approach transfers zero-shot across diverse T2V architectures (Lavie, VideoCrafter2, CogVideoX-5B) with up to 16.8% improvement, establishing that domain-specialized reinforcement learning with compositional curricula surpasses general-purpose scaling for physics-aware generation.

Shang Wu, Chenwei Xu, Zhuofan Xia +6

Northwestern University · Dolby Laboratories

Prompt Optimization for T2V GenerationPhysical Consistency in Video GenerationT2V GenerationVideo GenerationRL for Video Generation
AskPDF
Mar 3, 2026

BrandFusion: A Multi-Agent Framework for Seamless Brand Integration in Text-to-Video Generation

The rapid advancement of text-to-video (T2V) models has revolutionized content creation, yet their commercial potential remains largely untapped. We introduce, for the first time, the task of seamless brand integration in T2V: automatically embedding advertiser brands into prompt-generated videos while preserving semantic fidelity to user intent. This task confronts three core challenges: maintaining prompt fidelity, ensuring brand recognizability, and achieving contextually natural integration. To address them, we propose BrandFusion, a novel multi-agent framework comprising two synergistic phases. In the offline phase (advertiser-facing), we construct a Brand Knowledge Base by probing model priors and adapting to novel brands via lightweight fine-tuning. In the online phase (user-facing), five agents jointly refine user prompts through iterative refinement, leveraging the shared knowledge base and real-time contextual tracking to ensure brand visibility and semantic alignment. Experiments on 18 established and 2 custom brands across multiple state-of-the-art T2V models demonstrate that BrandFusion significantly outperforms baselines in semantic preservation, brand recognizability, and integration naturalness. Human evaluations further confirm higher user satisfaction, establishing a practical pathway for sustainable T2V monetization.

Zihao Zhu, Ruotong Wang, Siwei Lyu +2

The Chinese University of Hong Kong, Shenzhen · State University of New York at Buffalo · Harbin Institute of Technology +1

T2V GenerationControllable Video Generation
AskPDF
Mar 3, 2026

Does Semantic Noise Initialization Transfer from Images to Videos? A Paired Diagnostic Study

Semantic noise initialization has been reported to improve robustness and controllability in image diffusion models. Whether these gains transfer to text-to-video (T2V) generation remains unclear, since temporal coupling can introduce extra degrees of freedom and instability. We benchmark semantic noise initialization against standard Gaussian noise using a frozen VideoCrafter-style T2V diffusion backbone and VBench on 100 prompts. Using prompt-level paired tests with bootstrap confidence intervals and a sign-flip permutation test, we observe a small positive trend on temporal-related dimensions; however, the 95 percent confidence interval includes zero (p ~ 0.17) and the overall score remains on par with the baseline. To understand this outcome, we analyze the induced perturbations in noise space and find patterns consistent with weak or unstable signal. We recommend prompt-level paired evaluation and noise-space diagnostics as standard practice when studying initialization schemes for T2V diffusion.

Yixiao Jing, Chaoyu Zhang, Zixuan Zhong +1

University of Michigan · University College London

Neural Network InitializationVideo Diffusion ModelsT2V Generation
AskPDF
Mar 2, 2026

Retrieval, Refinement, and Ranking for Text-to-Video Generation via Prompt Optimization and Test-Time Scaling

While large-scale datasets have driven significant progress in Text-to-Video (T2V) generative models, these models remain highly sensitive to input prompts, demonstrating that prompt design is critical to generation quality. Current methods for improving video output often fall short: they either depend on complex, post-editing models, risking the introduction of artifacts, or require expensive fine-tuning of the core generator, which severely limits both scalability and accessibility. In this work, we introduce 3R, a novel RAG based prompt optimization framework. 3R utilizes the power of current state-of-the-art T2V diffusion model and vision language model. It can be used with any T2V model without any kind of model training. The framework leverages three key strategies: RAG-based modifiers extraction for enriched contextual grounding, diffusion-based Preference Optimization for aligning outputs with human preferences, and temporal frame interpolation for producing temporally consistent visual contents. Together, these components enable more accurate, efficient, and contextually aligned text-to-video generation. Experimental results demonstrate the efficacy of 3R in enhancing the static fidelity and dynamic coherence of generated videos, underscoring the importance of optimizing user prompts.

Zillur Rahman, Alex Sheng, Cristian Meo

Algoverse AI

Prompt Optimization for T2V GenerationRetrieval-Augmented Video GenerationT2V GenerationVideo Diffusion ModelsTemporal Consistency in Video Generation
AskPDF
Feb 27, 2026

SKeDA: A Generative Watermarking Framework for Text-to-video Diffusion Models

The rise of text-to-video generation models has raised growing concerns over content authenticity, copyright protection, and malicious misuse. Watermarking serves as an effective mechanism for regulating such AI-generated content, where high fidelity and strong robustness are particularly critical. Recent generative image watermarking methods provide a promising foundation by leveraging watermark information and pseudo-random keys to control the initial sampling noise, enabling lossless embedding. However, directly extending these techniques to videos introduces two key limitations: Existing designs implicitly rely on strict alignment between video frames and frame-dependent pseudo-random binary sequences used for watermark encryption. Once this alignment is disrupted, subsequent watermark extraction becomes unreliable; and Video-specific distortions, such as inter-frame compression, significantly degrade watermark reliability. To address these issues, we propose SKeDA, a generative watermarking framework tailored for text-to-video diffusion models. SKeDA consists of two components: (1) Shuffle-Key-based Distribution-preserving Sampling (SKe) employs a single base pseudo-random binary sequence for watermark encryption and derives frame-level encryption sequences through permutation. This design transforms watermark extraction from synchronization-sensitive sequence decoding into permutation-tolerant set-level aggregation, substantially improving robustness against frame reordering and loss; and (2) Differential Attention (DA), which computes inter-frame differences and dynamically adjusts attention weights during extraction, enhancing robustness against temporal distortions. Extensive experiments demonstrate that SKeDA preserves high video generation quality and watermark robustness.

Yang Yang, Xinze Zou, Zehua Ma +2

the School of Electronic and Information Engineering, Anhui University, Hefei, Anhui 230601, China · Anhui Province Key Laboratory of Digital Security and the CAS Key Laboratory of Electromagnetic Space Information, University of Science and Technology of China, Hefei 230026, China · the School of Computing, National University of Singapore, Singapore 117417

Diffusion Model WatermarkingRobust WatermarkingVideo Diffusion ModelsT2V Generation
AskPDF
Feb 26, 2026

SPATIALALIGN: Aligning Dynamic Spatial Relationships in Video Generation

Most text-to-video (T2V) generators prioritize aesthetic quality, but often ignoring the spatial constraints in the generated videos. In this work, we present SPATIALALIGN, a self-improvement framework that enhances T2V models capabilities to depict Dynamic Spatial Relationships (DSR) specified in text prompts. We present a zeroth-order regularized Direct Preference Optimization (DPO) to fine-tune T2V models towards better alignment with DSR. Specifically, we design DSR-SCORE, a geometry-based metric that quantitatively measures the alignment between generated videos and the specified DSRs in prompts, which is a step forward from prior works that rely on VLM for evaluation. We also conduct a dataset of text-video pairs with diverse DSRs to facilitate the study. Extensive experiments demonstrate that our fine-tuned model significantly out performs the baseline in spatial relationships. The code will be released in Link. Project page: https://fengming001ntu.github.io/SpatialAlign/

Fengming Liu, Tat-Jen Cham, Chuanxia Zheng

College of Computing and Data Science, Nanyang Technological University, 50 Nanyang Avenue, Singapore 639798.

T2V GenerationDirect Preference OptimizationVideo Generation Evaluation
AskPDF
Feb 23, 2026

PedaCo-Gen: Scaffolding Pedagogical Agency in Human-AI Collaborative Video Authoring

While advancements in Text-to-Video (T2V) generative AI offer a promising path toward democratizing content creation, current models are often optimized for visual fidelity rather than instructional efficacy. This study introduces PedaCo-Gen, a pedagogically-informed human-AI collaborative video generating system for authoring instructional videos based on Mayer's Cognitive Theory of Multimedia Learning (CTML). Moving away from traditional "one-shot" generation, PedaCo-Gen introduces an Intermediate Representation (IR) phase, enabling educators to interactively review and refine video blueprints-comprising scripts and visual descriptions-with an AI reviewer. Our study with 23 education experts demonstrates that PedaCo-Gen significantly enhances video quality across various topics and CTML principles compared to baselines. Participants perceived the AI-driven guidance not merely as a set of instructions but as a metacognitive scaffold that augmented their instructional design expertise, reporting high production efficiency (M=4.26) and guide validity (M=4.04). These findings highlight the importance of reclaiming pedagogical agency through principled co-creation, providing a foundation for future AI authoring tools that harmonize generative power with human professional expertise.

Injun Baek, Yearim Kim, Nojun Kwak

Samsung Electronics, Suwon, Republic of Korea

Human-AI Co-CreationGenerative AI in EducationVideo GenerationT2V Generation
AskPDF
Feb 18, 2026

CHAI: CacHe Attention Inference for text2video

Text-to-video diffusion models deliver impressive results but remain slow because of the sequential denoising of 3D latents. Existing approaches to speed up inference either require expensive model retraining or use heuristic-based step skipping, which struggles to maintain video quality as the number of denoising steps decreases. Our work, CHAI, aims to use cross-inference caching to reduce latency while maintaining video quality. We introduce Cache Attention as an effective method for attending to shared objects/scenes across cross-inference latents. This selective attention mechanism enables effective reuse of cached latents across semantically related prompts, yielding high cache hit rates. We show that it is possible to generate high-quality videos using Cache Attention with as few as 8 denoising steps. When integrated into the overall system, CHAI is 1.65x - 3.35x faster than baseline OpenSora 1.2 while maintaining video quality.

Joel Mathew Cherian, Ashutosh Muralidhara Bharadwaj, Vima Gupta +1

Georgia Institute of Technology.

T2V GenerationEfficient Video GenerationDiffusion Model Inference AccelerationVideo Diffusion ModelsDiffusion Model Caching
AskPDF
Feb 17, 2026

Consistency-Preserving Diverse Video Generation

Text-to-video generation is expensive, so only a few samples are typically produced per prompt. In this low-sample regime, maximizing the value of each batch requires high cross-video diversity. Recent methods improve diversity for image generation, but for videos they often degrade within-video temporal consistency and require costly backpropagation through a video decoder. We propose a joint-sampling framework for flow-matching video generators that improves batch diversity while preserving temporal consistency. Our approach applies diversity-driven updates and then removes only the components that would decrease a temporal-consistency objective. To avoid image-space gradients, we compute both objectives with lightweight latent-space models, avoiding video decoding and decoder backpropagation. Experiments on a state-of-the-art text-to-video flow-matching model show diversity close to strong joint-sampling baselines while substantially improving temporal consistency and color naturalness. Our code is available at https://github.com/XinshuangL/Diverse-Video.

Xinshuang Liu, Runfa Blark Li, Truong Nguyen

University of California, San Diego, CA, USA

Video GenerationT2V GenerationTemporal Consistency in Video Generation
AskPDF
Feb 2, 2026

PISCES: Annotation-free Text-to-Video Post-Training via Optimal Transport-Aligned Rewards

Text-to-video (T2V) generation aims to synthesize videos with high visual quality and temporal consistency that are semantically aligned with input text. Reward-based post-training has emerged as a promising direction to improve the quality and semantic alignment of generated videos. However, recent methods either rely on large-scale human preference annotations or operate on misaligned embeddings from pre-trained vision-language models, leading to limited scalability or suboptimal supervision. We present PISCES\texttt{PISCES}, an annotation-free post-training algorithm that addresses these limitations via a novel Dual Optimal Transport (OT)-aligned Rewards module. To align reward signals with human judgment, PISCES\texttt{PISCES} uses OT to bridge text and video embeddings at both distributional and discrete token levels, enabling reward supervision to fulfill two objectives: (i) a Distributional OT-aligned Quality Reward that captures overall visual quality and temporal coherence; and (ii) a Discrete Token-level OT-aligned Semantic Reward that enforces semantic, spatio-temporal correspondence between text and video tokens. To our knowledge, PISCES\texttt{PISCES} is the first to improve annotation-free reward supervision in generative post-training through the lens of OT. Experiments on both short- and long-video generation show that PISCES\texttt{PISCES} outperforms both annotation-based and annotation-free methods on VBench across Quality and Semantic scores, with human preference studies further validating its effectiveness. We show that the Dual OT-aligned Rewards module is compatible with multiple optimization paradigms, including direct backpropagation and reinforcement learning fine-tuning. Project page: https://roar-ai.github.io/pisces

Minh-Quan Le, Gaurav Mittal, Cheng Zhao +3

1Microsoft · 2Stony Brook University · †Currently at Dolby Labs. 1Microsoft 2Stony Brook University

T2V GenerationVideo GenerationVision-Language Alignment
AskPDF
Jan 4, 2026

Slot-ID: Identity-Preserving Video Generation from Reference Videos via Slot-Based Temporal Identity Encoding

Human identity-preserving text-to-video generation remains challenging under large changes in viewpoint, facial expression, illumination, and motion. Existing methods condition the generator on a single reference portrait, but a static image cannot capture how identity-bearing cues evolve across views and expressions, leading to face deformation, pose locking, identity drift, or over-smoothed faces. We observe that a short reference clip naturally provides richer temporal and multi-view identity cues than any single image, motivating a video-referential formulation. This richer signal, however, introduces a new challenge: identity evidence is distributed across many frames and must be distilled into a compact, stable representation under a limited token budget. To this end, we propose Slot-ID, a lightweight identity-conditioning framework built on a frozen text-to-video backbone. Slot-ID employs a slot-based temporal identity encoder with Sinkhorn-routed iterative reading to distill a compact, stable set of identity tokens from the reference clip, complemented by an image-anchor stream for dual-source conditioning. Extensive experiments demonstrate that Slot-ID outperforms state-of-the-art methods in identity preservation and visual naturalness while remaining competitive in prompt following, with particularly large gains under challenging pose, expression, and motion variations.

Yixuan Lai, He Wang, Kun Zhou +1

State Key Lab of CAD&CG Zhejiang University Hangzhou, China · Department of Computer Science University College London London, UK

Video Diffusion ModelsT2V GenerationIdentity-Preserving Video GenerationVideo Generation
AskPDF
Dec 30, 2025

PhyAVBench: A Challenging Audio Physics-Sensitivity Benchmark for Physically Grounded Text-to-Audio-Video Generation

Text-to-audio-video (T2AV) generation is central to applications such as filmmaking and world modeling. However, current models often fail to produce physically plausible sounds. Previous benchmarks primarily focus on audio-video temporal synchronization, while largely overlooking explicit evaluation of audio-physics grounding, thereby limiting the study of physically plausible audio-visual generation. To address this issue, we present PhyAVBench, the first benchmark that systematically evaluates the audio-physics grounding capabilities of T2AV, image-to-audio-video (I2AV), and video-to-audio (V2A) models. PhyAVBench offers PhyAV-Sound-11K, a new dataset of 25.5 hours of 11,605 audible videos collected from 184 participants to ensure diversity and avoid data leakage. It contains 337 paired-prompt groups with controlled physical variations that drive sound differences, each grounded with an average of 17 videos and spanning 6 audio-physics dimensions and 41 fine-grained test points. Each prompt pair is annotated with the physical factors underlying their acoustic differences. Importantly, PhyAVBench leverages paired text prompts to evaluate this capability. We term this evaluation paradigm the Audio-Physics Sensitivity Test (APST) and introduce a novel metric, the Contrastive Physical Response Score (CPRS), which quantifies the acoustic consistency between generated videos and their real-world counterparts. We conduct a comprehensive evaluation of 17 state-of-the-art models. Our results reveal that even leading commercial models struggle with fundamental audio-physical phenomena, exposing a critical gap beyond audio-visual synchronization and pointing to future research directions. We hope PhyAVBench will serve as a foundation for advancing physically grounded audio-visual generation. Prompts, ground-truth, and generated video samples are available at https://github.com/imxtx/PhyAVBench.

Tianxin Xie, Wentao Lei, Kai Jiang +27

HKUST(GZ) · Tencent

Audio-Video GenerationVideo Generation EvaluationT2V Generation
AskPDF
Dec 8, 2025

ContextAnyone: Context-Aware Diffusion for Character-Consistent Text-to-Video Generation

Text-to-video generation has advanced rapidly, yet preserving a character's holistic appearance from a single reference image remains challenging, particularly when the character undergoes large pose, motion, and scene changes. Existing reference-conditioned approaches primarily treat the reference image as a conditioning signal, which can weaken fine-grained appearance information as reference and noisy video tokens interact during denoising. We propose ContextAnyone, a context-aware diffusion framework that instead treats the reference as an explicitly preserved appearance anchor. Our key idea is to jointly reconstruct the reference image and generate the target video within a shared diffusion transformer, providing direct supervision for preserving identity and fine-grained appearance throughout denoising. To maintain the reference as a stable source of appearance information, we further introduce asymmetric information flow that allows video tokens to selectively access reference tokens while preventing noisy video features from propagating back to the reference branch. We complement this design with Gap-RoPE, which separates the positional representations of the reference and generated video tokens. Experiments on a benchmark constructed from OpenVid-HD demonstrate that ContextAnyone improves both identity and fine-grained appearance consistency over existing reference-conditioned baselines while maintaining motion characteristics close to the underlying text-to-video generator.

Ziyang Mai, Yu-Wing Tai

Dartmouth College

Video Diffusion ModelsDiffusion TransformerT2V GenerationIdentity-Preserving Video GenerationVideo Generation
AskPDF