Motion-Adaptive Temporal Attention for Lightweight Video Generation with Stable Diffusion

Rui Hong; Shuxue Quan

cs.CVMar 18, 2026

Motion-Adaptive Temporal Attention for Lightweight Video Generation with Stable Diffusion

Authors: Rui Hong, Shuxue Quan

Organizations: George Mason University, Fairfax, VA, USA · Independent Researcher

Abstract

We present a motion-adaptive temporal attention mechanism for parameter-efficient video generation built upon frozen Stable Diffusion models. Rather than treating all video content uniformly, our method dynamically adjusts temporal attention receptive fields based on estimated motion content: high-motion sequences attend locally across frames to preserve rapidly changing details, while low-motion sequences attend globally to enforce scene consistency. We inject lightweight temporal attention modules into all UNet transformer blocks via a cascaded strategy -- global attention in down-sampling and middle blocks for semantic stabilization, motion-adaptive attention in up-sampling blocks for fine-grained refinement. Combined with temporally correlated noise initialization and motion-aware gating, the system adds only 25.8M trainable parameters (2.9% of the base UNet) while achieving competitive results on WebVid validation when trained on 100K videos. We demonstrate that the standard denoising objective alone provides sufficient implicit temporal regularization, outperforming approaches that add explicit temporal consistency losses. Our ablation studies reveal a clear trade-off between noise correlation and motion amplitude, providing a practical inference-time control for diverse generation behaviors.

Figures & tables

Figure 1: Architecture overview. The overall pipeline follows the Latent Diffusion Model framework: video frames are encoded into latent space by a frozen VAE encoder ( $\mathcal{E}$ ), processed by the denoising UNet with injected temporal attention modules (shown in red), conditioned on text via CLIP, and the VAE decoder $\mathcal{D}$ reconstructs the frames back to pixel space. All spatial components (ResNet, Spatial Transformer) are frozen ( snowflake ); only temporal attention blocks are trained ( flame ). Down and mid blocks use global temporal attention; up blocks use motion-adaptive attention.

Figure 2: Temporal attention block variants. Left : Global mode (down/mid blocks) — uniform temporal self-attention, no motion conditioning. Right : Adaptive mode (up blocks) — motion-adaptive attention bias

\mathbf{b}_{m}

and motion-aware gate

g(m)

are both conditioned on the per-video motion score

m

Method	CLIP $\uparrow$	FC	MM $\uparrow$	FID $\downarrow$	FVD $\downarrow$
AnimateDiff $\dagger$	0.308	0.969	0.062	81.63	79.61
SD 1.5 $\ddagger$	0.267	0.777	0.326	180.48	211.02
Ours (SD 1.5)	0.302	0.980	0.051	89.96	85.99
Ours (SD 2.1)	0.303	0.983	0.040	92.09	92.49

Table 1: Comparison with AnimateDiff on WebVid validation (400 samples,

\rho{=}0.6

\lambda_{\text{temp}}{=}0

, 100K training videos).

\dagger

AnimateDiff retrained on the same 100K WebVid subset with

T{=}8

256{\times}256

, matching inference settings.

\ddagger

Base SD 1.5 without temporal modules; high MM reflects incoherent frame-independent generation.

$\lambda_{\text{temp}}$	CLIP $\uparrow$	FC	MM $\uparrow$	FID $\downarrow$	FVD $\downarrow$
2.0	0.305	0.990	0.025	95.82	97.21
0	0.303	0.983	0.040	92.09	92.49

Table 2: Effect of temporal consistency loss (

\lambda_{\text{temp}}

). SD 2.1, 100K,

\rho{=}0.6

$\rho$	CLIP $\uparrow$	FC	MM $\uparrow$	FID $\downarrow$	FVD $\downarrow$
0.4	0.278	0.929	0.158	105.85	108.22
0.6	0.303	0.983	0.040	92.09	92.49
0.85	0.298	0.999	0.008	130.87	140.11

Table 3: Effect of noise correlation

\rho

at inference. SD 2.1, 100K,

\lambda_{\text{temp}}{=}0

Base Model	CLIP $\uparrow$	FC	MM $\uparrow$	FID $\downarrow$	FVD $\downarrow$
SD 1.5	0.302	0.980	0.051	89.96	85.99
SD 2.1	0.303	0.983	0.040	92.09	92.49

Table 4: Effect of base model. 100K,

\rho{=}0.6

\lambda_{\text{temp}}{=}0

Panel of Figure 3 — Figure 3: Qualitative comparison on two WebVid validation prompts. Each pair shows frame 1 and frame 8 of the 8-frame generated sequence at $256{\times}256$ , highlighting temporal consistency across the full generation. AnimateDiff $\dagger$ retrained on 100K videos, $T{=}8$ .

Video Diffusion Models

Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.

See all 735 papers

TeDiO: Temporal Diagonal Optimization for Training-Free Coherent Video Diffusion
May 13, 2026Nurislam Tursynbek, Zhiqiang Lao, Heather Yu +2Video Diffusion Models Diffusion Transformer
UNC Chapel Hill1 · Futurewei Technologies Inc2 · UCSD3
Φ-Noise: Training-Free Temporal Video Conditioning via Phase-Based Noise Manipulation
May 23, 2026Ofir Abramovich, Nadav Z. Cohen, Adi Rosenthal +1Video Diffusion Models Diffusion Models for Motion Generation
Canvas-Lab Department of Computer Science Reichman University
AdaState: Self-Evolving Anchors for Streaming Video Generation
May 28, 2026Yusuf Dalva, Pinar YanardagVideo Diffusion Models Streaming Video Generation
Virginia Tech

May 13, 2026J/K move · Enter open · S save

TeDiO: Temporal Diagonal Optimization for Training-Free Coherent Video Diffusion

Nurislam Tursynbek, Zhiqiang Lao, Heather Yu, Gedas Bertasius+1

UNC Chapel Hill1 · Futurewei Technologies Inc2 · UCSD3

Video Diffusion Models Diffusion Transformer Temporal Attention Diffusion Models for Motion Generation Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Recent text-to-video diffusion transformers generate visually compelling frames, yet still struggle with temporal coherence, often producing flickering, drifting, or unstable motion. We show that these failures leave a clear imprint inside the model: incoherent videos consistently exhibit irregular, fragmented temporal diagonals in their intermediate self-attention maps, whereas stable motion corresponds to smooth, band-diagonal patterns. Building on this observation, we introduce TeDiO, a training-free, inference-time method that reinforces temporal consistency by regularizing these internal attention patterns. TeDiO estimates diagonal smoothness, identifies unstable regions, and performs lightweight latent updates that promote coherent frame-to-frame dynamics, without modifying model weights or using external motion supervision. Across multiple video diffusion models (e.g., Wan2.1, CogVideoX), TeDiO delivers markedly smoother motion while preserving per-frame visual quality, offering an efficient plug-and-play approach to improving dynamic realism in modern video generation systems.

Abstract

Video Diffusion Models

Explore similar work

TeDiO: Temporal Diagonal Optimization for Training-Free Coherent Video Diffusion

Φ-Noise: Training-Free Temporal Video Conditioning via Phase-Based Noise Manipulation

AdaState: Self-Evolving Anchors for Streaming Video Generation