Despite the substantial progress in long video generation, multi-prompt synthesis often encounters inconsistencies resulting from the training-inference gap caused by length extension techniques and coarse prompt interpolation. To overcome these issues, we propose the Video Diffusion with hidden states Editing (ViDE) framework, which consists of two key components. The first is the Time-frequency based Temporal Attention Reweighting (TiTAR) algorithm, which leverages the relationship between inconsistencies and diagonal elements of temporal attention. By reweighting the attention scores via the Discrete Short-Time Fourier Transform (DSTFT), TiTAR effectively reduces frame inconsistencies, a capability further corroborated by a Fourier-based analysis. The second component, PromptBlend, reduces inconsistencies in multi-prompt settings through fine-grained prompt alignment and adaptive interpolation, enabling smooth semantic transitions. Extensive experiments demonstrate the effectiveness of ViDE, demonstrating consistent and significant improvements over multiple baselines.
Figures & tables
Figure 1: Qualitative comparisons between ViDE and existing methods on long video generation . Our ViDE framework comprises two complementary modules, TiTAR and PromptBlend , which refine video and prompt hidden states, respectively, to enhance generation quality and consistency. Applying TiTAR to single-prompt generation (top left and top right figures) markedly improves object and background consistency, while employing ViDE, i.e. , TiTAR and PromptBlend , for multi-prompt generation (bottom figure) further enhances overall generation quality and coherence.
Figure 2: The ViDE framework comprises two complementary modules: TiTAR and PromptBlend . Left: To mitigate inconsistencies associated with excessive focus on individual frames during temporal self-attention, TiTAR introduces a reweighting matrix upon the attention map at each temporal step within the denoising U-Net. The diagonal elements of this matrix are dynamically determined via frequency-based motion intensity analysis of the generated video. Right: For multi-prompt video generation, PromptBlend improves prompt interpolation by first aligning prompts in the token space and interpolating their embeddings, then gradually blending the interpolated text conditioning into the inference process.
Figure 3: Analysis of motion and temporal attention maps. We analyze two segments within a 310-frame video as an example to illustrate the phenomenon. Up: The time-frequency map, where the left outlined segment shows consistent motion, while the right exhibits abnormal high-frequency components associated with poor generation. Middle: Averaged temporal attention maps, showing significantly stronger diagonal attention for poorly generated frames. Left : Sampled frames with normal motion. Right : Consecutive frames where inconsistencies occur, including the pop-up of a hot-air ballon and sudden transform of the horizon to clouds and sky. More examples can be found in Appendix J .
Figure 4: Qualitative comparisons of different reweighting schemes. (a) The original FIFO shows flag color inconsistencies, (b) The augmented versions with temporal attention reweighting only has blurred flag edges, while (c) The motion intensity adjusted reweighting— TiTAR maintains clarity and consistency.
Figure 5
SC ( ↑ )
BC ( ↑ )
TF ( ↑ )
MS ( ↑ )
CTS ( ↑ )
WE ( ↓ )
IQ ( ↑ )
CS ( ↑ )
FIFO-VC
92.38
94.55
94.24
96.48
99.72
0.026
60.09
19.94
FIFO-VC + TiTAR
94.13
95.16
96.85
98.10
99.87
0.010
60.11
19.95
ST2V
91.08
95.57
97.33
98.13
98.67
0.011
51.49
18.85
ST2V + TiTAR
92.95
96.41
98.09
98.79
98.82
0.006
52.62
18.83
FreeNoise
95.48
97.42
96.50
97.46
99.87
0.014
64.64
20.06
FreeNoise + TiTAR
98.35
98.64
97.35
97.90
99.91
0.008
65.92
20.07
Table 1: Quantitative results on single-prompt generation in FIFO-Videocrafter2 (FIFO-VC), StreamingT2V (ST2V), FreeNoise, FIFO-Open-Sora Plan (FIFO-OSP), and CogVideoX. We only adopt length extension part of existing methods and the TiTAR component of vide in the single-prompt generation.
Two-prompt Video Generation
SC ( ↑ )
BC ( ↑ )
TF ( ↑ )
MS ( ↑ )
CTS ( ↑ )
WE ( ↓ )
IQ ( ↑ )
CS ( ↑ )
FIFO-VC (MP)
85.25
90.66
95.83
97.63
99.78
0.012
54.99
21.61
FIFO-VC + vide
88.87
92.43
97.68
98.66
99.89
0.004
54.79
22.01
FreeNoise (MP)
89.44
94.48
97.29
98.00
99.89
0.006
64.87
22.37
FreeNoise + vide
91.01
95.16
98.25
98.61
99.95
0.002
66.28
22.24
Three-prompt Video Generation
Table 2: Quantitative results on multi-prompt generation in FIFO-VC and FreeNoise. FIFO-VC (MP) and FreeNoise (MP) refer to the multi-prompt versions from kim2024fifo and qiu2023freenoise , respectively. Specifically, they adopt d/swi and MI/interp to transition between prompts.
Figure 7: Ablation study on the reweighting scale α . The x-axis represents the value of the reweighting scale α while the y-axes represent the values of consistency metrics (left) and imaging quality (right).
Align
Interp
TiTAR
SC ( ↑ )
BC ( ↑ )
MS ( ↑ )
d/swi
85.25
90.66
97.63
✓
d/swi
85.74
91.17
97.71
d/interp
85.54
90.80
97.67
adapt/interp
86.23
91.00
97.71
✓
d/interp
85.98
91.27
97.71
✓
adapt/interp
86.52
91.36
97.73
Table 3: Ablation study on PromptBlend and TiTAR for multi-prompt generation with FIFO-VC. We decompose vide into three components: TiTAR and PromptBlend , the latter consisting of prompt alignment and the adaptive application of embeddings. In the second column, we compare three options: d/swi, d/interp, and adapt/interp (Ours). Here, d/interp naively interpolates the text embeddings of the two prompts across all layers and time steps.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: ChatGPT Instruction.
Figure 9: Multi-prompt test set.
Figure 10: Ablation study on the effect of prompt alignment on interpolation. The experiment is conducted using FIFO and VideoCrafter2. The first row of each example is the result with interpolation but without prompt alignment; the second row is the result with both interpolation and alignment.
Method
SC ↑
BC ↑
TF ↑
MS ↑
CTS ↑
WE ↓
IQ ↑
CS ↑
FIFO-VC
91.17
93.26
94.60
96.55
99.77
0.017
54.61
20.71
FIFO-VC + TiTAR
92.64
94.26
96.75
98.06
99.90
0.007
54.27
21.27
FreeNoise
96.04
96.71
96.66
97.41
99.89
0.012
59.33
21.08
FreeNoise + TiTAR
98.98
98.59
97.47
97.88
99.94
0.006
61.24
21.11
Appendix
Table 4: Quantitative results for single-prompt long-video generation using the VBench prompts. The baseline methods are FIFO-Videocrafter2 (FIFO-VC) and FreeNoise.
Method
SC ↑
BC ↑
TF ↑
MS ↑
CTS ↑
WE ↓
IQ ↑
CS ↑
FreeLong
92.75
95.54
96.79
97.71
99.89
0.013
52.91
20.85
FreeNoise+ TiTAR
98.98
98.59
97.47
97.88
99.94
0.006
61.24
21.11
Appendix
Table 5: Quantitative comparison of FreeLong and FreeNoise+ TiTAR on single-prompt long video generation using the VBench prompts.
Method
SC ↑
BC ↑
TF ↑
MS ↑
CTS ↑
WE ↓
IQ ↑
CS ↑
FIFO-VC
79.30
89.40
94.02
96.47
99.65
0.019
52.77
18.93
FIFO-VC + vide
82.42
90.32
96.10
98.10
99.78
0.007
52.14
19.03
Appendix
Table 6: Quantitative results on multi-prompt long video generation using DiTCtrl prompt set. We compare FIFO-VideoCrafter2 (FIFO) with and without vide .
Figure 11: Qualitative comparison on FIFO based on VideoCrafter. The first row of each example is the result with original FIFO; the second row is the result with FIFO augmented with TiTAR . The displayed frames are sampled at fixed intervals. The inconsistent parts in the videos are marked with red boxes.
Figure 12: Qualitative comparison on FreeNoise. The first row of each example is the result with original FreeNoise; the second row is the result with FreeNoise augmented with TiTAR . The displayed frames are sampled at fixed intervals. The inconsistent part in the videos are marked with red boxes.
Figure 13: Qualitative comparison on StreamingT2V (ST2V). The first row of each example is the result with original ST2V; the second row is the result with ST2V augmented with TiTAR . The displayed frames are sampled at fixed intervals. The inconsistent part in the videos are marked with red boxes.
SC ( ↑ )
BC ( ↑ )
TF ( ↑ )
MS ( ↑ )
CTS ( ↑ )
WE ( ↓ )
No reweighting
92.38
94.55
94.24
96.48
99.72
0.026
Corner reweighting
92.89
94.80
94.74
96.66
99.74
0.024
Diag reweighting
93.57
95.04
96.46
97.99
99.85
0.011
TiTAR
94.13
95.16
96.85
98.10
99.87
0.010
Appendix
Table 7: Ablation study on reweighting mask shape for single-prompt generation with FIFO-VC. “Corner” and “Diag” reweighting assign non - zero entries to Λ in ( 2 ) only at the corner and diagonal positions, respectively. In contrast, TiTAR sets Λ with values in both regions. “No reweighting” refers to the baseline without TiTAR .
SC ( ↑ )
BC ( ↑ )
TF ( ↑ )
MS ( ↑ )
IQ ( ↑ )
CS ( ↑ )
Zero Padding
84.91
91.49
95.36
97.45
57.92
21.86
Repeat Padding
86.68
92.30
95.66
97.62
58.06
21.88
Appendix
Table 8: Ablation study on padding scheme for multi-prompt interpolation with FIFO-VC. Zero padding extends the corresponding parts of the prompts to the same length using spaces, while repeat padding extends them by repeating the prompts.
SC ( ↑ )
BC ( ↑ )
TF ( ↑ )
MS ( ↑ )
IQ ( ↑ )
Original
92.16
94.88
94.24
96.47
60.05
TiTAR (LF-30-HF-90)
93.88
95.57
96.60
97.97
60.33
TiTAR (LF-50-HF-90)
94.07
95.59
96.76
98.06
59.88
TiTAR (LF-70-HF-90)
94.09
95.55
96.82
98.10
59.84
Appendix
Table 9: Ablation study on frequency threshold of TiTAR with FIFO-VC. “LF- a -HF- b ” means that we set thresholds for low and high frequencies corresponding to the top a% and b% of the highest frequencies, respectively. “Original” refers to the baseline without TiTAR .
Method
Runtime
Peak Host Memory
FIFO
28:44.46 (1724.46 s)
9.821 GiB
FIFO + TiTAR
29:14.66 (1754.66 s)
9.829 GiB
Absolute overhead
+30.20 s
+0.008 GiB
Relative overhead
+1.75%
+0.08%
Appendix
Table 10: Runtime and memory overhead of vide compared with the FIFO baseline. Both methods are evaluated under the same inference setting.
Figure 14: User preference studies . User preference ratio between our methods and baselines on Single- (S) and Multi- (M) prompt generation in FIFO, ST2V and FreeNoise.
Figure 15: Linear regression of WE and OFS.
Figure 16: Qualitative comparison on FIFO based on Open-Sora Plan. The first row of each example is the result with original FIFO; the second row is the result with FIFO augmented with TiTAR . The displayed frames are sampled at fixed intervals. The inconsistent part in the videos are marked with red boxes.
Figure 17: Qualitative comparison on multi-prompt video generation. The first two examples are generated with two prompts, and the third example is generated with three prompts. The first row of each example is the result with original multi-prompt FIFO; the second row is the result with FIFO + TiTAR + PromptBlend . The displayed frames are sampled at fixed intervals. The inconsistent part in the videos are marked with red boxes.
Figure 18: More examples on the frequency analysis and attention map.
Recent advances in text-to-video diffusion models have enabled high-fidelity and temporally coherent videos synthesis. However, current models are predominantly optimized for single-event generation. When handling multi-event prompts, without explicit temporal grounding, such models often produce blended or collapsed scenes that break the intended narrative. To address this limitation, we present SwitchCraft, a training-free framework for multi-event video generation. Our key insight is that uniform prompt injection across time ignores the correspondence between events and frames. To this end, we introduce Event-Aligned Query Steering (EAQS), which steers frame-level attention to align with relevant event prompts. Furthermore, we propose Auto-Balance Strength Solver (ABSS), which adaptively balances steering strength to preserve temporal consistency and visual fidelity. Extensive experiments demonstrate that SwitchCraft substantially improves prompt alignment, event clarity, and scene consistency compared with existing baselines, offering a simple yet effective solution for multi-event video generation.
Qianxun Xu, Chenxi Song, Yujun Cai +1
Westlake University · Duke Kunshan University · The University of Queensland
Streaming long-video generation faces a central challenge in continuous semantic switching, requiring adaptive memory to preserve coherent visual evolution. Current approaches rely on cache rebuilding at prompt boundaries or fixed memory budgets, but they introduce redundant computation and limit flexible semantic adaptation. This limitation arises from a mismatch between cached video history and prompt updates, as memory preserves visual continuity while prompt switches demand rapid semantic adaptation. Motivated by this observation, we present SWIFT, Semantic Windowing and Injection for Flexible Transitions, a training-free framework for multi-prompt long-video generation that enables efficient semantic switching while preserving temporal coherence in causal video diffusion models. SWIFT introduces a lightweight Semantic Injection Cache that augments cached video memory rather than reconstructing it from scratch at every prompt boundary. To avoid uniformly perturbing all attention channels, we further perform head-wise semantic injection, so that each attention head receives a prompt update proportional to its alignment with the current video state. In addition, we introduce an Adaptive Dynamic Window that allocates temporal memory according to prompt phase, using larger local context near switching boundaries and smaller windows during stable segments to reduce average inference cost. To preserve long-range semantic consistency under compressed local attention, we further maintain segment-level semantic anchors that summarize prompt-conditioned video history and reintroduce it as compact memory tokens. Compared with current state-of-the-art methods, SWIFT preserves generation quality while achieving 22.6 FPS on a single H100 GPU, establishing a substantially more efficient solution for multi-prompt long-video generation. Our code is available at https://github.com/ShanwenTan/SWIFT.
Shanwen Tan, Hao Li, Jingtao Zhang +4
University of Science and Technology of China · Fudan University · Georgia Institute of Technology +1
Recently, diffusion models have made great progress in video generation. However, most existing video diffusion models are trained with short videos, and degrade when extrapolated to long videos, struggling to maintain long-range temporal coherence while retaining diverse motions. To generate consistent, high-quality and dynamic long videos, we propose Diff-VF, a training-free, plug-and-play and model-agnostic framework that converts existing short-video diffusion backbones into long-video generators without modifying or fine-tuning the base model. Diff-VF couples three complementary strategies: Hybrid Noise Initialization (HNI) to constrain global semantics, Weighted Window Sampling (WWS) to remove inter-window discontinuities, and Temporal Extended Sampling (TES) to establish long-range dependencies with a timestep-varying fusion. We further extend Diff-VF to long-video enhancement via Skip Residual Guidance that balances fidelity and realism through timestep-dependent guidance. VBench-Long evaluation results show that Diff-VF achieves a more favorable balance between temporal coherence and motion diversity than base models and recent training-free long video generation baselines, including FreeNoise, FreeLong, and RIFLEx, while maintaining competitive frame-wise quality. Experiments on two base models demonstrate the applicability to video diffusion models with different spatial-temporal modeling strategies. Extensive ablations validate the contribution of each component and hyperparameters.
Haoning Yang, Xinyuan Chen, Yaohui Wang +1
Shanghai Jiao Tong University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China