Organizations: Grad. School of Artificial Intelligence, POSTECH · School of Electrical Engineering, KAIST · Dept. of Electrical Engineering, POSTECH · School of Computing, KAIST
Text-to-video (T2V) diffusion models have rapidly advanced, yet generations still occasionally fail in practice, such as low text-video alignment or low perceptual quality. Since diffusion sampling is non-deterministic, it is difficult to know during inference whether a generation will succeed or fail, incurring high computational cost due to trial-and-error regeneration. To address this, we propose an early failure detection and diagnostic intervention pipeline for latent T2V diffusion models. For detection, we design a Real-time Inspection (RI) module that converts latents into intermediate video previews, enabling the use of established text-video alignment scorers for inspection in the RGB space. The RI module completes the conversion and inspection process in just 39.2ms. This is highly efficient considering that CogVideoX-5B requires 4.3s per denoising step when generating a 480p, 49-frame video on an NVIDIA A100 GPU. Subsequently, we trigger a hierarchical and early-exit intervention pipeline only when failure is predicted. Experiments on CogVideoX-5B and Wan2.1-1.3B demonstrate consistency gains on VBench with up to 2.64 times less time overhead compared to post-hoc regeneration. Our method also generalizes to a higher-capacity setting, remaining effective on Wan2.1-14B with 720p resolution and 81-frame generation. Furthermore, our pipeline is plug-and-play and orthogonal to existing techniques, showing seamless compatibility with prompt refinement and sampling guidance methods. We also provide evidence that failure signals emerge early in the denoising process and are detectable within intermediate video previews using standard vision-language evaluators.
Figures & tables
Figure 2: Overall early failure detection and intervention pipeline. The Real-time Inspection (RI) module converts predicted clean latents into RGB video previews and evaluates their semantic alignment with the input prompt (19.7 ms + 19.5 ms). Intervention is triggered based on the semantic alignment score.
Figure 3: Real-time Latent-to-RGB (L2R) converter. (a) Our L2R architecture maps latents to an RGB video using causal 3D convolutional blocks, spatio-temporal learned upsampling, and a projection layer. It runs in 19.7 ms with only 0.059 M parameters. (b) The reconstruction result from the L2R converter shows reasonable visual quality compared to the result produced by the original decoder.
Figure 4: Diagnostic intervention framework. If the current generation is predicted to succeed, Trial 0 continues to generate video without intervention. When failure is predicted, the next Trials are applied, escalating from inexpensive to more costly fixes. Trial 1 performs a single-frame probe and injects its latent; Trial 2 refines the prompt via a VLM and restarts generation. If Trial 2 also fails, Trial 1 is attempted once more.
CogVideoX-5B
Wan2.1-1.3B
VBench Score
Time
VBench Score
Time
Method
Final
Quality
Semantic
Overhead (%)
Final
Quality
Semantic
Overhead (%)
Original prompt
0.778
0.803
0.680
0.00
0.789
0.811
0.700
0.00
+ Regeneration
0.793
0.813
0.716
43.73
0.815
0.823
0.783
51.25
+ Ours
0.800
0.816
0.735
16.55
0.805
0.817
0.756
20.30
Table 1: Comparison with regeneration. We report the Final, Quality, and Semantic VBench scores along with average time overhead relative to base generation. Our method achieves higher performance on CogVideoX-5B with 2.64× less additional time overhead, and demonstrates a consistent gain over the base generation on Wan2.1-1.3B while requiring 2.52× less additional time overhead.
Figure 5: Efficiency of selective early intervention. (a) Overall time overhead relative to base generation. Ours incurs only +16.6% additional overhead, substantially lower than the regeneration of failure samples (+43.7%). (b) Per-sample cost increases with deeper trials, yet remains cheaper than regeneration. Since deeper trials are triggered for only a subset of samples (red curve), the overall time overhead stays modest.
Method
CogVideoX-5B
Wan2.1-1.3B
Final
Quality
Semantic
Final
Quality
Semantic
Original prompt (baseline)
0.778
0.803
0.680
0.789
0.811
0.700
+ Ours
0.800
0.816
0.735
0.805
0.817
0.756
Refined prompt
0.787
0.818
0.663
0.815
0.820
0.796
+ Ours
0.813
0.822
0.776
0.820
0.822
0.809
Original prompt + Enhance-A-Video
0.766
0.794
0.656
0.794
0.813
0.718
Table 2: Plug-and-play orthogonality of our method. Our method consistently improves VBench scores across both CogVideoX and Wan2.1 when applied on top of VLM-refined prompts and training-free sampling enhancements (Enhance-A-Video [ 24 ] and STG [ 25 ] ). These gains come from early detection and intervention.
Figure 6: Qualitative results. Compared to baseline models, our method improves prompt adherence and visual fidelity by correcting early-failure generations.
Model
Setting
Visual Quality
Text-Video Alignment
Physical Consistency
CogVideoX-5B
Original Prompt (baseline)
3.86
3.82
3.66
Original Prompt + Ours
3.93 (+0.07)
3.96 (+0.14)
3.74 (+0.08)
Predicted Failure (wo. Intervention)
3.77
3.55
3.56
Predicted Failure + Ours
3.94 (+0.17)
3.91 (+0.35)
3.74 (+0.18)
Predicted Success (wo. Intervention)
3.92
4.00
3.74
Predicted Failure (wo. Intervention)
3.77
3.55
3.56
Table 3: Additional evaluation with VideoScore2. We report VideoScore2 scores on CogVideoX-5B and Wan2.1-1.3B under three comparisons. First, Original Prompt (baseline) and Original Prompt + Ours compare the overall performance before and after applying our method on the full evaluation set. Second, Predicted Failure (wo. Intervention) and Predicted Failure + Ours compare the initially detected failure subset before and after selective intervention. Third, Predicted Success (wo. Intervention) and Predicted Failure (wo. Intervention) compare the initial generations of samples predicted as success or failure by the dynamic failure detector, respectively.
Figure 7: Qualitative results on Wan2.1-14B at 720p and 81 frames. While the main experiments use 480p, 49-frame generation with CogVideoX-5B and Wan2.1-1.3B, here we show that our method also applies to a larger-model and more demanding setting. In both examples, the baseline Wan2.1-14B generations fail to match the prompts, whereas our intervention produces outputs that are substantially better aligned with the intended human actions.
Figure 8: Case visualization of diagnostic interventions. Top: although the early video preview poorly reflects the prompt, the single-frame preview captures the semantics. Our method applies single-frame semantic injection (Trial-1). Bottom: both of the video and the single-frame previews fail to satisfy the intended spatial relationship (TV on the left), indicating a persistent prompt misalignment. The intervention framework automatically invokes VLM-based observation-driven prompt refinement (Trial-2).
Figure 9: Sensitivity of video quality metrics to temporal subsampling and spatial downsampling. Compared to temporal subsampling, spatial downsampling causes larger deviations in VBench scores from the original videos.
Figure 10: Early predictability of final video quality. For CogVideoX and Wan2.1 (1,800 prompts), we compute VBench metrics on intermediate L2R previews and report the normalized MAE to the final metrics of the final videos using the original VAE. Errors drop sharply by ∼ 10 steps, notably for Overall Consistency.
Figure 11: Overall Consistency as a proxy for other VBench metrics. Using all VBench prompts, we measure PR-AUC for predicting whether a sample falls into the top- q or bottom- q group of each target metric ( q∈{0.1,0.2,0.3} ) using Overall Consistency. All scores exceed the random baseline (0.5), indicating that Overall Consistency captures quality variations reflected by other metrics.
Figure 12: Selecting the failure threshold τ for Overall Consistency. Left: Distributions of Overall Consistency for generated and real videos. Real videos concentrate at higher scores; we set τ=0.22 so that the retained generated samples better match the real-video regime. Right: A human study with 26 participants over 30 samples (780 responses) validates the perceptual boundary: scores of 0.18 are often judged as failures, whereas scores ≥ 0.22 are considered reasonable.
Figure 13: Intervention improves detected failure cases. CDFs of VBench metrics for samples predicted as failures, but forced generated, and our intervention. Despite using Overall Consistency for detection, ours shifts multiple Vbench metrics rightward.
Figure 14: Accuracy of early failure detection. Final Overall Consistency distributions for samples predicted to pass or fail (failures forced to complete without intervention). Strong separation across models demonstrates reliable early discrimination.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Architecture of Latent-to-RGB (L2R) converter. (a) Overall architecture. The converter takes a latent video tensor as input and produces an RGB video preview through two causal 3D convolution blocks, a spatio-temporal learned upsampling, and a projection layer. (b) Causal 3D convolution block with a main branch and a skip branch. (c) Illustration of spatio-temporal learned upsampling for a single red target location. Given the previous and current latents, learned parameters are used to predict the upsampled latent at the current step by weighted summation. In practice, this operation is implemented densely as a 3D convolution over the latent.
CogVideoX-5B
Wan2.1-1.3B
DAVIS
Pexels
DAVIS
Pexels
Decoder
Param. (M)
PSNR ↑
PSNR ↑
Time (s)
Mem (MB)
Param. (M)
PSNR ↑
PSNR ↑
Time (s)
Mem (MB)
Original
123.4
32.80
35.30
3.990
242.5
73.30
32.68
35.41
1.702
140.4
TinyAE
11.32
26.56
27.35
0.116
21.60
11.32
30.50
33.46
0.116
21.60
Turbo-VAED
40.69
32.03
35.03
0.599
114.6
-
-
-
-
-
Ours
0.059
30.00
32.50
0.0197
0.138
0.059
28.98
31.40
0.0197
0.138
Appendix
Table S1: Efficiency and quality of latent-to-RGB converter. We measure PSNR on DAVIS and Pexels. Then, we measure the decoding time/memory at 480 × 720. L2R offers moderate PSNR with the lowest latency and memory, providing previews suitable for inspection for both models.
Model
Method
Avg. Step
Normalized Error ( ×10−2↓ )
CogVideoX-5B
Fixed
11
6.67±0.45
Dynamic
11.05
5.73±0.41
Wan2.1-1.3B
Fixed
10
5.24±0.34
Dynamic
10.33
4.70±0.30
Appendix
Table S2: Overall Consistency between fixed-step vs. dynamic failure detector. Despite nearly identical average prediction steps, the dynamic strategy achieves lower normalized error across both models.
Figure S2: Semantic convergence varies across samples in video diffusion. We visualize the center frame (24th frame out of 49) decoded from intermediate denoising steps for two text-to-video generations. While the dog sample exhibits recognizable semantics at an early stage, the shark sample remains ambiguous until later steps. This illustrates that semantic convergence in video diffusion is highly sample-dependent, motivating the dynamic failure detector rather than relying on a fixed early step.
Figure S3: Efficiency of selective early intervention on Wan2.1-1.3B. (a) Overall time overhead relative to base generation. Ours incurs only +20.3% additional time overhead, substantially lower than the regeneration of failure samples (+51.25%). (b) Per-sample cost increases with deeper trials, yet remains cheaper than regeneration. Since deeper trials are triggered for only a subset of samples (red curve), the overall time overhead stays modest.
Dynamic failure detector
Execution time (s)
average step
Video
Video
Single-frame
Single-frame
VLM
Video
Single-frame
Models
denoising steps
preview inspections
denoising steps
preview inspections
calls
preview
preview
CogVideoX-5B
4.343
0.0392
0.405
0.0218
8.541
11.05
24.00
Wan2.1-1.3B
1.460
0.0392
0.103
0.0218
8.541
10.33
24.00
Appendix
Table S3: Measured execution time of each primitive operation. We report the average runtime of each primitive operation used in our framework, including video denoising, video preview inspection, single-frame denoising, single-frame preview inspection, and VLM calls. We also report the average detection step of the dynamic failure detector for video previews and single-frame previews. These measured runtimes are used together with the branch-wise execution counts in Table S4 to compute the per-sample and overall time overhead.
CogVideoX-5B: original generation time: 217.15 s (50 steps)
Execution statistics
Runtime
Time overhead
Branch
Samples / Total
Video denoising steps
Video preview inspections
Single-frame denoising steps
Single-frame preview inspections
VLM calls
Average time (s)
Additional time (s)
Per-sample time overhead (%)
Overall time overhead (%)
Trial 0
562 / 946
50
11.05
0
0
0
217.58
0.43
0.20
0.12
Trial 1
91 / 946
60.05
11.05
24
24
0
271.47
54.32
25.02
2.41
Trial 2
125 / 946
61.05
11.05
24
24
1
284.36
67.21
30.95
4.09
Trial 2 → Trial 1
168 / 946
71.10
22.10
48
48
1
338.68
121.53
55.97
9.94
Appendix
Table S4: Detailed breakdown of execution statistics and time overhead. We report the number of denoising steps, inspection operations, and wall-clock runtime for each execution branch. Per-sample time overhead measures the additional runtime relative to the original generation time when the branch is executed, while Overall time overhead represents the dataset-averaged contribution.
CogVideoX-5B
Wan2.1-1.3B
Method
Final
Quality
Semantic
Overhead (%)
Final
Quality
Semantic
Overhead (%)
Original prompt (baseline)
0.778
0.803
0.680
0.00
0.789
0.811
0.700
0.00
Best-of-3
0.787
0.802
0.728
200.00
0.796
0.812
0.730
200.00
Original prompt + Ours
0.800
0.816
0.735
16.55
0.805
0.817
0.756
20.30
Appendix
Table S5: Comparison with a simple Best-of-3 baseline. For Best-of-3 , we generate three videos per prompt from the original prompt using different random seeds, score them with our alignment scorer (ViCLIP), and select the sample with the highest semantic alignment score. Since this requires three full generations, it incurs 200% additional time overhead relative to single-sample generation.
CogVideoX-5B
Wan2.1-1.3B
Method
Final
Quality
Semantic
Overhead (%)
Final
Quality
Semantic
Overhead (%)
Original prompt (baseline)
0.778
0.803
0.680
0.00
0.789
0.811
0.700
0.00
Original prompt + Ours ( τ=0.22 )
0.800
0.816
0.735
16.55
0.805
0.817
0.756
20.30
Original prompt + Ours ( τ=0.26 )
0.807
0.824
0.740
32.61
0.807
0.823
0.743
35.96
Original prompt + Ours ( τ=0.30 )
0.806
0.827
0.725
39.47
0.812
0.826
0.756
42.77
Appendix
Table S6: Effect of the success and failure threshold. We compare the original prompt baseline with our selective early intervention under different failure thresholds τ . Higher thresholds make the detector more conservative, increasing the number of samples that receive intervention. As a result, VBench scores may improve further, but at the cost of substantially higher time overhead.
Figure S4: Effect of the single-frame-based intervention family on detected failure cases. We compare the CDFs of VBench metrics between predicted failure samples generated to completion without intervention and those corrected by the single-frame-based intervention family (Trial 1 and Trial 2 → 1). Across both CogVideoX-5B and Wan2.1-1.3B, the intervention shifts the distributions toward better scores over multiple VBench dimensions.
Figure S5: Effect of observation-driven prompt refinement on detected failure cases. We compare the CDFs of VBench metrics between predicted failure samples generated to completion without intervention and those corrected by observation-driven prompt refinement (Trial 2). Across both CogVideoX-5B and Wan2.1-1.3B, this branch also shifts the distributions toward better scores over multiple VBench dimensions.
Figure S6: Additional qualitative results of Trial 1 (single-frame semantic injection). We present additional qualitative results from CogVideoX-5B (top) and Wan2.1-1.3B (bottom). For generations predicted as failures from early video previews, we further examine a single-frame preview. If the single-frame preview is better than the video preview, we apply semantic injection. These examples show that Trial 1 improves prompt fidelity by recovering missing objects, attributes, and subject composition in the final outputs across different models and failure cases.
Figure S7: Additional qualitative results of Trial 2 (observation-driven prompt refinement). We present additional qualitative results from CogVideoX-5B (top) and Wan2.1-1.3B (bottom). For generations predicted as failures based on early video previews, Trial 2 uses a vision–language model (VLM) to refine the prompt according to the observed failure patterns. The refined prompt strengthens the desired semantics, while the negative prompt suppresses undesired content or artifacts. These examples show that Trial 2 improves prompt fidelity by correcting subject identity, object count, composition, and visual attributes across different models and failure cases.
Figure S8: Qualitative comparison between the native CogVideoX decoder and L2R converter. We compare the reconstruction results between two methods. Across diverse scenes and multiple frames, L2R produces lightweight video previews that preserve the key semantic content and visual structure of the native CogVideoX decoder. These previews are sufficiently informative for real-time inspection and failure detection.
Figure S9: Qualitative comparison between the native Wan2.1 decoder and L2R converter. We compare the reconstruction results between two methods. Across diverse scenes and multiple frames, L2R produces lightweight video previews that preserve the key semantic content and visual structure of the native Wan2.1 decoder. These previews are sufficiently informative for real-time inspection and failure detection.
Dimension
CogVideoX-5B
Wan2.1-1.3B
Original Prompt
+ Ours
Original Prompt
+ Ours
Subject Consistency
0.957
0.967
0.949
0.964
Background Consistency
0.953
0.964
0.959
0.972
Temporal Flickering
0.969
0.973
0.991
0.992
Motion Smoothness
0.981
0.985
0.970
0.979
Dynamic Degree
0.542
0.542
0.542
0.403
Appendix
Table S7: All VBench results across all dimensions. We report all individual VBench dimension scores, along with the aggregated Semantic, Quality, and Final scores, for the original prompt baseline and our method on CogVideoX-5B and Wan2.1-1.3B. Overall, our method improves most VBench dimensions on both models, indicating that selective early intervention benefits not only alignment-related metrics but also broader perceptual and temporal aspects of video quality.
Figure S10: Prompt template for observation-driven prompt refinement. The VLM receives the original prompt, preview video, and the predicted preview score, and returns a refined prompt with an optional negative prompt in JSON format.
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.
We present the first systematic study of the resilience of text-to-video (T2V) diffusion models under random hardware-level faults. While T2V models are widely used for automated video generation due to their ability to produce high-quality, temporally coherent, and realistic videos, their iterative denoising process and spatiotemporal dependencies introduce unique failure modes. We perform an extensive fault-injection study covering both computational and memory faults across three T2V models and a representative benchmark. Our results show that (1) a single fault can degrade overall performance by up to 3.7%, with semantic correctness more affected than perceptual quality; (2) memory faults are more damaging than computational faults, high-order exponent bits are particularly vulnerable, and the widely-used bfloat16 is more susceptible than alternative formats; and (3) 7-28% of faults cause visible artifacts, including semantic changes such as added objects, suggesting that single faults are sufficient to alter output semantics. Our findings reveal reliability risks in deployed T2V systems and motivate further research on improving fault resilience. Code: https://github.com/ztcoalson/T2V-Resilience.
Text-to-video diffusion models generate temporally coherent content from natural language, yet when a prompt describes an early scene that persists while a new event emerges on top of it---such as "a tall sandcastle standing on a beach where a wave rushes in and washes it away"---generation frequently fails to realize the late-segment event in the corresponding frames. We identify this failure as Temporal Prior Suppression (TPS): the dominant prior of the early segment captures the cross-attention trajectory across the temporal axis and suppresses the guidance signal needed for late-segment realization, a competing tendency existing guidance mechanisms do not model. We introduce Temporal Prior Decoupling (TPD), a training-free framework that restores suppressed late-segment signals during diffusion sampling. TPD constructs a temporal counterfactual by conditioning on the early segment alone, and defines the discrepancy between the full-prompt and counterfactual trajectories as a suppressed signal direction. Rather than removing this direction as in prior subtractive projection methods, TPD restores it through a frame-selective lower-bound constraint resolved jointly over diffusion timestep and video frame, realizing the suppressed event in the late frames without disrupting early-segment coherence: where prior work enforces upper-bound feasibility to remove unwanted semantics, TPD enforces lower-bound feasibility to guarantee suppressed-signal contribution. TPD runs entirely within standard diffusion sampling without retraining, and is defined purely in classifier-free guidance space, making it backbone-agnostic by construction. Experiments show that TPD significantly improves late-concept realization while preserving temporal coherence and visual fidelity, and that the targeted suppression recurs across distinct text-to-video backbones.
Taewon Kang, Matthias Zwicker
University of Maryland at College Park, United States
Text-to-video diffusion models deliver impressive results but remain slow because of the sequential denoising of 3D latents. Existing approaches to speed up inference either require expensive model retraining or use heuristic-based step skipping, which struggles to maintain video quality as the number of denoising steps decreases. Our work, CHAI, aims to use cross-inference caching to reduce latency while maintaining video quality. We introduce Cache Attention as an effective method for attending to shared objects/scenes across cross-inference latents. This selective attention mechanism enables effective reuse of cached latents across semantically related prompts, yielding high cache hit rates. We show that it is possible to generate high-quality videos using Cache Attention with as few as 8 denoising steps. When integrated into the overall system, CHAI is 1.65x - 3.35x faster than baseline OpenSora 1.2 while maintaining video quality.
Joel Mathew Cherian, Ashutosh Muralidhara Bharadwaj, Vima Gupta +1