cs.CVMar 15, 2026
SaveEarly Failure Detection and Intervention in Video Diffusion Models
Organizations: Grad. School of Artificial Intelligence, POSTECH · School of Electrical Engineering, KAIST · Dept. of Electrical Engineering, POSTECH · School of Computing, KAIST
Abstract
Text-to-video (T2V) diffusion models have rapidly advanced, yet generations still occasionally fail in practice, such as low text-video alignment or low perceptual quality. Since diffusion sampling is non-deterministic, it is difficult to know during inference whether a generation will succeed or fail, incurring high computational cost due to trial-and-error regeneration. To address this, we propose an early failure detection and diagnostic intervention pipeline for latent T2V diffusion models. For detection, we design a Real-time Inspection (RI) module that converts latents into intermediate video previews, enabling the use of established text-video alignment scorers for inspection in the RGB space. The RI module completes the conversion and inspection process in just 39.2ms. This is highly efficient considering that CogVideoX-5B requires 4.3s per denoising step when generating a 480p, 49-frame video on an NVIDIA A100 GPU. Subsequently, we trigger a hierarchical and early-exit intervention pipeline only when failure is predicted. Experiments on CogVideoX-5B and Wan2.1-1.3B demonstrate consistency gains on VBench with up to 2.64 times less time overhead compared to post-hoc regeneration. Our method also generalizes to a higher-capacity setting, remaining effective on Wan2.1-14B with 720p resolution and 81-frame generation. Furthermore, our pipeline is plug-and-play and orthogonal to existing techniques, showing seamless compatibility with prompt refinement and sampling guidance methods. We also provide evidence that failure signals emerge early in the denoising process and are detectable within intermediate video previews using standard vision-language evaluators.
Figures & tables
Figure 2: Overall early failure detection and intervention pipeline. The Real-time Inspection (RI) module converts predicted clean latents into RGB video previews and evaluates their semantic alignment with the input prompt (19.7 ms + 19.5 ms). Intervention is triggered based on the semantic alignment score.
Figure 3: Real-time Latent-to-RGB (L2R) converter. (a) Our L2R architecture maps latents to an RGB video using causal 3D convolutional blocks, spatio-temporal learned upsampling, and a projection layer. It runs in 19.7 ms with only 0.059 M parameters. (b) The reconstruction result from the L2R converter shows reasonable visual quality compared to the result produced by the original decoder.
Figure 4: Diagnostic intervention framework. If the current generation is predicted to succeed, Trial 0 continues to generate video without intervention. When failure is predicted, the next Trials are applied, escalating from inexpensive to more costly fixes. Trial 1 performs a single-frame probe and injects its latent; Trial 2 refines the prompt via a VLM and restarts generation. If Trial 2 also fails, Trial 1 is attempted once more.
| CogVideoX-5B | Wan2.1-1.3B | |||||||
| VBench Score | Time | VBench Score | Time | |||||
| Method | Final | Quality | Semantic | Overhead (%) | Final | Quality | Semantic | Overhead (%) |
| Original prompt | 0.778 | 0.803 | 0.680 | 0.00 | 0.789 | 0.811 | 0.700 | 0.00 |
| + Regeneration | 0.793 | 0.813 | 0.716 | 43.73 | 0.815 | 0.823 | 0.783 | 51.25 |
| + Ours | 0.800 | 0.816 | 0.735 | 16.55 | 0.805 | 0.817 | 0.756 | 20.30 |
Table 1: Comparison with regeneration. We report the Final, Quality, and Semantic VBench scores along with average time overhead relative to base generation. Our method achieves higher performance on CogVideoX-5B with less additional time overhead, and demonstrates a consistent gain over the base generation on Wan2.1-1.3B while requiring less additional time overhead.
Figure 5: Efficiency of selective early intervention. (a) Overall time overhead relative to base generation. Ours incurs only +16.6% additional overhead, substantially lower than the regeneration of failure samples (+43.7%). (b) Per-sample cost increases with deeper trials, yet remains cheaper than regeneration. Since deeper trials are triggered for only a subset of samples (red curve), the overall time overhead stays modest.
| Method | CogVideoX-5B | Wan2.1-1.3B | ||||
| Final | Quality | Semantic | Final | Quality | Semantic | |
| Original prompt (baseline) | 0.778 | 0.803 | 0.680 | 0.789 | 0.811 | 0.700 |
| + Ours | 0.800 | 0.816 | 0.735 | 0.805 | 0.817 | 0.756 |
| Refined prompt | 0.787 | 0.818 | 0.663 | 0.815 | 0.820 | 0.796 |
| + Ours | 0.813 | 0.822 | 0.776 | 0.820 | 0.822 | 0.809 |
| Original prompt + Enhance-A-Video | 0.766 | 0.794 | 0.656 | 0.794 | 0.813 | 0.718 |
Table 2: Plug-and-play orthogonality of our method. Our method consistently improves VBench scores across both CogVideoX and Wan2.1 when applied on top of VLM-refined prompts and training-free sampling enhancements (Enhance-A-Video [ 24 ] and STG [ 25 ] ). These gains come from early detection and intervention.
Figure 6: Qualitative results. Compared to baseline models, our method improves prompt adherence and visual fidelity by correcting early-failure generations.
| Model | Setting | Visual Quality | Text-Video Alignment | Physical Consistency |
| CogVideoX-5B | Original Prompt (baseline) | 3.86 | 3.82 | 3.66 |
| Original Prompt + Ours | 3.93 (+0.07) | 3.96 (+0.14) | 3.74 (+0.08) | |
| Predicted Failure (wo. Intervention) | 3.77 | 3.55 | 3.56 | |
| Predicted Failure + Ours | 3.94 (+0.17) | 3.91 (+0.35) | 3.74 (+0.18) | |
| Predicted Success (wo. Intervention) | 3.92 | 4.00 | 3.74 | |
| Predicted Failure (wo. Intervention) | 3.77 | 3.55 | 3.56 |
Table 3: Additional evaluation with VideoScore2. We report VideoScore2 scores on CogVideoX-5B and Wan2.1-1.3B under three comparisons. First, Original Prompt (baseline) and Original Prompt + Ours compare the overall performance before and after applying our method on the full evaluation set. Second, Predicted Failure (wo. Intervention) and Predicted Failure + Ours compare the initially detected failure subset before and after selective intervention. Third, Predicted Success (wo. Intervention) and Predicted Failure (wo. Intervention) compare the initial generations of samples predicted as success or failure by the dynamic failure detector, respectively.
Figure 7: Qualitative results on Wan2.1-14B at 720p and 81 frames. While the main experiments use 480p, 49-frame generation with CogVideoX-5B and Wan2.1-1.3B, here we show that our method also applies to a larger-model and more demanding setting. In both examples, the baseline Wan2.1-14B generations fail to match the prompts, whereas our intervention produces outputs that are substantially better aligned with the intended human actions.
Figure 8: Case visualization of diagnostic interventions. Top: although the early video preview poorly reflects the prompt, the single-frame preview captures the semantics. Our method applies single-frame semantic injection (Trial-1). Bottom: both of the video and the single-frame previews fail to satisfy the intended spatial relationship (TV on the left), indicating a persistent prompt misalignment. The intervention framework automatically invokes VLM-based observation-driven prompt refinement (Trial-2).
Figure 9: Sensitivity of video quality metrics to temporal subsampling and spatial downsampling. Compared to temporal subsampling, spatial downsampling causes larger deviations in VBench scores from the original videos.
Figure 10: Early predictability of final video quality. For CogVideoX and Wan2.1 (1,800 prompts), we compute VBench metrics on intermediate L2R previews and report the normalized MAE to the final metrics of the final videos using the original VAE. Errors drop sharply by 10 steps, notably for Overall Consistency.
Figure 11: Overall Consistency as a proxy for other VBench metrics. Using all VBench prompts, we measure PR-AUC for predicting whether a sample falls into the top- or bottom- group of each target metric ( ) using Overall Consistency. All scores exceed the random baseline (0.5), indicating that Overall Consistency captures quality variations reflected by other metrics.
Figure 12: Selecting the failure threshold for Overall Consistency. Left: Distributions of Overall Consistency for generated and real videos. Real videos concentrate at higher scores; we set so that the retained generated samples better match the real-video regime. Right: A human study with 26 participants over 30 samples (780 responses) validates the perceptual boundary: scores of 0.18 are often judged as failures, whereas scores 0.22 are considered reasonable.
Figure 13: Intervention improves detected failure cases. CDFs of VBench metrics for samples predicted as failures, but forced generated, and our intervention. Despite using Overall Consistency for detection, ours shifts multiple Vbench metrics rightward.
Figure 14: Accuracy of early failure detection. Final Overall Consistency distributions for samples predicted to pass or fail (failures forced to complete without intervention). Strong separation across models demonstrates reliable early discrimination.
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Architecture of Latent-to-RGB (L2R) converter. (a) Overall architecture. The converter takes a latent video tensor as input and produces an RGB video preview through two causal 3D convolution blocks, a spatio-temporal learned upsampling, and a projection layer. (b) Causal 3D convolution block with a main branch and a skip branch. (c) Illustration of spatio-temporal learned upsampling for a single red target location. Given the previous and current latents, learned parameters are used to predict the upsampled latent at the current step by weighted summation. In practice, this operation is implemented densely as a 3D convolution over the latent.
| CogVideoX-5B | Wan2.1-1.3B | |||||||||
| DAVIS | Pexels | DAVIS | Pexels | |||||||
| Decoder | Param. (M) | PSNR | PSNR | Time (s) | Mem (MB) | Param. (M) | PSNR | PSNR | Time (s) | Mem (MB) |
| Original | 123.4 | 32.80 | 35.30 | 3.990 | 242.5 | 73.30 | 32.68 | 35.41 | 1.702 | 140.4 |
| TinyAE | 11.32 | 26.56 | 27.35 | 0.116 | 21.60 | 11.32 | 30.50 | 33.46 | 0.116 | 21.60 |
| Turbo-VAED | 40.69 | 32.03 | 35.03 | 0.599 | 114.6 | - | - | - | - | - |
| Ours | 0.059 | 30.00 | 32.50 | 0.0197 | 0.138 | 0.059 | 28.98 | 31.40 | 0.0197 | 0.138 |
Table S1: Efficiency and quality of latent-to-RGB converter. We measure PSNR on DAVIS and Pexels. Then, we measure the decoding time/memory at 480 720. L2R offers moderate PSNR with the lowest latency and memory, providing previews suitable for inspection for both models.
| Model | Method | Avg. Step | Normalized Error ( ) |
| CogVideoX-5B | Fixed | 11 | |
| Dynamic | 11.05 | ||
| Wan2.1-1.3B | Fixed | 10 | |
| Dynamic | 10.33 |
Table S2: Overall Consistency between fixed-step vs. dynamic failure detector. Despite nearly identical average prediction steps, the dynamic strategy achieves lower normalized error across both models.
Figure S2: Semantic convergence varies across samples in video diffusion. We visualize the center frame (24th frame out of 49) decoded from intermediate denoising steps for two text-to-video generations. While the dog sample exhibits recognizable semantics at an early stage, the shark sample remains ambiguous until later steps. This illustrates that semantic convergence in video diffusion is highly sample-dependent, motivating the dynamic failure detector rather than relying on a fixed early step.
Figure S3: Efficiency of selective early intervention on Wan2.1-1.3B. (a) Overall time overhead relative to base generation. Ours incurs only +20.3% additional time overhead, substantially lower than the regeneration of failure samples (+51.25%). (b) Per-sample cost increases with deeper trials, yet remains cheaper than regeneration. Since deeper trials are triggered for only a subset of samples (red curve), the overall time overhead stays modest.
| Dynamic failure detector | |||||||
| Execution time (s) | average step | ||||||
| Video | Video | Single-frame | Single-frame | VLM | Video | Single-frame | |
| Models | denoising steps | preview inspections | denoising steps | preview inspections | calls | preview | preview |
| CogVideoX-5B | 4.343 | 0.0392 | 0.405 | 0.0218 | 8.541 | 11.05 | 24.00 |
| Wan2.1-1.3B | 1.460 | 0.0392 | 0.103 | 0.0218 | 8.541 | 10.33 | 24.00 |
Table S3: Measured execution time of each primitive operation. We report the average runtime of each primitive operation used in our framework, including video denoising, video preview inspection, single-frame denoising, single-frame preview inspection, and VLM calls. We also report the average detection step of the dynamic failure detector for video previews and single-frame previews. These measured runtimes are used together with the branch-wise execution counts in Table S4 to compute the per-sample and overall time overhead.
| CogVideoX-5B: original generation time: 217.15 s (50 steps) | ||||||||||
| Execution statistics | Runtime | Time overhead | ||||||||
| Branch | Samples / Total | Video denoising steps | Video preview inspections | Single-frame denoising steps | Single-frame preview inspections | VLM calls | Average time (s) | Additional time (s) | Per-sample time overhead (%) | Overall time overhead (%) |
| Trial 0 | 562 / 946 | 50 | 11.05 | 0 | 0 | 0 | 217.58 | 0.43 | 0.20 | 0.12 |
| Trial 1 | 91 / 946 | 60.05 | 11.05 | 24 | 24 | 0 | 271.47 | 54.32 | 25.02 | 2.41 |
| Trial 2 | 125 / 946 | 61.05 | 11.05 | 24 | 24 | 1 | 284.36 | 67.21 | 30.95 | 4.09 |
| Trial 2 Trial 1 | 168 / 946 | 71.10 | 22.10 | 48 | 48 | 1 | 338.68 | 121.53 | 55.97 | 9.94 |
Table S4: Detailed breakdown of execution statistics and time overhead. We report the number of denoising steps, inspection operations, and wall-clock runtime for each execution branch. Per-sample time overhead measures the additional runtime relative to the original generation time when the branch is executed, while Overall time overhead represents the dataset-averaged contribution.
| CogVideoX-5B | Wan2.1-1.3B | |||||||
| Method | Final | Quality | Semantic | Overhead (%) | Final | Quality | Semantic | Overhead (%) |
| Original prompt (baseline) | 0.778 | 0.803 | 0.680 | 0.00 | 0.789 | 0.811 | 0.700 | 0.00 |
| Best-of-3 | 0.787 | 0.802 | 0.728 | 200.00 | 0.796 | 0.812 | 0.730 | 200.00 |
| Original prompt + Ours | 0.800 | 0.816 | 0.735 | 16.55 | 0.805 | 0.817 | 0.756 | 20.30 |
Table S5: Comparison with a simple Best-of-3 baseline. For Best-of-3 , we generate three videos per prompt from the original prompt using different random seeds, score them with our alignment scorer (ViCLIP), and select the sample with the highest semantic alignment score. Since this requires three full generations, it incurs 200% additional time overhead relative to single-sample generation.
| CogVideoX-5B | Wan2.1-1.3B | |||||||
| Method | Final | Quality | Semantic | Overhead (%) | Final | Quality | Semantic | Overhead (%) |
| Original prompt (baseline) | 0.778 | 0.803 | 0.680 | 0.00 | 0.789 | 0.811 | 0.700 | 0.00 |
| Original prompt + Ours ( ) | 0.800 | 0.816 | 0.735 | 16.55 | 0.805 | 0.817 | 0.756 | 20.30 |
| Original prompt + Ours ( ) | 0.807 | 0.824 | 0.740 | 32.61 | 0.807 | 0.823 | 0.743 | 35.96 |
| Original prompt + Ours ( ) | 0.806 | 0.827 | 0.725 | 39.47 | 0.812 | 0.826 | 0.756 | 42.77 |
Table S6: Effect of the success and failure threshold. We compare the original prompt baseline with our selective early intervention under different failure thresholds . Higher thresholds make the detector more conservative, increasing the number of samples that receive intervention. As a result, VBench scores may improve further, but at the cost of substantially higher time overhead.
Figure S4: Effect of the single-frame-based intervention family on detected failure cases. We compare the CDFs of VBench metrics between predicted failure samples generated to completion without intervention and those corrected by the single-frame-based intervention family (Trial 1 and Trial 2 1). Across both CogVideoX-5B and Wan2.1-1.3B, the intervention shifts the distributions toward better scores over multiple VBench dimensions.
Figure S5: Effect of observation-driven prompt refinement on detected failure cases. We compare the CDFs of VBench metrics between predicted failure samples generated to completion without intervention and those corrected by observation-driven prompt refinement (Trial 2). Across both CogVideoX-5B and Wan2.1-1.3B, this branch also shifts the distributions toward better scores over multiple VBench dimensions.
Figure S6: Additional qualitative results of Trial 1 (single-frame semantic injection). We present additional qualitative results from CogVideoX-5B (top) and Wan2.1-1.3B (bottom). For generations predicted as failures from early video previews, we further examine a single-frame preview. If the single-frame preview is better than the video preview, we apply semantic injection. These examples show that Trial 1 improves prompt fidelity by recovering missing objects, attributes, and subject composition in the final outputs across different models and failure cases.
Figure S7: Additional qualitative results of Trial 2 (observation-driven prompt refinement). We present additional qualitative results from CogVideoX-5B (top) and Wan2.1-1.3B (bottom). For generations predicted as failures based on early video previews, Trial 2 uses a vision–language model (VLM) to refine the prompt according to the observed failure patterns. The refined prompt strengthens the desired semantics, while the negative prompt suppresses undesired content or artifacts. These examples show that Trial 2 improves prompt fidelity by correcting subject identity, object count, composition, and visual attributes across different models and failure cases.
Figure S8: Qualitative comparison between the native CogVideoX decoder and L2R converter. We compare the reconstruction results between two methods. Across diverse scenes and multiple frames, L2R produces lightweight video previews that preserve the key semantic content and visual structure of the native CogVideoX decoder. These previews are sufficiently informative for real-time inspection and failure detection.
Figure S9: Qualitative comparison between the native Wan2.1 decoder and L2R converter. We compare the reconstruction results between two methods. Across diverse scenes and multiple frames, L2R produces lightweight video previews that preserve the key semantic content and visual structure of the native Wan2.1 decoder. These previews are sufficiently informative for real-time inspection and failure detection.
| Dimension | CogVideoX-5B | Wan2.1-1.3B | ||
| Original Prompt | + Ours | Original Prompt | + Ours | |
| Subject Consistency | 0.957 | 0.967 | 0.949 | 0.964 |
| Background Consistency | 0.953 | 0.964 | 0.959 | 0.972 |
| Temporal Flickering | 0.969 | 0.973 | 0.991 | 0.992 |
| Motion Smoothness | 0.981 | 0.985 | 0.970 | 0.979 |
| Dynamic Degree | 0.542 | 0.542 | 0.542 | 0.403 |
Table S7: All VBench results across all dimensions. We report all individual VBench dimension scores, along with the aggregated Semantic, Quality, and Final scores, for the original prompt baseline and our method on CogVideoX-5B and Wan2.1-1.3B. Overall, our method improves most VBench dimensions on both models, indicating that selective early intervention benefits not only alignment-related metrics but also broader perceptual and temporal aspects of video quality.
Figure S10: Prompt template for observation-driven prompt refinement. The VLM receives the original prompt, preview video, and the predicted preview score, and returns a refined prompt with an optional negative prompt in JSON format.
More in this topic
Video Diffusion Models
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.
See all 697 papers