We present the first systematic study of the resilience of text-to-video (T2V) diffusion models under random hardware-level faults. While T2V models are widely used for automated video generation due to their ability to produce high-quality, temporally coherent, and realistic videos, their iterative denoising process and spatiotemporal dependencies introduce unique failure modes. We perform an extensive fault-injection study covering both computational and memory faults across three T2V models and a representative benchmark. Our results show that (1) a single fault can degrade overall performance by up to 3.7%, with semantic correctness more affected than perceptual quality; (2) memory faults are more damaging than computational faults, high-order exponent bits are particularly vulnerable, and the widely-used bfloat16 is more susceptible than alternative formats; and (3) 7-28% of faults cause visible artifacts, including semantic changes such as added objects, suggesting that single faults are sufficient to alter output semantics. Our findings reveal reliability risks in deployed T2V systems and motivate further research on improving fault resilience. Code: https://github.com/ztcoalson/T2V-Resilience.
Figures & tables
Figure 1: High-level architecture of a text-to-video diffusion model. The DiT backbone operates in latent space and is reused across denoising steps. In our study, we assess resilience by injecting random bitwise faults into the backbone’s weights and activations .
Figure 2: Impact of hardware faults on overall video quality and semantics. Average Total, Quality, and Semantic Scores over 100 trials for the fault-free (baseline), 2-bit memory fault, and 1-bit computational fault models. 95% CIs are within ± 0.23 across all bars.
Figure 3: Resilience of model-level components to 2-bit memory faults for Latte. The average change across individual VBench metrics, categorized by maximum bit location, Transformer block index, and layer type.
Figure 4: Impact of computational faults across diffusion timesteps . The average change in individual VBench metrics for each (normalized) diffusion step at which faults were injected.
Figure 5: Impact of numerical representation for CogVideoX. The average Total Score under the fault-free (baseline) and 2-bit memory fault models. 95% CIs are within ± 0.08 for all bars.
Figure 6: Example of qualitative fault outcomes for representative videos generated from OpenSora. We show 5 evenly spaced frames from each video. Prompt: “a dilapidated phone booth stood as a relic of a bygone era on the sidewalk, frozen in time.”
Figure 7: Distribution of qualitative fault outcomes for the 2-bit memory and 1-bit computational fault models (150 samples each).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Impact of hardware faults on individual video quality and semantics metrics. For each metric, we report the average score over 100 trials for the fault-free (baseline), 2-bit memory fault, and 1-bit computational fault models. 95% CIs are within ± 0.95.
Figure 9: Resilience of model-level components to 2-bit memory faults for OpenSora. The average change across individual VBench metrics, categorized by maximum bit location, Transformer block index, and layer type.
Figure 10: Resilience of model-level components to 2-bit memory faults for CogVideoX. The average change across individual VBench metrics, categorized by maximum bit location, Transformer block index, and layer type.
Video diffusion models generate or edit temporally coherent video by iteratively denoising latent sequences conditioned on text, images, motion, or other signals.
Text-to-video (T2V) diffusion models have rapidly advanced, yet generations still occasionally fail in practice, such as low text-video alignment or low perceptual quality. Since diffusion sampling is non-deterministic, it is difficult to know during inference whether a generation will succeed or fail, incurring high computational cost due to trial-and-error regeneration. To address this, we propose an early failure detection and diagnostic intervention pipeline for latent T2V diffusion models. For detection, we design a Real-time Inspection (RI) module that converts latents into intermediate video previews, enabling the use of established text-video alignment scorers for inspection in the RGB space. The RI module completes the conversion and inspection process in just 39.2ms. This is highly efficient considering that CogVideoX-5B requires 4.3s per denoising step when generating a 480p, 49-frame video on an NVIDIA A100 GPU. Subsequently, we trigger a hierarchical and early-exit intervention pipeline only when failure is predicted. Experiments on CogVideoX-5B and Wan2.1-1.3B demonstrate consistency gains on VBench with up to 2.64 times less time overhead compared to post-hoc regeneration. Our method also generalizes to a higher-capacity setting, remaining effective on Wan2.1-14B with 720p resolution and 81-frame generation. Furthermore, our pipeline is plug-and-play and orthogonal to existing techniques, showing seamless compatibility with prompt refinement and sampling guidance methods. We also provide evidence that failure signals emerge early in the denoising process and are detectable within intermediate video previews using standard vision-language evaluators.
Kwon Byung-Ki, Sohwi Lim, Nam Hyeon-Woo +2
Grad. School of Artificial Intelligence, POSTECH · School of Electrical Engineering, KAIST · Dept. of Electrical Engineering, POSTECH +1
Semantic noise initialization has been reported to improve robustness and controllability in image diffusion models. Whether these gains transfer to text-to-video (T2V) generation remains unclear, since temporal coupling can introduce extra degrees of freedom and instability. We benchmark semantic noise initialization against standard Gaussian noise using a frozen VideoCrafter-style T2V diffusion backbone and VBench on 100 prompts. Using prompt-level paired tests with bootstrap confidence intervals and a sign-flip permutation test, we observe a small positive trend on temporal-related dimensions; however, the 95 percent confidence interval includes zero (p ~ 0.17) and the overall score remains on par with the baseline. To understand this outcome, we analyze the induced perturbations in noise space and find patterns consistent with weak or unstable signal. We recommend prompt-level paired evaluation and noise-space diagnostics as standard practice when studying initialization schemes for T2V diffusion.
Yixiao Jing, Chaoyu Zhang, Zixuan Zhong +1
University of Michigan · University College London
Diffusion models have recently advanced text-to-video (T2V) generation, yet they still struggle with fine-grained compositional alignment, such as attribute binding, spatial relations, and object interactions. While reward-based fine-tuning improves alignment, it is susceptible to reward hacking and adapts poorly to new prompt distributions. In this work, we propose NoisEasier, a test-time scaling framework that improves T2V generation through differentiable reward-guided noise optimization without modifying the underlying model. By combining efficient short-step generators with a multi-objective reward formulation, NoisEasier enables stable and practical test-time optimization under realistic inference budgets. Our key insight is that jointly optimizing the entire stochastic trajectory accelerates reward convergence and improves compositional alignment over optimizing only the initial latent, with negligible additional computational and time cost. Experiments on VBench and T2V-CompBench demonstrate consistent improvements across multiple backbones, achieving over 10% average gains on challenging dimensions such as attribute binding, object interaction, and numeracy. Overall, NoisEasier serves as both a flexible alternative and a complementary enhancement to reward-based fine-tuning, establishing test-time scaling as an effective paradigm for controllable text-to-video generation.
Yujiang Pu, Yu Kong
Michigan State University, East Lansing, MI 48824, USA