Taming Noise Initialization for Zero-shot Video Restoration with Image Diffusion Priors
Authors: Chong Wang, Zixuan Fu, Hao Cheng, Siyuan Yang, Alex C. Kot, Bihan Wen
Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Artificial Intelligence, Hebei University of Technology, Tianjin, China · Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Stockholm, Sweden · Faculty of Engineering , Shenzhen MSU-BIT University, China
Recent advances in diffusion models have achieved remarkable success in restoring high-quality images in a zero-shot manner. Unfortunately, extending this success to video restoration is challenging due to the substantial computational resources required to pre-train video diffusion models. A rising trend seeks to address video restoration by utilizing a pre-trained image diffusion model, initiating restoration from either frame-wise independent or identical Gaussian noise. Yet, these approaches often fall short, as such simple noise initiations fail to capture and utilize the inherent temporal correlations within degraded video frames. In this work, we propose taming noise initialization for zero-shot video restoration, which preserves the intrinsic temporal correlations by generating initial noises directly using DDIM inversion on the corrupted video frames themselves. Building on this, we introduce an adaptive noise blending strategy that suppresses degradation patterns encoded during DDIM inversion while gradually steering the noise toward a Gaussian distribution. To further emphasize temporal consistency, we propose measurement-guided trajectory attention to better align features across frames. Extensive experiments on multiple video restoration tasks demonstrate that our method outperforms existing zero-shot approaches, consistently achieving superior results on the DAVIS, REDS4, and Set8 datasets.
Figures & tables
Fig. 1: (a) Existing method [ 1 ] utilizes a simple identical Gaussian noise initialization but fails to ensure frame consistency. (b) Our method tames the DDIM inversion noise with adaptive filtering while capturing the inherent temporal correlation in the degraded video, achieving results with guaranteed temporal consistency.
Fig. 2: Overview of the proposed TINDER framework, where the initial noise is first obtained by performing DDIM inversion on the degraded video to capture the inherent temporal correlation. Subsequently, an adaptive noise blending strategy is employed to mitigate the degradation pattern in the inversion noise XT , while directing the noise toward a Gaussian distribution. After that, the temporal correlation is further exploited by measurement-guided trajectory attention during the sampling process.
Fig. 3: Visual comparison of restoration using (a) DDIM inversion noise and (b) DDIM inversion with adaptive noise blending. Directly using inversion noise for restoration encodes the degradation pattern into restored results, as shown in red boxes in (a).
Fig. 4: A visual example of (a) the initial inversion noise obtained directly from the degraded frame, which exhibits significant degradation patterns and distorted frequency components. With adaptive noise blending, (b) the blended noise increasingly resembles (c) the ground truth inverted noise, both in the image domain and in terms of energy distribution across the frequency domain.
Fig. 5: Example of various video restoration results including Gaussian deblur, super-resolution, and inpainting. Please zoom in for better visual details.
Fig. 6: Visualization of ablation study by removing each key component of Gaussian deblur on the DAVIS dataset.
Zero-shot image restoration methods with text-to-image latent diffusion models have achieved great success in universal image restoration tasks without training. However, applying them to video restoration will result in severe temporal flickering. In this paper, we propose a novel framework for zero-shot video restoration and enhancement which uses a text-to-image latent diffusion model and multi-modal references. Through the proposed dual prompt tuning inversion and sampling, the inference time can be reduced to nearly 1/3 of the original. The performance and temporal consistency can be also significantly stregthened. By using the proposed texture-aware video token merging, the temporal correlation between frames can be further utilized to improve the temporal consistency. We futher propose the referenced self-attention and referenced token merging to support image reference. Experimental results demonstrate the superiority of the proposed method in restoring and enhancing temporally consistent videos.
Cong Cao, Huanjing Yue, Xin Liu +1
School of Electrical and Information Engineering, Tianjin University, Tianjin, China · Computer Vision and Pattern Recognition Laboratory, School of Engineering Science, Lappeenranta-Lahti University of Technology LUT, Lappeenranta, Finland
Recent progress in video diffusion models has enabled remarkable generative fidelity, yet leveraging these priors for restoration remains limited by the strong coupling between conditional and unconditional branches in standard classifier-free guidance. We introduce a training-free framework that enhances distorted and low-resolution videos by decoupling these signals in time. Our proposed Decoupled Time Guidance (DTG) evaluates the unconditional branch at a cleaner diffusion timestep, providing a lookahead prior that preserves geometry while suppressing replication of warped content. This temporal bias is annealed throughout sampling, allowing the model to transition from structure correction to detail refinement without retraining. Combined with any off-the-shelf restoration module in a plug-and-play manner, our approach improves perceptual coherence and restores plausible structure in AIgenerated and real-world videos alike. To facilitate evaluation, we curate GenWarp480, a benchmark of 4,400 distorted 480p videos synthesized from diverse text-to-video models. GenWarp480 focuses on characteristic generative degradations such as warped faces, body misalignments, and spatial artifacts, providing a purpose-built testbed for assessing robustness to generative errors. Extensive experiments demonstrate that our method achieves significant improvements in structural fidelity and temporal stability without any model training.
Video face restoration aims to enhance degraded face videos into high-quality results with realistic facial details, stable identity, and temporal coherence. Recent diffusion-based methods have brought strong generative priors to restoration and enabled more realistic detail synthesis. However, existing approaches for face videos still rely heavily on generic diffusion priors and multi-step sampling, which limit both facial adaptation and inference efficiency. These limitations motivate the use of one-step diffusion for video face restoration, yet achieving faithful facial recovery alongside temporally stable outputs remains challenging. In this paper, we propose, DVFace, a one-step diffusion framework for real-world video face restoration. Specifically, we introduce a spatio-temporal dual-codebook design to extract complementary spatial and temporal facial priors from degraded videos. We further propose an asymmetric spatio-temporal fusion module to inject these priors into the diffusion backbone according to their distinct roles. Evaluation on various benchmarks shows that DVFace delivers superior restoration quality, temporal consistency, and identity preservation compared to recent methods. Code: https://github.com/zhengchen1999/DVFace.