Zero-Shot Video Restoration and Enhancement with Text-to-Image Latent Diffusion Models and Multi-Modal References

Cong Cao; Huanjing Yue; Xin Liu; Jingyu Yang

cs.CVAug 27, 2026

Zero-Shot Video Restoration and Enhancement with Text-to-Image Latent Diffusion Models and Multi-Modal References

Authors: Cong Cao, Huanjing Yue, Xin Liu, Jingyu Yang

Organizations: School of Electrical and Information Engineering, Tianjin University, Tianjin, China · Computer Vision and Pattern Recognition Laboratory, School of Engineering Science, Lappeenranta-Lahti University of Technology LUT, Lappeenranta, Finland

Abstract

Zero-shot image restoration methods with text-to-image latent diffusion models have achieved great success in universal image restoration tasks without training. However, applying them to video restoration will result in severe temporal flickering. In this paper, we propose a novel framework for zero-shot video restoration and enhancement which uses a text-to-image latent diffusion model and multi-modal references. Through the proposed dual prompt tuning inversion and sampling, the inference time can be reduced to nearly 1/3 of the original. The performance and temporal consistency can be also significantly stregthened. By using the proposed texture-aware video token merging, the temporal correlation between frames can be further utilized to improve the temporal consistency. We futher propose the referenced self-attention and referenced token merging to support image reference. Experimental results demonstrate the superiority of the proposed method in restoring and enhancing temporally consistent videos.

Figures & tables

Figure 2: Visual quality comparison for video super-resolution and low-light video enhancement. Zoom in for better observation.

Methods	Backbone	PSNR $\uparrow$	SSIM $\uparrow$	WE( $10^{-2}$ ) $\downarrow$
PSLD	SDv1.5	24.85	0.6313	2.0854
ZVRM (no Ref.)	SDv1.5	25.16	0.6599	0.5003
ZVRM (text Ref.)	SDv1.5	25.32	0.6642	0.4755
ZVRM (text Ref.+image Ref. 1)	SDv1.5	25.61	0.6785	0.3871
VISION-XL	SDXL	25.91	0.7487	1.5578
ZVRM (no Ref.)	SDXL	26.13	0.7594	0.9345

Table 1: Quantitative comparison with state-of-the-art methods for 4

\times

video super-resolution. The best results are highlighted in bold.

Methods	Backbone	PSNR $\uparrow$	SSIM $\uparrow$	WE( $10^{-2}$ ) $\downarrow$
PSLD	SDv1.5	18.17	0.8174	0.6895
ZVRM (no Ref.)	SDv1.5	18.40	0.8303	0.2170
ZVRM (text Ref.)	SDv1.5	18.45	0.8369	0.2032
ZVRM (text Ref.+image Ref. 1)	SDv1.5	18.79	0.8552	0.1924

Table 2: Quantitative comparison with state-of-the-art methods for low-light video enhancement. The best results are highlighted in bold.

Methods	Backbone	PSNR $\uparrow$	SSIM $\uparrow$	WE( $10^{-2}$ ) $\downarrow$
PSLD	SDv1.5	22.91	0.6241	1.9198
ZVRM (no Ref.)	SDv1.5	23.14	0.6339	0.8215
ZVRM (image Ref. 1)	SDv1.5	23.46	0.6552	0.8069
ZVRM (image Ref. 2)	SDv1.5	23.33	0.6439	0.7720
ZVRM (image Ref. 3)	SDv1.5	23.37	0.6461	0.8013

Table 3: Quantitative comparison of 4

\times

video super-resolution using different reference image sources on the RealMCVSR dataset [ 16 ] . The best results are highlighted in bold.

Methods	Backbone	PSNR $\uparrow$	SSIM $\uparrow$	WE( $10^{-2}$ ) $\downarrow$
DiffBIR	SDv2.1	24.47	0.6727	0.6907
DiffIR2VR	SDv2.1	24.49	0.6718	0.6818
DiffBIR+ZVRD	SDv2.1	24.55	0.6799	0.4923
DiffBIR+ZVRM (no Ref.)	SDv2.1	24.71	0.6839	0.4397
DiffBIR+ZVRM (text Ref.)	SDv2.1	24.80	0.6860	0.4375
DiffBIR+ZVRM (text Ref.+image Ref. 1)	SDv2.1	24.96	0.6971	0.4288

Table 4: Quantitative comparison with state-of-the-art methods for 4

\times

blind video super-resolution on the DAVIS dataset. The best results are highlighted in bold.

Figure 3: Different kinds of reference images for zero-shot video super-resolution on the RealMCVSR dataset.

DPTT	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
DPTS	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
TSTM	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
TTTM	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
RSA	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$
RTM	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$
PSNR $\uparrow$	24.85	24.88	25.03	25.05	25.16	25.31	25.49

Table 5: Ablation study for Dual Prompt Tuning Inversion (DPTI), Dual Prompt Tuning Sampling (DPTS), Texture-Aware Spatial Token Merging (TSTM), Texture-Aware Temporal Token Merging (TTTM), Referenced Self-Attention (RSA), and Referenced Token Merging (RTM) on 4

\times

video super-resolution task.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	Backbone	PSNR $\uparrow$	SSIM $\uparrow$	WE( $10^{-2}$ ) $\downarrow$
PSLD	SDv1.5	19.33	0.4755	5.4533
ZVRM (no Ref.)	SDv1.5	20.07	0.4852	0.8183
ZVRM (text Ref.)	SDv1.5	20.21	0.4831	0.7697
ZVRM (text Ref.+image Ref. 1)	SDv1.5	20.45	0.4910	0.7009

Appendix

Table 1: Quantitative comparison with state-of-the-art methods for video deblurring. The best results are highlighted in bold.

Figure 1: Visual quality comparison for video super-resolution using an SDXL backbone. Zoom in for better observation.

Figure 2: Visual quality comparison for video deblurring. Zoom in for better observation.

Figure 3: Different kinds of reference images for zero-shot video super-resolution.

Figure 4: Visual quality comparison for blind video super-resolution. Zoom in for better observation..

DPTI	Cond.	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$
DPTI	Uncond.	$\times$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$
DPTS	Cond.	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$
DPTS	Uncond.	$\times$	$\times$	$\times$	$\times$	$\checkmark$
PSNR $\uparrow$		24.85	24.87	24.88	24.96	25.03
SSIM $\uparrow$		0.6313	0.6457	0.6501	0.6552	0.6585
WE( $10^{-2}$ ) $\downarrow$		2.0854	2.0197	1.9646	1.3880	1.0721

Appendix

Table 2: Ablation study for dual prompt tuning inversion, dual prompt tuning sampling, spatial-temporal self-attention and temporal latent fusion on 4

\times

video super-resolution task. Cond. and Uncond. respectively represent the optimization of conditional and unconditional embedding.

Video Restoration

Video restoration reconstructs clearer, more faithful, temporally consistent footage from blur, noise, compression, weather, and other degradations.

See all 58 papers

FastVR: Efficient Streaming Video Restoration with One-Step Diffusion
Sep 29, 2026Xiaoxu Chen, Qin Yang, Haoran Bai +2Diffusion Transformer Video Restoration
Alibaba Group · Xidian University
DTG-Restore: Training-Free Diffusion Refinement for Generative Video Super-Resolution
May 28, 2026Hidir Yesiltepe, Koutilya PNVR, Gaurav Pathak +4Diffusion Model Guidance Video Diffusion Models
1Virginia Tech · 2Adobe
DVFace: Spatio-Temporal Dual-Prior Diffusion for Video Face Restoration
Apr 16, 2026Zheng Chen, Bowen Chai, Rongjun Gao +7Video Diffusion Models Image Restoration
Shanghai Jiaotong University · Meituan Inc

Sep 29, 2026J/K move · Enter open · S save

FastVR: Efficient Streaming Video Restoration with One-Step Diffusion

Xiaoxu Chen, Qin Yang, Haoran Bai, Sibin Deng+1

Alibaba Group · Xidian University

Diffusion Transformer Video Restoration Diffusion Model Inference Acceleration Diffusion Models

Open paper AskPDF arXiv

Diffusion-based video restoration recovers realistic details, but its practical deployment is limited by two efficiency bottlenecks: costly VAE encoding and decoding, and the quadratic cost of full self-attention in diffusion transformers (DiTs). This paper presents FastVR, a streaming video restoration framework built on a one-step diffusion model, which delivers strong restoration quality and temporal consistency while processing 1080p video at 11 FPS on a single H20 GPU. To improve inference efficiency, FastVR combines a lightweight VAE with chunk-wise causal attention, which substantially reduces the computational cost. During training, it further adopts velocity consistency regularization and continuous trajectory learning, which improve restoration quality. Extensive experiments show that FastVR is more efficient than the evaluated diffusion baselines while achieving state-of-the-art performance on synthetic and real-world benchmarks. We hope that this work supports further progress in the community.

Abstract

Video Restoration

Explore similar work

FastVR: Efficient Streaming Video Restoration with One-Step Diffusion

DTG-Restore: Training-Free Diffusion Refinement for Generative Video Super-Resolution

DVFace: Spatio-Temporal Dual-Prior Diffusion for Video Face Restoration