Organizations: The University of Hong Kong · Shenzhen Loop Area Institute · INSAIT, Sofia University “St. Kliment Ohridski” · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · The Chinese University of Hong Kong · Snap Inc. · Shenzhen University of Advanced Technology
Recent image generation models produce photo-realistic results with rich details, but transferring their powerful priors to video restoration remains difficult due to temporal inconsistency from stochastic detail synthesis. Existing video restoration methods improve temporal coherence but often sacrifice visual fidelity, leaving a gap between image-level quality and video-level stability. We propose a generative video restoration method that bridges this gap by enabling temporally consistent detail synthesis from image generation priors. Our method combines noise energy rebalancing to suppress structure-disturbing low-frequency stochasticity, motion-aligned noise warping to propagate details along object motion, a self-supervised denoising encoder to mitigate encoder-induced temporal uncertainty, and decoupled temporal attention to model cross-frame dependencies. Together, these components preserve fidelity, enhance fine-grained realism, and maintain temporal consistency. Extensive experiments show that our method substantially outperforms existing video restoration approaches in visual quality, detail richness, and temporal stability.
Figures & tables
Figure 1: We propose HYPI2VR, which transfers powerful image generation priors to video restoration. Motivated by the stronger generative capacity of image models over video models, HYPI2VR synthesizes fine-grained details while preserving temporal consistency.
Figure 2
Figure 5: Effect of noise energy distribution on generative image restoration. Allocating noise energy to low frequencies degrades fidelity, while concentrating it in high frequencies preserves structure and enables detailed synthesis.
Figure 6: Warping noise along motion (8-pixel translation) preserves temporal correspondence, whereas independent or fixed noise causes flickering or repetitive artifacts.
Figure 7: Overview of the proposed video restoration framework. Noise energy rebalancing, motion-aligned noise propagation, and temporal attention are jointly applied in a sequence-to-sequence manner to achieve temporally consistent restoration.
Figure 8: Qualitative and temporal profile comparison. HYPI2VR restores semantically meaningful details while preserving high temporal coherence and visual stability across frames.
Datasets
Metrics
RealViformer
MGLD-VSR
UAV
STAR
SeedVR2-7B
DLoRAL
FlashVSR
HYPI2VR (Ours)
REDS30
PSNR ↑
24.73
24.57
22.56
22.29
24.25
22.77
21.59
23.56
SSIM ↑
0.6491
0.6486
0.5690
0.5764
0.6714
0.5643
0.5431
0.6524
LPIPS ↓
0.2132
0.2033
0.5438
0.5211
0.2931
0.2956
0.3163
0.2469
MUSIQ ↑
62.30
63.08
25.13
21.80
51.46
68.41
55.68
64.13
CLIP-IQA ↑
0.3510
0.3753
0.1940
0.1651
0.2265
0.5828
0.3075
0.4496
DOVER ↑
0.3585
0.4117
0.1927
0.1727
0.3144
0.3547
0.3906
0.4284
Table 1: Quantitative comparisons of different methods on various synthetic and real benchmarks. The best and second performances are marked in red and blue , respectively. HYPI2VR consistently achieves competitive or superior results across fidelity, perceptual, and temporal metrics, demonstrating a balanced restoration capability in both synthetic and real-world scenarios.
Noise
Noise
Temp.
Temp.
Feat.
PSNR ↑
SSIM ↑
LPIPS ↓
MUSIQ ↑
CLIP-IQA ↑
DOVER ↑
Ewarp∗↓
Rebal.
Align.
Encode
Block
Prop.
23.90
0.6826
0.3421
61.80
0.5035
0.6896
10.061
✓
23.89
0.6791
0.3474
61.69
0.5069
0.6956
7.648
✓
25.80
0.7428
0.2663
62.08
0.4729
0.6590
7.048
✓
✓
25.81
0.7428
0.2663
62.01
0.4725
0.6653
5.670
✓
✓
✓
25.59
0.7571
0.2547
61.70
0.4603
0.6858
5.340
Table 2: Ablation study of key modules on the POND25 dataset. 3D-SW and 1D denote the 3D shift-window attention block and the 1D full attention block used for temporal modeling. The best and second performances are marked in red and blue , respectively. Each component contributes to the overall performance trade-off across fidelity, perceptual quality, and temporal consistency.
Figure 9: The proposed modules improve temporal consistency while preserving text fidelity and high-quality texture synthesis, showing that better alignment does not compromise perceptual detail or generative capability.
Figure 10
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Visualization of encoder-induced temporal inconsistency in latent space. We project 16-channel latents to 3 channels using PCA, followed by per-sample min–max normalization for visualization. The original encoder produces temporally inconsistent noise patterns across frames, while our temporal encoder yields more stable and consistent representations. The difference maps further highlight frame-dependent noise variations.
Figure 13: A snapshot from the POND25 dataset. Scenes exhibit richer composition compared to previous validation datasets.
Figure 14: A snapshot from the POND25 dataset. Scenes exhibit higher motion complexity compared to previous validation datasets.
Figure 14
Method
Steps
Runtime (ms)
Params
UAV
30
2080
1.4B
MGLD-VSR
50
2369
1.4B
STAR
15
1617
2.5B
SeedVR2-7B
1
480
7B
DLoRAL
1
198
1.3B
FlashVSR
1
31
1.8B
Appendix
Table 3: Inference speed comparison on a single NVIDIA H200 GPU for ×4 VSR with 50 frames of resolution 512×512 . We report the number of sampling steps, runtime (ms), and model parameters.
Figure 15: Temporal consistency profiles of different model variants in the ablation study on the POND25 dataset. The progressively added modules elevate the temporal consistency of HYPIR to a level comparable with other video restoration methods.
Figure 16: Comparison of temporal modeling strategies under dynamic scenes with camera motion, where 3D attention introduces artifacts, feature propagation blurs details, and our method achieves a better balance between temporal consistency and detail preservation.
Metric
τ=0
τ=0.5
τ=1.0
τ=+∞
PSNR ↑
23.90
25.09
25.80
28.30
SSIM ↑
0.6826
0.7217
0.7428
0.8301
LPIPS ↓
0.3421
0.2869
0.2663
0.2515
MUSIQ ↑
61.80
60.76
62.08
37.88
CLIP-IQA ↑
0.5035
0.4806
0.4729
0.3048
DOVER ↑
0.6896
0.6571
0.6590
0.6143
Appendix
Table 4: Quantitative evaluation under different τ values on POND25. τ=1.0 achieves a favorable trade-off across fidelity, perceptual quality, and temporal consistency.
Figure 17: Visual comparison of restoration results with different τ values. Smaller τ introduces excessive low-frequency perturbations, while τ→+∞ suppresses stochasticity and leads to overly smooth results. τ=1.0 provides a better balance between structural fidelity and texture richness.
Figure 18: Effect of frame-based pre-training on dynamic scenes. Training with image frames allows the model to capture natural motion cues and reproduce realistic dynamic blur, outperforming models trained only on static images.
Figure 19: Additional qualitative comparisons on the POND25 dataset. HYPI2VR accurately reconstructs complex spatial structures and reproduces realistic textures under dynamic motion.
Figure 20: Additional qualitative comparisons on the POND25 dataset. HYPI2VR accurately reconstructs complex spatial structures and reproduces realistic textures under dynamic motion.
Figure 21: Qualitative comparison on the POND25 dataset. HYPI2VR accurately reconstructs complex spatial structures and reproduces realistic textures under challenging dynamic motion.
Figure 22: Qualitative comparison on the real-world VideoLQ dataset. HYPI2VR preserves structure and avoids distortion under challenging degradations.
Figure 23: Qualitative restoration on historical film footage. HYPI2VR recovers facial details and background textures while maintaining temporal coherence.
Figure 24: Qualitative restoration on historical film footage. HYPI2VR reconstructs clear facial features and background details with stable visual quality.
Recent advances in diffusion models have achieved remarkable success in restoring high-quality images in a zero-shot manner. Unfortunately, extending this success to video restoration is challenging due to the substantial computational resources required to pre-train video diffusion models. A rising trend seeks to address video restoration by utilizing a pre-trained image diffusion model, initiating restoration from either frame-wise independent or identical Gaussian noise. Yet, these approaches often fall short, as such simple noise initiations fail to capture and utilize the inherent temporal correlations within degraded video frames. In this work, we propose taming noise initialization for zero-shot video restoration, which preserves the intrinsic temporal correlations by generating initial noises directly using DDIM inversion on the corrupted video frames themselves. Building on this, we introduce an adaptive noise blending strategy that suppresses degradation patterns encoded during DDIM inversion while gradually steering the noise toward a Gaussian distribution. To further emphasize temporal consistency, we propose measurement-guided trajectory attention to better align features across frames. Extensive experiments on multiple video restoration tasks demonstrate that our method outperforms existing zero-shot approaches, consistently achieving superior results on the DAVIS, REDS4, and Set8 datasets.
Chong Wang, Zixuan Fu, Hao Cheng +3
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · School of Artificial Intelligence, Hebei University of Technology, Tianjin, China · Division of Robotics, Perception and Learning, KTH Royal Institute of Technology, Stockholm, Sweden +1
Video face restoration aims to enhance degraded face videos into high-quality results with realistic facial details, stable identity, and temporal coherence. Recent diffusion-based methods have brought strong generative priors to restoration and enabled more realistic detail synthesis. However, existing approaches for face videos still rely heavily on generic diffusion priors and multi-step sampling, which limit both facial adaptation and inference efficiency. These limitations motivate the use of one-step diffusion for video face restoration, yet achieving faithful facial recovery alongside temporally stable outputs remains challenging. In this paper, we propose, DVFace, a one-step diffusion framework for real-world video face restoration. Specifically, we introduce a spatio-temporal dual-codebook design to extract complementary spatial and temporal facial priors from degraded videos. We further propose an asymmetric spatio-temporal fusion module to inject these priors into the diffusion backbone according to their distinct roles. Evaluation on various benchmarks shows that DVFace delivers superior restoration quality, temporal consistency, and identity preservation compared to recent methods. Code: https://github.com/zhengchen1999/DVFace.
Diffusion-based video restoration recovers realistic details, but its practical deployment is limited by two efficiency bottlenecks: costly VAE encoding and decoding, and the quadratic cost of full self-attention in diffusion transformers (DiTs). This paper presents FastVR, a streaming video restoration framework built on a one-step diffusion model, which delivers strong restoration quality and temporal consistency while processing 1080p video at 11 FPS on a single H20 GPU. To improve inference efficiency, FastVR combines a lightweight VAE with chunk-wise causal attention, which substantially reduces the computational cost. During training, it further adopts velocity consistency regularization and continuous trajectory learning, which improve restoration quality. Extensive experiments show that FastVR is more efficient than the evaluated diffusion baselines while achieving state-of-the-art performance on synthetic and real-world benchmarks. We hope that this work supports further progress in the community.