Video inverse problems are fundamental to streaming, telepresence, and AR/VR, where high perceptual quality must coexist with tight latency constraints. Diffusion-based priors currently deliver state-of-the-art reconstructions, but existing approaches either adapt image diffusion models with ad hoc temporal regularizers - leading to temporal artifacts - or rely on native video diffusion models whose iterative posterior sampling is far too slow for real-time use. We introduce InstantViR, an amortized inference framework for ultra-fast video reconstruction powered by a pre-trained video diffusion prior. We distill a powerful bidirectional video diffusion model (teacher) into a causal autoregressive student that maps a degraded video directly to its restored version in a single forward pass, inheriting the teacher's strong temporal modeling while completely removing iterative test-time optimization. The distillation is prior-driven: it only requires the teacher diffusion model and known degradation operators, and does not rely on externally paired clean/noisy video data. To further boost throughput, we replace the video-diffusion backbone VAE with a high-efficiency LeanVAE via an innovative teacher-space regularized distillation scheme, enabling low-latency latent-space processing. Across streaming random inpainting, Gaussian deblurring and super-resolution, InstantViR matches or surpasses the reconstruction quality of diffusion-based baselines while running at over 35 FPS on NVIDIA A100 GPUs, achieving up to 100 times speedups over iterative video diffusion solvers. These results show that diffusion-based video reconstruction is compatible with real-time, interactive, editable, streaming scenarios, turning high-quality video restoration into a practical component of modern vision systems.
Figures & tables
Figure 1: We introduce InstantViR, a real-time video inverse problem solver that drastically outperforms slow sampling-based methods in both speed and quality. Bottom-right : At 832 × 480 resolution, our amortized framework is over 100 × faster than sampling-based baselines like SVI [ 10 ] , achieving over 35 FPS and the excellent quality. Left and Bottom-left : Qualitative examples demonstrate versatile, high-fidelity reconstruction for inpainting and deblurring, along with optional text-guided control (e.g., "pink lips", "light-blue collar").
Figure 2: Overview of the InstantViR framework. (Top) Training: We train a single-step solver qϕ using only degraded measurements y . The solver is optimized with two objectives: a data fidelity loss (ensuring the reconstruction matches the measurement y ) and a prior distillation loss (using a frozen video diffusion prior [ 24 ] , sθ , to enforce temporal consistency and realism). (Bottom) Inference: The trained solver qϕ operates as a fast, feed-forward network, processing the video in a causal, block-wise, and autoregressive manner. This enables real-time, single-step streaming reconstruction with optional text guidance.
Figure 3: Qualitative comparison for video random inpainting. (Top) On the Open-Sora dataset [ 12 ] , our model reconstructs a high-fidelity and temporally consistent video from a 50% masked measurement. The zoomed-in face sequence demonstrates the stability and fine detail of our single-step result. (Bottom) We demonstrate strong zero-shot generalization on the REDS dataset [ 33 ] , our method generates a sharp, coherent video that is perceptually close to the ground truth.
Figure 4: Results for text-guided video reconstruction. InstantViR not only restores high-fidelity details from severely masked measurements (top row) but also uses text prompts to control the reconstruction. (Left) Given a masked input, our method can add specific details like a "black headband" or "wear glasses" as guided by the text. (Right) The framework can generate multi-modal, semantically distinct, and plausible outputs from the same masked input, such as reconstructing the subject with either "close eyes" or "open eyes" , demonstrating fine-grained controllability.
Method
FVD ↓
Avg. FPS ↑
Inpainting
Super-Res.
Deblur
DPS
375.81
711.61
783.10
<0.02
DiffIR2VR
-
311.61
-
0.12
SVI
219.90
176.60
154.38
0.29
VISION-XL
224.74
172.79
138.79
<0.17
InstantViR (Ours)
136.06
153.13
110.51
13.91
Table 1: Quantitative results of temporal quality and inference speed. The former is evaluated with FVD ↓ and the latter is with FPS ↑ . Best results are in bold , suboptimal are underlined .
Method
50% Random Inpainting
4 × Super-Resolution
Gaussian Deblur
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
PSNR ↑
SSIM ↑
LPIPS ↓
DPS
27.68
0.92
0.32
22.78
0.91
0.46
23.54
0.88
0.46
DiffIR2VR
-
-
-
33.44
0.92
0.33
-
-
-
SVI
29.42
0.90
0.17
33.85
0.96
0.17
26.93
0.89
0.31
VISION-XL
30.83
0.95
0.25
35.69
0.98
0.24
30.03
0.93
0.28
InstantViR (Ours)
30.54
0.97
0.12
34.91
0.96
0.23
31.85
0.97
0.17
Table 2: Quantitative results of spatial quality. Metrics include PSNR, SSIM, LPIPS. Best results are in bold , suboptimal are underlined .
Figure 5: Qualitative comparison for video Gaussian deblurring.
Figure 6: Qualitative comparison for video super-resolution (4x).
Figure 1: Video Inpainting qualitative comparison. Each row shows a complete sequence reconstructed by a specific method. InstantViR (both WanVAE Ours and LeanVAE Ours † variants) produces coherent content for every frame while requiring only a single feed-forward pass.
Figure 2: Video Deblurring qualitative comparison. Rows correspond to different methods; columns show consecutive frames covering the entire clip. InstantViR (both WanVAE Ours and LeanVAE Ours † variants) restores fine structures consistently across time, outperforming slower diffusion-based baselines.
Figure 3: Video Super-Resolution (4 × ) qualitative comparison. InstantViR restores temporally consistent structures, outperforming slower diffusion-based baselines in both sharpness and coherence.
Figure 4: Text-Guided Video Deblurring. InstantViR can generate diverse outcomes from the same blurred input based on text prompts, adding specific features like glasses or changing eye color while maintaining temporal coherence.
Diffusion models provide powerful priors for zero-shot video inverse problems, but their real-time deployment is hindered by two inefficiencies: high initial latency caused by holistic video restoration, and low throughput resulting from multiple VAE passes to enforce measurement consistency in pixel space. To overcome these limitations, we propose Autoregressive Video Inverse problem Solver (AVIS). The AVIS framework leverages autoregressive video diffusion models to restore videos in a streaming manner, naturally eliminating latency bottlenecks. Specifically, AVIS initializes reverse diffusion with a measurement-consistent estimate, reducing the required sampling steps. Compared to leading non-autoregressive solvers, AVIS drastically reduces initial latency from 114s to 4s and increases throughput from 0.71 to 1.18 FPS while achieving superior restoration quality. We further introduce a highly accelerated variant, dubbed AVIS Flash, that enforces measurement consistency solely on the first chunk. AVIS Flash substantially boosts throughput to 5.91 FPS on a single RTX 4090 GPU while maintaining competitive performance and achieving a favorable efficiency-performance trade-off, paving the way toward real-time deployment.
Diffusion-based video restoration recovers realistic details, but its practical deployment is limited by two efficiency bottlenecks: costly VAE encoding and decoding, and the quadratic cost of full self-attention in diffusion transformers (DiTs). This paper presents FastVR, a streaming video restoration framework built on a one-step diffusion model, which delivers strong restoration quality and temporal consistency while processing 1080p video at 11 FPS on a single H20 GPU. To improve inference efficiency, FastVR combines a lightweight VAE with chunk-wise causal attention, which substantially reduces the computational cost. During training, it further adopts velocity consistency regularization and continuous trajectory learning, which improve restoration quality. Extensive experiments show that FastVR is more efficient than the evaluated diffusion baselines while achieving state-of-the-art performance on synthetic and real-world benchmarks. We hope that this work supports further progress in the community.
Video super-resolution (VSR) seeks to reconstruct high-resolution frames from low-resolution inputs. While diffusion-based methods have substantially improved perceptual quality, extending them to video remains challenging for two reasons: strong generative priors can introduce temporal instability, and multi-frame diffusion pipelines are often too expensive for practical deployment. To address both challenges simultaneously, we propose InstaVSR, a lightweight diffusion framework for efficient video super-resolution. InstaVSR combines three ingredients: (1) a pruned one-step diffusion backbone that removes several costly components from conventional diffusion-based VSR pipelines, (2) recurrent training with flow-guided temporal regularization to improve frame-to-frame stability, and (3) dual-space adversarial learning in latent and pixel spaces to preserve perceptual quality after backbone simplification. On an NVIDIA RTX 4090, InstaVSR processes a 30-frame video at 2K×2K resolution in under one minute with only 7 GB of memory usage, substantially reducing the computational cost compared to existing diffusion-based methods while maintaining favorable perceptual quality with significantly smoother temporal transitions.
Jintong Hu, Bin Chen, Zhenyu Hu +3
Insta360 Research · Peking University · Wuhan University