ReCaVSR: One-Step Streaming Diffusion Video Super-Resolution with Recycled Latents and Learned Cache Routing

Xijun Wang; Xin Li; Suhang Yao; Zirui Lang; Bingchen Li; Zhibo Chen

cs.CVSep 29, 2026

ReCaVSR: One-Step Streaming Diffusion Video Super-Resolution with Recycled Latents and Learned Cache Routing

Authors: Xijun Wang, Xin Li, Suhang Yao, Zirui Lang, Bingchen Li, Zhibo Chen

Organizations: University of Science and Technology of China

Abstract

Real-time diffusion-based video super-resolution (VSR) is in high demand for online streaming, yet stringent latency requirements often compromise generative fidelity. We propose ReCaVSR, a Wan2.2-based, one-step framework for streaming VSR that builds on two observations: recycled SR latents retain local temporal context, reducing the need for full historical Key-Value (KV) caches; and individual transformer layers benefit from distinct temporal scopes. ReCaVSR combines three complementary designs: (i) layer-wise cache routing with recycled SR latents: each DiT layer learns its KV-cache temporal scope under a cache budget and exports a static inference schedule, while recycled SR latents propagate local context by conditioning each new block on the model's own preceding predictions. (ii) Multi-Scope Query (MSQ) Discriminator: a compositional discriminator combining global, spatial-window, and temporal-tube feedback for holistic realism, local texture generation, and temporal stability. (iii) LR-conditioned adaptation of FlashDecoder: a VAE decoder that incorporates LR observations for efficient latent decoding. ReCaVSR enables streaming VSR without iterative sampling or full historical KV-cache materialization. Experiments on synthetic and real-world VSR benchmarks show better perceptual quality, temporal consistency, and streaming efficiency than representative VSR baselines. At $1080{\times}1920$ output resolution on a single NVIDIA A100-80GB, ReCaVSR achieves 21.20 FPS with 15.16 GB peak allocated GPU memory, running 2.72 $\times$ faster while using 38.0% less peak allocated memory than FlashVSR Tiny. The code is available at https://github.com/kopperx/ReCaVSR.

Figures & tables

Figure 1: Streaming inference with ReCaVSR. Recycled SR latents convey local temporal context from the preceding block, while layer-wise cache routing retains only the historical KV states required by each DiT layer.

Figure 2: Overview of ReCaVSR. The generator conditions on LR observations and recycled SR latents, with layer-wise routing selecting historical KV states. In Stage 1, cache allocation is learned under budget and entropy regularization and exported as a static per-layer schedule. In Stage 2, the generator performs one-step sequential rollout under this schedule, supervised by the MSQ discriminator through global, spatial-window, and temporal-tube queries.

Dataset	Metric	RealViformer	UAV	STAR	DOVE	SeedVR2	SwiftVR	FlashVSR	Ours
REDS30	PSNR $\uparrow$	23.32	21.15	22.14	23.28	22.17	21.33	21.41	21.67
	SSIM $\uparrow$	0.5967	0.5143	0.5432	0.6103	0.5860	0.5234	0.5399	0.5569
	LPIPS $\downarrow$	0.3043	0.4036	0.4967	0.3773	0.3158	0.3564	0.3311	0.3129
	NIQE $\downarrow$	3.0804	3.0017	5.2237	4.2282	3.5157	3.4153	2.9504	2.9368
	MUSIQ $\uparrow$	59.12	60.02	37.95	50.44	57.65	63.77	56.01	60.42
	CLIP-IQA $\uparrow$	0.3236	0.3698	0.2132	0.2904	0.3041	0.3914	0.3160	0.3396

Table 1: Quantitative comparison on synthetic and real-world VSR benchmarks. The best and second performances are marked in red and blue , respectively.

Figure 3: Qualitative comparisons on challenging synthetic and real-world VSR examples. ReCaVSR restores sharper structures, more natural textures, and cleaner face, text, and logo details.

Method	NIQE $\downarrow$	MUSIQ $\uparrow$	CLIP-IQA $\uparrow$	DOVER $\uparrow$	SC $\uparrow$	BC $\uparrow$	MS $\uparrow$
RealViformer	5.0770	46.17	0.3819	0.5053	0.8846	0.9198	0.9848
Stream-DiffVSR	4.0491	54.52	0.4694	0.5704	0.8830	0.9161	0.9813
SwiftVR	4.2822	53.89	0.4640	0.5512	0.8831	0.9177	0.9824
FlashVSR	3.8160	55.15	0.4747	0.5895	0.8829	0.9144	0.9802
Ours	3.9921	57.89	0.4932	0.6075	0.8834	0.9133	0.9828

Table 2: Quantitative comparison on LongVSR60 (Real + AIGC). The best and second-best results are marked in red and blue , respectively.

Metric	DOVE	SeedVR2-3B	SparkVSR	Stream-DiffVSR	FlashVSR	ReCaVSR
First-output latency (s) $\downarrow$	356.801	255.199	356.482	0.705	2.830	0.982
FPS $\uparrow$	0.563	0.788	0.564	0.849	7.799	21.202
Peak Mem. (GB) $\downarrow$	41.079	73.889	41.296	25.923	24.447	15.159

Table 3: GPU inference efficiency at

1080{\times}1920

output resolution on 201 real input frames using a single NVIDIA A100-80GB GPU.

Variant	DOVER $\uparrow$	MS $\uparrow$	KV (GB) $\downarrow$	FLOPs $\downarrow$
Uniform ( $3R$ )	0.5601	0.9801	6.44	$1.00\times$
Uniform ( $R$ )	0.5261	0.9722	2.15	$\mathbf{0.27}\times$
Ours †	0.5130	0.9748	2.17	$\underline{0.28\times}$
Ours	0.5587	0.9785	2.17	$\underline{0.28\times}$

Table 7

Decoder	PSNR $\uparrow$	FPS $\uparrow$	Mem. (GB)
Wan2.1 VAE	38.5332	4.51	22.13
Wan2.2 VAE	39.1866	3.77	25.80
TCDecoder	36.8953	33.06	5.56
SwiftVR ReAE	30.9792	141.35	18.11
Ours ‡	34.4523	93.20	1.28
Ours	36.2525	91.71	1.33

Table 6: Decoder quality and efficiency.

\ddagger

: without LR conditioning. Best and second-best values are bold and underlined.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1: Blockwise streaming inference with ReCaVSR. Each block uses one DiT forward pass to generate SR latents, which are decoded with LR observations from the same block. Green dotted paths recycle the generated SR latents; orange paths carry the historical KV selected for each layer; gray paths preserve the LR projector and decoder states. The first block starts with zero recycled input and an empty DiT KV cache.

Generator		LR-conditioned decoder
Backbone	Wan2.2-TI2V-5B	Architecture	FlashDecoder
DiT layers	30	Backbone layers	12
Hidden dimension	3072	Refinement layers	2
Latent channels	48	Hidden dimension	512
Patch size $(T,H,W)$	$1\times 2\times 2$	Latent projection	$48\rightarrow 512$
LoRA rank	512	LR stem	$768\rightarrow 48$

Appendix

Table B.1: Architecture configurations of ReCaVSR.

Metric	DOVE	SeedVR2-3B	SwiftVR	FlashVSR Tiny	ReCaVSR
MOS-Q $\uparrow$	3.15	3.40	3.32	3.78	3.84
MOS-D $\uparrow$	3.17	3.35	3.15	3.72	3.81
MOS-T $\uparrow$	3.62	3.29	2.98	3.69	3.87

Appendix

Table C.2: Human evaluation on 35 VideoLQ videos with 15 raters. MOS-Q, MOS-D, and MOS-T measure overall visual quality, fine-detail quality, and temporal stability, respectively. Best results are shown in bold.

	720p		1080p		1440p		4K
Method	FPS $\uparrow$	GB $\downarrow$	FPS $\uparrow$	GB $\downarrow$	FPS $\uparrow$	GB $\downarrow$	FPS $\uparrow$	GB $\downarrow$
RealViformer	31.88	5.44	14.87	12.36	8.69	21.66	3.88	39.03
DOVE	1.53	30.52	0.56	41.08	0.24	57.12	0.15 †	42.11 †
SeedVR2-3B	2.65	75.05	0.79 †	73.89 †	0.44 †	74.88 †	0.15 †	61.46 †
Stream-DiffVSR	1.76	10.24	0.85	25.92	0.47	61.67	0.17 †	44.39 †
FlashVSR Tiny	16.52	12.90	7.80	24.45	4.41	40.61	1.30	67.99

Appendix

Table C.3: Resolution scaling on one A100-80GB. GPU throughput and peak allocated memory for 201 input frames. Memory is reported in decimal GB.

Action	DiT layers	Layers	Slots per layer
$\emptyset$	1, 2, 4, 6, 9, 22, 25, 27, 29, 30	10	0
$W_{1}$	3, 28	2	1
$W_{2}$	5, 7, 13, 17, 21, 24, 26	7	2
$W_{4}$	8, 11, 20	3	4
$A$	15	1	2
$W_{2}+A$	10, 19, 23	3	4

Appendix

Table C.4: Deployed layer-wise historical cache policy. Layer indices are one-based; reserved slots count latent positions per layer.

Figure C.2: Additional visual comparisons. Each example shows the input frame with a marked region and the corresponding crops from VSR methods. The scenes include building facades, masonry, and construction machinery.

Explore similar work

Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion
Dec 29, 2025Hau-Shiang Shiu, Chin-Yang Lin, Zhixiang Wang +4Autoregressive Diffusion Iterative Denoising Methods
National Yang Ming Chiao Tung University · Shanda AI Research Tokyo · MediaTek Inc.
RelayVSR: Large-Small Model Collaboration for Efficient Real-World Video Super-Resolution
Sep 29, 2026Xijun Wang, Xin Li, Zirui Lang +3
University of Science and Technology of China
TRaM-VSR: Importance-Aware Token Routing and Merging for One-Step Diffusion Video Super-Resolution
Jul 24, 2026Sicheng Gao, Zhuyun Zhou, Yixuan Liu +3Diffusion Transformers Spatial Tokens
Advanced Micro Devices Inc. · Computer Vision Lab, CAIDAS & IFI, University of Würzburg

Dec 29, 2025, cs.CVJ/K move · Enter open · S save

Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion

Hau-Shiang Shiu, Chin-Yang Lin, Zhixiang Wang, Chi-Wei Hsiao+3

National Yang Ming Chiao Tung University · Shanda AI Research Tokyo · MediaTek Inc.

Autoregressive Diffusion Iterative Denoising Methods

Open paper AskPDF arXiv

Diffusion-based video super-resolution (VSR) methods deliver strong perceptual quality but are often unsuitable for latency-sensitive scenarios due to reliance on future frames and expensive multi-step denoising. We propose Stream-DiffVSR, a causally conditioned diffusion framework for efficient online VSR. Operating strictly on past frames, Stream-DiffVSR integrates a four-step distilled denoiser for fast inference, an Auto-regressive Temporal Guidance (ARTG) module that injects motion-aligned cues during latent denoising, and a lightweight temporal-aware decoder with a Temporal Processor Module (TPM) to enhance detail and temporal coherence. Unlike chunk-wise streaming inference, our strictly frame-by-frame causal design avoids sequence-level waiting, substantially reducing time-to-first-frame and end-to-end latency. Stream-DiffVSR processes 720p frames in 0.328 seconds on an RTX 4090 and consistently outperforms prior diffusion-based baselines. Compared with the online state-of-the-art TMP, it improves perceptual quality (LPIPS +0.095). Compared with prior diffusion-based VSR methods such as MGLD-VSR, it reduces per-frame runtime by over 130x. Moreover, Stream-DiffVSR substantially lowers time-to-first-frame for diffusion-based VSR, reducing initial delay from over 4600 seconds to 0.328 seconds, making diffusion-based VSR markedly more practical for low-latency online and streaming deployment. Project page: https://jamichss.github.io/stream-diffvsr-project-page/

Abstract

Explore similar work

Stream-DiffVSR: Low-Latency Streamable Video Super-Resolution via Auto-Regressive Diffusion

RelayVSR: Large-Small Model Collaboration for Efficient Real-World Video Super-Resolution

TRaM-VSR: Importance-Aware Token Routing and Merging for One-Step Diffusion Video Super-Resolution