ScoutCache: Efficient Few-Step Autoregressive Video Diffusion via Conditional Readout and Selective Caching
Authors: Hao Tan, Jinghui Qin, Xuming Hu
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Guangdong University of Technology · The Hong Kong University of Science and Technology
Few-step autoregressive video diffusion enables efficient long video generation, but inference still incurs substantial computational costs from repeated denoising and KV cache operations. Existing acceleration methods reduce parts of this overhead, while historical KV reading and cache construction remain costly across video chunks. To address these challenges, we propose ScoutCache, a training-free framework that jointly optimizes denoising, historical KV reading, and cache construction. We observe that shallow feature changes provide useful information for estimating clean latent updates, while historical KV dependence varies across Transformer layers and attention heads but remains relatively stable across video chunks. Based on these observations, we develop Conditional Denoising Readout (CDR) to estimate intermediate denoising updates from shallow feature changes and known noise variations. We further introduce Output Risk Guided Historical KV Reading (ORHR) to allocate historical reading budgets based on attention output perturbations, and Content and Coverage Guided Sparse Cache Construction (CCSC) to reduce cache construction costs by selecting informative spatial tokens while maintaining spatial coverage. Extensive experiments on Self-Forcing and LongLive using VBench and MovieGen demonstrate that ScoutCache achieves up to 2.28× and 1.86× speedups, respectively, while consistently improving generation quality without modifying pretrained model parameters.
Figures & tables
Figure 1: Motivating observations for ScoutCache. (a) Changes in shallow features exhibit spatial correspondence with changes in clean latent estimates, while shuffling spatial positions disrupts this correlation. (b) Dependence on the historical KV cache varies across Transformer layers and attention heads but remains relatively stable across video chunks.
Figure 2: The top shows the overall inference pipeline, integrating denoising acceleration and KV cache optimization. The bottom illustrates three components. (a) CDR estimates clean latent updates from noise variations and shallow feature changes. (b) ORHR allocates historical KV reading budgets based on attention output perturbations. (c) CCSC reduces cache construction costs through content and coverage guided spatial token selection.
Method
VBench (5s)
MovieGen (30s)
MovieGen (60s)
MovieGen (120s)
Latency ↓
FPS ↑
Speedup ↑
Total ↑
Latency ↓
FPS ↑
Speedup ↑
Total ↑
Latency ↓
FPS ↑
Speedup ↑
Total ↑
Latency ↓
FPS ↑
Speedup ↑
Total ↑
Self-Forcing (NeurIPS’25)
9.00
8.89
1.00 ×
81.52
65.59
7.32
1.00 ×
79.10
131.90
7.28
1.00 ×
77.76
264.32
7.26
1.00 ×
75.03
FlowCache (ICLR’26)
7.22
11.08
1.25 ×
81.56 ▲
52.82
9.09
1.24 ×
78.45
105.71
9.08
1.25 ×
76.60
212.24
9.05
1.25 ×
74.62
TeaCache (CVPR’25)
7.39
10.83
1.22 ×
81.07
54.05
8.88
1.21 ×
78.54
109.33
8.78
1.21 ×
77.20
218.52
8.79
1.21 ×
74.77
MotionCache-Slow (ICML’26)
7.53
10.62
1.20 ×
68.92
55.35
8.67
1.19 ×
68.97
111.68
8.60
1.18 ×
68.71
223.17
8.60
1.18 ×
68.56
MotionCache-Fast (ICML’26)
6.54
12.24
1.38 ×
66.86
46.76
10.27
1.40 ×
68.72
94.80
10.13
1.39 ×
68.57
186.26
10.31
1.42 ×
68.91
Table 1: Quantitative comparison on Self-Forcing across different video durations. Latency is measured in seconds. ▲ indicates a Total Score higher than the original backbone. The best and second best results are highlighted in bold and underline , respectively.
Method
VBench (5s)
MovieGen (30s)
MovieGen (60s)
MovieGen (120s)
Latency ↓
FPS ↑
Speedup ↑
Total ↑
Latency ↓
FPS ↑
Speedup ↑
Total ↑
Latency ↓
FPS ↑
Speedup ↑
Total ↑
Latency ↓
FPS ↑
Speedup ↑
Total ↑
LongLive (ICLR’26)
8.58
9.32
1.00 ×
79.63
54.77
8.76
1.00 ×
80.35
108.38
8.86
1.00 ×
80.04
215.83
8.90
1.00 ×
80.06
FlowCache (ICLR’26)
7.66
10.44
1.12 ×
79.70 ▲
48.59
9.88
1.13 ×
80.09
96.64
9.93
1.12 ×
79.99
193.45
9.92
1.12 ×
79.88
TeaCache (CVPR’25)
6.91
11.57
1.24 ×
80.14 ▲
44.07
10.89
1.24 ×
79.89
87.95
10.91
1.23 ×
79.58
175.25
10.96
1.23 ×
79.68
SVG2 (NeurIPS’25)
11.14
7.18
0.77 ×
79.79 ▲
66.57
7.21
0.82 ×
79.37
132.91
7.22
0.82 ×
79.37
266.31
7.21
0.81 ×
79.42
Dummy Forcing (arXiv’26)
6.43
12.44
1.33 ×
79.50
37.70
12.73
1.45 ×
79.64
74.98
12.80
1.45 ×
79.43
149.04
12.88
1.45 ×
79.30
Table 2: Quantitative comparison on LongLive across different video durations. Latency is measured in seconds. ▲ indicates a Total Score higher than the original backbone. The best and second best results are highlighted in bold and underline , respectively.
Configuration
Latency ↓
Speedup ↑
Total ↑
Main Ablation: Self-Forcing, 30s
Original Backbone
65.59
1.00 ×
79.10
Fixed History (w/o All)
39.47
1.66 ×
80.09
CDR Only
32.23
2.04 ×
79.82
w/o CDR
36.29
1.81 ×
80.22
w/o ORHR
30.58
2.15 ×
79.83
Table 3: Ablation studies of ScoutCache on MovieGen. Additional evaluations verify the component contributions across video durations and backbones. Latency is measured in seconds.
Figure 3: Qualitative comparison of video generation results. We show four sampled frames for each prompt, comparing ScoutCache with the original backbones, Dummy Forcing, and Forcing-KV.
Figure 4: Chunkwise diffusion latency on Self-Forcing and LongLive. (a,b) Latency over the first 12 chunks, showing calibration and subsequent acceleration. (c,d) Latency across 160 chunks.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Hyperparameter analysis of ScoutCache. (a,b) Tradeoff between Total Score and speedup under different normalized historical reading budgets B/(LH) on Self-Forcing and LongLive. (c-e) Effects of shallow feature projection dimension rs , staleness threshold A , and regularization scale λ/m on generation quality and speedup using Self-Forcing.
Method
Denoising ↓
KV Build ↓
Other ↓
Total ↓
VBench (5s) – Self-Forcing
Dummy Forcing
5.132
1.297
0.050
6.48
Forcing-KV
5.155
1.327
0.027
6.51
ScoutCache
4.385
1.167
0.090
5.64
VBench (5s) – LongLive
Dummy Forcing
5.052
1.278
0.104
6.43
Appendix
Table 4: Breakdown of diffusion generation time across different video durations on Self-Forcing and LongLive. All times are measured in seconds. Denoising includes historical KV reading, while Other includes online calibration and remaining overhead.
Strategy
Tokens ↓
Self-Forcing ↓
LongLive ↓
Dense Construction
4,680
194.55
194.06
Post Compression
4,680
201.41
201.25
CCSC (Ours)
3,510
151.23
151.27
Appendix
Table 5: Comparison of KV cache construction strategies. Latency is measured in milliseconds.
Figure 6: Mechanism analysis of CDR. (a) Spatial distribution of prediction error reduction compared with no update. (b) Contributions of known noise variations and shallow feature changes to error reduction. (c) Normalized update errors under different input configurations with the same frozen readout, showing the benefit of combining both inputs.
Figure 7: Mechanism analysis of ORHR. (a) Unequal historical KV dependence across Transformer layers and heads. (b) Attention output errors under uniform reading and ORHR with the same budget of 720 frame accesses. (c) Output NMSE across subsequent video chunks using the early calibrated allocation, demonstrating consistently lower errors without recalibration.
Figure 8: Mechanism analysis of CCSC. (a) Proxy mismatch under fixed spatial positions and content guided selection. (b) Spatial coverage under equal selection budgets, showing how spatial grouping avoids uncovered regions. (c) Accumulated coverage across latent frames, demonstrating how coverage priority refreshes previously unvisited positions.
Figure 9: Analysis of historical KV reading effects on generation quality. (a) Consecutive reference frames and spatial changes in previous-frame bias after restoring omitted historical KV reads. Blue and orange indicate increased and decreased bias, respectively. (b) Distribution of bias changes on Self-Forcing and LongLive, with percentages indicating the proportion of videos showing increased bias.
Metric
Self- Forcing
Flow Cache
Tea Cache
MotionCache (Slow)
MotionCache (Fast)
SVG2
Light Forcing
Dummy Forcing
Forcing- KV
ScoutCache
VBench (5s)
Quality Score
84.06
84.04
83.61
72.96
72.73
83.41
83.44
83.56
84.34
84.38
Semantic Score
71.36
71.64
70.92
52.75
43.40
70.70
71.66
70.95
70.92
71.28
Total Score
81.52
81.56
81.07
68.92
66.86
80.87
81.08
81.04
81.65
81.76
MovieGen (30s)
Subject Consistency
96.74
96.15
96.36
92.43
91.18
95.16
96.47
96.42
95.91
96.01
Appendix
Table 6: Detailed video quality scores on Self-Forcing. The 5s results report the VBench Quality, Semantic, and Total scores. The 30s, 60s, and 120s results report six VBench-Long dimensions and the Total Score.
Metric
LongLive
Flow Cache
Tea Cache
SVG2
Dummy Forcing
Forcing- KV
ScoutCache
VBench (5s)
Quality Score
81.83
81.78
82.36
82.03
81.55
82.66
82.14
Semantic Score
70.84
71.36
71.23
70.78
71.25
71.28
70.71
Total Score
79.63
79.70
80.14
79.79
79.50
80.39
79.85
MovieGen (30s)
Subject Consistency
97.59
97.46
97.54
96.91
97.92
97.58
98.04
Appendix
Table 7: Detailed video quality scores on LongLive. The 5s results report the VBench Quality, Semantic, and Total scores. The 30s, 60s, and 120s results report six VBench-Long dimensions and the Total Score.
Figure 10: Additional qualitative comparison of video generation results.
Autoregressive video generation paradigms offer theoretical promise for long video synthesis, yet their practical deployment is hindered by the computational burden of sequential iterative denoising. While cache reuse strategies can accelerate generation by skipping redundant denoising steps, existing methods rely on coarse-grained chunk-level skipping that fails to capture fine-grained pixel dynamics. This oversight is critical: pixels with high motion require more denoising steps to prevent error accumulation, while static pixels tolerate aggressive skipping. We formalize this insight theoretically by linking cache errors to residual instability, and propose MotionCache, a motion-aware cache framework that exploits inter-frame differences as a lightweight proxy for pixel-level motion characteristics. MotionCache employs a coarse-to-fine strategy: an initial warm-up phase establishes semantic coherence, followed by motion-weighted cache reuse that dynamically adjusts update frequencies per token. Extensive experiments on state-of-the-art models like SkyReels-V2 and MAGI-1 demonstrate that MotionCache achieves significant speedups of 6.28× and 1.64× respectively, while effectively preserving generation quality (VBench: 1%↓ and 0.01%↓ respectively). The code is available at https://github.com/ywlq/MotionCache.
Jing Xu, Yuexiao Ma, Xuzhe Zheng +7
1Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China · 2ByteDance · 3Max Planck Institute for Intelligent Systems, 4ELLIS Institute Tbingen, 5Tbingen AI Center
Autoregressive (AR) video generation enables long-horizon synthesis but requires repeated forward passes through large diffusion transformers. Training-free caching reduces this cost by reusing intermediate computation, yet existing controllers often couple two decisions: when to synchronize the cache exactly and how much computation to repair between synchronizations. We introduce Frequency-Step Cache (FSC), which separates these roles. FSC schedules exact full-computation anchors on a dense-to-sparse trajectory, while a lightweight latent-space risk based on frequency-resolved drift and token age controls inter-anchor repair through whole-step replay, selective token recomputation, or full repair. On SkyReels-V2, whole-step reuse provides most of the acceleration, while selective token computation is most effective as fidelity repair. A controlled ablation shows that using one reactive signal for both synchronization and repair reduces fidelity at matched latency, with the same separation advantage observed on FramePack and Matrix-Game. Across multiple autoregressive video backbones, FSC improves speed-quality trade-offs. On SkyReels-V2, FSC-Speed reaches 2.36x speedup versus MotionCache's 2.275x while improving SSIM by 0.019 and PSNR by 0.5 dB. At MotionCache-matched speed, FSC-Quality improves SSIM by 0.031 and PSNR by 1.0 dB and lowers LPIPS by 0.024. Without retuning, FSC-Quality transfers to SkyReels-V2 14B at 2.12x speedup versus 1.79x for MotionCache.
Few-step autoregressive video diffusion generates a long video by splitting the video into temporal chunks and generating chunk-by-chunk, each through a short sequence of denoising stages. To memorize chunks that are already generated, previous methods reconstruct a clean or less-noisy key--value (KV) cache by additional forwards to build the cache without advancing an output latent. However, every denoising forward itself already computes the in-flight KV of the current chunk. We introduce FlashForward, which directly reuses this cache to avoid the heavy cache-update-only model forwards. After the current chunk completes one denoising stage, its stage-specific cache is already available for the next chunk. Assigning one GPU to each stage therefore lets different chunks occupy different stages concurrently. This early availability has a quality cost: the resulting stage-matched history is noisy, causing appearance and motion drift among chunks. To complement it, FlashForward produces sparse auxiliary clean anchor latents before the corresponding region is generated so the generation trajectories can be stabilized by this two-sided conditioning. The two memories operate at different temporal scales: sparse clean anchor KV supplies coarse, long-range two-sided structural guidance, while dense stage-matched history preserves fine, recent evolution. With up to four GPUs, FlashForward runs 1.16--1.69× faster than HiAR and 1.42--2.92× faster than Self-Forcing for 16 FPS videos of 20 seconds or longer across 1.3B and 14B backbone scales at 480p and 720p. On VBench, for the 1.3B model at 480p, it achieves higher scores and remains stable at longer durations, demonstrating that FlashForward generates high-quality and temporally consistent videos across durations of 20s, 35s and 65s at a much faster generation speed.