ScoutCache: Efficient Few-Step Autoregressive Video Diffusion via Conditional Readout and Selective Caching
Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Guangdong University of Technology · The Hong Kong University of Science and Technology
Abstract
Few-step autoregressive video diffusion enables efficient long video generation, but inference still incurs substantial computational costs from repeated denoising and KV cache operations. Existing acceleration methods reduce parts of this overhead, while historical KV reading and cache construction remain costly across video chunks. To address these challenges, we propose ScoutCache, a training-free framework that jointly optimizes denoising, historical KV reading, and cache construction. We observe that shallow feature changes provide useful information for estimating clean latent updates, while historical KV dependence varies across Transformer layers and attention heads but remains relatively stable across video chunks. Based on these observations, we develop Conditional Denoising Readout (CDR) to estimate intermediate denoising updates from shallow feature changes and known noise variations. We further introduce Output Risk Guided Historical KV Reading (ORHR) to allocate historical reading budgets based on attention output perturbations, and Content and Coverage Guided Sparse Cache Construction (CCSC) to reduce cache construction costs by selecting informative spatial tokens while maintaining spatial coverage. Extensive experiments on Self-Forcing and LongLive using VBench and MovieGen demonstrate that ScoutCache achieves up to 2.28 and 1.86 speedups, respectively, while consistently improving generation quality without modifying pretrained model parameters.
Figures & tables
| Method | VBench (5s) | MovieGen (30s) | MovieGen (60s) | MovieGen (120s) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Latency | FPS | Speedup | Total | Latency | FPS | Speedup | Total | Latency | FPS | Speedup | Total | Latency | FPS | Speedup | Total | |
| Self-Forcing (NeurIPS’25) | 9.00 | 8.89 | 1.00 | 81.52 | 65.59 | 7.32 | 1.00 | 79.10 | 131.90 | 7.28 | 1.00 | 77.76 | 264.32 | 7.26 | 1.00 | 75.03 |
| FlowCache (ICLR’26) | 7.22 | 11.08 | 1.25 | 81.56 | 52.82 | 9.09 | 1.24 | 78.45 | 105.71 | 9.08 | 1.25 | 76.60 | 212.24 | 9.05 | 1.25 | 74.62 |
| TeaCache (CVPR’25) | 7.39 | 10.83 | 1.22 | 81.07 | 54.05 | 8.88 | 1.21 | 78.54 | 109.33 | 8.78 | 1.21 | 77.20 | 218.52 | 8.79 | 1.21 | 74.77 |
| MotionCache-Slow (ICML’26) | 7.53 | 10.62 | 1.20 | 68.92 | 55.35 | 8.67 | 1.19 | 68.97 | 111.68 | 8.60 | 1.18 | 68.71 | 223.17 | 8.60 | 1.18 | 68.56 |
| MotionCache-Fast (ICML’26) | 6.54 | 12.24 | 1.38 | 66.86 | 46.76 | 10.27 | 1.40 | 68.72 | 94.80 | 10.13 | 1.39 | 68.57 | 186.26 | 10.31 | 1.42 | 68.91 |
| Method | VBench (5s) | MovieGen (30s) | MovieGen (60s) | MovieGen (120s) | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Latency | FPS | Speedup | Total | Latency | FPS | Speedup | Total | Latency | FPS | Speedup | Total | Latency | FPS | Speedup | Total | |
| LongLive (ICLR’26) | 8.58 | 9.32 | 1.00 | 79.63 | 54.77 | 8.76 | 1.00 | 80.35 | 108.38 | 8.86 | 1.00 | 80.04 | 215.83 | 8.90 | 1.00 | 80.06 |
| FlowCache (ICLR’26) | 7.66 | 10.44 | 1.12 | 79.70 | 48.59 | 9.88 | 1.13 | 80.09 | 96.64 | 9.93 | 1.12 | 79.99 | 193.45 | 9.92 | 1.12 | 79.88 |
| TeaCache (CVPR’25) | 6.91 | 11.57 | 1.24 | 80.14 | 44.07 | 10.89 | 1.24 | 79.89 | 87.95 | 10.91 | 1.23 | 79.58 | 175.25 | 10.96 | 1.23 | 79.68 |
| SVG2 (NeurIPS’25) | 11.14 | 7.18 | 0.77 | 79.79 | 66.57 | 7.21 | 0.82 | 79.37 | 132.91 | 7.22 | 0.82 | 79.37 | 266.31 | 7.21 | 0.81 | 79.42 |
| Dummy Forcing (arXiv’26) | 6.43 | 12.44 | 1.33 | 79.50 | 37.70 | 12.73 | 1.45 | 79.64 | 74.98 | 12.80 | 1.45 | 79.43 | 149.04 | 12.88 | 1.45 | 79.30 |
| Configuration | Latency | Speedup | Total |
|---|---|---|---|
| Main Ablation: Self-Forcing, 30s | |||
| Original Backbone | 65.59 | 1.00 | 79.10 |
| Fixed History (w/o All) | 39.47 | 1.66 | 80.09 |
| CDR Only | 32.23 | 2.04 | 79.82 |
| w/o CDR | 36.29 | 1.81 | 80.22 |
| w/o ORHR | 30.58 | 2.15 | 79.83 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Denoising | KV Build | Other | Total |
|---|---|---|---|---|
| VBench (5s) – Self-Forcing | ||||
| Dummy Forcing | 5.132 | 1.297 | 0.050 | 6.48 |
| Forcing-KV | 5.155 | 1.327 | 0.027 | 6.51 |
| ScoutCache | 4.385 | 1.167 | 0.090 | 5.64 |
| VBench (5s) – LongLive | ||||
| Dummy Forcing | 5.052 | 1.278 | 0.104 | 6.43 |
| Strategy | Tokens | Self-Forcing | LongLive |
|---|---|---|---|
| Dense Construction | 4,680 | 194.55 | 194.06 |
| Post Compression | 4,680 | 201.41 | 201.25 |
| CCSC (Ours) | 3,510 | 151.23 | 151.27 |
| Metric | Self- Forcing | Flow Cache | Tea Cache | MotionCache (Slow) | MotionCache (Fast) | SVG2 | Light Forcing | Dummy Forcing | Forcing- KV | ScoutCache |
|---|---|---|---|---|---|---|---|---|---|---|
| VBench (5s) | ||||||||||
| Quality Score | 84.06 | 84.04 | 83.61 | 72.96 | 72.73 | 83.41 | 83.44 | 83.56 | 84.34 | 84.38 |
| Semantic Score | 71.36 | 71.64 | 70.92 | 52.75 | 43.40 | 70.70 | 71.66 | 70.95 | 70.92 | 71.28 |
| Total Score | 81.52 | 81.56 | 81.07 | 68.92 | 66.86 | 80.87 | 81.08 | 81.04 | 81.65 | 81.76 |
| MovieGen (30s) | ||||||||||
| Subject Consistency | 96.74 | 96.15 | 96.36 | 92.43 | 91.18 | 95.16 | 96.47 | 96.42 | 95.91 | 96.01 |
| Metric | LongLive | Flow Cache | Tea Cache | SVG2 | Dummy Forcing | Forcing- KV | ScoutCache |
|---|---|---|---|---|---|---|---|
| VBench (5s) | |||||||
| Quality Score | 81.83 | 81.78 | 82.36 | 82.03 | 81.55 | 82.66 | 82.14 |
| Semantic Score | 70.84 | 71.36 | 71.23 | 70.78 | 71.25 | 71.28 | 70.71 |
| Total Score | 79.63 | 79.70 | 80.14 | 79.79 | 79.50 | 80.39 | 79.85 |
| MovieGen (30s) | |||||||
| Subject Consistency | 97.59 | 97.46 | 97.54 | 96.91 | 97.92 | 97.58 | 98.04 |
Diffusion Model Inference Acceleration
Diffusion model inference acceleration reduces the time and compute needed to generate or reconstruct outputs while preserving quality.
See all 534 papers