cs.CVNov 16, 2026

ScoutCache: Efficient Few-Step Autoregressive Video Diffusion via Conditional Readout and Selective Caching

Authors: Hao Tan, Jinghui Qin, Xuming Hu

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Guangdong University of Technology · The Hong Kong University of Science and Technology

Abstract

Few-step autoregressive video diffusion enables efficient long video generation, but inference still incurs substantial computational costs from repeated denoising and KV cache operations. Existing acceleration methods reduce parts of this overhead, while historical KV reading and cache construction remain costly across video chunks. To address these challenges, we propose ScoutCache, a training-free framework that jointly optimizes denoising, historical KV reading, and cache construction. We observe that shallow feature changes provide useful information for estimating clean latent updates, while historical KV dependence varies across Transformer layers and attention heads but remains relatively stable across video chunks. Based on these observations, we develop Conditional Denoising Readout (CDR) to estimate intermediate denoising updates from shallow feature changes and known noise variations. We further introduce Output Risk Guided Historical KV Reading (ORHR) to allocate historical reading budgets based on attention output perturbations, and Content and Coverage Guided Sparse Cache Construction (CCSC) to reduce cache construction costs by selecting informative spatial tokens while maintaining spatial coverage. Extensive experiments on Self-Forcing and LongLive using VBench and MovieGen demonstrate that ScoutCache achieves up to 2.28×\times and 1.86×\times speedups, respectively, while consistently improving generation quality without modifying pretrained model parameters.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Motion-Aware Caching for Efficient Autoregressive Video Generation

    May 3, 2026Jing Xu, Yuexiao Ma, Xuzhe Zheng +7Video Diffusion ModelsKV-Cache Management

  2. In-Flight KV Cache with Clean Anchors for Faster Autoregressive Video Diffusion

    Sep 26, 2026Yikai Wang, Xiao Han, Mengmeng Xu +9Long-Horizon Video GenerationDiffusion Model Caching