In-Flight KV Cache with Clean Anchors for Faster Autoregressive Video Diffusion

Yikai Wang; Xiao Han; Mengmeng Xu; Juan Camilo Perez; Yiannis Douratsos; Sen He; Zijian Zhou; Fei Zhang

cs.CVSep 26, 2026

In-Flight KV Cache with Clean Anchors for Faster Autoregressive Video Diffusion

Authors: Yikai Wang, Xiao Han, Mengmeng Xu, Juan Camilo Perez, Yiannis Douratsos, Sen He, Zijian Zhou, Fei Zhang, +4 more

Organizations: Meta · S-Lab, Nanyang Technological University

Abstract

Few-step autoregressive video diffusion generates a long video by splitting the video into temporal chunks and generating chunk-by-chunk, each through a short sequence of denoising stages. To memorize chunks that are already generated, previous methods reconstruct a clean or less-noisy key--value (KV) cache by additional forwards to build the cache without advancing an output latent. However, every denoising forward itself already computes the in-flight KV of the current chunk. We introduce FlashForward, which directly reuses this cache to avoid the heavy cache-update-only model forwards. After the current chunk completes one denoising stage, its stage-specific cache is already available for the next chunk. Assigning one GPU to each stage therefore lets different chunks occupy different stages concurrently. This early availability has a quality cost: the resulting stage-matched history is noisy, causing appearance and motion drift among chunks. To complement it, FlashForward produces sparse auxiliary clean anchor latents before the corresponding region is generated so the generation trajectories can be stabilized by this two-sided conditioning. The two memories operate at different temporal scales: sparse clean anchor KV supplies coarse, long-range two-sided structural guidance, while dense stage-matched history preserves fine, recent evolution. With up to four GPUs, FlashForward runs $1.16$ -- $1.69\times$ faster than HiAR and $1.42$ -- $2.92\times$ faster than Self-Forcing for 16 FPS videos of 20 seconds or longer across 1.3B and 14B backbone scales at 480p and 720p. On VBench, for the 1.3B model at 480p, it achieves higher scores and remains stable at longer durations, demonstrating that FlashForward generates high-quality and temporally consistent videos across durations of 20s, 35s and 65s at a much faster generation speed.

Figures & tables

Figure 1: FlashForward redesigns the autoregressive diffusion generation pipeline to remove heavy cache-update-only forwards, significantly reducing generation latency . (a) Compared with pipelines that reconstruct clean (Self-Forcing [ 21 ] ) or less-noisy (HiAR [ 67 ] ) KV through context encoding that does not advance an output latent, FlashForward substantially reduces latency while preserving generation quality and coherence, even in long videos. (b) FlashForward adopts a planner to autoregressively produce sparse auxiliary anchors for two-sided temporal conditioning. A renderer can then run in standard autoregressive form conditioned on these anchors such that every ordinary forward advances one chunk and publishes stage-matched history for later chunks, allowing different denoising stages to run on different GPUs. (c) Quality versus latency for 1.3B models on 480p videos, using each method’s fastest setup with up to four GPUs. See the project page for an interactive demo.

Method	Memory noise level	When it becomes usable	Cache-update-only forwards	Chunk execution
Self-Forcing	$\sigma_{0}$	After the chunk is re-encoded	$1$ per chunk	Serial
N-C-Causal-rCM	$\sigma_{1}\rightarrow\sigma_{0}$	After the last denoising stage	$0$	Serial
HiAR	$\sigma_{k-1}$	After each stage-specific re-encoding	$S$ per chunk	Overlapped
FlashForward	$\sigma_{k}\rightarrow\sigma_{k-1}$	Immediately after each denoising stage	$0$	Overlapped

Table 1: How memory determines execution for

S

denoising stages with noise levels

\sigma_{S}>\cdots>\sigma_{0}=0

. FlashForward additionally uses sparse clean anchor KV prepared before generation.

Figure 2: (a) The planner produces sparse auxiliary anchor latents; local three-anchor windows guide the renderer. (b) One renderer forward reads clean anchor KV and stage-matched renderer history, predicts its latent update, and publishes its own KV. (c) These dependencies form a wavefront that completes one chunk per pipeline round after warm-up. See the concrete generation example in Sec. 3 . See the project page for an interactive demo.

Figure 3: (a) Shared backbone with role-specific embeddings and LoRA adapters. (b) Packed supervised fine-tuning (SFT) trains both roles in one forward. (c) Self-rollout distillation adapts the model to long generated contexts. See the project page for an interactive demo.

Figure 4: Average denoising latency on four-GPUs and our five-GPU streaming version with standard deviations. Tab. 4 reports exact numbers. See the project page for an interactive demo.

VBench-1.0	LTX-	Wan2.1-	NOVA	Pyramid	SkyReels-	MAGI-1-	CausVid	Self-	Causal	HiAR	FlashForward
Metric	Video	T2V-1.3B	NOVA	Flow	V2-1.3B	4.5B	CausVid	Forcing	Forcing	HiAR	FlashForward
Total $\uparrow$	0.766	0.802	0.773	0.775	0.788	0.757	0.764	0.805	0.810	0.821	0.838
Quality $\uparrow$	0.789	0.813	0.777	0.804	0.808	0.785	0.771	0.829	0.837	0.846	0.859
Semantic $\uparrow$	0.685	0.766	0.757	0.670	0.707	0.647	0.740	0.708	0.701	0.723	0.753

Table 2: VBench-1.0 on 20 s videos (top; 5 s for bidirectional models) and VBench-Long (bottom).

Figure 5: Qualitative comparison at 20, 35, and 65 s. Blue boxes indicate anchor frames. The baselines have no anchors. See full prompt and video comparisons on the project page .

	SFT phase (50-step sampling)			Distillation phase (4-step sampling)
Ablation	Total $\uparrow$	Quality $\uparrow$	Semantic $\uparrow$	Total $\uparrow$	Quality $\uparrow$	Semantic $\uparrow$
Dual-role SFT	0.8172	0.8371	0.7376	0.8184	0.8332	0.7593
$+$ time-rebased supervision	0.8150	0.8345	0.7371	0.8198	0.8338	0.7638
$+$ role-specific embedding and LoRA	0.8205	0.8377	0.7516	0.8380	0.8593	0.7528

Table 3: Top: training ablations. Bottom: conditioning-state ablations.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

	Wan2.1-T2V-1.3B (H100)
	480p, $1{\times}$ GPU				480p, multi-GPU				720p, multi-GPU
Schedule	5	20	35	65	5	20	35	65	5	20	35	65
Bidirectional [-0.15em]	3.31 [-0.15em] $\pm$ 0.00	32.02 [-0.15em] $\pm$ 0.01	91.22 [-0.15em] $\pm$ 0.04	295.83 [-0.15em] $\pm$ 0.12	1.13 [-0.15em] $\pm$ 0.01	8.95 [-0.15em] $\pm$ 0.01	24.01 [-0.15em] $\pm$ 0.02	75.28 [-0.15em] $\pm$ 0.05	3.74 [-0.15em] $\pm$ 0.00	39.59 [-0.15em] $\pm$ 0.06	115.79 [-0.15em] $\pm$ 0.04	379.17 [-0.15em] $\pm$ 0.64
Self-Forcing [-0.15em]	3.69 [-0.15em] $\pm$ 0.01	17.44 [-0.15em] $\pm$ 0.11	31.48 [-0.15em] $\pm$ 0.14	57.89 [-0.15em] $\pm$ 0.20	5.10 [-0.15em] $\pm$ 0.05	20.25 [-0.15em] $\pm$ 0.10	35.40 [-0.15em] $\pm$ 0.31	65.14 [-0.15em] $\pm$ 0.55	6.04 [-0.15em] $\pm$ 0.05	27.17 [-0.15em] $\pm$ 0.11	47.71 [-0.15em] $\pm$ 0.27	97.68 [-0.15em] $\pm$ 0.48
HiAR [-0.15em]	5.57 [-0.15em] $\pm$ 0.04	27.86 [-0.15em] $\pm$ 0.03	50.41 [-0.15em] $\pm$ 0.20	94.74 [-0.15em] $\pm$ 0.15	2.06 [-0.15em] $\pm$ 0.05	7.67 [-0.15em] $\pm$ 0.00	13.31 [-0.15em] $\pm$ 0.01	24.67 [-0.15em] $\pm$ 0.03	6.49 [-0.15em] $\pm$ 0.01	26.82 [-0.15em] $\pm$ 0.02	46.94 [-0.15em] $\pm$ 0.03	86.85 [-0.15em] $\pm$ 0.06
FlashForward [-0.15em]	4.56 [-0.15em] $\pm$ 0.07	17.77 [-0.15em] $\pm$ 0.14	32.80 [-0.15em] $\pm$ 0.21	59.64 [-0.15em] $\pm$ 0.46	2.21 [-0.15em] $\pm$ 0.08	6.32 [-0.15em] $\pm$ 0.02	10.79 [-0.15em] $\pm$ 0.22	21.29 [-0.15em] $\pm$ 0.08	5.71 [-0.15em] $\pm$ 0.05	19.33 [-0.15em] $\pm$ 0.27	35.14 [-0.15em] $\pm$ 0.13	66.53 [-0.15em] $\pm$ 0.12

Appendix

Table 4: Per-video denoising-path latency in seconds (lower is better). Each cell is the mean over videos with its standard deviation beneath it. Each setting spans four output lengths: 5, 20, 35, and 65 seconds, which are 21, 81, 141, and 261 latents at 16 FPS. The two FlashForward planner allocations are both measured in every four-GPU cell. All multi-GPU columns use four GPUs unless marked otherwise; FlashForward-Streaming uses one planner GPU and four renderer GPUs. Boldface identifies the fastest schedule at an equal device count and therefore excludes the five-GPU row.

	Wan2.1-T2V-1.3B (H100)
	480p, $1{\times}$ GPU				480p, $4{\times}$ GPU				720p, $4{\times}$ GPU
Schedule	5	20	35	65	5	20	35	65	5	20	35	65
Bidirectional	22.1	31.7	41.2	60.3	15.7	18.4	21.1	26.5	17.0	23.2	29.4	41.8
Self-Forcing	24.9	24.9	25.0	25.1	20.6	20.6	20.7	20.7	28.1	28.2	28.3	28.5
HiAR	24.9	24.9	25.0	25.1	24.9	24.9	25.0	25.1	32.9	33.0	33.1	33.3
FlashForward	24.9	25.0	25.2	25.4	24.9	24.9	25.0	25.1	32.9	33.0	33.1	33.3

Appendix

Table 5: Peak allocated memory per rank in GiB.

Figure 6: Forward-count break-even boundary between FlashForward and HiAR when the planner uses context parallelism, evaluated in the sixteen four-GPU cells of Tab. 4 . Marker shape denotes the system setting; marker color reports whether FlashForward is the fastest measured schedule among FlashForward, HiAR, and Self-Forcing, using the same two colors as the predicted regions containing the markers. The two gray markers are the cells in which FlashForward does not win. A marker whose color disagrees with its region is a cell in which the pairwise boundary alone does not recover the three-way winner. Both outliers occur at a length of 21 latents, suggesting that FlashForward is not well amortized for short sequences in these settings.

Figure 7: Color seams at the anchor positions of 20-second videos. The two splice rows keep the planner anchor latents in the final video and build the anchor KV with the planner or with the renderer, as in Panel (a) of Tab. 6 ; FlashForward uses the planner anchors only as conditioning and lets the renderer generate every output position. In each block, the boxed column is the last frame decoded from an anchor latent, and the outer columns are the nearest non-anchor frames before and after it. Between each non-anchor frame and the anchor frame, we show their per-channel absolute RGB difference, computed after a Gaussian blur and amplified $16\times$ . The blur suppresses edges and fine texture, leaving mostly low-frequency color changes; large moving regions can remain visible.

(a) Compared modes: KV source and final anchor source
Denoising steps	Final anchor	Anchor KV	Anchor seam	Chunk seam	Interior	Periodic
50	Planner	Planner	3.93	2.71	1.32	7.66
50	Planner	Renderer	3.34	2.37	1.29	5.99
50	Renderer (ours)	Planner	1.83	2.27	1.32	3.54
4	Planner	Renderer	6.20	1.68	1.40	9.55
4	Renderer (ours)	Planner	2.43	2.50	1.26	5.61

Appendix

Table 6: All-position renderer output leads to smaller seam measurements. Panel (a) compares anchor-KV and final-anchor sources. All seam columns are normalized adjacent-frame luminance-jump statistics, and lower is better: Anchor seam is measured at anchor/non-anchor transitions, Chunk seam at renderer-chunk boundaries away from anchors, and Interior at transitions of neither type. “Periodic” sums the excess over Interior across the transitions in one anchor period. Panel (b) reports mean luminance levels for planner-generated anchor frames and their neighboring renderer-generated frames. Panel (c) compares real and generated anchors over paired clips; chromatic share is the fraction of the RGB offset orthogonal to the grayscale direction

(1,1,1)

, and spatial excess is

\sqrt{s_{\mathrm{anchor}}^{2}-s_{\mathrm{control}}^{2}}

, where

s

is the spatial standard deviation of the per-cell RGB change. The paired

t

-statistic compares each clip’s anchor-boundary offset with its mid-chunk control.

Explore similar work

Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion
May 18, 2026Peiliang Cai, Evelyn Zhang, Jiacheng Liu +8Autoregressive Video Diffusion Models Keyframe Selection
1SJTU · 2SDU · 3HUST +4
Forcing-KV: Hybrid KV Cache Compression for Efficient Autoregressive Video Diffusion Models
May 10, 2026Yicheng Ji, Zhizhou Zhong, Jun Zhang +7Autoregressive Video Diffusion Models Video Diffusion Models
ZJU · Video Rebirth · HKUST +1
TetherCache: Stabilizing Autoregressive Long-Form Video Generation with Gated Recall and Trusted Alignment
Jun 11, 2026Yu Meng, Xiangyang Luo, Letian Li +5Autoregressive Video Diffusion Models Autoregressive Video Generation
Tsinghua University · D-INFK, ETH Zürich

May 18, 2026, cs.CVJ/K move · Enter open · S save

Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion

Peiliang Cai, Evelyn Zhang, Jiacheng Liu, Hao Lin+7

1SJTU · 2SDU · 3HUST+4

Autoregressive Video Diffusion Models Keyframe Selection Key-Value Cache

Open paper AskPDF arXiv

Recent advances in autoregressive video diffusion have enabled sequential and streaming video generation. However, long-horizon generation requires increasingly large KV caches, making efficient compression without sacrificing quality challenging. Existing methods mostly select historical frames based on attention scores, but their context decisions remain coarse. When multiple frames are generated in the same chunk, these methods often apply a shared history selection to the whole chunk, score historical frames solely by attention, and assign head-wise budgets either uniformly or by attention-pattern heuristics rather than explicit head-importance estimation. We show that frames within the same generated chunk can depend on distinct historical frames, that the same historical frame can receive different attention scores as its relative temporal distance to the current frames changes, and that masking different heads induces unequal generation degradation. Motivated by these findings, we propose \textbf{Focused Forcing}, a training-free KV selection method that focuses cached history along both generated-frame and head dimensions. For each generated frame, Focused Forcing preserves the most relevant and distinctive historical frames by combining attention scores with diversity scores of historical frames, while assigning larger budgets to heads with higher estimated importance. Across multiple autoregressive generation paradigms, Focused Forcing achieves up to

\textbf{1.48}\times

end-to-end acceleration without training, while \textbf{improving visual quality and text alignment}. \textit{Our code will be released on GitHub.}

Abstract

Explore similar work

Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion

Forcing-KV: Hybrid KV Cache Compression for Efficient Autoregressive Video Diffusion Models

TetherCache: Stabilizing Autoregressive Long-Form Video Generation with Gated Recall and Trusted Alignment