dKV Forcing: Separating Generation and Memory Representations through a Differentiable KV Cache

Zhiqi Li; Bo Zhu

cs.CVNov 1, 2026

dKV Forcing: Separating Generation and Memory Representations through a Differentiable KV Cache

Authors: Zhiqi Li, Bo Zhu

Organizations: Georgia Institute of Technology

Abstract

A causal video generator produces two key--value (KV) states for every segment: the KV it uses while denoising that segment, and the KV it writes into the cache for later segments to read. In a pretrained generator the two already differ substantially, later segments attend to them differently, and replacing the written KV with the generation KV degrades the continuation. We propose dKV Forcing (dKV denotes differentiable KV) to learn a dedicated memory representation from its future use: a writer-specific low-rank residual on the shared backbone separates memory writing from generation and is trained through the future reads of what it writes. Learning a write from its future use needs gradients through the cache, which the mutable cache in common use cannot provide. dKV Forcing replaces the training-time cache by persistent differentiable pages with local rematerialization and, in one temporal traversal, computes exactly the gradient that a two-pass reconstruction of the same objective computes. One traversal computes the full memory gradient at $7\%$ generator-update overhead over detached-write training, with $28\%$ less resident memory and $20.8\%$ shorter training cycles than the two-pass reconstruction. Over Self Gradient Forcing (SGF), the same objective trained with the same traversal, writer adaptation raises VBench-Long aesthetic quality by $6.6$ points at 60,s and $8.5$ points at 240,s with the highest subject and background consistency, and rollout dropout of the residual keeps most of this gain while restoring the motion of training without write gradients.

Figures & tables

Panel of Figure 1 — Figure 1: Trained on 5 s clips, dKV Forcing generates up to four minutes. The top two rows are 60 s generations and the bottom two are 240 s generations, 12 and 46 times the training clip, with the time stamp drawn in each frame. The prompts are a cat and a dog in a sword fight, a child blowing bubbles, a cat drinking water, and a man wiping a kitchen counter. Subject, layout and lighting hold throughout while the motion continues; the same generations are compared with CF++ and SGF in Figures 4 and 6 and Appendix C.5 .

Panel of Figure 2 — Figure 2: dKV Forcing: generation, memory writing and the memory gradient in one temporal traversal. For each segment the model runs a generation forward $G_{\theta}$ , whose prediction $\tilde{x}_{i}^{t_{s}}$ enters the video loss, and a memory write $\mathcal{K}_{\theta,\phi}$ , which turns the detached prediction $\operatorname{sg}(x_{i}^{\mathrm{ctx}})$ into the KV of that segment; $\theta$ is the shared backbone and $\phi$ the writer LoRA on the K and V projections, active only at the write. The written KV is committed once to a fresh slot of the page table and never modified afterwards. Later segments read the stored prefix through handles (teal). Both forwards are checkpointed in groups of blocks (gold ticks). In backward (purple, dashed) the gradient of the loss reaches every prediction, flows through the handles to every page it read, and on to the writer that produced that page; the stop-gradient keeps it from re-entering the prediction, and replay only reads the pages. Sampling substeps without gradient and the final, never-read write are omitted. Implementation details of each element are in Appendix B.4 .

Panel of Figure 3 — Figure 3: Autograd graphs and backward replay under three cache execution strategies. Top: forward computation and retained graphs. Bottom: operations replayed during backward. (a) With a mutable cache, writes modify a shared tensor in place. Checkpoint replay repeats these writes, accumulating graphs until backward runs out of memory. (b) Two-pass reconstruction builds the rollout without a graph, then recomputes the entire sequence in a differentiable forward, with layer-wise replay during backward. (c) dKV Forcing commits each write to an immutable page outside replay. The graph retains handles and checkpoint boundaries, while backward reads the pages and reconstructs one block group at a time.

Panel of Figure 4 — Figure 4: Comparison on 60 s video generation. Frames at $0$ – $59$ s for CF++, SGF and dKV Forcing on two MovieGen prompts, “cat and dog in a sword fight” (top) and “eye with the earth as its iris” (bottom). CF++ loses the scene within the minute and SGF drifts to another layout or breaks up the subject, while dKV Forcing keeps subject, layout and motion. More prompts are in Appendix C.5 .

method	cycle	gen. update	critic rollout	vs. two-pass	gen. update peak	whole-run peak	resident
	time (s)				GPU memory (GiB)
two-pass reconstruction	122.8	35.7	11.6	–	69.9	75.4	40.7
CF++ (writes detached)	104.7	31.8	8.9	$-14.8\%$	55.9	55.9	40.7
mutable cache (write differentiated)	out of memory in the first generator backward
dKV Forcing (default)	97.3	33.9	7.0	$-20.8\%$	62.1	62.1	29.2
dKV Forcing (fused)	95.9	32.4	6.8	$-21.9\%$	65.6	65.6	29.2

Table 1: Training cost on one node (4 H100, batch 2 per GPU, bf16, 21-frame clips; median over steady cycles; max over ranks). Cycle = 5 critic + 1 generator update. “Gen. update” = rollout, gradient-bearing forwards, teacher/critic scoring and backward of one generator update; “critic rollout” = the gradient-free generator rollout of one critic update (its critic loss and backward,

5.2

s, are identical across rows). Bold and underline: best and second best among the methods that carry the memory gradient; CF++ carries none and is listed as the reference.

Panel of Figure 5 — Figure 5: Page storage and gradient flow in one transformer layer. The page table (bottom row) stores each write in a separate, write-once slot outside the autograd graph (dashed frame), committed once (teal solid arrow). A data-free handle (purple circle) preserves the gradient connection to the writer without retaining the page data. Each forward reads a snapshot of the visible pages (teal dashed arrows), while backward returns an independent, contiguous gradient buffer (purple rectangles) to each page’s writer (purple dashed arrows).

Panel of Figure 6 — Figure 6: Comparison on 240 s video generation. Frames at $0$ – $238$ s, $46$ times the training clip, for CF++, SGF and dKV Forcing on the MovieGen prompt “man wiping a kitchen counter” (top) and the VBench-Long standard prompt “a cat drinking water” (bottom). CF++ has lost the scene, SGF drifts, and dKV Forcing keeps subject, layout and lighting across the full four minutes with the motion intact. More prompts are in Appendix C.5 .

	$\mathrm{KV}^{\mathrm{mem}}$ vs. $\mathrm{KV}^{\mathrm{gen}}$		readout by later frames
	K	V	attention mass	read-out vector
relative difference $\\|A-B\\|/\\|B\\|$	0.35–0.37	0.70–0.72	$1.17$ – $1.40\times$	0.65–0.68
token-wise cosine	0.93	0.73	–	0.76–0.77
per-head linear CKA	0.89–0.91	0.72–0.74	–	–

Table 2: Generation and memory KV differ in representation and in future attention readout: the written K/V against the frame’s own generation K/V (same frame, same layer; layer means over 30 layers, 48 videos

\times

21 frames, ranges over the four checkpoints). “Readout” swaps only frame

i

’s cache entry while every later query is held fixed.

checkpoint	next-frame change (other seed)	$\Delta$ consistency, frames 0–2	$\Delta$ consistency, all frames
CD initialization (untrained)	0.55 (0.97)	$-0.231\pm 0.024$	$-0.597\pm 0.018$
SGF	0.57 (1.08)	$-0.059\pm 0.010$	$-0.183\pm 0.017$
writer LoRA, $\alpha=1$	0.56 (1.07)	$-0.070\pm 0.009$	$-0.166\pm 0.014$
writer LoRA, $\alpha=0.5$	0.57 (1.05)	$-0.073\pm 0.008$	$-0.166\pm 0.019$

Table 3: Replacing written KV changes the autoregressive continuation. With checkpoint, prompt and sampling randomness fixed, the written K/V of frames 0–2 or of all 21 frames is replaced by their generation K/V at 5 s (48 prompts per checkpoint). Next-frame change: latent relative

L_{2}

to the unmodified rollout, with an independent seed in parentheses as the reference scale.

\Delta

: paired change of the prefix

\to

later DINO consistency (mean

\pm

s.e.m.).

configuration	memory gradient	residual	aesthetic	dynamic	imaging	subject
CF++, detached writes	none	none	$51.6\pm 2.4$	$\mathbf{74.9}\pm 15.8$	$62.7\pm 1.8$	$94.8\pm 1.1$
SGF	backbone	none	$57.2\pm 2.6$	$38.7\pm 16.3$	$68.0\pm 1.8$	$\underline{96.7}\pm 0.8$
writer LoRA + rollout dropout $0.5$ (ours) ∗	backbone + residual	write	$\underline{60.1}\pm 1.2$	$\underline{74.0}\pm 28.7$	$\underline{69.0}\pm 1.1$	$96.6\pm 1.0$
writer LoRA without rollout dropout	backbone + residual	write	$\mathbf{62.1}\pm 1.4$	$22.6\pm 14.4$	$\mathbf{69.6}\pm 0.6$	$\mathbf{97.9}\pm 0.6$
separate writer network (second 1.3B copy, backbone learning rate)	backbone + writer network	none (full network)	$64.9\pm 0.6$	$26.0\pm 18.2$	$71.6\pm 0.7$	$98.6\pm 0.4$

Table 4: Disentangling the effects of differentiable memory writing and writer adaptation: VBench-Long 60 s, step 1500, frame-wise, mean

\pm

SD over three to five training seeds. “Residual” says where a rank-64 residual is active; “memory gradient” says which parameters receive gradient from future reads. The dedicated writer row replaces the residual by a fully separate writer network (three seeds). ∗ Our most balanced configuration: the writer residual with rollout dropout

0.5

; the next row is the same residual without dropout. Best value per column among the first four rows in bold, second best underlined.

frame-wise (one latent frame per segment)
	VBench 5 s				VBench-Long 60 s
configuration	Total	Quality	Semantic	dynamic	aesthetic	dynamic	imaging	subject	backgr.
CF++	$\mathbf{83.5}\pm 0.7$	$\mathbf{84.8}\pm 0.5$	$78.1\pm 1.5$	$\mathbf{84.0}\pm 12.9$	$52.7\pm 1.4$	$\mathbf{76.5}\pm 19.0$	$63.3\pm 1.6$	$94.7\pm 1.3$	$94.5\pm 0.7$
SGF	$\underline{83.1}\pm 0.4$	$\underline{84.1}\pm 0.3$	$\underline{79.4}\pm 1.1$	$59.4\pm 7.6$	$56.2\pm 1.4$	$38.5\pm 6.9$	$67.4\pm 2.2$	$96.5\pm 0.4$	$\underline{95.7}\pm 0.5$
writer LoRA	$\underline{83.1}\pm 1.0$	$83.8\pm 1.2$	$\mathbf{80.4}\pm 0.2$	$36.7\pm 19.6$	$\mathbf{62.8}\pm 0.4$	$21.8\pm 17.5$	$\mathbf{69.7}\pm 0.6$	$\mathbf{98.2}\pm 0.2$	$\mathbf{96.9}\pm 0.2$
+ rollout dropout $0.5$	$82.9\pm 0.1$	$83.8\pm 0.2$	$79.3\pm 0.9$	$\underline{76.2}\pm 28.1$	$\underline{59.8}\pm 1.3$	$\underline{71.4}\pm 34.5$	$\underline{68.8}\pm 1.3$	$\underline{96.6}\pm 1.2$	$94.8\pm 1.1$

Table 5: Main results at 5 s and 60 s, mean

\pm

SD over training seeds

\{1,10,20\}

, step 1500. 5 s: official VBench Total / Quality / Semantic scores and dynamic degree. 60 s: VBench-Long standard prompts (aesthetic, dynamic, imaging, subject, background). Results at 240 s are in Table 6 . Per block, best value per column in bold, second best underlined.

frame-wise
configuration	aesthetic	dynamic	imaging	subject	background	smoothness	flicker
CF++	$49.0\pm 2.5$	$\mathbf{81.7}\pm 14.9$	$61.7\pm 2.7$	$93.8\pm 1.0$	$94.4\pm 0.7$	$97.5\pm 1.5$	$98.4\pm 0.3$
SGF	$51.3\pm 1.7$	$38.9\pm 7.4$	$64.6\pm 3.5$	$95.9\pm 0.3$	$\underline{95.3}\pm 0.4$	$\underline{98.6}\pm 0.2$	$98.0\pm 0.7$
writer LoRA	$\mathbf{59.8}\pm 1.4$	$20.5\pm 19.2$	$\underline{67.7}\pm 1.3$	$\mathbf{97.7}\pm 0.3$	$\mathbf{96.4}\pm 0.3$	$\mathbf{99.0}\pm 0.2$	$\mathbf{99.2}\pm 0.3$
+ rollout dropout $0.5$	$\underline{57.4}\pm 0.4$	$\underline{72.0}\pm 33.8$	$\mathbf{68.2}\pm 0.5$	$\underline{96.2}\pm 1.0$	$94.8\pm 0.7$	$97.7\pm 0.9$	$\underline{98.5}\pm 0.1$ ( $n{=}2$ )
chunk-wise

Table 6: 240 s results, VBench-Long standard prompts with

963

latent frames (four times the 60 s horizon), mean

\pm

SD over training seeds

\{1,10,20\}

, step 1500, except rollout dropout, whose entire row uses the two-seed subset marked in the table. Per block, best value per column in bold, second best underlined.

Figure 7: Fidelity versus motion. Top: the frame-wise configurations of Secs. 5.4 and 5.6 at 60 s (VBench-Long standard prompts; mean

\pm

SD over training seeds; numbers index the key below the plot). Bottom: the reported frame-wise and chunk-wise configurations at 240 s (seeds 1, 10, 20, except the dropout row marked in Table 6 ).

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Cycle, no LoRA (s)	vs. two-pass	Cycle, LoRA (s)	vs. two-pass	Gen. peak	Critic peak	Resident	Step-0 peak	All-step peak	Reserved
Two-pass	78.2 $\pm$ 1.1	+0.0%	81.0 $\pm$ 1.0	+0.0%	36.1	29.3	20.4	59.0	59.0	60.8
Unoptimized	88.8 $\pm$ 1.2	+13.6%	96.3 $\pm$ 1.9	+18.9%	36.4	23.5	14.7	20.2	36.4	58.7
default	66.9 $\pm$ 0.9	$-$ 14.5%	67.9 $\pm$ 2.8	$-$ 16.2%	36.1	23.5	14.7	20.2	36.1	58.3
fused	$\mathbf{59.5}$ $\pm$ 1.2	$-$ 23.9%	$\mathbf{61.8}$ $\pm$ 0.5	$-$ 23.7%	36.6	23.5	14.7	20.2	36.6	60.6
hidden	69.4 $\pm$ 1.9	$-$ 11.2%	75.2 $\pm$ 1.0	$-$ 7.2%	$\mathbf{31.8}$	23.6	14.7	20.2	$\mathbf{31.8}$	$\mathbf{56.2}$
fused+hidden	$\underline{62.6}$ $\pm$ 0.6	$-$ 20.0%	$\underline{65.7}$ $\pm$ 0.7	$-$ 19.0%	$\underline{33.3}$	23.6	14.7	20.2	$\underline{33.3}$	$\underline{57.9}$

Appendix

Table 7: Training cycles at 8 GPUs

\times

batch 1 (bf16). Steady-cycle medians

\pm

half the min–max range over six cycles. Memory in GiB (no-LoRA maximum over the eight ranks, resident memory at critic-rollout entry). Bold and underline: best and second best per column.

Figure 8: Measured component accounting of one generator update for the four training schemes. Panel titles give the generator-update time. CF++ (detached writes) keeps only exit-forward boundaries and its backward is short. The naive mutable cache keeps both forwards’ boundaries and the sampled trajectory, and its backward exhausts memory while replay attaches new graph to the external cache (purple: engine-allocated gradients and temporaries). The two-pass reconstruction keeps no graph in the rollout but retains the whole second-pass forward (violet) and replays it layer by layer in backward (cyan), with the cache resident between steps (red). dKV Forcing allocates the page table once (green), keeps only group entry states during the rollout, and in backward adds one contiguous gradient per page (light green) and the local replay of five-block groups.

configuration	$n$	aesthetic	dynamic	imaging	subject
CF++ (writes detached)	4	$51.6\pm 2.4$	$74.9\pm 15.8$	$62.7\pm 1.8$	$94.8\pm 1.1$
SGF ( $\alpha=1$ )	5	$57.2\pm 2.6$	$38.7\pm 16.3$	$68.0\pm 1.8$	$96.7\pm 0.8$
writer LoRA, $\alpha=0.5$	4	$62.5\pm 2.7$	$33.9\pm 24.0$	$69.5\pm 1.1$	$98.1\pm 0.7$
SGF, $\alpha=0.25$	4	$54.8\pm 1.8$	$48.6\pm 32.6$	$66.4\pm 1.5$	$96.0\pm 1.8$
SGF, $\alpha=0.5$	4	$54.9\pm 2.5$	$46.1\pm 21.2$	$65.0\pm 1.3$	$96.0\pm 1.3$
writer LoRA, $\alpha=0.25$	4	$58.6\pm 2.8$	$14.5\pm 9.1$	$66.9\pm 3.0$	$97.5\pm 0.6$

Appendix

Table 8: Every frame-wise configuration, VBench-Long standard prompts at 60 s, step 1500, mean

\pm

SD over

n

training seeds (same-seed repeats averaged first).

	configuration	$n$	aesthetic	dynamic	imaging
no residual	$\alpha=0$ (CF++, writes detached)	4	$51.6\pm 2.4$	$\mathbf{74.9}\pm 15.8$	$62.7\pm 1.8$
	$\alpha=0.25$	4	$54.8\pm 1.8$	$\underline{48.6}\pm 32.6$	$\underline{66.4}\pm 1.5$
	$\alpha=0.5$	4	$\underline{54.9}\pm 2.5$	$46.1\pm 21.2$	$65.0\pm 1.3$
	$\alpha=1$ (SGF)	5	$\mathbf{57.2}\pm 2.6$	$38.7\pm 16.3$	$\mathbf{68.0}\pm 1.8$
writer LoRA	$\alpha=0.25$	4	$58.6\pm 2.8$	$14.5\pm 9.1$	$66.9\pm 3.0$
	$\alpha=0.5$	4	$\mathbf{62.5}\pm 2.7$	$\mathbf{33.9}\pm 24.0$	$69.5\pm 1.1$

Appendix

Table 11: Strength of the writing-path training versus 60 s fidelity and motion (VBench-Long standard prompts, frame-wise, mean

\pm

SD over

n

seeds).

\alpha

scales the backward gradient at each writer-output interface. Upper block: SGF with a scaled memory gradient. Middle block: writer LoRA. Lower block: writer LoRA (

\alpha=1

) with a control that changes how the residual is trained. Per block, best value per column in bold, second best underlined.

Figure 9: Cache intervention. Top: paired change of the prefix

\to

later DINO consistency at 5 s when the memory of frames 0–2, or of all 21 frames, is replaced by generation K/V, with an independent seed as the reference scale (mean

\pm

s.e.m. over 48 prompts). Bottom: 60 s rollouts (16 prompts), similarity to the opening by latent-frame window, unchanged (solid) versus memory of frames 0–2 replaced (dashed).

Panel of Figure 10 — Figure 10: “Mother duck and six ducklings”. Prompt: A serene watercolor painting of a mother duck leading her six ducklings across a tranquil pond. The mother duck has a gentle expression, her feathers glistening in the sunlight, and she frequently glances back to ensure all her ducklings are safely following in a neat little line. The ducklings follow closely behind, their small heads bobbing up and down as they waddle along. The background features a peaceful pond with lily pads and ducks floating nearby, creating a harmonious and natural scene. A mid-shot from a slightly elevated angle captures the mother duck and her ducklings in motion.

Panel of Figure 13 — Figure 13: “Bird made of oranges”. Prompt: A dynamic and vibrant illustration in the style of a modern digital painting, depicting a bird crafted entirely from fresh oranges in vivid hues of orange and green. The bird is in flight, its wings spread wide and agile, as if rushing out of a large pile of freshly sliced oranges. The bird’s body is composed of various sizes and shapes of oranges, with some segments showing the inner fruit flesh, adding a sense of freshness and vitality. Its eyes are small and black, giving it a lively and spirited expression. The background is a blurred orchard with rows of orange trees, some leaves fluttering in the wind. The scene captures the bird in mid-flight, with a slight tilt of its head and a spread of its wings, creating a sense of movement and energy. A close-up shot from a low angle, emphasizing the intricate details of the bird’s form.

Panel of Figure 16 — Figure 16: “Red panda eating pizza”. Prompt: A vibrant and lively illustration in a whimsical cartoon style of a red panda taking a bite of a pepperoni pizza. The red panda has a playful expression, with large, round eyes and a mischievous grin. It has reddish-brown fur and black markings around its eyes and muzzle. The panda is sitting upright on a wooden table, one paw holding the pizza slice while it takes a big bite. The background features a cozy kitchen setting with a few scattered plates and utensils, and a window letting in warm sunlight. A close-up shot from a slightly lower angle, capturing the panda’s joyful moment.

Panel of Figure 19 — Figure 19: “Broom morphing into a peacock”. Prompt: A dynamic motion shot of a broom morphing surreally and magically into a peacock. The broom handle shortens and curves into a slender neck, while the bristles fan out into a magnificent tail. Vibrant colors and eye-shaped patterns emerge on the expanding feathers as the camera moves around, capturing every detail. A small head forms at the top, complete with a delicate crest. The transformation completes as the peacock proudly displays its newly formed plumage, standing tall and regal. The background features a soft, pastel garden with blooming flowers and gentle sunlight filtering through, enhancing the ethereal quality of the scene.

Panel of Figure 22 — Figure 22: “Coral reef with fish and a turtle”. Prompt: A vibrant coral reef teeming with colorful fish and marine life, captured in a dynamic underwater scene. The reef is adorned with a variety of coral shapes and sizes, creating a lively and intricate landscape. Schools of fish dart between the corals, their vibrant hues ranging from electric blues and greens to fiery oranges and yellows. A schools of clownfish can be seen swimming among the anemones, while a majestic turtle glides gracefully nearby. The water has a clear, turquoise hue, allowing the viewer to see the rich biodiversity beneath the surface. The camera angle is slightly from below, emphasizing the vertical depth and the bustling activity of the reef.

Panel of Figure 25 — Figure 25: “Young man hugging a friend”. Prompt: A vibrant and lively photograph capturing a moment of genuine joy, depicting a young man with a radiant smile as he hugs a dear friend. His face is filled with happiness, with his eyes sparkling and cheeks flushed. He wears a casual shirt and jeans, and his hair is tousled, adding to the natural and relaxed feeling. His friend, equally joyful, returns the embrace with equal warmth. The background is a blurred setting, hinting at a park or a sunny outdoor space, with patches of green grass and trees in the distance. The photo has a warm and candid feel, with soft lighting and natural shadows. A close-up shot from a slightly angled perspective, emphasizing their heartfelt connection.

Memory-Augmented Video Generation

Memory-augmented video generation gives video models persistent context to preserve identities, scenes, and events across long sequences.

See all 121 papers

Aug 28, 2026

DensityKV: Density-Guided KV Cache Compression for Long Video Generation

Autoregressive video diffusion models enable streaming generation through sliding-window attention, but each generated block is conditioned on previously generated content, causing appearance and motion errors to propagate recursively over time. Historical key-value (KV) memory preserves earlier subject and scene states and helps maintain long-horizon consistency. However, retaining every generated state creates a historical archive that grows continuously with the rollout, while recurrent states repeatedly add redundant coverage. To address this problem, we propose DensityKV, a training-free historical KV bank management strategy. DensityKV maintains a separate token-level KV bank for each attention head and measures local redundancy among the post-RoPE keys that directly parameterize attention routing using Soft-Riesz density. By constraining neighborhood-density growth after states enter the bank, DensityKV limits repeated historical accumulation while preserving coherent states from each completed generation block. Experiments across three autoregressive video generation backbones and multiple generation lengths show that, at the same upper bound on historical KV capacity, DensityKV improves long-horizon consistency and generation stability while keeping persistent historical storage bounded independently of rollout length.

Wenqu Zhao, Xuemin Chi, Xin Zhang +6

Manifold AI · Tsinghua University

KV-Cache Compression Long-Horizon Video Generation Autoregressive Video Generation Temporal Consistency in Video Generation

Ask PDF

Oct 11, 2026

TempoKV: Predictive KV Cache Quantization for Autoregressive Video Generation

Autoregressive (AR) video diffusion models enable streaming generation of long videos but suffer from significant memory overhead due to the rapidly growing KV cache. Existing KV cache quantization methods are primarily designed for LLMs and transfer poorly to video models, where activations exhibit different statistics and strong spatiotemporal redundancy. We observe that the video KV cache is largely predictable over time, with a clear asymmetry. Keys are highly predictable from their temporal history, while values depend primarily on the current key and need a short history of preceding values. Based on these observations, we propose \method, a training-free predictive KV cache quantization framework for AR video generation. \method predicts keys and values according to their distinct dependencies and stores only the residuals at low precision. Both predictors are lightweight, calibrated once offline by linear regression on a small set of BF16 samples, and remain fixed across prompts and generation runs. Experiments on Self-Forcing, Causal Forcing, and LongCat-Video-13B show that \method consistently achieves higher generation quality than existing KV cache quantization methods at comparable or higher compression ratios, reaching up to

6.71\times

KV cache compression with the best overall quality--efficiency tradeoff. Our code will be released on GitHub.

Jiahui Hu, Weile Mo, Peiliang Cai +5

Shanghai Jiao Tong University · South China University of Technology · Alibaba Group

KV-Cache Quantization Video Diffusion Models Autoregressive Video Generation

Ask PDF

May 18, 2026

Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion

Recent advances in autoregressive video diffusion have enabled sequential and streaming video generation. However, long-horizon generation requires increasingly large KV caches, making efficient compression without sacrificing quality challenging. Existing methods mostly select historical frames based on attention scores, but their context decisions remain coarse. When multiple frames are generated in the same chunk, these methods often apply a shared history selection to the whole chunk, score historical frames solely by attention, and assign head-wise budgets either uniformly or by attention-pattern heuristics rather than explicit head-importance estimation. We show that frames within the same generated chunk can depend on distinct historical frames, that the same historical frame can receive different attention scores as its relative temporal distance to the current frames changes, and that masking different heads induces unequal generation degradation. Motivated by these findings, we propose Focused Forcing, a training-free KV selection method that focuses cached history along both generated-frame and head dimensions. For each generated frame, Focused Forcing preserves the most relevant and distinctive historical frames by combining attention scores with diversity scores of historical frames, while assigning larger budgets to heads with higher estimated importance. Across multiple autoregressive generation paradigms, Focused Forcing achieves up to 1.48x end-to-end acceleration without training, while improving visual quality and text alignment. Our code is available at https://github.com/caipeiliang2004/Focused-Forcing.

Peiliang Cai, Evelyn Zhang, Jiacheng Liu +8

SJTU · SDU · HUST +4

Video Diffusion Models Autoregressive Diffusion KV Caching Diffusion Model Caching KV-Cache Compression Autoregressive Video Generation+1

Ask PDF

Abstract

Memory-Augmented Video Generation

Explore similar work

DensityKV: Density-Guided KV Cache Compression for Long Video Generation

TempoKV: Predictive KV Cache Quantization for Autoregressive Video Generation

Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusion