FrameFold: Response-Guided KV Cache Consolidation for Autoregressive Long Video Generation

Muyu Liu; Mingming Liu; Hong Gu; Yuyao Zhang; Qin Lu

cs.CVOct 10, 2026

FrameFold: Response-Guided KV Cache Consolidation for Autoregressive Long Video Generation

Authors: Muyu Liu, Mingming Liu, Hong Gu, Yuyao Zhang, Qin Lu

Organizations: ShanghaiTech University · vivo BlueImage Lab, vivo Mobile Communication Co., Ltd., China

Abstract

In autoregressive video generation, retaining all historical key-value (KV) states increases memory usage and per-step attention costs as videos grow longer. A fixed cache budget bounds these costs but raises the challenge of preserving historical content that supports consistent, high-quality generation. Existing methods control cache size through history selection or consolidation, yet deciding what to retain and how to limit the loss of useful content remains a shared challenge. Our analysis shows that spatial queries depend on different historical content and that higher current attention coverage does not necessarily yield better long-video quality. We introduce FrameFold, a training-free KV cache consolidation method that compares the content retrieved by the same recent historical queries before and after merging to guide both representative construction and merge selection. Exploiting temporal redundancy between adjacent historical entries, FrameFold merges their KV states into shared representatives while retaining complete spatial grids. It fits value-merging coefficients in closed form by matching conditional attention responses, then selects the merge with the lowest remaining response distortion. Under matched KV cache budgets, experiments on Self Forcing and Causal Forcing at 30 and 60 seconds and on LongLive at 120 seconds demonstrate improved consistency and visual quality. Continuous eight-minute Self Forcing rollouts further show less quality degradation than the baselines.

Figures & tables

Figure 1: Token selection and shared representation. (a) Attention-based Top-K selection retains subsets of historical tokens; (b) FrameFold consolidates adjacent history into a representative grid. Actual Deep Forcing selection maps are shown in Appendix A.1 .

Figure 2: Shared Top-K leaves query-level coverage gaps; higher coverage need not yield better generation quality. (a) Candidate-region attention coverage for Top-K, Bottom-K, and a per-query oracle (solid: mean; dashed: P10). (b) VBench-Long score differences (Bottom-K minus Top-K). Despite lower coverage, Bottom-K improves five consistency and quality metrics on both models, with lower dynamic degree. (c) First-frame references and final frames for Self Forcing (top) and Causal Forcing (bottom). See Appendix A.2 for experimental details.

Figure 3: Overview of FrameFold . When the KV cache exceeds its budget, FrameFold consolidates adjacent entries in the candidate history while protecting sink and recent entries. Recent historical queries guide the evaluation of candidate merges, and the selected pair is replaced by a shared KV representative. Here, $s_{i}$ denotes the number of original latent frames represented by entry $i$ .

Config	Method	Subject Consist. $\uparrow$	Background Consist. $\uparrow$	Motion Smooth. $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$
30s	Self Forcing	96.38	95.89	97.77	59.11	57.40	66.80
	+ Deep Forcing	96.72	96.03	98.09	56.67	57.84	67.27
	+ FadeMem	96.94	96.15	98.08	56.35	58.01	68.29
	+ FrameFold	97.16	96.29	98.25	60.42	58.27	68.52
	Causal Forcing	95.65	95.52	96.95	75.73	55.80	66.31
	+ Deep Forcing	95.53	95.48	96.93	78.59	56.35	65.53

Table 1: Quantitative comparison of FrameFold with baselines on 30- and 60-second video generation. We evaluate Self Forcing and Causal Forcing on 128 MovieGenBench prompts with a cache budget of 12 frame-equivalent KV entries. Higher is better; best scores within each backbone–duration group are in bold .

Figure 4: Qualitative comparison of FrameFold with baselines on 60-second video generation. Self Forcing and Causal Forcing examples are shown in the top and bottom groups, respectively. Methods within each example use the same prompt and seed, with a cache budget of 12 frame-equivalent KV entries.

Figure 5: Quantitative comparison of FrameFold with baselines on video generation up to 480 seconds. We evaluate Self Forcing on 24 prompts with a cache budget of 12 frame-equivalent KV entries, using nested prefixes of continuous rollouts. Colored labels show

\Delta=s_{480\mathrm{s}}-s_{30\mathrm{s}}

; the duration axis uses a base-2 log scale.

Selection	Value merging	Subject Consist. $\uparrow$	Background Consist. $\uparrow$	Motion Smooth. $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$
V diff	Span	97.31	96.26	98.17	62.36	60.33	70.58
V diff	Optim	97.37	96.31	98.19	60.41	60.39	70.57
K diff	Span	97.40	96.25	98.17	61.11	60.34	70.56
K diff	Optim	97.45	96.33	98.20	59.44	60.38	70.69
O diff	Span	97.48	96.33	98.25	57.64	60.73	70.67
O diff	Optim	97.66	96.43	98.28	59.31	60.66	70.80

Table 2: Ablation on merge selection and value fitting. Self Forcing on 48 prompts. K/V/O diff measure key/value/conditional-response differences; Span/Optim use span-weighted/fitted values. Best scores are in bold .

Response objective	Subject Consist. $\uparrow$	Background Consist. $\uparrow$	Motion Smooth. $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$
Full-cache	96.94	96.15	98.08	61.56	57.55	68.45
Conditional	97.16	96.29	98.25	60.42	58.27	68.52

Table 3: Ablation on the response objective. Self Forcing on 128 prompts. Each response objective is used for both value fitting and merge selection. Best scores are in bold .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Visualization of query-specific spatial attention patterns. Each row shows a query at the upper-left, center, or lower-right spatial location, followed by its attention maps in Transformer layers 0, 10, 20, and 29.

Figure 7: Visualization of spatial selection patterns in Deep Forcing. A generated frame is shown alongside selection-frequency maps for seven layers: 7, 11, 15, 18, 22, 26, and 29. Color indicates selection frequency at each spatial location.

Figure 8: Qualitative comparison of Top-K and Bottom-K over 60 seconds of generation. Results are shown for Self Forcing and Causal Forcing. Both policies retain 50% of the candidate history using the same scoring procedure, with matched prompts, seeds, and sampling settings.

Figure 9: Additional qualitative comparisons on 60-second video generation. Self Forcing and Causal Forcing examples are shown in the top and bottom groups, respectively, with frames sampled at 0, 15, 30, 45, and 60 seconds. Methods within each example use the same prompt, seed, and sampling settings, with a cache budget of 12 frame-equivalent KV entries.

Duration	Method	Subject Consist. $\uparrow$	Background Consist. $\uparrow$	Motion Smooth. $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$
30s	Deep Forcing	97.27	96.26	98.49	55.99	61.40	69.50
	FadeMem	97.22	96.29	98.50	67.44	61.77	71.77
	FrameFold	97.36	96.35	98.62	67.45	62.00	72.31
60s	Deep Forcing	97.02	96.09	98.33	54.97	60.43	68.74
	FadeMem	97.00	96.13	98.49	70.29	61.27	71.07
	FrameFold	97.20	96.28	98.62	68.68	61.68	72.18

Appendix

Table 4: Quantitative comparison of FrameFold with baselines on video generation up to 480 seconds. We evaluate Self Forcing on 24 prompts with a cache budget of 12 frame-equivalent KV entries. At each duration, we evaluate a nested prefix of the same continuous 480-second rollout. Higher is better; best scores within each duration group are in bold .

Figure 10: Qualitative comparison of FrameFold and FadeMem over 480 seconds of generation. FadeMem (top) and FrameFold (bottom) use Self Forcing with the same prompt and a cache budget of 12 frame-equivalent KV entries.

Figure 11: Ablation on merge selection and value fitting. Qualitative results are shown for Self Forcing with 30-second videos and a cache budget of 12 frame-equivalent KV entries. O diff + Optim corresponds to FrameFold .

Figure 12: Ablation on the response objective. Qualitative results show 30-second Self Forcing generation with a cache budget of 12 frame-equivalent KV entries. Full-cache (top) and conditional (bottom, FrameFold ) use the same prompt; each objective is applied to both value fitting and merge selection.

Config	Method	Subject Consist. $\uparrow$	Background Consist. $\uparrow$	Motion Smooth. $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$
120s	LongLive	97.24	96.22	98.43	47.38	57.61	66.93
	Deep Forcing	97.58	96.25	98.53	39.59	57.88	67.35
	FadeMem	97.45	96.39	98.47	45.96	58.53	67.65
	FrameFold	97.76	96.75	98.50	46.32	59.09	68.10

Appendix

Table 5: Quantitative comparison on LongLive for 120-second video generation. All methods use the same 128 prompts as the main comparison and a cache budget of 12 frame-equivalent KV entries. We report six VBench-Long dimensions; higher is better, and the best scores are in bold .

Figure 13: Qualitative comparison on LongLive over 120 seconds of generation. From top to bottom: the original LongLive model, Deep Forcing, FadeMem, and FrameFold . Frames are shown at 0, 30, 60, 90, and 120 seconds. All methods use the same prompt and random seed, with a cache budget of 12 frame-equivalent KV entries.

Method	Cache Budget	Subject Consist. $\uparrow$	Background Consist. $\uparrow$	Motion Smooth. $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$
Deep Forcing	12	96.75	95.98	98.09	61.32	59.63	68.85
	15	96.65	95.73	98.02	66.32	58.90	68.34
	18	96.64	95.83	97.98	61.74	58.88	68.11
	21	96.47	95.65	97.96	59.37	58.44	67.24
FrameFold	12	97.15	96.27	98.27	67.08	60.55	70.90
	15	97.19	96.21	98.28	64.37	60.28	70.74

Appendix

Table 6: Ablation on KV cache budget. Experiments on Self Forcing use 48 prompts and 60-second videos. FrameFold uses a pooling factor of 4. Cache budgets are measured in frame-equivalent KV entries; higher scores are better. The default FrameFold configuration is highlighted in blue.

Backbone	Pooling Factor	Subject Consist. $\uparrow$	Background Consist. $\uparrow$	Motion Smooth. $\uparrow$	Dynamic Degree $\uparrow$	Aesthetic Quality $\uparrow$	Imaging Quality $\uparrow$
Self Forcing	1	97.12	96.19	98.29	66.39	60.29	70.97
	2	97.20	96.20	98.28	65.69	60.24	71.30
	4	97.15	96.27	98.27	67.08	60.55	70.90
	8	97.09	96.11	98.17	67.01	60.28	70.37

Appendix

Table 7: Ablation on spatial pooling factor. Experiments on Self Forcing use 48 prompts, 60-second videos, and a cache budget of 12 frame-equivalent KV entries. A pooling factor of 1 uses the original spatial resolution. Higher scores are better. The default configuration is highlighted in blue.

Figure 14: Quantitative comparison of generation throughput. Throughput is measured in output frames per second (FPS) on Self Forcing with an NVIDIA H20, excluding VAE decoding and video saving. All four methods use a cache budget of 12 frame-equivalent KV entries; FIFO and Deep Forcing are additionally evaluated with 21 entries.

Autoregressive Video Generation

Autoregressive video generation creates videos frame by frame or in chunks, conditioning each prediction on prior context for streaming and interactive use.

See all 211 papers

FadeMem: Distance-Aware Memory Consolidation for Autoregressive Video Diffusion
Jun 9, 2026Yu Lu, Junjie Yang, Piotr Koniusz +2KV Caching Long-Horizon Video Generation
Zhejiang University · University of New South Wales (UNSW) · Data61/CSIRO +1
TempoKV: Predictive KV Cache Quantization for Autoregressive Video Generation
Oct 11, 2026Jiahui Hu, Weile Mo, Peiliang Cai +5KV-Cache Quantization Video Diffusion Models
Shanghai Jiao Tong University · South China University of Technology · Alibaba Group
Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation
May 28, 2026Jiayi Luo, Qiyan Liu, Tengyang Wang +8KV Caching Long-Horizon Video Generation
BUAA · ZGCA · PKU +3

Jun 9, 2026J/K move · Enter open · S save

FadeMem: Distance-Aware Memory Consolidation for Autoregressive Video Diffusion

Yu Lu, Junjie Yang, Piotr Koniusz, YuXin Song+1

Zhejiang University · University of New South Wales (UNSW) · Data61/CSIRO+1

KV Caching Long-Horizon Video Generation Diffusion Model Caching KV-Cache Compression Autoregressive Video Generation Temporal Consistency in Video Generation

Open paper AskPDF arXiv

Autoregressive video generators synthesize long videos by generating successive temporal segments, but their historical KV cache grows with video length. Existing bounded-cache methods reduce this cost with local windows, sink tokens, or compressed memory states, yet they usually assign fixed roles to different parts of the history. We propose FadeMem, a distance-aware KV memory consolidation mechanism that organizes historical KV blocks into a temporal hierarchy under a fixed cache budget. This design is motivated by frequency-dependent temporal decay: fine details decorrelate quickly, while coarse scene structure and identity remain useful over longer horizons. During generation, new history is inserted as fine-grained entries, while older adjacent entries are progressively merged under a power-law temporal allocation schedule, yielding a dense-near, sparse-far memory within one cache. Without architectural changes, FadeMem preserves recent context for short-term dynamics and compact long-range anchors for identity and scene coherence. Experiments show improved subject consistency, background stability, and temporal coherence over existing bounded-cache strategies.

Abstract

Autoregressive Video Generation

Explore similar work

FadeMem: Distance-Aware Memory Consolidation for Autoregressive Video Diffusion

TempoKV: Predictive KV Cache Quantization for Autoregressive Video Generation

Future Forcing: Future-aware Training-free KV Cache Policy for Autoregressive Video Generation