Drift Says When, Budget Says How Much: Rate-Controlled KV Re-selection for Long-Form Multimodal Generation

Bin Hu; Yingbo Xiao

cs.LGOct 22, 2026

Drift Says When, Budget Says How Much: Rate-Controlled KV Re-selection for Long-Form Multimodal Generation

Authors: Bin Hu, Yingbo Xiao

Organizations: Tsinghua University · The Chinese University of Hong Kong, Shenzhen

Abstract

KV cache compression for multimodal LLMs usually selects the visual tokens to keep once, at prefill, and reuses that selection for the whole generation. We show that this one-shot selection is severely limited on long-form generation: on detailed image description, even an oracle that selects with the attention of the uncompressed answer reaches only 0.44-0.52 ROUGE-L agreement with the uncompressed output, because the evidence the model reads moves as the description unfolds. Re-selecting whenever attention drifts out of the kept set is the natural remedy, but we find that a fixed drift threshold fails systematically: drift magnitudes are not comparable across samples, so low-drift samples are almost never refreshed although they benefit the most. We propose Rate-Controlled Refresh (RCR), which lets drift decide when and where to re-select while a per-sample controller regulates how often, combined with a low-rank drift detector, candidate-restricted re-selection, and a decision that stays on the GPU. With one setting for all models and tasks, RCR outperforms the per-step retrieval method InfiniGen by 1.6-3.8 ROUGE-L points on five of six model-task settings with no significant difference on the sixth, while staying within 1.7 points of exact per-step re-selection, with comparable estimated data movement per step and a 4x smaller GPU-resident detector. On DetailCaps, drift-chosen timing beats random timing at the same refresh rate and periodic timing even at a higher rate, by 2.0-5.0 points. On pre-registered video description, where drift is homogeneous across samples, RCR is on par with InfiniGen on Qwen2-VL and exceeds it on Qwen3-VL but no longer beats a fixed threshold at the nearest tested rates, consistent with the diagnosis; object-level evaluation shows that RCR achieves higher object F1 against the uncompressed output than InfiniGen on two of three models.

Figures & tables

Figure 1: Illustration (not measured). While describing the image, attention moves from the skateboarder to the ramp and the trash bin. Drift $d_{t}$ , the attention mass on evicted prefix tokens, stays low most of the time; when it crosses the threshold, the current query re-selects tokens from the full KV kept on the CPU and overwrites the GPU cache. In RCR the threshold is adapted online per sample so that a fraction $\rho$ of layer-steps refresh.

DetailCaps fidelity	Qwen2	LLaVA	Qwen3
SnapKV	.389	.362	–
BACON	.388	.371	.409
Last-query	.367	.386	–
Look-ahead 8 steps	.448	.412	–
Prefill oracle (answer attn.)	.516	.440	.507
Exact re-selection every step	.613	.478	.571

Table 1: One-shot selection falls short on long descriptions even with oracle scores; re-selecting during decoding is not. –: not run.

Figure 2: Qwen2-VL DetailCaps split into terciles by each sample’s mean drift under the fixed threshold (

\tau=0.03

). Numbers above bars are refresh rates. The fixed threshold matches exact re-selection on high-drift samples but starves low-drift ones; RCR closes this gap.

	Qwen2-VL-7B		LLaVA-NeXT-7B		Qwen3-VL-8B
Method	DetailCaps	TextCaps	DetailCaps	TextCaps	DetailCaps	TextCaps
BACON (one-shot)	.388	.626	.371	.680	.409	.505
Prefill oracle (one-shot)	.516	.774	.440	.810	.507	.735
Periodic, $T{=}4$	.521 (25%)	.738 (28%)	.448 (25%)	.751 (29%)	.535 (25%)	.687 (27%)
Quest, page 8	.408	.644	.419	.735	.440	.565
ArkVale, page 8	.347	.535	.401	.697	.424	.510

Table 2: Fidelity (ROUGE-L to the uncompressed output) at a budget of 64 prefix KV entries per head (generated tokens are kept). Parentheses: share of layer-steps that re-select (per-step methods: 100%). Bold: best method other than exact every step.

\Delta

: paired difference in points with 95% bootstrap CI.

Figure 4: Fidelity vs. average number of 128-d vectors read from CPU per step and KV head on DetailCaps (log scale; cost model of § 5 ). At the traffic of a per-step method that fetches only the selected tokens, RCR reaches the highest fidelity.

Variant (dev split)	Refresh	Fid.	$\Delta$ vs. IG
Fixed $\tau{=}.03$ , page-mean	15%	.498	$-$ 5.0
+ per-head trigger	15%	.536	$-$ 1.2
Fixed $\tau{=}.02$ , page-mean	26%	.535	$-$ 1.3
Fixed $\tau{=}.03$ , low-rank	28%	.548	0.0
Fixed $\tau{=}.02$ , low-rank + cand.	37%	.558	+0.9
Rate $\rho{=}.2$ , page-mean	20%	.529	$-$ 2.0

Table 3: Component analysis on the 100-image development split (Qwen2-VL DetailCaps).

\Delta

: paired difference to InfiniGen in points. Full list with intervals in Appendix A .

Trigger (same detector, cand.)	Refresh	Qwen2	LLaVA
Random	40%	.548 $-$ 5.0	.462 $-$ 2.3
Periodic, $T{=}3$	33%	.552 $-$ 4.6	–
Periodic, $T{=}2$	50%	.578 $-$ 2.0	–
Drift, rate-controlled ( RCR )	40%	.598	.485

Table 4: Does drift decide when ? DetailCaps fidelity when only the trigger changes; small numbers: paired difference to RCR (all 95% CIs exclude zero; Appendix G ).

Figure 5: Post-hoc sweeps on Qwen2-VL DetailCaps (300 test images, budget 64). RCR points vary only

\rho

(the main configuration uses

\rho{=}0.4

, fixed beforehand); fixed-threshold and periodic points vary

\tau

and

T

and scan the full prefix at each refresh.

Figure 6: Where refreshes go (61 Qwen2-VL DetailCaps samples, layer-step traces). Left: refresh rate by position in the output. Right: refresh rate per layer. The fixed threshold (

\tau{=}0.03

) spends its refreshes at the start of generation; RCR spreads them evenly over time while keeping the same preference over layers.

Figure 7: Drift, threshold and refreshes (ticks) for layer 11 on one Qwen2-VL DetailCaps image. RCR ’s threshold is replayed exactly from the traced decisions. The fixed threshold stops refreshing once drift decays; RCR keeps refreshing at a steady rate. Fidelity: fixed .36, RCR .55.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Tag	Configuration	Refresh	Fidelity	$\Delta$ vs. InfiniGen
b64_bacon	BACON	–	.367	–
grow3_px	threshold $=3\times$ residual drift after last refresh, $\tau_{\min}{=}.005$	0.5%	.396	–
grow2h_px	threshold $=2\times$ residual drift, per head	0.6%	.430	–
async_px003	fixed $\tau{=}.03$ , page-mean (fixed-threshold version)	15.4%	.498	$-$ 5.0 [ $-$ 8.4, $-$ 1.8]
head_px003	+ per-head trigger	15.3%	.536	$-$ 1.2 [ $-$ 4.5, +1.9]
rate20_px	rate control $\rho{=}.2$ , page-mean	19.7%	.529	$-$ 2.0 [ $-$ 5.8, +1.6]

Appendix

Table 5: All development-split results (Qwen2-VL DetailCaps, 100 images), sorted by refresh rate, except that the per-head variant directly follows its base configuration. “–”: interval not computed.

Setting	RCR ( $\rho{=}.4$ )	$\tau{=}.01$ + low-rank + cand.	$\tau{=}.02$ + low-rank + cand. + per head
Qwen2 DetailCaps	+2.8 [+0.8, +4.8] (40%)	+2.8 [+0.9, +4.6] (54%)	+1.9 [ $-$ 0.1, +3.8] (31%)
Qwen2 TextCaps	+3.7 [+2.1, +5.4] (48%)	+5.1 [+3.3, +6.8] (100%)	+4.8 [+3.0, +6.6] (92%)
LLaVA DetailCaps	+3.4 [+2.1, +4.8] (40%)	+3.5 [+2.1, +4.8] (96%)	+3.4 [+2.1, +4.8] (66%)
LLaVA TextCaps	+3.8 [+1.8, +5.9] (48%)	+4.4 [+2.4, +6.4] (100%)	+4.6 [+2.6, +6.7] (92%)
Qwen3 DetailCaps	+0.5 [ $-$ 1.1, +2.1] (40%)	+0.9 [ $-$ 0.7, +2.6] (40%)	+1.1 [ $-$ 0.5, +2.9] (21%)
Qwen3 TextCaps	+1.6 [+0.2, +3.2] (41%)	+3.5 [+2.1, +4.9] (99%)	+3.6 [+2.1, +5.0] (69%)

Appendix

Table 6: Paired fidelity difference to InfiniGen (points, 95% CI) and refresh rate for the three confirmed candidates.

Configuration	DetailCaps	TextCaps
BACON	.409	.505
Prefill oracle	.507	.735
Periodic $T{=}8$	.507 (13%)	.633 (14%)
Periodic $T{=}4$	.535 (25%)	.687 (27%)
Quest / ArkVale, page 8	.440 / .424	.565 / .510
Two-stage Quest, page 8	.550	.684

Appendix

Table 7: Pre-registered Qwen3-VL-8B matrix (all runs complete). Bold: headline settings chosen by the registered rule.

Qwen2-VL DetailCaps	Refresh	Fidelity	$\Delta$ vs. IG	Ref. R-L
Budget 64
InfiniGen	100%	.570	0	.281
RCR $\rho{=}0.1$	9.8%	.538	$-$ 3.2 [ $-$ 5.3, $-$ 1.0]	.277
RCR $\rho{=}0.2$	19.7%	.560	$-$ 1.0 [ $-$ 2.9, +1.0]	.279
RCR $\rho{=}0.4$	39.6%	.598	+2.8 [+0.8, +4.8]	.285
RCR $\rho{=}0.6$	59.5%	.606	+3.6 [+1.6, +5.6]	.279

Appendix

Table 8: Post-hoc

\rho

, budget and trigger sweeps on DetailCaps (300 test images; Qwen2-VL unless noted).

\Delta

is against InfiniGen except in the last block. Reference ROUGE-L was computed only for the first two blocks.

CIDEr / ROUGE-L	Qwen2-VL-7B		LLaVA-NeXT-7B		Qwen3-VL-8B
Method	DetailCaps	TextCaps	DetailCaps	TextCaps	DetailCaps	TextCaps
Uncompressed	.067 / .287	1.524 / .550	.099 / .285	.777 / .445	.009 / .261	.376 / .384
BACON	.044 / .261	1.209 / .506	.073 / .262	.578 / .416	.004 / .249	.318 / .348
Two-stage Quest	.057 / .280	1.462 / .545	.091 / .279	.741 / .444	.005 / .256	.390 / .382
InfiniGen	.061 / .281	1.550 / .554	.099 / .279	.715 / .438	.004 / .256	.403 / .386
Fixed threshold	.054 / .281	1.439 / .538	.096 / .279	.716 / .433	.007 / .258	.384 / .385

Appendix

Table 9: Reference-based CIDEr / ROUGE-L (budget 64). Dynamic methods are not separated reliably by references.

Image 2475 (forest with a dog), favourable to RCR
Uncompressed	[…] The sunlight appears to be coming from behind the trees, creating a silhouette effect on the left side of the image, where a dog is visible. The dog is standing still, facing towards the right side of the image , and appears to be observing something in the distance. […]
BACON (.33)	[…] with a dog prominently featured in the foreground. The dog appears to be a medium to large breed, possibly a Golden Retriever […] capturing a moment of quiet companionship […]
InfiniGen (.60)	[…] creating a silhouette effect on the left side of the image. The trees are mostly coniferous […] The forest floor is not clearly visible […] (no dog is mentioned)
RCR (1.00)	Identical to the uncompressed output.
Image 3142 (wristwatch), unfavourable to RCR
Uncompressed	[…] The watch face is circular and also silver-colored, with a white background. The hour markers on the watch face are small, circular, and also silver-colored […] The watch hands are thin and silver-colored […]

Appendix

Table 10: Outputs on two images; numbers are fidelity to the uncompressed output.

VideoDetailCaption	Qwen2-VL	Qwen3-VL
BACON	.363 $-$ 7.9 [ $-$ 9.2, $-$ 6.7]	.266 $-$ 12.6 [ $-$ 13.5, $-$ 11.7]
Two-stage Quest, page 8	.423 $-$ 2.0 [ $-$ 3.3, $-$ 0.7]	.365 $-$ 2.7 [ $-$ 3.6, $-$ 1.9]
InfiniGen	.442	.392
Fixed $\tau{=}.12$	.423 (7%) $-$ 1.9 [ $-$ 3.1, $-$ 0.7]	.369 (6%) $-$ 2.3 [ $-$ 3.2, $-$ 1.3]
Fixed $\tau{=}.08$	.438 (18%) $-$ 0.5 [ $-$ 1.7, +0.9]	.394 (12%) +0.2 [ $-$ 0.7, +1.1]
Fixed $\tau{=}.05$	.461 (35%) +1.9 [+0.6, +3.2]	.406 (23%) +1.4 [+0.6, +2.2]

Appendix

Table 11: Video fidelity (300 videos, budget 64). Parentheses: refresh rate; small numbers: paired difference to InfiniGen with 95% CI. Bold: fixed-threshold headline chosen by the registered rule.

Setting	CV	p90/p10	refresh $<$ 5%
Qwen2-VL DetailCaps	0.71	7.0	21%
Qwen3-VL DetailCaps	0.78	8.0	45%
LLaVA-NeXT DetailCaps	0.16	1.5	0%
Qwen2-VL video	0.18	1.5	1%
Qwen3-VL video	0.14	1.4	0%

Appendix

Table 12: Spread of per-sample mean drift: coefficient of variation, 90th/10th percentile ratio, and share of samples refreshing on fewer than 5% of layer-steps.

	Qwen2-VL-7B			LLaVA-NeXT-7B			Qwen3-VL-8B
Method	CAPTURE	P/R ref	P/R/F1 full	CAPTURE	P/R ref	P/R/F1 full	CAPTURE	P/R ref	P/R/F1 full
BACON	.602	.659/.634	.733/.702/.708	.554	.652/.539	.698/.632/.651	.611	.602/.720	.779/.721/.744
Two-stage Quest	.631	.684/.676	.802/.790/.786	.593	.694/.586	.774/.736/.745	.624	.594/.770	.818/.813/.812
InfiniGen	.631	.680/.676	.823/.815/.811	.590	.716/.573	.785/.708/.733	.627	.596/.778	.821/.819/.816
Fixed threshold	.634	.682/.686	.809/.811/.802	.592	.693/.594	.772/.737/.745	.626	.592/.777	.808/.809/.805
RCR	.635	.687/.682	.835/.827/.824	.598	.694/.597	.778/.756/.758	.626	.594/.777	.825/.827/.823

Appendix

Table 13: Object-level evaluation on DetailCaps (300 images, budget 64). “ref”: against the human references; “full”: against the uncompressed output. LLaVA’s uncompressed row is below 1 because one of its outputs yields no parsed objects and scores 0.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,084 papers

KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression
Sep 23, 2026Linfeng DongKV-Cache Management KV-Cache Compression
InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context
Mar 5, 2026Xin Teng, Canyu Zhang, Shaoyi Zheng +3Retrieval-Augmented Generation KV Caching
New York University · New York University Shanghai · Duke University +1
RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training
Oct 22, 2026Dor Tsur, Maor Ashkenazi, Maximilian Golub +7KV-Cache Compression LLM Inference Acceleration
NVIDIA

Sep 23, 2026J/K move · Enter open · S save

KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression

Linfeng Dong

KV-Cache Management KV-Cache Compression

Open paper AskPDF arXiv

As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing methods fall into three families: score-based eviction, summary compensation, and offload-and-recall. Yet all three decide what to keep or recall by content relevance to the current query. We show this shared design is structurally incomplete. A cache supports two access modes: associative lookup by content and sequential traversal by position; current compressors implement only the first. The gap matters in practice: retrieval-augmented generation, code completion, and structured-data extraction all require the model to reproduce identifiers, field values, or code tokens verbatim from the context. Under compression, content-based eviction retains the head of such a sequence but discards its continuation, causing verbatim copying to break irreversibly midway, a failure we call sequential forgetting. This failure resists better scoring, larger budgets, summary compensation, and dynamic re-scoring; it is the dominant source of remaining quality loss under compression. We propose KVFetch, a training-free, drop-in framework that opens a temporal recall channel for any score-based compressor. It demotes evicted candidates to a quantized cold tier, detects active copying through a monotone read pointer, and prefetches positional successors into fixed-size hot-tier slots without increasing attention cost. On RULER-16K under an iso-budget control, KVFetch recovers verbatim copying from 0.8 to 78.4 and raises the 13-task average by +8.4, with gains concentrating on tasks that require sequential access. On LongBench, where no task requires sequential access, the channel remains dormant and imposes no cost.

Abstract

Multimodal Large Language Models

Explore similar work

KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression

InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context

RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training