Reasoning vision-language models (VLMs) think for thousands of tokens before they answer, and retention policies keep the KV cache affordable by pruning it as the reasoning grows. We find that this pruning sacrifices the image in multi-turn conversations: the image and the reasoning share one budget, so the growing reasoning crowds the visual entries out. A chart question that loses 8 points to the full cache when asked first loses 24 points when asked after a round of reasoning. Protecting a small visual reserve from eviction recovers most of this loss, while adding the same slots to the shared budget recovers considerably less. The image therefore lacks protection, not capacity. Based on this finding, we propose ShelfKV, a training-free method that works on top of existing retention policies. It keeps every visual entry in CPU memory and, at each decoding step, swaps the visual entries that the current query scores highest into the attention slots of the text entries it scores lowest. The image is never evicted, yet it occupies attention slots only while it is read. We evaluate ShelfKV on four benchmarks, three of them conversational, with two retention policies and two reasoning VLMs. In conversations, it improves the strongest existing policy, DBTrimKV, by up to 20.3 points at matched GPU memory, and still leads by 8.7 points when DBTrimKV uses twice the memory. It also consistently outperforms a fixed visual reserve of matched GPU memory in these settings, without hurting accuracy on standalone questions. These results show that the image should be protected from the reasoning but read only on demand.
Figures & tables
Figure 2: Reasoning crowds out visual KV; protection beats extra shared capacity. (a) Cache composition during the first-turn output under DBTrimKV ( B=256 ). (b) Follow-up accuracy gains (percentage points) over the same host (61.3%); the dashed line marks the unconstrained full-cache gain.
Figure 3: ShelfKV: keep the image, read it on demand. Every visual entry stays on the shelf in CPU memory. A sketch proposes candidates, and the current query swaps those that outscore text entries onto the desk on the GPU. Displaced text entries are parked, not deleted, so the desk keeps the host’s entry count.
Method
Host budget B
GPU cache entries/head
CharXiv follow-ups Acc. (%) ↑
CharXiv conversations Acc. (%) ↑
MMMU-Pro Acc. (%) ↑
MMDU Score ↑
Full cache
–
all
85.7
87.3
54.0
5.2
DSTP + R-KV
512
512
15.3
16.8
6.3
3.6
DSTP + R-KV
1024
1024
48.0
49.2
21.3
4.1
Qwen3-VL-8B-Thinking, R-KV host
Host
256
256
4.7
5.3
7.0
3.1
Host
384
384
14.0
13.5
11.0
3.5
Table 1: Main results. Rows above the 4B block use Qwen3-VL-8B-Thinking; bold marks the best compressed 8B result. GPU cache: KV entries per head, including auxiliary data. CharXiv conversations score turns 2–4; MMDU reports the judge score (1–10). ‡ CharXiv values; 320/383 on MMMU-Pro and 296/383 on MMDU.
Figure 4: A follow-up that needs a region the reasoning never used. The first turn asks about the top panel, and the follow-up asks for the x-axis ticks of the bottom panel (blue box). The crops show the visual entries that DBTrimKV still keeps when the follow-up begins and those that ShelfKV reads while answering, at the same GPU memory. Quotes are excerpts from the model outputs.
Configuration
GPU
CharXiv
MMMU-Pro
Host alone
256
61.3
47.7
Shared budget ( B=384 )
384
71.3
54.0
Unprotected ( V=128 )
384
68.7
51.7
Reserve ( V=128 )
384
79.0
55.3
Reserve ( V=32 )
288
74.3
47.3
Reserve ( V=64 )
320
81.0
53.0
Table 2: A small host-selected reserve suffices. Accuracy (%) with DBTrimKV at B=256 , except for the shared-budget control. GPU counts equivalent KV entries per head; Nv is the visual token count. Random selection on CharXiv is averaged over three draws.
Figure 5: On-demand reading versus fixed allocation at matched GPU cache budgets. CharXiv follow-ups with DBTrimKV. Best fixed split: the best text/visual allocation we ran that fits within each budget. 16-bit keys: full-precision GPU candidate keys; prev. query: CPU candidates scored with the previous query.
(a) Mechanism
Variant
CharXiv
MMMU-Pro
ShelfKV, 16-bit keys (reference)
81.3
52.7
No host (all entries on the shelf)
80.0
47.3
Exact scores, no sketch
81.3
51.3
(b) Candidate-key storage
Storage
CharXiv
MMMU-Pro
Table 3: Ablations. Accuracy (%) on the CharXiv follow-ups and MMMU-Pro. (a) Variants of ShelfKV with 16-bit candidate keys on DBTrimKV at B=256 . (b) Rows in each column use matched GPU memory.
Peak (GB)
ms per step
CX
MM
CharXiv
MMMU-Pro
Method
64
64
1
16
64
1
16
64
R-KV, B=384
21.1
21.1
41.0
38.3
41.8
42.9
38.0
41.6
R-KV, B=512
22.8
22.8
43.1
38.8
43.4
36.2
38.1
43.4
R-KV + ShelfKV
20.8
21.4
48.3
60.4
103.2
48.1
68.6
152.2
CPU at step start
20.8
21.4
50.8
73.1
126.8
54.7
78.8
174.4
Table 4: Decoding memory and latency on an 80 GB A100. Qwen3-VL-8B-Thinking; ShelfKV uses B=338 on both benchmarks. Decoding peak memory includes model weights; batch sizes are shown in the header. CX/MM: CharXiv/MMMU-Pro. CPU at step start: sketch scoring without background overlap.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
CharXiv
Conv.
MMMU
MMDU
GPU entries per head
128
256
384
384
384
384
Best fixed split
54.0
76.7
81.0
81.9
55.3
4.6
ShelfKV, prev. query
63.0
79.7
83.7
83.3
53.7
4.7
ShelfKV, 16-bit keys
45.3
75.7
81.3
83.9
52.7
4.8
ShelfKV, 4-bit keys
66.7
81.7
84.0
85.0
56.7
4.9
Appendix
Table S1 : Matched-GPU comparison on all benchmarks (DBTrimKV as the host; the CharXiv columns are Fig. 5 of the paper). Best fixed split: the best split between text budget and reserve that we ran within each budget (128: 96 + 32 and 64 + 64; 256: 192 + 64, 224 + 32 and 128 + 128; 384: 256 + 128, 320 + 64, 352 + 32 and, with 320 entries, 256 + 64). The ShelfKV rows differ in how the swap scores the candidates: with the previous query and every key in CPU memory, or with the current query and the candidate keys on the GPU in 16 or 4 bits. Conv.: turns 2 to 4 of the CharXiv conversations; MMMU: MMMU-Pro; MMDU: judge score. With 4-bit keys, ShelfKV holds 131.6, 258.6 and 385.6 entries per head on CharXiv and 385.6 on the conversations, and runs at B=320 on MMMU-Pro and B=296 on MMDU to hold 383.2 and 383.1 ( Appendix A ).
All 56
11 fixed
Visual entries deleted by the host
97.9 [97.3, 98.4]
98.7
Borrow steps on deleted entries
91.4 [90.5, 92.3]
93.5
Appendix
Table S2: Overlap between the host and ShelfKV (%, CharXiv follow-ups, B=256 ). The first row is the 97.9% of Sec. 3.2 of the paper. “11 fixed”: the questions that ShelfKV answers correctly and the host alone does not.
r
b
c
refresh
CharXiv
MMMU-Pro
32
8
64
every step
71.9
54.7
32
8
256
every 4 steps
73.4
54.7
16
8
256
every 4 steps
78.1
48.4
32
4
128
every 4 steps
82.8
45.3
32
4
256
every 4 steps
75.0
54.7
DBTrimKV, B=256 / 512
67.2 / 73.4
45.3 / 54.7
Appendix
Table S3: Sketch design on the development splits (64 questions each, B=256 ). The fifth configuration is the one we use.
CharXiv
MMMU-Pro
Candidate keys
128
256
384
128
256
384
Mean
Previous query (CPU)
65.6
76.6
79.7
31.2
45.3
54.7
58.9
16-bit keys (GPU)
46.9
71.9
75.0
35.9
50.0
54.7
55.7
4-bit keys (GPU)
65.6
78.1
73.4
43.8
51.6
56.2
61.5
Appendix
Table S4: Storage of the candidate keys on the development splits (64 questions each, DBTrimKV as the host). Columns: GPU entries per head, matched across rows; on MMMU-Pro, the 4-bit rows hold about 20 more for the minima and maxima of the visual groups.
CharXiv question
MMMU-Pro question
Method
9
10
16
20
21
24
25
27
Median
Pha. 287
His. 76
Psy. 35
Lit. 19
Median
ShelfKV, B=338
375
361
362
154
165
41
193
55
179
34
121
82
99
90.5
R-KV alone, B=256
252
147
307
141
253
43
184
103
165.5
148
195
255
80
171.5
Appendix
Table S5: First token at which free generation differs , per question, greedy decoding up to 512 tokens, R-KV as the host. The first row compares the physical implementation of ShelfKV with the evaluation framework; the second compares the physical and the mask-based implementations of R-KV alone. Questions are numbered by their CharXiv index and by their MMMU-Pro subject and index (Pha.: Pharmacy, His.: History, Psy.: Psychology, Lit.: Literature).
Method
1
2
3
4
5
2–4
Full cache
50.7
85.0
88.0
89.0
79.7
87.3
DBTrimKV, B=256
45.0
66.7
59.0
62.3
45.3
62.7
DBTrimKV, B=384
46.3
71.3
69.7
68.0
51.3
69.7
DBTrimKV, B=512
43.3
71.7
72.7
72.0
48.3
72.1
+ reserve ( V=128 )
46.7
82.0
82.0
81.7
60.0
81.9
+ ShelfKV, 4-bit keys
46.3
83.3
84.7
87.0
68.0
85.0
Appendix
Table S6: Accuracy (%) by turn in the 300 five-turn CharXiv conversations (Qwen3-VL-8B-Thinking, DBTrimKV as the host; the reserve holds 384 and ShelfKV 386 GPU entries per head). Rows with + add the reserve or ShelfKV to DBTrimKV. Of each method’s 1,200 answers in turns 2 to 5, one each for DBTrimKV at B=512 and ShelfKV reaches the 12,288-token limit of a turn, and none for the other methods.
Method
Rule
LLM judge
Agreement
Full cache
85.7
85.7
98.7
DBTrimKV, B=256
61.3
61.3
98.0
DBTrimKV, B=512
73.0
73.3
99.0
ShelfKV, 16-bit keys, B=256
81.3
82.3
97.7
Appendix
Table S7: CharXiv follow-ups scored by our rule and by an LLM judge with the official prompts (%). Agreement: answers that both score the same.