CanopyKV: Remember beyond the Crop for Fast Active Visual Reasoning
Authors: Lei Yang, Dayan Wu, Yiliang Zhu, Xinze Liu, Hengjie Zhu, Zihao Zhang, Ding Wang, Peng Fu, Zheng Lin
Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Department of Applied Mathematics and Statistics, John Hopkins University, Baltimore, MD, USA
Active visual reasoning enables multimodal models to answer challenging visual questions by gathering fine-grained evidence through iterative image inspection and tool use. However, accumulating crops and reasoning tokens expands the key-value (KV) cache, making compression important for reducing memory and inference costs. Existing KV compression methods typically select historical tokens based on attention, recency, or redundancy. These criteria do not explicitly distinguish locally replaced evidence from complementary information needed for future exploration: each crop refreshes a local region, while earlier views may still support subsequent decisions. Discarding such evidence can impair accuracy or require additional search, while scoring and cache reorganization can offset the computational savings from compression. We introduce CanopyKV, a training-free KV compression framework that uses parent-child crop relationships to retain complementary evidence, keeps retained history stable, and protects the state needed for ongoing reasoning. Tool metadata provides retention signals without additional attention scoring, while compact KV storage reduces subsequent attention costs. Across VisualProbe, V*, HR-Bench, and MME-Lite, CanopyKV achieves state-of-the-art answer accuracy and end-to-end efficiency among the evaluated KV compression methods. On MME-Lite, it retains approximately one-quarter of Dense's total KV while improving accuracy by 10.5 percentage points and accelerating complete reasoning by 1.64×.
Figures & tables
Figure 1: Local replacement and persistent complementarity. (a) Dense retains all acquired views, growing its KV cache. A1 refreshes its footprint but leaves C outside it. (b,c) At equal parent-KV budgets, CanopyKV retains complementary C-region evidence across A2 and B, while token-score pruning may discard it. Insets show parent history, not a newly acquired C.
Figure 2: CanopyKV retains complementary evidence in compact KV. (a) Source quotas and crop lineage favor context outside the incoming crop. (b) One-time retirement protects continuation state and fixes selected history across branch visits. (c) Incremental prefill is followed by visual pruning and KV compaction, preserving current text and logical positions.
Figure 3: Parent evidence outside the new crop is harder to replace. After removing matched amounts of parent KV inside or outside the incoming crop, we supply the full crop and read out the corresponding facts. The horizontal axes show KV retention within each target fact region.
Method
Venue
VisualProbe
V*
HR-Bench 4K
HR-Bench 8K
MME-Lite
Acc. ↑
Speed ↑
Acc. ↑
Speed ↑
Acc. ↑
Speed ↑
Acc. ↑
Speed ↑
Acc. ↑
Speed ↑
Dense
Reference
51.26
1.00
82.20
1.00
72.38
1.00
67.38
1.00
43.04
1.00
H 2 O [ 38 ]
NeurIPS ’23
50.10
0.38
31.94
0.10
66.25
0.30
66.88
0.37
42.68
0.44
LOOK-M [ 25 ]
EMNLP-F ’24
46.41
0.55
81.15
0.53
67.75
0.51
64.38
0.53
39.97
0.74
VL-Cache [ 24 ]
ICLR ’25
53.59
1.18
81.68
1.04
70.75
0.91
67.63
0.97
46.27
1.25
AirCache [ 12 ]
ICCV ’25
46.21
0.83
78.53
0.58
71.00
0.89
64.25
0.90
37.99
0.93
Table 1: Answer accuracy and reasoning speed. Acc. is in percent, and Speed is relative to Dense ( × ). Bold marks column maxima. Compressed methods target B≈0.25MDense total KV. Actual total-KV retention spans 24.07–26.84% of Dense, except LOOK-M (30.97%) and AirCache (30.18%) on MME-Lite. Full per-dataset retentions are listed in Appendix F , Tab. S2 .
Figure 4: Compression changes both memory and search effort. Budget sweep on the fixed VisualProbe subset. Dashed lines denote Dense.
Configuration
Acc.
Total KV
Rounds
Time
(%) ↑
(%) ↓
(s) ↓
Dense
51.26
100.00
7.84
30.43
CanopyKV (full)
55.53
24.34
7.18
23.23
w/o source quotas
53.79
26.43
7.10
26.44
w/o crop lineage
46.99
26.22
7.14
22.45
w/o fixed history
52.23
24.10
7.27
24.81
Table 2: Single-component ablation on VisualProbe. Each variant removes one component. KV is normalized to Dense.
Figure 5: Crop lineage preserves complementary evidence. Delayed readout at matched source quotas and KV capacity for (a) synthetic facts inside/outside the incoming crop and (b) natural-image questions.
Figure 6: Stable history survives branch changes. Fixed versus uniformly resampled history on 128 synthetic scenes at matched KV capacity.
Figure 7: Low cache overhead and fast execution. Cache-operation costs (a) and execution time (b) on identical long MME-Lite search streams.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Intervention
Cases
Full (%)
Control (%)
Lineage (both regions, 16 visits)
128
91.41
77.93
Protected handoff tail
192
72.40
65.63
Appendix
Table S1 : Lineage and handoff protection at equal cache capacity. Readout accuracy over 128 synthetic scenes and answer accuracy over 192 natural-image questions.
Figure S1 : Lineage reduces repeated coverage across MME crops. Mean fraction of selected parent tokens overlapping the incoming crop, using equal parent quotas and averaging all saved heads.
Figure S2 : Compact history reduces trajectory and per-token cost. Top: request-time breakdown. Bottom: backend time per generated token. Each method uses the complete VisualProbe evaluation set.
Figure S3 : Spatial selections at an equal parent-token budget. Each method retains 28 parent tokens. Purple marks the incoming crop.
Method
VP
V*
4K
8K
MME
Dense
100.00
100.00
100.00
100.00
100.00
H 2 O
26.03
25.72
24.77
25.21
25.33
LOOK-M
24.75
24.55
24.75
25.86
30.97
VL-Cache
24.76
26.45
24.91
25.56
25.32
AirCache
26.84
26.15
24.56
25.75
30.18
MixKV
26.03
25.72
24.76
25.21
24.97
Appendix
Table S2 : Actual total-KV retention for all main-table methods. Values are percentages of each dataset's Dense mean.
Method
Hard
Medium
Easy
Dense
43.40
47.76
63.83
H 2 O
46.23
46.64
59.57
LOOK-M
34.91
44.03
59.57
VL-Cache
50.94
48.13
65.96
AirCache
40.57
44.03
54.61
MixKV
43.40
47.39
66.67
Appendix
Table S3: VisualProbe accuracy by difficulty in the main comparison. Percentages use all 106 Hard, 268 Medium, and 141 Easy questions. Bold marks the best reported accuracy.
Method
VP
V*
4K
8K
MME
Dense
7.84
3.80
3.97
4.07
10.01
H 2 O
8.50
18.52
5.92
4.25
9.18
LOOK-M
8.40
4.80
5.35
4.36
10.00
VL-Cache
7.12
3.60
3.50
3.80
6.80
AirCache
9.19
6.57
4.42
4.80
11.25
MixKV
6.61
3.10
3.19
3.68
5.32
Appendix
Table S4: Reasoning rounds in the main comparison. VP denotes VisualProbe, 4K/8K denote HR-Bench, and MME denotes MME-Lite. Rounds follow the main-comparison speed boundaries defined in Appendix B.1 .
AirCache
LOOK-M
Capacity (rows)
128
2
256
2
Total KV (%)
21.10
28.41
26.20
26.85
Text KV (%)
19.48
28.39
24.29
24.40
Visual KV (%)
1.61
0.02
1.91
2.45
Accuracy (%)
28.21
5.31
10.06
3.07
Rounds
12.96
19.19
17.07
18.28
Appendix
Table S5 : Stronger visual pruning can increase realized total KV. Capacity is the native visual-row budget. All KV percentages use Dense’s mean total KV on the same budget-sweep subset as the denominator.
Configuration
Hard
Medium
Easy
Dense
46/106 (43.40)
128/268 (47.76)
90/141 (63.83)
CanopyKV (full)
55/106 (51.89)
136/268 (50.75)
95/141 (67.38)
w/o source quotas
54/106 (50.94)
135/268 (50.37)
88/141 (62.41)
w/o crop lineage
42/106 (39.62)
115/268 (42.91)
85/141 (60.28)
w/o fixed history
53/106 (50.00)
127/268 (47.39)
89/141 (63.12)
w/o tail protection
44/106 (41.51)
115/268 (42.91)
85/141 (60.28)
Appendix
Table S6 : Single-component accuracy by difficulty. Each cell gives correct questions divided by the full difficulty population, followed by accuracy in percent.
Figure S4 : Exploration increases cache size and per-token cost. Dense on the 112 VisualProbe questions described in Appendix D.2 . Top: mean post-prefill KV. Bottom: median per-token backend time, normalized to each question’s original-image call. After six crops, KV grows 4.02× and normalized per-token time rises by 13.3%.
Reasoning vision-language models (VLMs) impose substantial KV cache demands from both large multimodal contexts and growing reasoning traces. Low-rank KV compression has emerged as a promising paradigm that stores low-dimensional latents of KV states and can be combined with quantization. However, existing low-rank methods often induce growing predictive policy drift from the dense model when compressing reasoning KV states, resulting in reasoning failures. We identify two key causes of this degradation: calibration-state mismatch between static contexts and on-policy reasoning states, and unequal error sensitivity across KV states. To address these challenges, we propose TraceKV, a training-free low-rank KV compression framework that unifies on-policy calibration, sensitivity-aware SVD, and global rank allocation under a single policy-preserving principle, substantially reducing policy-KL divergence during reasoning. Experiments across multiple VLMs and visual reasoning benchmarks demonstrate superior accuracy--memory trade-offs, achieving near-lossless reasoning accuracy with 5.64× KV cache compression when combined with quantization. Optimized low-rank attention kernels further deliver up to 1.83× end-to-end decoding speedup. Code will be available at https://github.com/AK-DREAM/TraceKV.
Ke Li, Zishan Shao, Yiran Chen
Department of Electrical and Computer Engineering, Duke University
Large Reasoning Models (LRMs) achieve superior problem-solving through extended Chain-of-Thought (CoT) generation, but the resulting key-value (KV) cache grows linearly with sequence length and creates severe memory bottlenecks, often exceeding GPU capacity for long reasoning traces. Existing KV cache compression methods rely on recent queries to estimate future token importance, implicitly assuming these serve as reliable proxies for future attention patterns. We demonstrate that this assumption fails in long-horizon reasoning: certain decoding steps generate Thought Revisiting Tokens (TRT) that re-attend to distant previous context, such as task-solving plans formulated early in the trace. Through systematic analysis, we discover that queries corresponding to the TRT cluster into a small number of similarity groups in the embedding space. Based on this insight, we propose BeaconKV, a training-free KV cache compression method that maintains beacon queries, compact representatives for each global query cluster, to anticipate which KV pairs will be revisited without storing the entire query history. Across four open-source LRMs and diverse reasoning benchmarks, BeaconKV generally outperforms existing compression methods, achieving up to 5.8× memory reduction while nearly preserving full cache accuracy and improving throughput by over 4.3×.
While Vision Large Language Models (VLLMs) have achieved remarkable success in multimodal reasoning, their long-context inference remains prohibitively expensive due to the massive computation and memory overhead of visual Key-Value (KV) caches. Existing KV compression methods often apply uniform pruning across visual tokens and layers, leading to substantial information loss and degraded performance.To address this challenge, we propose VisCache, a plug-and-play framework for coarse-to-fine Visual KV Cache pruning without training, which consists of two synergistic stages. First, a lightweight VLM filters temporal redundancy by selectively forwarding semantically informative keyframes. Second, we introduce {PruneKV}, a surgical KV compression algorithm tailored to the attention dynamics of VLLMs. Unlike rigid pruning strategies, PruneKV adopts a parabolic layer-wise budget allocation together with an asymmetric update mechanism that selectively prunes keys while fusing values, thereby preserving critical contextual information. Extensive experiments demonstrate that VisCache substantially improves inference efficiency, achieving up to {2.35× speedup} and significant memory reduction while maintaining competitive performance with only {19--28%} KV cache retention. VisCache consistently outperforms existing baselines, establishing a new Pareto frontier between efficiency and performance for long-context VLLM inference. Code is available at https://github.com/Wlklk/VisCache
Lyuke Wang, Zhuo Li, Guangxu Zhu
Shenzhen International Center for Industrial and Applied Mathematics · Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong, Shenzhen +1