CanopyKV: Remember beyond the Crop for Fast Active Visual Reasoning
Authors: Lei Yang, Dayan Wu, Yiliang Zhu, Xinze Liu, Hengjie Zhu, Zihao Zhang, Ding Wang, Peng Fu, Zheng Lin
Organizations: Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China · Department of Applied Mathematics and Statistics, John Hopkins University, Baltimore, MD, USA
Active visual reasoning enables multimodal models to answer challenging visual questions by gathering fine-grained evidence through iterative image inspection and tool use. However, accumulating crops and reasoning tokens expands the key-value (KV) cache, making compression important for reducing memory and inference costs. Existing KV compression methods typically select historical tokens based on attention, recency, or redundancy. These criteria do not explicitly distinguish locally replaced evidence from complementary information needed for future exploration: each crop refreshes a local region, while earlier views may still support subsequent decisions. Discarding such evidence can impair accuracy or require additional search, while scoring and cache reorganization can offset the computational savings from compression. We introduce CanopyKV, a training-free KV compression framework that uses parent-child crop relationships to retain complementary evidence, keeps retained history stable, and protects the state needed for ongoing reasoning. Tool metadata provides retention signals without additional attention scoring, while compact KV storage reduces subsequent attention costs. Across VisualProbe, V*, HR-Bench, and MME-Lite, CanopyKV achieves state-of-the-art answer accuracy and end-to-end efficiency among the evaluated KV compression methods. On MME-Lite, it retains approximately one-quarter of Dense's total KV while improving accuracy by 10.5 percentage points and accelerating complete reasoning by 1.64×.
Figures & tables
Figure 1: Local replacement and persistent complementarity. (a) Dense retains all acquired views, growing its KV cache. A1 refreshes its footprint but leaves C outside it. (b,c) At equal parent-KV budgets, CanopyKV retains complementary C-region evidence across A2 and B, while token-score pruning may discard it. Insets show parent history, not a newly acquired C.
Figure 2: CanopyKV retains complementary evidence in compact KV. (a) Source quotas and crop lineage favor context outside the incoming crop. (b) One-time retirement protects continuation state and fixes selected history across branch visits. (c) Incremental prefill is followed by visual pruning and KV compaction, preserving current text and logical positions.
Figure 3: Parent evidence outside the new crop is harder to replace. After removing matched amounts of parent KV inside or outside the incoming crop, we supply the full crop and read out the corresponding facts. The horizontal axes show KV retention within each target fact region.
Method
Venue
VisualProbe
V*
HR-Bench 4K
HR-Bench 8K
MME-Lite
Acc. ↑
Speed ↑
Acc. ↑
Speed ↑
Acc. ↑
Speed ↑
Acc. ↑
Speed ↑
Acc. ↑
Speed ↑
Dense
Reference
51.26
1.00
82.20
1.00
72.38
1.00
67.38
1.00
43.04
1.00
H 2 O [ 38 ]
NeurIPS ’23
50.10
0.38
31.94
0.10
66.25
0.30
66.88
0.37
42.68
0.44
LOOK-M [ 25 ]
EMNLP-F ’24
46.41
0.55
81.15
0.53
67.75
0.51
64.38
0.53
39.97
0.74
VL-Cache [ 24 ]
ICLR ’25
53.59
1.18
81.68
1.04
70.75
0.91
67.63
0.97
46.27
1.25
AirCache [ 12 ]
ICCV ’25
46.21
0.83
78.53
0.58
71.00
0.89
64.25
0.90
37.99
0.93
Table 1: Answer accuracy and reasoning speed. Acc. is in percent, and Speed is relative to Dense ( × ). Bold marks column maxima. Compressed methods target B≈0.25MDense total KV. Actual total-KV retention spans 24.07–26.84% of Dense, except LOOK-M (30.97%) and AirCache (30.18%) on MME-Lite. Full per-dataset retentions are listed in Appendix F , Tab. S2 .
Figure 4: Compression changes both memory and search effort. Budget sweep on the fixed VisualProbe subset. Dashed lines denote Dense.
Configuration
Acc.
Total KV
Rounds
Time
(%) ↑
(%) ↓
(s) ↓
Dense
51.26
100.00
7.84
30.43
CanopyKV (full)
55.53
24.34
7.18
23.23
w/o source quotas
53.79
26.43
7.10
26.44
w/o crop lineage
46.99
26.22
7.14
22.45
w/o fixed history
52.23
24.10
7.27
24.81
Table 2: Single-component ablation on VisualProbe. Each variant removes one component. KV is normalized to Dense.
Figure 5: Crop lineage preserves complementary evidence. Delayed readout at matched source quotas and KV capacity for (a) synthetic facts inside/outside the incoming crop and (b) natural-image questions.
Figure 6: Stable history survives branch changes. Fixed versus uniformly resampled history on 128 synthetic scenes at matched KV capacity.
Figure 7: Low cache overhead and fast execution. Cache-operation costs (a) and execution time (b) on identical long MME-Lite search streams.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Intervention
Cases
Full (%)
Control (%)
Lineage (both regions, 16 visits)
128
91.41
77.93
Protected handoff tail
192
72.40
65.63
Appendix
Table S1 : Lineage and handoff protection at equal cache capacity. Readout accuracy over 128 synthetic scenes and answer accuracy over 192 natural-image questions.
Figure S1 : Lineage reduces repeated coverage across MME crops. Mean fraction of selected parent tokens overlapping the incoming crop, using equal parent quotas and averaging all saved heads.
Figure S2 : Compact history reduces trajectory and per-token cost. Top: request-time breakdown. Bottom: backend time per generated token. Each method uses the complete VisualProbe evaluation set.
Figure S3 : Spatial selections at an equal parent-token budget. Each method retains 28 parent tokens. Purple marks the incoming crop.
Method
VP
V*
4K
8K
MME
Dense
100.00
100.00
100.00
100.00
100.00
H 2 O
26.03
25.72
24.77
25.21
25.33
LOOK-M
24.75
24.55
24.75
25.86
30.97
VL-Cache
24.76
26.45
24.91
25.56
25.32
AirCache
26.84
26.15
24.56
25.75
30.18
MixKV
26.03
25.72
24.76
25.21
24.97
Appendix
Table S2 : Actual total-KV retention for all main-table methods. Values are percentages of each dataset's Dense mean.
Method
Hard
Medium
Easy
Dense
43.40
47.76
63.83
H 2 O
46.23
46.64
59.57
LOOK-M
34.91
44.03
59.57
VL-Cache
50.94
48.13
65.96
AirCache
40.57
44.03
54.61
MixKV
43.40
47.39
66.67
Appendix
Table S3: VisualProbe accuracy by difficulty in the main comparison. Percentages use all 106 Hard, 268 Medium, and 141 Easy questions. Bold marks the best reported accuracy.
Method
VP
V*
4K
8K
MME
Dense
7.84
3.80
3.97
4.07
10.01
H 2 O
8.50
18.52
5.92
4.25
9.18
LOOK-M
8.40
4.80
5.35
4.36
10.00
VL-Cache
7.12
3.60
3.50
3.80
6.80
AirCache
9.19
6.57
4.42
4.80
11.25
MixKV
6.61
3.10
3.19
3.68
5.32
Appendix
Table S4: Reasoning rounds in the main comparison. VP denotes VisualProbe, 4K/8K denote HR-Bench, and MME denotes MME-Lite. Rounds follow the main-comparison speed boundaries defined in Appendix B.1 .
AirCache
LOOK-M
Capacity (rows)
128
2
256
2
Total KV (%)
21.10
28.41
26.20
26.85
Text KV (%)
19.48
28.39
24.29
24.40
Visual KV (%)
1.61
0.02
1.91
2.45
Accuracy (%)
28.21
5.31
10.06
3.07
Rounds
12.96
19.19
17.07
18.28
Appendix
Table S5 : Stronger visual pruning can increase realized total KV. Capacity is the native visual-row budget. All KV percentages use Dense’s mean total KV on the same budget-sweep subset as the denominator.
Configuration
Hard
Medium
Easy
Dense
46/106 (43.40)
128/268 (47.76)
90/141 (63.83)
CanopyKV (full)
55/106 (51.89)
136/268 (50.75)
95/141 (67.38)
w/o source quotas
54/106 (50.94)
135/268 (50.37)
88/141 (62.41)
w/o crop lineage
42/106 (39.62)
115/268 (42.91)
85/141 (60.28)
w/o fixed history
53/106 (50.00)
127/268 (47.39)
89/141 (63.12)
w/o tail protection
44/106 (41.51)
115/268 (42.91)
85/141 (60.28)
Appendix
Table S6 : Single-component accuracy by difficulty. Each cell gives correct questions divided by the full difficulty population, followed by accuracy in percent.
Figure S4 : Exploration increases cache size and per-token cost. Dense on the 112 VisualProbe questions described in Appendix D.2 . Top: mean post-prefill KV. Bottom: median per-token backend time, normalized to each question’s original-image call. After six crops, KV grows 4.02× and normalized per-token time rises by 13.3%.
Shenzhen International Center for Industrial and Applied Mathematics · Shenzhen Research Institute of Big Data · The Chinese University of Hong Kong, Shenzhen +1