Multimodal Large Language Models (MLLMs) are increasingly deployed as multi-step agents, where explicit reasoning supports task decomposition and tool coordination but also accumulates self-generated text. Over long trajectories, this text can dominate the context and suppress visual evidence, creating textual debt. We observe that reasoning becomes redundant once task-relevant visual evidence is grounded, while stale hypotheses can misguide later inference when grounding remains uncertain. Pruning must therefore remove redundant text without discarding visual evidence. We propose SPARE, a Kullback-Leibler (KL)-guided framework for pruning accumulated reasoning in multimodal tool-use agents. SPARE uses a compact task-state summary as privileged diagnostic context. For each candidate segment, it replays the same model under the original and summary-conditioned contexts. Reverse-KL divergence from on-policy self-distillation (OPSD) then tests whether the summary sufficiently covers the segment without disrupting future reasoning. We further fine-tune the summarizer with supervised fine-tuning (SFT), enabling more compact summaries, broader coverage, and more aggressive pruning. Across multi-step visual tool-use benchmarks, SPARE achieves the highest average accuracy among pruning methods while removing 37.89-64.58% of reasoning tokens. This favorable accuracy-context trade-off shows that reducing textual dominance restores reliance on visual evidence and mitigates over-conditioning on self-generated language.
Figures & tables
Figure 1: Overview of SPARE. (a) In the full reasoning trace, accumulated self-generated text reinforces a stale hypothesis and obscures visual evidence returned by the tools, leading to an incorrect answer. (b) SPARE invokes a compact task-state summary as a transient probe that is not written into the persistent trajectory. It replays each historical reasoning segment with and without summary conditioning and measures its token-level residual sensitivity using normalized reverse KL. A count-based rule prunes summary-covered segments with insufficient high-KL residue while retaining evidence-critical segments. Each pruned segment is replaced by concise structured visual evidence extracted from its original reasoning, whereas planning-dominated text is removed and the original tool calls, observations, and images remain unchanged. The resulting context reduces textual debt, refocuses subsequent inference on visual evidence, and produces the correct answer.
Model / Metric
GTA ↑
V ∗ ↑
B-Jig. ↑
VTB ↑
MNMS ↑
Acc. ↑
Pr. %
Qwen3-VL-30B-A3B-Instruct
Tool Baseline (Full Trace)
45.76
55.85
67.33
27.31
36.84
44.30
0.00
− Tools (Direct Answer)
33.90
52.36
70.67
21.43
–
43.42
0.00
+ Delete All Reasoning
50.85
37.17
57.33
26.05
24.56
35.22
99.86
+ Visual Evidence-Only
59.32
61.78
64.00
26.47
25.44
42.73
77.09
+ SPARE (ours)
59.32
62.30
64.00
26.47
49.56
49.19
63.70
Table 1: Task performance and average reasoning pruning across five visual tool-use benchmarks. Benchmark columns report accuracy. Acc. and Pr. denote average accuracy and reasoning-token pruning rate, respectively. Note that MNMS evaluates tool planning and therefore is inapplicable for Direct Answer mode.
Figure 2: Accuracy–pruning trade-off averaged across three models on BLINK-Jigsaw, GTA, and MNMS. Accuracy and reasoning-pruning rates are weighted by the number of evaluation samples. The staircase connects the Pareto-optimal comparison methods, and the upper-right region indicates a better trade-off. SPARE lies above this frontier, attaining 50.34% accuracy while pruning 57.81% of reasoning tokens.
Model / Metric
VTB
MNMS
Base (Qwen3-VL-8B)
26.83
60.61
+ SFT summarizer
28.73
65.44
More pruned tokens vs. Base
23.8%
13.5%
Table 2: SFT results on VTB and MNMS.
Ablation Setting
Acc.
Pr.
Selective Pruning and Evidence Reconstruction
Full trace baseline
40.68
0.00
Summary probe only
35.59
0.00
+ KL selection
40.68
0.00
+ Evidence
37.29
56.08
+ KL selection + Evidence (ours)
42.37
62.42
Table 3: Ablations on GTA. Accuracy and pruning rates are percentages.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
BLINK-Jigsaw
GTA
MNMS
Model
Method
Acc.
Pr.
Acc.
Pr.
Acc.
Pr.
Qwen3-VL-8B
Full Trace
74.00
0.00
42.37
0.00
60.96
0.00
Random-33
70.67
34.97
38.98
46.59
57.46
34.49
Random-50
60.67
46.31
44.07
54.99
58.33
50.37
KL-guided
67.33
15.66
40.68
36.08
60.96
14.15
Delete All
62.00
100.00
40.68
100.00
57.02
100.00
Appendix
Table 4: Complete per-model results on the three benchmarks. Acc. denotes accuracy, and Pr. denotes the reasoning-pruning rate. All values are percentages. Figure 2 aggregates the results using benchmark-size weighting: 150 samples for BLINK-Jigsaw, 59 answer-scored samples for GTA, and 228 samples for MNMS.
Model
Method
Acc. ↑
SCR ↓
Qwen3-VL-8B
Full Trace
63.64
36.36
Random-1
72.73
27.27
SPARE
90.91
9.09
Qwen3-VL-30B-A3B
Full Trace
54.55
45.45
Random-1
72.73
22.73
SPARE
81.82
18.18
Appendix
Table 5: Controlled textual-interference results on the constructed GTA subset. All values are percentages. Random-1 results are averaged over random draws; cases without an eligible deletion remain unchanged. SPARE is evaluated as an automatic procedure rather than with a manually specified deletion mask.
Figure 3: Text-to-image attention across decoder layers, averaged over 10 tool-use trajectories. Blue and orange denote attention before and after pruning, respectively, and the shaded region shows the gain. Pruning summary-covered reasoning consistently increases attention to image tokens, indicating stronger reliance on visual evidence.
Visual token pruning reduces the inference cost of multimodal large language models, but a fixed token ratio is poorly matched to text-rich inputs. In OCR-centric tasks, decisive evidence can be a small number, label, or field whose relevance is specified by the question; indiscriminate pruning can erase that evidence while retaining visually salient but irrelevant regions. We present ET-Prune, a training-free framework that casts pruning as evidence allocation. It derives question-conditioned evidence from a decoder-side partial query-key block, safeguards text-like spatial regions, and converts evidence uncertainty and density into a sample-specific token floor. Three progressive middle-layer events then move the sequence toward this budget, retaining more tokens for diffuse or text-dense evidence and pruning concentrated evidence more aggressively. At the observed point estimates from one deterministic pass per configuration, ET-Prune leads or ties among pruned methods in all six backbone-benchmark comparisons at roughly half tokens. On OCRBench-v2, it leads the strongest pruned baselines by 1.80 and 0.68 percentage points on Qwen3-VL-8B and InternVL3.5-8B, respectively, while retaining about half of the visual tokens; on MMBench v1.1, it reaches 0.8467 circular exact-matching accuracy versus 0.8437 for Vanilla at 54.45% average visual-token retention. These results show a favorable observed quality-cost trade-off for evidence-aware dynamic budgeting in text-rich multimodal inference.
Visual-token pruning can substantially reduce the inference cost of multimodal large language models (MLLMs), yet existing methods largely rely on fixed, handcrafted heuristics and costly expert trial and error. As pruning objectives, budgets, and model architectures diversify, manually navigating the expanding design space becomes increasingly difficult. This paper aims to build an AI4AI framework for visual-token pruning by addressing a natural question: Can large language models automatically design effective visual-token reduction algorithms? Although LLMs possess broad algorithmic knowledge and strong reasoning capabilities, translating such general knowledge into effective solutions for a specialized task remains nontrivial. We argue that the key lies in designing an appropriate search-state representation that connects the internal knowledge of LLMs with the structural requirements and constraints of visual-token pruning. Based on this insight, we propose AutoPrune, a training-free framework for LLM-driven visual-token pruning policy design. At its core, AutoPrune introduces a Token Pruning Domain-Specific Language (TPDSL) comprising 131 reusable atoms for budget control, token scoring, selection constraints, and token reassembly. A key property of TPDSL is that it represents each search state as a residual modification of a strong base policy. This residual formulation narrows the search space and directs the LLM's attention toward the policy components that are most consequential for performance. Experiments on 14 multimodal benchmarks and three MLLM backbones demonstrate the effectiveness, efficiency, and transferability of AutoPrune. Even when removing 94.4% of visual tokens, AutoPrune preserves more than 99% of full-token performance while reducing FLOPs by 9.9x and prefill latency by 6.4x.
Zhen Liu, Wenli Huang, Wei Song +3
Xi’an Jiaotong University · Ningbo University of Technology · 3North China University of Technology +3
Visual token pruning speeds up multimodal large language models (MLLMs) by keeping a small subset of the visual tokens, and pruning methods are compared by the accuracy they retain. We study what pruning does to the confidence of these models, across common selectors, several MLLMs, and different output formats. Pruning errors concentrate on questions whose evidence the selector removed, and confidence does not register the removal. When the queried object loses all of its tokens, accuracy on these questions drops from 59% to 17%, while confidence stays at the unpruned level. Returning a few object tokens to the kept set recovers most of the lost accuracy. Selectors that keep the most attended tokens remove such evidence most often and produce confident errors, which temperature scaling cannot re-rank. Selectors that avoid keeping redundant tokens stay close to the calibration of the unpruned model. We then use the confidence of the pruned model to set a per-question token budget. The model answers with few tokens first and again with all tokens when its confidence is low. Conformal risk control sets the threshold to bound the expected deviation from the unpruned model. With coverage-based selection, this cascade needs about a third of the prefill tokens of the unpruned model, while with FastV it needs more than the unpruned model. The savings come mainly from how well the confidence ranks the answers that differ from the unpruned ones.
Kaizhen Tan, Yang Feng, Heqing Du +3
Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA