Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents

Yuchen Huang; Sijia Li; Jun Zhang; Yi R. Fung

cs.AIAug 24, 2026

Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents

Authors: Yuchen Huang, Sijia Li, Jun Zhang, Yi R. Fung

Organizations: Hong Kong University of Science and Technology

Abstract

Multimodal Large Language Models (MLLMs) are increasingly deployed as multi-step agents, where explicit reasoning supports task decomposition and tool coordination but also accumulates self-generated text. Over long trajectories, this text can dominate the context and suppress visual evidence, creating textual debt. We observe that reasoning becomes redundant once task-relevant visual evidence is grounded, while stale hypotheses can misguide later inference when grounding remains uncertain. Pruning must therefore remove redundant text without discarding visual evidence. We propose SPARE, a Kullback-Leibler (KL)-guided framework for pruning accumulated reasoning in multimodal tool-use agents. SPARE uses a compact task-state summary as privileged diagnostic context. For each candidate segment, it replays the same model under the original and summary-conditioned contexts. Reverse-KL divergence from on-policy self-distillation (OPSD) then tests whether the summary sufficiently covers the segment without disrupting future reasoning. We further fine-tune the summarizer with supervised fine-tuning (SFT), enabling more compact summaries, broader coverage, and more aggressive pruning. Across multi-step visual tool-use benchmarks, SPARE achieves the highest average accuracy among pruning methods while removing 37.89-64.58% of reasoning tokens. This favorable accuracy-context trade-off shows that reducing textual dominance restores reliance on visual evidence and mitigates over-conditioning on self-generated language.

Figures & tables

Figure 1: Overview of SPARE. (a) In the full reasoning trace, accumulated self-generated text reinforces a stale hypothesis and obscures visual evidence returned by the tools, leading to an incorrect answer. (b) SPARE invokes a compact task-state summary as a transient probe that is not written into the persistent trajectory. It replays each historical reasoning segment with and without summary conditioning and measures its token-level residual sensitivity using normalized reverse KL. A count-based rule prunes summary-covered segments with insufficient high-KL residue while retaining evidence-critical segments. Each pruned segment is replaced by concise structured visual evidence extracted from its original reasoning, whereas planning-dominated text is removed and the original tool calls, observations, and images remain unchanged. The resulting context reduces textual debt, refocuses subsequent inference on visual evidence, and produces the correct answer.

Model / Metric	GTA $\uparrow$	V ∗ $\uparrow$	B-Jig. $\uparrow$	VTB $\uparrow$	MNMS $\uparrow$	Acc. $\uparrow$	Pr. $\%$
Qwen3-VL-30B-A3B-Instruct
Tool Baseline (Full Trace)	45.76	55.85	67.33	27.31	36.84	44.30	0.00
$-$ Tools (Direct Answer)	33.90	52.36	70.67	21.43	–	43.42	0.00
+ Delete All Reasoning	50.85	37.17	57.33	26.05	24.56	35.22	99.86
+ Visual Evidence-Only	59.32	61.78	64.00	26.47	25.44	42.73	77.09
+ SPARE (ours)	59.32	62.30	64.00	26.47	49.56	49.19	63.70

Table 1: Task performance and average reasoning pruning across five visual tool-use benchmarks. Benchmark columns report accuracy. Acc. and Pr. denote average accuracy and reasoning-token pruning rate, respectively. Note that MNMS evaluates tool planning and therefore is inapplicable for Direct Answer mode.

Figure 2: Accuracy–pruning trade-off averaged across three models on BLINK-Jigsaw, GTA, and MNMS. Accuracy and reasoning-pruning rates are weighted by the number of evaluation samples. The staircase connects the Pareto-optimal comparison methods, and the upper-right region indicates a better trade-off. SPARE lies above this frontier, attaining 50.34% accuracy while pruning 57.81% of reasoning tokens.

Model / Metric	VTB	MNMS
Base (Qwen3-VL-8B)	26.83	60.61
+ SFT summarizer	28.73	65.44
More pruned tokens vs. Base	23.8%	13.5%

Table 2: SFT results on VTB and MNMS.

Ablation Setting	Acc.	Pr.
Selective Pruning and Evidence Reconstruction
Full trace baseline	40.68	0.00
Summary probe only	35.59	0.00
${}+{}$ KL selection	40.68	0.00
${}+{}$ Evidence	37.29	56.08
${}+{}$ KL selection ${}+{}$ Evidence (ours)	42.37	62.42

Table 3: Ablations on GTA. Accuracy and pruning rates are percentages.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

		BLINK-Jigsaw		GTA		MNMS
Model	Method	Acc.	Pr.	Acc.	Pr.	Acc.	Pr.
Qwen3-VL-8B	Full Trace	74.00	0.00	42.37	0.00	60.96	0.00
	Random-33	70.67	34.97	38.98	46.59	57.46	34.49
	Random-50	60.67	46.31	44.07	54.99	58.33	50.37
	KL-guided	67.33	15.66	40.68	36.08	60.96	14.15
	Delete All	62.00	100.00	40.68	100.00	57.02	100.00

Appendix

Table 4: Complete per-model results on the three benchmarks. Acc. denotes accuracy, and Pr. denotes the reasoning-pruning rate. All values are percentages. Figure 2 aggregates the results using benchmark-size weighting: 150 samples for BLINK-Jigsaw, 59 answer-scored samples for GTA, and 228 samples for MNMS.

Model	Method	Acc. $\uparrow$	SCR $\downarrow$
Qwen3-VL-8B	Full Trace	63.64	36.36
	Random-1	72.73	27.27
	SPARE	90.91	9.09
Qwen3-VL-30B-A3B	Full Trace	54.55	45.45
	Random-1	72.73	22.73
	SPARE	81.82	18.18

Appendix

Table 5: Controlled textual-interference results on the constructed GTA subset. All values are percentages. Random-1 results are averaged over random draws; cases without an eligible deletion remain unchanged. SPARE is evaluated as an automatic procedure rather than with a manually specified deletion mask.

Figure 3: Text-to-image attention across decoder layers, averaged over 10 tool-use trajectories. Blue and orange denote attention before and after pruning, respectively, and the shaded region shows the gain. Pruning summary-covered reasoning consistently increases attention to image tokens, indicating stronger reliance on visual evidence.

Multimodal Agents

Multimodal agents perceive and reason across text, images, audio, and video to use tools, interact with interfaces, and act in the world.

See all 389 papers

ET-Prune: Evidence-Aware Dynamic Budgeting for Visual Token Pruning in Text-Rich MLLMs
Aug 3, 2026Zizhong Ding, Junxian Li, Kai Liu +4Efficient Multimodal Inference Multimodal Large Language Models
Shanghai Jiao Tong University · Xidian University
An AI4AI Framework for Visual Token Pruning
Aug 7, 2026Zhen Liu, Wenli Huang, Wei Song +3Automated Algorithm Discovery Efficient VLM Inference
Xi’an Jiaotong University · Ningbo University of Technology · 3North China University of Technology +3
Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs
Apr 13, 2026Kaizhen Tan, Yang Feng, Heqing Du +3VLM Evaluation Confidence Estimation for VLMs
Carnegie Mellon University, Pittsburgh, PA, USA · Columbia University, New York, NY, USA

Aug 3, 2026J/K move · Enter open · S save

ET-Prune: Evidence-Aware Dynamic Budgeting for Visual Token Pruning in Text-Rich MLLMs

Zizhong Ding, Junxian Li, Kai Liu, Shaoqiu Zhang+3

Shanghai Jiao Tong University · Xidian University

Efficient Multimodal Inference Multimodal Large Language Models Visual Token Pruning

Open paper AskPDF arXiv

Visual token pruning reduces the inference cost of multimodal large language models, but a fixed token ratio is poorly matched to text-rich inputs. In OCR-centric tasks, decisive evidence can be a small number, label, or field whose relevance is specified by the question; indiscriminate pruning can erase that evidence while retaining visually salient but irrelevant regions. We present ET-Prune, a training-free framework that casts pruning as evidence allocation. It derives question-conditioned evidence from a decoder-side partial query-key block, safeguards text-like spatial regions, and converts evidence uncertainty and density into a sample-specific token floor. Three progressive middle-layer events then move the sequence toward this budget, retaining more tokens for diffuse or text-dense evidence and pruning concentrated evidence more aggressively. At the observed point estimates from one deterministic pass per configuration, ET-Prune leads or ties among pruned methods in all six backbone-benchmark comparisons at roughly half tokens. On OCRBench-v2, it leads the strongest pruned baselines by 1.80 and 0.68 percentage points on Qwen3-VL-8B and InternVL3.5-8B, respectively, while retaining about half of the visual tokens; on MMBench v1.1, it reaches 0.8467 circular exact-matching accuracy versus 0.8437 for Vanilla at 54.45% average visual-token retention. These results show a favorable observed quality-cost trade-off for evidence-aware dynamic budgeting in text-rich multimodal inference.

Abstract

Multimodal Agents

Explore similar work

ET-Prune: Evidence-Aware Dynamic Budgeting for Visual Token Pruning in Text-Rich MLLMs

An AI4AI Framework for Visual Token Pruning

Confidence under Visual Token Pruning: Removed Evidence and Risk-Controlled Token Budgets for MLLMs