ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models

Guangzong Si; Dong Wang; Yifan Yu; Miao Liu; Panwang Pan; Wentao Zhu

cs.CVOct 18, 2026

ReTrace: Visual Backtracking for Efficient Multimodal Large Language Models

Authors: Guangzong Si, Dong Wang, Yifan Yu, Miao Liu, Panwang Pan, Wentao Zhu

Organizations: University of Science and Technology of China · Eastern Institute of Technology, Ningbo · Tsinghua University · ETH Zurich ·

Abstract

Multimodal Large Language Models (MLLMs) encode images into dense visual token sequences, creating substantial attention and KV-cache overhead. Existing visual token compression methods reduce this cost by pruning low-saliency tokens in shallow layers, but this practice couples temporary computational deactivation with irreversible information deletion. We show that this coupling breaks an intrinsic inference pattern in MLLMs. Through layer-wise attention analysis, we identify Visual Backtracking: visual pivots remain spatially stable, yet in middle layers they actively retrieve context from tokens that appeared unimportant earlier. Thus, early low-saliency tokens can be inactive for computation while still being useful for later reasoning. Motivated by this finding, we propose ReTrace, a backtracking-aware compression framework that introduces a dormant token state. ReTrace caches non-pivot tokens outside expensive layer-wise computation, restores them during the backtracking window, and prunes only after their information has been absorbed. Experiments on Qwen2.5-VL and InternVL3 demonstrate nearly 90% reductions in FLOPs and KV cache, a 2.83 $\times$ prefilling speedup, and strong performance retention across complex vision-language benchmarks.

Figures & tables

Figure 1: Conceptual paradigm of the Visual Backtracking phenomenon. In shallow layers, pivot tokens interact internally to build clear feature representations. In mid layers, a “backtracking” mechanism emerges, where pivot tokens strongly query unimportant visual tokens to absorb global context. Finally, in deep layers, forms Enriched Visual Tokens essential for complex reasoning.

Figure 2: Visualization of pivot tokens across different layers. We highlight the top 10% visual tokens with the highest saliency scores. The selected regions remain highly consistent from shallow to deep layers, intuitively demonstrating the spatial stability of the model’s visual attention.

Panel of Figure 3 — Figure 3: Quantitative evidence of visual backtracking. We analyze the layer-wise behavior of text-guided visual pivot tokens. (a) The selected pivots remain spatially stable across layers, indicating that the model establishes persistent visual pivots early. (b) Pivot-to-visual attention exhibits a distinct middle-layer expansion band, where pivots attend broadly to non-pivot visual tokens. (c) The normalized attention flow shows that pivots shift from pivot-internal consolidation in shallow layers to pivot-to-context retrieval in middle layers, revealing the visual backtracking window.

Figure 4: Performance comparison of different feature integration paradigms. Under identical retention ratios (10%–50%), backtracking-aware injection consistently outperforms rhythm-agnostic fusion at moderate ratios. At the extreme 10% retention rate, injecting into an undersized buffer triggers Information Overload , revealing the need for a multi-stage design.

Figure 5: Overall architecture of ReTrace. ReTrace follows a cache–inject–prune schedule. (1) Pivot Buffer Construction builds a compact active receiver buffer at the encoder output while caching the remaining tokens as dormant context. (2) Backtracking-Aware Context Injection restores the missing pivot-to-context retrieval route at the backtracking layer through similarity-based routing. (3) Post-Absorption Task Pruning removes task-irrelevant enriched pivots after context absorption.

Method	Venue	MMB	MMB CN	MME	POPE	SQA I	MMStar	MRW	MRW CN	Avg.
Qwen2.5VL-7B	-	83.5	83.4	2347	87.7	88.1	63.9	57.4	50.2	100.0%
Token Reduction Setting: Retain 33.3% Tokens ( $\downarrow$ 66.7%)
FastV Chen et al. (2024a)	ECCV’24	75.7	78.8	2072	86.3	85.1	49.4	46.1	42.5	88.9%
VisionZip Yang et al. (2025)	CVPR’25	83.2	80.7	2160	86.7	86.0	53.8	48.4	44.2	92.7%
Dart Wen et al. (2025)	EMNLP’25	80.1	78.3	2166	85.1	84.3	48.9	45.7	39.0	88.6%
HoloV Zou et al. (2025)	NeurIPS’25	78.3	78.3	2093	84.9	85.8	48.1	35.3	34.7	84.7%

Table 1: Performance comparisons on Qwen2.5VL-7B across 8 benchmarks. The “Avg.” metric denotes the mean relative performance retention across all datasets compared to the baseline.

Method	Venue	MMB	MMB CN	MME	POPE	SQA I	MMStar	MRW	MRW CN	Avg.
InternVL3-8B	-	83.5	83.4	2415	90.7	98.2	63.9	62.0	60.8	100.0%
Token Reduction Setting: Retain 22.2% Tokens ( $\downarrow$ 77.8%)
FastV Chen et al. (2024a)	ECCV’24	83.8	83.0	2324	89.4	95.4	58.2	46.3	51.9	92.7%
VisionZip Yang et al. (2025)	CVPR’25	80.3	82.0	2242	88.8	95.0	56.8	44.8	49.5	90.4%
HoloV Zou et al. (2025)	NeurIPS’25	80.6	82.3	2201	88.7	95.4	57.0	44.7	50.0	90.5%
Ours	-	83.6	84.3	2319	90.2	96.7	59.7	48.7	53.3	94.4%

Table 2: Performance comparisons on InternVL3-8B across 8 benchmarks. The “Avg.” metric denotes the mean relative performance retention across all datasets compared to the baseline.

Method	MMB	MME	MMStar	MRW CN	Avg.
Qwen2.5VL-3B	79.1	2157	55.9	45.1	100.0%
FastV Chen et al. (2024a)	71.0	1698	36.6	32.2	76.3%
VisionZip Yang et al. (2025)	73.2	1698	40.1	31.9	78.4%
Dart Wen et al. (2025)	72.8	1640	36.8	29.4	74.8%
HoloV Zou et al. (2025)	70.8	1673	38.6	25.1	72.9%
VLMPruner Wu et al. (2026)	68.2	1735	36.0	26.7	72.6%

Table 3: Compact comparison on Qwen2.5VL-3B at 11.1% token retention.

Panel of Figure 6 — Figure 6: Performance degradation trends across different token retention ratios. We compare the degradation curves on Qwen2.5-VL-7B and InternVL3-8B under different visual token retention ratios.

Method	MMB	MME	MMStar	MRW CN	Avg.
InternVL3-2B	80.8	2221	60.7	52.1	100.0%
FastV Chen et al. (2024a)	54.0	1398	36.4	25.9	59.9%
VisionZip Yang et al. (2025)	67.7	1747	42.5	32.9	73.9%
HoloV Zou et al. (2025)	70.1	1743	44.8	34.8	76.5%
Ours	74.3	1965	47.9	37.1	82.6%

Table 4: Compact comparison on InternVL3-2B at 11.1% token retention.

Figure 7: Effect of backtracking-aware injection. We report the relative gains brought by context injection under different retention ratios.

Figure 8: Ablation on injection layer placement. We evaluate different injection layers and observe the most balanced performance around the middle-layer backtracking window.

Method	Tokens $\downarrow$	FLOPs $\downarrow$	KV $\downarrow$	MME $\uparrow$	Prefill Speedup $\uparrow$
Qwen2.5VL-7B	100%	1043.83	211.4 MB	2347	1.00 $\times$
+ Ours	22.2%	279.82	37.0 MB	2160	2.28 $\times$
+ Ours	11.1%	148.06	26.7 MB	2093	2.83 $\times$

Table 5: Compact efficiency results on Qwen2.5VL-7B. Runtime is evaluated on a single NVIDIA A100-80GB GPU with FlashAttention.

Panel of Figure 9 — Figure 9: Qualitative result at 11.1% visual-token retention.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Average Retention Ratio	$\rho_{1}$	$\rho_{2}$
11.1%	13.7%	33.3%
22.2%	27.4%	33.3%
33.3%	41.17%	33.3%

Appendix

Table 6: Hyperparameter configurations for different target compression levels. The late-stage retention

\rho_{2}

is strictly fixed, while the mid-stage retention

\rho_{1}

adaptively scales to meet the overall average retention ratio.

Panel of Figure 10 — Figure 10: Cross-layer spatial stability of pivot tokens under varying pivot ratios. The cross-layer stability of pivot tokens remains consistently high across different top- $k$ ratio.

Panel of Figure 11 — Figure 11: Attention heatmap depicting the backtracking band. The attention heatmap from pivot tokens to all visual tokens consistently reveals a distinct vertical bright band in the middle layers, regardless of the pivot ratio .

Panel of Figure 12 — Figure 12: Temporal evolution of normalized attention proportions. The temporal evolution shows an abrupt “Visual Backtracking” peak (orange curve) where pivot tokens strongly query the background across different top- $k$ ratio settings .

Figure 13: Layer-wise visualization of the visual backtracking phenomenon. We present the attention heatmaps from pivot tokens to all visual tokens across all transformer layers. The visualization clearly demonstrates a distinct dynamic: while attention remains highly localized on main subjects in the shallow and deep layers, it undergoes a dramatic spatial expansion in the mid-layers to actively query seemingly unimportant background tokens.

Figure 14: Layer-wise visualization of the visual backtracking phenomenon.

Panel of Figure 15 — Figure 15: Evolution of 3D feature representations across intermediate layers.

Method	Venue	MMB	MMB CN	MME	POPE	SQA I	MMStar	MRW	MRW CN	Avg.
InternVL3-8B	-	83.5	83.4	2415	90.7	98.2	63.9	62.0	60.8	100.0%
Token Reduction Setting: Retain 33.3% Tokens ( $\downarrow$ 66.7%)
FastV Chen et al. (2024a)	ECCV’24	84.2	84.3	2347	90.0	96.2	59.2	50.2	54.7	95.1%
VisionZip Yang et al. (2025)	CVPR’25	81.3	84.3	2290	89.4	97.1	60.5	50.8	53.8	94.2%
HoloV Zou et al. (2025)	NeurIPS’25	82.4	84.2	2285	89.7	97.3	60.6	50.2	53.8	94.4%
Ours	-	85.1	85.7	2351	90.5	97.5	61.2	53.0	55.9	96.8%

Appendix

Table 7: Performance comparisons on InternVL3-8B across 8 benchmarks.

Method	Venue	MMB	MMB CN	MME	POPE	SQA I	MMStar	MRW	MRW CN	Avg.
Qwen2.5VL-3B	-	79.1	78.1	2157	87.3	81.0	55.9	53.1	45.1	100.0%
Token Reduction Setting: Retain 33.3% Tokens ( $\downarrow$ 66.7%)
FastV Chen et al. (2024a)	ECCV’24	73.7	73.4	2026	84.2	80.3	44.4	39.2	36.7	88.9%
VisionZip Yang et al. (2025)	CVPR’25	76.0	73.6	1964	84.2	79.8	48.0	40.2	37.7	90.2%
Dart Wen et al. (2025)	EMNLP’25	75.3	73.7	1950	82.8	78.9	36.8	40.3	36.0	86.7%
HoloV Zou et al. (2025)	NeurIPS’25	76.1	74.0	2015	83.5	81.2	44.1	36.3	33.5	87.7%

Appendix

Table 8: Performance comparisons on Qwen2.5VL-3B across 8 benchmarks. The “Avg.” metric denotes the mean relative performance retention across all datasets compared to the baseline.

Method	Venue	MMB	MMB CN	MME	POPE	SQA I	MMStar	MRW	MRW CN	Avg.
InternVL3-2B	-	80.8	79.0	2221	89.9	95.7	60.7	53.8	52.1	100.0%
Token Reduction Setting: Retain 33.3% Tokens ( $\downarrow$ 66.7%)
FastV Chen et al. (2024a)	ECCV’24	67.2	65.5	1570	75.3	79.6	43.5	25.5	33.2	73.3%
VisionZip Yang et al. (2025)	CVPR’25	78.9	76.3	2041	87.2	93.2	51.8	33.5	41.9	88.6%
HoloV Zou et al. (2025)	NeurIPS’25	79.2	76.0	1997	88.0	92.9	52.8	34.2	42.6	88.9%
Ours	-	80.6	76.9	2085	89.2	94.2	53.7	37.1	44.9	91.5%

Appendix

Table 9: Performance comparisons on InternVL3-2B across 8 benchmarks. The “Avg.” metric denotes the mean relative performance retention across all datasets compared to the baseline.

Model	Method	11%	22%	33%	50%	100%
32B	FastV	70.13	78.09	81.01	85.40	87.37
	PDrop	73.22	79.04	82.74	86.03	87.37
	Vscan	78.09	82.47	85.31	86.17	87.37
	Ours	82.21	85.48	86.42	86.85	87.37

Appendix

Table 10: MMBench accuracy under different visual token retention ratios. Higher is better.

Model	Method	11%	22%	33%	50%	100%
32B	FastV	67.61	76.12	79.81	83.42	86.00
	PDrop	71.31	77.65	80.72	84.28	86.00
	Vscan	77.23	80.76	84.45	84.78	86.00
	Ours	80.67	83.93	84.53	84.96	86.00

Appendix

Table 11: MMBench-CN accuracy under different visual token retention ratios. Higher is better.

Model	Method	11%	22%	33%	50%	100%
32B	FastV	1536	1924	2124	2316	2432
	PDrop	1536	1797	2078	2279	2432
	Vscan	2015	2189	2288	2378	2432
	Ours	2037	2306	2373	2414	2432

Appendix

Table 12: MME total scores under different visual token retention ratios. Higher is better.

Retention	Method	MMB	MMB-CN	MME	POPE	SQA	MMStar	MRW	MRW-CN	Avg.
22.2%	HiMap	78.3	77.2	1935	83.7	84.2	48.3	40.9	39.5	85.7%
22.2%	PDrop	79.7	79.4	2017	83.3	85.3	49.3	42.3	39.4	87.2%
22.2%	G-Search	80.3	78.9	2103	84.6	83.9	50.5	41.8	39.9	87.9%
22.2%	ReTrace	81.7	80.8	2160	86.3	86.1	52.1	44.4	42.0	90.7%
11.1%	HiMap	77.2	75.1	1847	79.9	81.1	43.3	34.2	33.1	79.7%
11.1%	PDrop	76.9	75.5	1932	80.2	81.2	44.0	35.0	34.9	81.0%

Appendix

Table 13: Comparison under different visual-token retention ratios. ReTrace consistently outperforms representative visual-token compression methods across both retention settings.

Table 14: Ablation studies on injection layer placement and pivot extraction strategies (Qwen2.5-VL-7B).

Method	Retention: 33.3%				Retention: 22.2%
Method	MME	POPE	SQA	MMStar	MME	POPE	SQA	MMStar
w/o inject	2173	85.7	86.6	54.4	2104	85.0	85.7	51.6
w/ inject (Ours)	2240	87.5	87.1	55.3	2160	86.3	86.1	52.1

Appendix

Table 15: Ablation study on the injection mechanism.

Layer Index	MME	MMStar
9	1786	41.9
11	1789	41.8
13	1800	41.6
15	2006	44.3
17	2053	45.7
19 (Ours)	2093	46.3

Appendix

Table 16: Ablation study on layer selection. Performance on MME and MMStar benchmarks when applying our extraction strategy at different intermediate layers.

$p_{2}$	MME	SQA I	MMStar
1/5	2054	80.9	44.9
1/4	2069	81.4	43.3
1/3	2093	82.3	46.3
1/2	2050	82.2	43.4

Appendix

Table 17: Ablation study on the intermediate layer pruning

p_{2}

. The total token retention budget is strictly fixed at 11.1%. Setting the coefficient

p_{2}

to 1/3 yields the optimal balance across multiple benchmarks.

Method	Tokens $\downarrow$	Total Time $\downarrow$	Prefilling Time $\downarrow$	FLOPs $\downarrow$	KV Cache $\downarrow$	MME $\uparrow$	Speedup $\uparrow$
Method	Tokens $\downarrow$	(Sec)	(Sec)	FLOPs $\downarrow$	(MB)	(Score)	Total	Prefilling
Qwen2.5VL-7B	100%	557.443	239.833	1043.83	211.4	2347	1.00 $\times$	1.00 $\times$
+ Ours	22.2%	401.919	105.234	279.82	37.0	2160	1.38 $\times$	2.28 $\times$
+ Ours	11.1%	373.961	84.809	148.06	26.7	2093	1.49 $\times$	2.83 $\times$

Appendix

Table 18: Efficiency results on Qwen2.5VL-7B. Performance, runtime, FLOPs, and memory overhead are evaluated on a single NVIDIA A100-80GB GPU with FlashAttention.

Tokens $\downarrow$	Total Time $\downarrow$	$T_{d}$ Extraction		Mid-stage Injection		Overhead Proportion $\downarrow$
Tokens $\downarrow$	(Sec)	Total (s)	Avg. (ms)	Total (s)	Avg. (ms)	$T_{d}$ Ext.	Inject
33.3%	426.85	1.78	0.8	0.20	0.1	0.42%	0.05%
22.2%	401.92	1.66	0.7	0.21	0.1	0.41%	0.05%
11.1%	373.96	1.58	0.7	0.23	0.1	0.42%	0.06%

Appendix

Table 19: Algorithmic overhead analysis under different token retention settings on Qwen2.5-VL-7B. The absolute execution times and their relative proportions explicitly demonstrate that the spatial diversity extraction (

T_{d}

) and mid-stage injection introduce negligible latency to the overall inference.

Panel of Figure 16 — Figure 16: Visualization of pivot buffer construction under different sparsity levels. As the token retention ratio increases from 10% to 50%, the buffer progressively covers more visual regions while consistently preserving core semantic objects, local structures, and distributed context. This balanced receiver set supports later backtracking-aware context injection.

Panel of Figure 17 — Figure 17: More qualitative results on retain 22.2% visual tokens. Using LLaVA-bench, we have demonstrated the effectiveness of our method on OpenQA, where the model’s hallucination-free responses fully showcase the validity of our method.

Panel of Figure 18 — Figure 18: Qualitative comparison.

Panel of Figure 19 — Figure 19: Qualitative comparison.

Panel of Figure 20 — Figure 20: Qualitative comparison.

Panel of Figure 21 — Figure 21: Qualitative comparison.

Panel of Figure 22 — Figure 22: Qualitative comparison.

Panel of Figure 23 — Figure 23: Qualitative comparison.

Panel of Figure 24 — Figure 24: Qualitative comparison.

Efficient VLM Inference

Efficient vision-language model inference reduces the compute, memory, latency, and energy needed to process images, video, and text.

See all 485 papers

DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction
Oct 13, 2026Haolan Wei, Jiahui Wang, Junli Zha +4VLM Distillation Efficient VLM Inference
SF Technology Co., Ltd, Shenzhen, China
Rescuing Your Pruning: Rethinking Visual Token Pruning and Post-Training Recovery in MLLMs
Oct 24, 2026Peiran Yin, Hao Wu, Zihao Yang +3Visual Token Pruning Efficient VLM Inference
EIT-NLP Lab, Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University · Uppsala University
Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models
Sep 28, 2026Jingdi lei, Junxian Li, Di Zhang +3Efficient Multimodal Inference Multimodal Large Language Models
Nanyang Technological University · Shanghai Jiao Tong University · Fudan University +2

Oct 13, 2026J/K move · Enter open · S save

DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction

Haolan Wei, Jiahui Wang, Junli Zha, Xinkai Lu+3

SF Technology Co., Ltd, Shenzhen, China

VLM Distillation Efficient VLM Inference

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have achieved strong performance on diverse vision-language tasks, but their computational demands limit deployment under constrained budgets, particularly on edge devices. Many works focus on Knowledge Distillation (KD) and try to transfer knowledge from powerful teacher models to students. In addition to reducing model size, visual token compression provides a complementary route to efficiency, but introduces mismatched visual token counts that complicate direct token-wise knowledge transfer. To mitigate this, we propose DVTR-KD, a knowledge distillation framework based on Dense Visual Token Reconstruction, which reconstructs the teacher's dense visual tokens from the student's compressed tokens. We reconstruct visual states after the final LLM layer and at the input to an intermediate layer identified as important for visual-information readout. At the intermediate layer, reconstruction is supervised through the teacher's response predictions rather than by direct token-wise KL or MSE matching. An auxiliary attention-readout loss complements this response-level supervision by aligning how the student and the original teacher integrate contextual information when predicting responses. Evaluations on diverse benchmarks demonstrate that DVTR-KD improves the performance of token-compressed students while retaining the inference efficiency afforded by visual token compression.

Abstract

Efficient VLM Inference

Explore similar work

DVTR-KD: Distilling Vision-Language Models via Intermediate Visual Token Reconstruction

Rescuing Your Pruning: Rethinking Visual Token Pruning and Post-Training Recovery in MLLMs

Just MLPs: Efficient Visual State Reconstruction for Multimodal Language Models