Where and How to Prune: An Empirical Study of Visual Token Pruning for GUI Agent Navigation

Daiqiang Li; Zihao Pan; Zeyu Zhang; Xuyang Liu; Shijia Xu; Ronghao Chen; Huacan Wang; Honggang Chen

cs.CVMar 27, 2026

Where and How to Prune: An Empirical Study of Visual Token Pruning for GUI Agent Navigation

Authors: Daiqiang Li, Zihao Pan, Zeyu Zhang, Xuyang Liu, Shijia Xu, Ronghao Chen, Huacan Wang, Honggang Chen,

Organizations: Shanghai Jiao Tong University · Sichuan University · The University of Hong Kong · University of California, Berkeley ·

Abstract

In recent years, GUI agents have demonstrated strong potential in navigation tasks. However, preserving complete historical screenshots introduces substantial computational overhead. This paper investigates how token pruning, a plug-and-play inference acceleration technique, can be effectively applied to GUI agent navigation scenarios. Firstly, we address the question of where pruning should occur. We identify a system-level redundancy overlooked by existing methods: as the same screenshot is repeatedly fed into the model across different steps, its ViT encoding is redundantly recomputed each time. We show that its ViT-encoded embeddings can be fully cached and reused across steps, substantially reducing FLOPs while preserving model performance. This finding suggests that inference acceleration efforts should focus on the subsequent Large Language Model (LLM). Building on this, we further address the question of how to prune within the LLM, and distill two key insights: (i) from a semantic perspective, the token budget should be balanced between foreground and background regions; (ii) from a spatial perspective, the spatial uniformity of retained tokens should be maintained to preserve the model's global spatial perception. These findings provide practical guidance for the design of inference acceleration and token pruning for GUI agent navigation.

Figures & tables

Figure 1: Performance and computational cost comparison of Qwen3-VL-2B on GUI agent navigation benchmarks with and without historical screenshots.

Figure 2: (a) The ViT feature caching mechanism , which caches and reuses ViT-encoded features of repeated screenshots across steps. For Qwen3-VL with DeepStack, the intermediate ViT features are cached, but are omitted from the illustration for clarity. (b) Two approaches that compress historical screenshots before ViT encoding: patch removal and resolution reduction .

Strategies	FLOPs $\downarrow$	Memory $\downarrow$	Wall-clock time $\downarrow$	Encoder latency $\downarrow$	LLM prefill latency $\downarrow$	Performance $\uparrow$
Strategies	(T)	(MB)	(min)	(ms/step)	(ms/step)	Performance $\uparrow$
With HI	14.9	5067	108.54	212.08	193.05	0.7035
Without HI	4.17 ( $\downarrow$ 72.0%)	4289 ( $\downarrow$ 15.4%)	76.62 ( $\downarrow$ 29.4%)	67.90 ( $\downarrow$ 68.0%)	151.70 ( $\downarrow$ 21.4%)	0.6453 ( $\downarrow$ 8.27%)
ViT Cache	10.2 ( $\downarrow$ 31.5%)	5144 ( $\uparrow$ 1.52%)	95.70 ( $\downarrow$ 11.8%)	69.82 ( $\downarrow$ 67.1%)	202.54 ( $\uparrow$ 4.92%)	0.7030 ( $\downarrow$ 0.07%)
Resize ( $\times$ 0.8)	10.9 ( $\downarrow$ 26.8%)	4795 ( $\downarrow$ 5.37%)	97.69 ( $\downarrow$ 10.0%)	151.73 ( $\downarrow$ 28.5%)	144.63 ( $\downarrow$ 25.1%)	0.6965 ( $\downarrow$ 1.00%)
Patch Removal	11.0 ( $\downarrow$ 26.2%)	4606 ( $\downarrow$ 9.10%)	98.02 ( $\downarrow$ 9.69%)	162.11 ( $\downarrow$ 23.6%)	149.47 ( $\downarrow$ 22.6%)	0.6859 ( $\downarrow$ 2.50%)

Table 1: Comparison of FLOPs, GPU peak memory, wall-clock time, encoder latency, LLM prefill latency, and step success rate for different strategies on the AITW dataset, based on Qwen3-VL-2B. HI denotes historical images. Parenthesized values report relative changes with respect to With HI.

	AITW	M-d	M-t	M-w	AndCtrl	GUI-Ody
Original	0.7035	0.4863	0.5363	0.4858	0.7069	0.8128
ViT Cache	0.7030	0.4884	0.5363	0.4829	0.7061	0.8136

Table 2: Performance comparison of Qwen3-VL-2B with and without the proposed ViT cache-and-reuse mechanism on four benchmarks.

Qwen2.5-VL-3B
Res	AITW	M-d	M-t	M-w	AndCtrl	GUI-Ody
$\times$ 1.0	0.6806	0.4889	0.5580	0.4697	0.7113	0.7790
$\times$ 0.9	0.6794	0.4865	0.5482	0.4710	0.7110	0.7730
$\times$ 0.8	0.6806	0.4818	0.5382	0.4687	0.7094	0.7670
Qwen3-VL-2B
Res	AITW	M-d	M-t	M-w	AndCtrl	GUI-Ody

Table 3: Performance under different resolution compression ratios for Qwen2.5-VL-3B and Qwen3-VL-2B.

Figure 3: Left: Step success rate of different token retention strategies (KF: keep foreground only; KB: keep background only; KB † : keep background subsampled to match KF budget; KA: keep all) across four benchmarks. Right: Proportion of foreground and background patches across four GUI agent benchmarks, where background patches account for 63.63% on average.

Figure 4: Illustration of the edge-based foreground–background partition on a GUI screenshot.

Figure 5: A toy example illustrating the effect of token pruning on spatial perception . Removing background tokens causes the model to lose global spatial references and produce an incorrect answer, whereas random pruning preserves spatial uniformity and maintains correct spatial perception.

Qwen2.5-VL-3B
Method	Domain	Task	Website	Average
Upper Bound	0.4889	0.5580	0.4697	0.5055
FastV	0.4829	0.5483	0.4624	0.4979
SparseVLM	0.4815	0.5498	0.4573	0.4962
DivPrune	0.4855	0.5430	0.4654	0.4980
DART	0.4776	0.5305	0.4588	0.4890

Table 4: Comparison of token pruning methods on Mind2Web (

R=50\%

Qwen2.5-VL-3B
Method	type acc	grounding acc	step acc
Upper Bound	0.8532	0.7670	0.7113
FastV	0.8514	0.7645	0.7084
SparseVLM	0.8492	0.7602	0.7055
DivPrune	0.8513	0.7657	0.7062
DART	0.8487	0.7614	0.7039

Table 5: Comparison of token pruning methods on AndroidControl (

R=50\%

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visualization of edge-based foreground–background segmentation on Mind2Web screenshots. The same parameter settings ( $\tau_{e}=50$ , $\tau_{r}=0.01$ ) as in the main paper produce accurate results across diverse web interface layouts.

Figure 8: Effect of pruning layer selection under different retention ratios. Upper Bound denotes the setting without pruning.

30% Token Retention
Method	General	Single	Webshopping	Install	GoogleApps	Average
Upper Bound	0.6496	0.7844	0.6579	0.7366	0.6891	0.7035
FastV	0.6211	0.7784	0.6591	0.7294	0.6911	0.6958
SparseVLM	0.6211	0.7575	0.6603	0.7287	0.6931	0.6921
DivPrune	0.6211	0.7665	0.6591	0.7294	0.6812	0.6915
DART	0.6140	0.7725	0.6292	0.7255	0.6871	0.6857

Appendix

Table 7: Comparison of token pruning methods on AITW under 10% and 30% token-retention budgets. Upper Bound denotes the model without token pruning.

Method	General	Single	Webshopping	Install	GoogleApps	Avg.	Wall-clock Time (min)
No Cache	0.6496	0.7844	0.6579	0.7366	0.6891	0.7035	108.54
ViT Feature Cache	0.6390	0.7784	0.6615	0.7310	0.7050	0.7030	95.70
ViT + Strict LLM KV Cache	0.6532	0.7814	0.6579	0.7310	0.6832	0.7013	92.89
ViT + Segment-level LLM KV Cache	0.5190	0.7665	0.3708	0.5196	0.5406	0.5433	105.48

Appendix

Table 8: Ablation on KV cache reuse on AITW. ViT feature caching and strict LLM KV caching largely preserve task performance while reducing wall-clock inference time. In contrast, segment-level LLM KV reuse substantially degrades performance because the reused states do not correspond to exact shared prefixes.

Step	Prompt structure with $H=2$	Safe LLM KV reuse from previous step
$t=0$	$[U,O_{0}]$	–
$t=1$	$[U,O_{0},A_{0},O_{1}]$	$[U,O_{0}]$
$t=2$	$[U,O_{0},A_{0},O_{1},A_{1},O_{2}]$	$[U,O_{0},A_{0},O_{1}]$
$t=3$	$[U,O_{1},A_{1},O_{2},A_{2},O_{3}]$	only the initial text prefix before $O_{1}$
$t=4$	$[U,O_{2},A_{2},O_{3},A_{3},O_{4}]$	only the initial text prefix before $O_{2}$

Appendix

Table 9: Illustration of strict LLM KV cache reuse with a 2-history window. Before the window slides, the prompt grows by appending new action and screenshot segments, so the previous prompt is an exact prefix of the current prompt. After the window slides at

t=3

, the oldest pair

(O_{0},A_{0})

is removed, and the remaining screenshots appear under different causal prefixes. Therefore, their previous KV states are no longer valid.

Method	Domain	Task	Website	Avg.
No Cache	0.4863	0.5363	0.4858	0.5028
ViT Feature Cache	0.4884	0.5363	0.4829	0.5025
ViT + Strict LLM KV Cache	0.4918	0.5368	0.4902	0.5063
ViT + Segment-level LLM KV Cache	0.2584	0.2362	0.2210	0.2385

Appendix

Table 10: Ablation on KV cache reuse on Mind2Web. ViT feature caching and strict LLM KV caching preserve performance, whereas segment-level LLM KV reuse causes a substantial performance drop.

Method	General	Single	Webshopping	Install	GoogleApps	Average
Upper Bound	0.6496	0.7844	0.6579	0.7366	0.6891	0.7035
FastV-0.5	0.6200	0.7246	0.6226	0.7031	0.6356	0.6612
FastV-0.4	0.5903	0.7216	0.5927	0.6680	0.6178	0.6381
FastV-0.3	0.5606	0.7066	0.5694	0.6097	0.5941	0.6081
FastV-0.2	0.5131	0.6407	0.4928	0.5435	0.5406	0.5461

Appendix

Table 11: Performance of FastV when applied to the current screenshot on AITW under different visual-token retention ratios. The suffix denotes the token-retention ratio. Upper Bound denotes the model without token pruning.

Figure 9: Performance of FastV when applied to the current screenshot on Mind2Web under different visual-token retention ratios

Strategy	General	Single	Webshopping	Install	GoogleApps	Avg.	Wall-clock (min)	Encoder (ms/step)	LLM Prefill (ms/step)
No Cache	0.6496	0.7844	0.6579	0.7366	0.6891	0.7035	108.54	212.08	193.05
ViT Feature Cache	0.6389	0.7784	0.6614	0.7310	0.7049	0.7029	95.70	69.82	202.54
Random-0.1	0.6140	0.7605	0.6274	0.7223	0.6970	0.6842	100.37	192.08	103.67
ViT + Random-0.1	0.6152	0.7574	0.6280	0.7287	0.6970	0.6853	89.22	70.04	105.69

Appendix

Table 12: Compatibility of ViT feature caching and LLM-side random token pruning on AITW using Qwen3-VL-2B. Random-0.1 retains 10% of the visual tokens from each historical screenshot. The combined strategy preserves the performance and LLM prefill speed of LLM-side pruning while additionally avoiding redundant visual encoding.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 275 papers

STP-GUI: Spatially-Aware Token Pruning for Efficient Web-GUI Agents
Oct 14, 2026Shuo Wang, Sifan Zhou, Xiaobo Wang +6Visual Token Pruning Efficient VLM Inference
Mininglamp · Southeast University · Shenzhen University of Advanced Technology
Spatio-Temporal Token Pruning for Efficient High-Resolution GUI Agents
Feb 26, 2026Zhou Xu, Bowen Zhou, Qi Wang +2GUI Agents Visual Token Pruning
Tsinghua University, Shenzhen, China · Xidian University, Xi’an, China · The Chinese University of Hong Kong, Hong Kong, China
AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees
May 19, 2026Yuankai Li, Tinghui Zhu, Ha Min Son +3Efficient Multimodal Inference GUI Agents
UC Davis

Oct 14, 2026J/K move · Enter open · S save

STP-GUI: Spatially-Aware Token Pruning for Efficient Web-GUI Agents

Shuo Wang, Sifan Zhou, Xiaobo Wang, Tianyu Fu+5

Mininglamp · Southeast University · Shenzhen University of Advanced Technology

Visual Token Pruning Efficient VLM Inference Web Agents

Open paper AskPDF arXiv

With the rapid development of vision-language models (VLMs), Web-GUI agents have achieved promising performance in interacting with real-world interfaces through screenshot-based grounding. However, high-resolution screenshots and multi-turn histories introduce excessive visual tokens, causing substantial inference overhead. Existing token pruning methods, mainly designed for natural images, often select tokens independently based on importance scores, which can fragment the local visual structures required for recognizing small GUI elements such as buttons, icons, and hyperlinks. In this work, we propose STP-GUI (Spatially-aware Token Pruning for GUI agents), a query-guided token pruning framework that preserves fine-grained instruction-conditioned token scoring while performing spatially coherent window-level selection under aggressive token reduction. A lightweight selector is optimized through policy learning while the VLM backbone remains frozen. Experiments on Online-Mind2Web show that \papername achieves a favorable efficiency-performance tradeoff under low token budgets. At 21% token retention, it preserves 77.7% of full-token performance while reducing inference latency by 31% and FLOPs by 57%. Results on Android in the Wild further suggest that the method transfers to mobile GUI environments.

Abstract

Visual Token Pruning

Explore similar work

STP-GUI: Spatially-Aware Token Pruning for Efficient Web-GUI Agents

Spatio-Temporal Token Pruning for Efficient High-Resolution GUI Agents

AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees