STP-GUI: Spatially-Aware Token Pruning for Efficient Web-GUI Agents

Shuo Wang; Sifan Zhou; Xiaobo Wang; Tianyu Fu; Chenxu Zhao; Anyang Su; Hanning Wang; Ping Jiang

cs.CVOct 14, 2026

STP-GUI: Spatially-Aware Token Pruning for Efficient Web-GUI Agents

Authors: Shuo Wang, Sifan Zhou, Xiaobo Wang, Tianyu Fu, Chenxu Zhao, Anyang Su, Hanning Wang, Ping Jiang,

Organizations: Mininglamp · Southeast University · Shenzhen University of Advanced Technology

Abstract

With the rapid development of vision-language models (VLMs), Web-GUI agents have achieved promising performance in interacting with real-world interfaces through screenshot-based grounding. However, high-resolution screenshots and multi-turn histories introduce excessive visual tokens, causing substantial inference overhead. Existing token pruning methods, mainly designed for natural images, often select tokens independently based on importance scores, which can fragment the local visual structures required for recognizing small GUI elements such as buttons, icons, and hyperlinks. In this work, we propose STP-GUI (Spatially-aware Token Pruning for GUI agents), a query-guided token pruning framework that preserves fine-grained instruction-conditioned token scoring while performing spatially coherent window-level selection under aggressive token reduction. A lightweight selector is optimized through policy learning while the VLM backbone remains frozen. Experiments on Online-Mind2Web show that \papername achieves a favorable efficiency-performance tradeoff under low token budgets. At 21% token retention, it preserves 77.7% of full-token performance while reducing inference latency by 31% and FLOPs by 57%. Results on Android in the Wild further suggest that the method transfers to mobile GUI environments.

Figures & tables

Figure 1: Compared to existing VLM token pruning methods, our framework for Web - GUI agents aggregates fine - grained scores into non - overlapping spatial windows and uses a Plackett–Luce policy during training to select complete, contiguous patches. This helps suppress background noise while preserving local semantic and geometric structure, enabling the frozen VLM backbone to retain action - relevant evidence for downstream decisions.

Figure 2: Overview of the STP-GUI pipeline. Given the task instruction and visual tokens, STP-GUI scores token relevance and performs window-level selection under a token budget before the retained tokens are fed into the LLM. The resulting compact visual representation is then used for downstream action prediction.

Model	Method	KR	SR ( $\uparrow$ )	ACC( $\uparrow$ )
Qwen3-VL-2B	Baseline	100%	0.373	100%
	HiPrune	21%	0.100	26.8%
	HiPrune	50%	0.183	49.1%
	HiPrune*	21%	0.140	37.5%
	HiPrune*	50%	0.193	51.7%
	PDrop	41%	0.120	32.2%

Table 1: Task success rate (SR) on Online-Mind2Web. Keep ratio (KR) denotes the fraction of visual tokens retained. The baseline means without token pruning. Best results are in bold . * denotes our re-training implementation for training-free methods. ACC represents the ratio relative to the full-token model. Partial baselines are reported at their native setting, while direct same-ratio comparisons are available where matched keep ratios are reported.

Method	KR	General	Web Shopping
Baseline	100%	0.782	0.525
FlashVLM*	50%	0.176	0.081
PDrop*	41%	0.178	0.131
HiPrune*	50%	0.252	0.214
Attention	50%	0.338	0.338
STP-GUI	21%	0.718	0.436

Table 2: Results on AitW. Following Bai et al. (2024) ; Feng et al. (2025) , we report General and Web Shopping splits.

Method	$N_{cc}\downarrow$	$R_{\mathrm{largest}}\uparrow$	$A_{\mathrm{mean}}\uparrow$	$H_{cc}\downarrow$
FlashVLM	124.56 $\pm$ 16.55	0.176 $\pm$ 0.059	3.24 $\pm$ 0.49	3.885 $\pm$ 0.343
HiPrune	128.95 $\pm$ 11.80	0.083 $\pm$ 0.030	3.10 $\pm$ 0.27	4.293 $\pm$ 0.179
Attention	94.95 $\pm$ 17.68	0.272 $\pm$ 0.118	4.47 $\pm$ 0.89	3.234 $\pm$ 0.475
STP-GUI	5.71 $\pm$ 2.29	0.780 $\pm$ 0.180	125.1 $\pm$ 76.1	0.724 $\pm$ 0.461

Table 3: Spatial coherence of retained token on Mind2Web at 21% budget.

N_{cc}

: #connected components;

R_{\mathrm{largest}}

: ratio of tokens in the largest component;

A_{\mathrm{mean}}

: mean component area;

H_{cc}

: component-size entropy. Lower

N_{cc}

H_{cc}

and higher

R_{\mathrm{largest}}

A_{\mathrm{mean}}

indicate more coherent retention.

KR	Method	$W=1$	$W=4$	$\Delta$ (pp)
21%	Attention	0.220	0.230	+1.0
	STP-GUI	0.247	0.290	+4.3
50%	Attention	0.280	0.297	+1.7
	STP-GUI	0.313	0.350	+3.7

Table 4: Effect of selection granularity on Online-Mind2Web with Qwen3-VL-2B. KR denotes the visual token keep ratio. We report task success rate (SR) for token-wise selection (

W=1

) and window-wise selection (

W=4

\Delta

denotes the SR improvement in percentage points. Attention uses scores from layer 16. The best SR at each keep ratio is bolded.

Figure 3: SR vs. KR on Online-Mind2Web. The dashed red line denotes the full-token baseline. STP-GUI achieves 77.7% of baseline performance at KR = 0.21.

Window strategy	Mind2Web	AitW General .	AitW WebShop
Overlap ( $W/2$ )	0.283	0.711	0.418
Non-overlap	0.290	0.718	0.436

Table 5: Effect of window overlap on task success rate.

Optimizer	SR ( $\uparrow$ )
Naive REINFORCE	0.247
PPO (clipped objective)	0.290

Table 6: Comparison between PPO and naive REINFORCE with EMA baseline on Online-Mind2Web.

Method	KR	Lat. (ms) $\downarrow$	Mem. (MB) $\downarrow$	FLOPs (G) $\downarrow$
Baseline	100%	714.1	1914	48216.0
FlashVLM	21%	895.2	9174	22207.4
HiPrune	21%	789.2	15270	22207.4
PDrop	41%	557.1	956	26430.7
IVC	50%	723.5	984	39337.8
Attention	21%	707.7	6840	35328.4

Table 7: Inference efficiency on an H20 GPU (3-image)

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Method	KR	Layer	SR ( $\uparrow$ )
IVC	0.50	4	0.130
		8	0.177
		16	0.217
Attention	0.10	4	0.060
		8	0.100
		16	0.130

Appendix

Table 8: SR across layer configurations on Qwen3-VL-2B. Best result per group in bold .

Method	KR	Training Time (h)
FlashVLM	0.21	5.3
FlashVLM	0.50	6.2
HiPrune	0.21	10.3
HiPrune	0.50	14.1
PDrop	0.41	2.1
IVC	0.50	7.0

Appendix

Table 9: Tuning cost on Qwen3-VL-2B (61,427 samples, 8

\times

A100 GPUs). All times are wall-clock hours. STP-GUI optimizes only the 2M-parameter selector with the VLM backbone frozen.

Method	$N_{\mathrm{cc}}$ $\downarrow$	$R_{\mathrm{largest}}$ $\uparrow$	$A_{\mathrm{mean}}$ $\uparrow$	$H_{\mathrm{cc}}$ $\downarrow$
STP-GUI	5.29 $\pm$ 2.16	0.794 $\pm$ 0.180	133.4 $\pm$ 75.4	0.665 $\pm$ 0.448
FlashVLM	82.82 $\pm$ 12.49	0.215 $\pm$ 0.077	4.91 $\pm$ 0.86	3.383 $\pm$ 0.399
Attention	71.45 $\pm$ 11.63	0.311 $\pm$ 0.127	5.86 $\pm$ 1.03	2.894 $\pm$ 0.439
HiPrune	97.88 $\pm$ 9.41	0.104 $\pm$ 0.038	4.08 $\pm$ 0.39	3.961 $\pm$ 0.171

Appendix

Table 10: Connected-component statistics under 8-connectivity on Online-Mind2Web (Qwen3-VL-2B, keep ratio 0.21). Mean

\pm

std. over 300 trajectories.

Dataset	KR	SR ( $\uparrow$ )	ACC ( $\uparrow$ )
Mind2Web	100%	0.42	100%
	21%	0.319	75.95%
	50%	0.387	92.14%
AitW-General	100%	0.8356	100%
AitW-General	21%	0.7083	84.76%
	50%	0.7546	90.31%

Appendix

Table 11: Results on Qwen3.5-9B. Baseline uses all visual tokens; ACC denotes the ratio relative to the full-token baseline.

Method	KR	Lat. (ms) $\downarrow$	Mem. (MB) $\downarrow$	FLOPs (G) $\downarrow$
Baseline	100%	244.6	712	15016.8
FlashVLM	21%	298.5	4759	8196.3
HiPrune	21%	286.6	6791	8196.3
PDrop	41%	204.1	318	9302.0
IVC	50%	236.4	388	12601.9
Attention	21%	232.2	1005	11513.3

Appendix

Table 12: Inference efficiency at keep ratio

\approx

0.21 on an H20 GPU (1-image scenario, 2420 tokens). STP-GUI uses

W=4

. Lower is better for all metrics.

Method	KR	Lat. (ms) $\downarrow$	Mem. (MB) $\downarrow$	FLOPs (G) $\downarrow$
Baseline	100%	476.8	1312	30649.2
FlashVLM	21%	576.0	6966	15160.0
HiPrune	21%	538.0	11031	15160.0
PDrop	41%	379.1	636	17668.6
IVC	50%	472.1	683	25294.4
Attention	21%	458.5	3275	22850.2

Appendix

Table 13: Inference efficiency at keep ratio

\approx

0.21 on an H20 GPU (2-image scenario, 4468 tokens). STP-GUI uses

W=4

. Lower is better for all metrics.

Figure 4: SR vs. training iterations for different window sizes on Online-Mind2Web.

W=4

achieves best-performing window granularity under our GUI settings (0.290).

$\lambda_{\mathrm{gt}}$	$\lambda_{\mathrm{full}}$	SR( $\uparrow$ )	ACC( $\uparrow$ )
Baseline		0.373	100%
1.0	0.0	0.280	75.1%
0.0	1.0	0.283	75.9%
0.7	0.3	0.290	77.7%
0.3	0.7	0.287	76.9%
0.5	0.5	0.256	68.6%

Appendix

Table 14: Ablation on reward composition

(\lambda_{\mathrm{gt}},\,\lambda_{\mathrm{full}})

. Qwen3-VL-2B with

W=4

and ratio = 0.21.

Figure 5: Case-study visualization of STP-GUI token selection across four steps of a successful Online-Mind2Web task (“Compare AeroAPI’s available plans”). Shaded and highlighted areas indicate pruned and retained tokens, respectively, while red boxes mark manually annotated step-level targets. Although such annotations are not available benchmark-wide, this representative example shows that STP-GUI preserves spatially coherent regions around key UI elements (e.g., menus, pricing tables) despite significant token reduction.

Figure 6: Step-wise visualization of retained visual tokens for FlashVLM over a complete GUI interaction episode at 50 % keep ratio. Each panel corresponds to one execution step in the same task. The retained-token pattern appears spatially fragmented across steps, which may weaken the local structural and contextual evidence needed for GUI grounding. In this example, the method does not complete the task within 10 steps.

Figure 7: Step-wise visualization of retained visual tokens for HiPrune over a complete GUI interaction episode under a fixed token budget. Each panel corresponds to one execution step in the same task. In this example, the trajectory includes multiple scrolling steps and does not reach task completion.

Figure 8: Step-wise visualization of retained visual tokens for Attention over a complete GUI interaction episode at 50 % keep ratio. Each panel corresponds to one execution step in the same task. This method completes the task in this example, but its retained-token pattern remains more fragmented than that of STP-GUI, while appearing more continuous than some other baselines.

Figure 9: Step-wise visualization of retained visual tokens for IVC over a complete GUI interaction episode at 50 % keep ratio. Each panel corresponds to one execution step in the same task. In this example, the method executes the first two steps correctly, but later steps exhibit less continuous retained evidence over the webpage, and the task is not completed.

Figure 10: Step-wise visualization of retained visual tokens for PDrop over a complete GUI interaction episode at 41 % keep ratio. Each panel corresponds to one execution step in the same task. The retained-token pattern appears spatially fragmented across steps, which may weaken the local structural and contextual evidence needed for GUI grounding. In this example, the trajectory returns to the initial page and does not complete the task.

Visual Token Pruning

Visual token pruning reduces the image and video patches processed by transformers to lower inference cost while preserving task-relevant information.

See all 275 papers

Where and How to Prune: An Empirical Study of Visual Token Pruning for GUI Agent Navigation
Mar 27, 2026Daiqiang Li, Zihao Pan, Zeyu Zhang +8Visual Token Pruning Efficient VLM Inference
Shanghai Jiao Tong University · Sichuan University · The University of Hong Kong +5
AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees
May 19, 2026Yuankai Li, Tinghui Zhu, Ha Min Son +3Efficient Multimodal Inference GUI Agents
UC Davis
Spatio-Temporal Token Pruning for Efficient High-Resolution GUI Agents
Feb 26, 2026Zhou Xu, Bowen Zhou, Qi Wang +2GUI Agents Visual Token Pruning
Tsinghua University, Shenzhen, China · Xidian University, Xi’an, China · The Chinese University of Hong Kong, Hong Kong, China

Mar 27, 2026J/K move · Enter open · S save

Where and How to Prune: An Empirical Study of Visual Token Pruning for GUI Agent Navigation

Daiqiang Li, Zihao Pan, Zeyu Zhang, Xuyang Liu+7

Shanghai Jiao Tong University · Sichuan University · The University of Hong Kong+5

Visual Token Pruning Efficient VLM Inference GUI Agents

Open paper AskPDF arXiv

In recent years, GUI agents have demonstrated strong potential in navigation tasks. However, preserving complete historical screenshots introduces substantial computational overhead. This paper investigates how token pruning, a plug-and-play inference acceleration technique, can be effectively applied to GUI agent navigation scenarios. Firstly, we address the question of where pruning should occur. We identify a system-level redundancy overlooked by existing methods: as the same screenshot is repeatedly fed into the model across different steps, its ViT encoding is redundantly recomputed each time. We show that its ViT-encoded embeddings can be fully cached and reused across steps, substantially reducing FLOPs while preserving model performance. This finding suggests that inference acceleration efforts should focus on the subsequent Large Language Model (LLM). Building on this, we further address the question of how to prune within the LLM, and distill two key insights: (i) from a semantic perspective, the token budget should be balanced between foreground and background regions; (ii) from a spatial perspective, the spatial uniformity of retained tokens should be maintained to preserve the model's global spatial perception. These findings provide practical guidance for the design of inference acceleration and token pruning for GUI agent navigation.

Abstract

Visual Token Pruning

Explore similar work

Where and How to Prune: An Empirical Study of Visual Token Pruning for GUI Agent Navigation

AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees

Spatio-Temporal Token Pruning for Efficient High-Resolution GUI Agents