GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents

Ye Li; Haoran Ding; Huanan Liu; Yuan Meng; Yutian Shen; Yuansong Wang; Kangye Ji; Hanyun Cui

cs.CVOct 26, 2026

GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents

Authors: Ye Li, Haoran Ding, Huanan Liu, Yuan Meng, Yutian Shen, Yuansong Wang, Kangye Ji, Hanyun Cui,

Organizations: Tsinghua University · The Chinese University of Hong Kong · Joy Future Academy, JD Group, Beijing, China

Abstract

GUI agents built on vision-language models interact with continuously evolving interfaces, yet conventional inference repeatedly recomputes representations of largely overlapping observations and histories. This mismatch makes multi-step interaction costly, while changes in context, interface content, and action arguments prevent straightforward cache reuse. We introduce GUIFlow, a framework for full-pipeline incremental inference in multi-image GUI interaction. Our central insight is that advancing a decision need not require rebuilding all supporting representations: each interaction step can instead selectively update computational states maintained across the trajectory. To support new decisions as the context evolves, GUIFlow adapts how past computation is reused: historical KV states are position-aligned and injected into the current context; change signals guide selective updates of current-observation states in the vision encoder and language model; and earlier action responses serve as speculative drafts verified by the target model. Together, these mechanisms enable cross-step reuse throughout visual encoding, prefill, and decoding without additional training, a separate draft model, or changes to the model-visible inputs and action interface. Experiments demonstrate up to 3.39x geometric-mean speedup across offline multi-image benchmarks and up to 3.49x in online, closed-loop interaction on OSWorld, while maintaining task performance broadly comparable to the dense baseline.

Figures & tables

Figure 1: Incremental changes in GUI observations and internal representations. (a–c) Image-level changes. A local edit modifies only a small part of successive screenshots; across the evaluated tasks, more than $70\%$ of observation pairs have RGB-image error at most $5\%$ at each temporal gap $\Delta t\in\{1,2,3,4\}$ . (d–f) Representation-level changes. For the same observation pair, KV-drift maps at layer 6 and the final layer 36 show broadly similar spatial concentrations of change. Along the illustrated trajectory, a large share of tokens has small KV drift ( $d_{K},d_{V}\leq 0.1$ ) during stable-interface segments, with drops around substantial interface changes. Together, these observations motivate selectively updating changing states rather than rebuilding all representations at every step.

Figure 2: Overview of GUIFlow. GUIFlow treats successive GUI decisions as incremental updates to existing computation rather than independent inference passes. (A) Historical State Integration. Historical KV states are positionally aligned and injected into the current context. (B) Current-Observation Refinement. Pixel changes guide visual-encoder updates, while source-aware KV deviations guide deeper language-model updates. (C) Verified Incremental Decoding. Conditioned on the prefill state formed by (A) and (B), the target model verifies historical response drafts, accepting matching tokens and correcting mismatches. Together, these mechanisms exploit temporal redundancy across visual encoding, prefill, and decoding without additional training or a separate draft model.

		AITW			Mind2Web
Model	Method	Action Matching (%) $\uparrow$	FLOPs (T) $\downarrow$	Speedup ( $\times$ ) $\uparrow$	Step SR (%) $\uparrow$	FLOPs (T) $\downarrow$	Speedup ( $\times$ ) $\uparrow$	Geo. Mean ( $\times$ ) $\uparrow$
Qwen3-VL 4B-Instruct	Dense	61.20	31.94	1.00	49.77	43.97	1.00	1.00
	FastV	60.26	22.14	1.44	48.31	30.10	1.46	1.45
	VisionZip	60.23	18.91	1.69	42.09	24.90	1.77	1.73
	GUI-KV	58.59	31.94	1.00	49.25	43.96	1.00	1.00
	STaR-KV	60.34	31.93	1.00	49.61	43.95	1.00	1.00

Table 1: Main results on offline multi-image navigation. FLOPs (T) are averaged per step; speedup is the Dense/method FLOPs ratio, with an equal-weight geometric mean across datasets. GUIFlow achieves

3.37

–

3.39\times

geometric-mean speedups while maintaining scores close to Dense.

Model	Method	Mean Score (%) $\uparrow$	Strict SR (%) $\uparrow$	FLOPs (T) $\downarrow$	Speedup ( $\times$ ) $\uparrow$
Qwen3-VL 8B-Instruct	Dense	24.30	23.55	247.01	1.00
	FastV	23.16	22.16	165.50	1.49
	VisionZip	21.22	20.22	161.48	1.53
	GUI-KV	22.81	21.61	246.54	1.00
	HistPrune	20.82	19.94	194.52	1.27
	DivPrune	22.93	22.71	185.89	1.33

Table 2: Main results on online OSWorld interaction. FLOPs (T) are averaged per step/request for each evaluated run; speedups are Dense/method mean FLOPs ratios across rollout trajectories. GUIFlow achieves

3.32

–

3.49\times

speedups relative to Dense.

Figure 3: Qualitative comparison of task execution time on an OSWorld task. Dense inference and GUIFlow complete the same seven interaction stages. GUIFlow reduces task execution time from $25.76$ to $11.95$ seconds, corresponding to a $2.16\times$ speedup and $13.81$ seconds saved. The horizontal axis measures cumulative task execution time.

Benchmark	Model	FLOPs (T) $\downarrow$	Total Latency (ms) $\downarrow$	Speedup
				FLOPs	Latency
				( $\times$ ) $\uparrow$	( $\times$ ) $\uparrow$
AITW	Qwen3-VL-4B	9.23	470.64	3.46	1.96
AITW	UI-TARS-1.5-7B	18.33	547.23	3.41	1.94
Mind2Web	Qwen3-VL-4B	13.37	633.91	3.29	1.97
Mind2Web	UI-TARS-1.5-7B	25.49	603.08	3.37	2.32

Table 3: FLOPs and wall-clock efficiency of GUIFlow. Total latency covers per-step processing and inference, excluding environment execution. Speedups use Dense/GUIFlow ratios of unrounded means; online records come from different trajectories.

Enabled Components				Task Performance		Inference Cost
History KV	Sparse Vision	Token Lease	Spec. Decode	Mean Score (%) $\uparrow$	Strict SR (%) $\uparrow$	FLOPs (T/task) $\downarrow$	Total (s/task) $\downarrow$
–	–	–	–	24.30	23.55	2981.19	45.43
$\checkmark$	–	–	–	27.33	26.32	1010.29	29.54
$\checkmark$	$\checkmark$	–	–	26.21	24.93	962.02	29.35
$\checkmark$	$\checkmark$	$\checkmark$	–	25.43	24.65	819.07	27.51
$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	27.90	26.87	841.33	17.85

Table 4: Cumulative component ablation on Qwen3-VL-8B in OSWorld. Checkmarks indicate enabled components; quality uses 361 tasks, with missing results scored as zero. TokenLease uses

L_{p}=6

and

M_{\mathrm{cur}}=3

in both enabled configurations. FLOPs and Total average valid task measurements; Total sums server processing time and excludes environment execution.

Sweep	Setting	Mean Score (%) $\uparrow$	Strict SR (%) $\uparrow$	FLOPs (T/step) $\downarrow$	Total (s/step) $\downarrow$
Probe Depth ( $M_{\mathrm{cur}}=2$ )	$L_{p}=3$	24.28	23.27	224.70	3.59
	$L_{p}=6^{\dagger}$	27.35	26.59	228.22	3.56
	$L_{p}=9$	26.48	25.48	230.43	3.58
	$L_{p}=12$	25.25	24.65	231.01	3.62
Reuse Limit ( $L_{p}=6$ )	$M_{\mathrm{cur}}=1$	25.93	24.65	226.28	3.57
	$M_{\mathrm{cur}}=2$	27.35	26.59	228.22	3.56

Table 5: Sensitivity to TokenLease probe depth and reuse limit on Qwen3-VL-8B in OSWorld. The layer sweep fixes

M_{\mathrm{cur}}=2

, and the reuse-limit sweep fixes

L_{p}=6

\dagger

marks the shared default value of the swept parameter; quality uses all 361 tasks, with missing results scored as zero. Bold marks the best value within each sweep.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method	Cross-Task Step SR $\uparrow$	Cross-Website Step SR $\uparrow$	Cross-Domain Step SR $\uparrow$	Avg. Step SR $\uparrow$	Speedup ( $\times$ ) $\uparrow$
Qwen3-VL 4B-Instruct	Dense	51.11	48.37	49.84	49.77	1.00
	FastV	49.55	46.73	48.64	48.31	1.46
	VisionZip	42.36	41.23	42.69	42.09	1.77
	GUI-KV	51.59	47.06	49.09	49.25	1.00
	STaR-KV	51.40	47.81	49.63	49.61	1.00
	Op-Skip	54.18	52.48	53.56	53.41	1.29

Appendix

Table A1: Split-level results on Mind2Web. Step SR (%) is macro-averaged within each split, and Avg. is the unweighted mean across the three splits. Speedup is the Dense/method mean FLOPs ratio; bold denotes the best result for each model.

Application	Tasks	Request change (%)	FLOPs/request speedup ( $\times$ )	Task FLOPs speedup ( $\times$ )	Score change (pp)
Chrome	46	$+40.4$	3.27	2.33	$-0.01$
GIMP	26	$+14.2$	3.44	3.01	$-3.85$
LibreOffice Calc	47	$-5.6$	3.23	3.42	$-6.38$
LibreOffice Impress	47	$-16.8$	3.67	4.41	$+4.26$
LibreOffice Writer	23	$+45.5$	3.45	2.37	$-8.70$
Multi-Apps	93	$+5.7$	3.19	3.02	$-1.08$

Appendix

Table A2: Category-level efficiency and quality on OSWorld with GUI-Owl-1.5-8B. Request and score changes are measured relative to Dense; FLOPs speedups are computed as Dense/GUIFlow. Overall covers all 361 tasks; efficiency uses valid measurements.

Sweep	Setting	Effective Tasks	Lease Acceptance (%)	Reused Image Tokens (%)	FLOPs Reduction (%)	Latency Speedup ( $\times$ )
Probe Depth ( $M_{\mathrm{cur}}=2$ )	$L_{p}=3$	359	91.66	30.66	9.07	1.05
	$L_{p}=6^{\dagger}$	359	91.59	29.95	7.53	1.06
	$L_{p}=9$	359	91.66	25.46	6.73	1.05
	$L_{p}=12$	359	91.56	23.64	6.23	1.04
Reuse Limit ( $L_{p}=6$ )	$M_{\mathrm{cur}}=1$	359	91.54	25.60	8.08	1.05
	$M_{\mathrm{cur}}=2$	359	91.59	29.95	7.53	1.06

Appendix

Table A3: Mechanistic statistics of TokenLease on Qwen3-VL-8B in OSWorld. The probe-depth sweep fixes

M_{\mathrm{cur}}=2

, and the reuse-limit sweep fixes

L_{p}=6

. FLOPs reduction and latency speedup are paired against Dense on common task-step keys;

\dagger

marks the shared default value of the swept parameter.

Panel of Figure 4 — Figure 4: Additional presentation-editing trajectories. GUIFlow preserves all interaction stages while reducing cumulative task execution time by $11.11$ and $20.62$ seconds, respectively.

Panel of Figure 5 — Figure 5: Additional file- and email-organization trajectories. GUIFlow saves $14.05$ and $17.53$ seconds while retaining the completed task sequence.

Panel of Figure 6 — Figure 6: Additional document- and image-editing trajectories. GUIFlow saves $8.97$ and $13.17$ seconds while completing the same editing stages as Dense inference.

Panel of Figure 7 — Figure 7: Additional system-configuration and code-editing trajectories. GUIFlow saves $22.61$ and $14.39$ seconds, respectively, while preserving all interaction stages.

Efficient Inference

Efficient inference reduces the compute, memory, latency, or energy needed to produce accurate predictions, posterior estimates, and decisions.

See all 109 papers

GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
Oct 1, 2025Kung-Hsiang Huang, Haoyi Qiu, Yutong Dai +2KV-Cache Compression Efficient Inference
Salesforce AI Research · University of California, Los Angeles
AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees
May 19, 2026Yuankai Li, Tinghui Zhu, Ha Min Son +3Efficient Multimodal Inference GUI Agents
UC Davis
PastForward: Faster On-Device GUI Agents via Computational Experience Reuse
Sep 26, 2026Taehwan Park, Changmin Lee, Hayeon Lee +1Mobile GUI Agents On-Device Inference
Meta Superintelligence Labs

Oct 1, 2025J/K move · Enter open · S save

GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness

Kung-Hsiang Huang, Haoyi Qiu, Yutong Dai, Caiming Xiong+1

Salesforce AI Research · University of California, Los Angeles

KV-Cache Compression Efficient Inference GUI Agents KV-Cache Pruning

Open paper AskPDF arXiv

Graphical user interface (GUI) agents built on vision-language models have emerged as a promising approach to automate human-computer workflows. However, they also face the inefficiency challenge as they process long sequences of high-resolution screenshots and solving long-horizon tasks, making inference slow, costly and memory-bound. While key-value (KV) caching can mitigate this, storing the full cache is prohibitive for image-heavy contexts. Existing cache-compression methods are sub-optimal as they do not account for the spatial and temporal redundancy of GUIs. In this work, we first analyze attention patterns in GUI agent workloads and find that, unlike in natural images, attention sparsity is uniformly high across all transformer layers. This insight motivates a simple uniform budget allocation strategy, which we show empirically outperforms more complex layer-varying schemes. Building on this, we introduce GUI-KV, a plug-and-play KV cache compression method for GUI agents that requires no retraining. GUI-KV combines two novel techniques: (i) spatial saliency guidance, which augments attention scores with the L2 norm of hidden states to better preserve semantically important visual tokens, and (ii) temporal redundancy scoring, which projects previous frames' keys onto the current frame's key subspace to preferentially prune redundant history. Across standard GUI agent benchmarks and models, GUI-KV outperforms competitive KV compression baselines, closely matching full-cache accuracy at modest budgets. Notably, in a 5-screenshot setting on the AgentNetBench benchmark, GUI-KV reduces decoding FLOPs by 38.9% while increasing step accuracy by 4.1% over the full-cache baseline. These results demonstrate that exploiting GUI-specific redundancies enables efficient and reliable agent performance.

Abstract

Efficient Inference

Explore similar work

GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness

AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees

PastForward: Faster On-Device GUI Agents via Computational Experience Reuse