Graphical User Interface (GUI) agents increasingly act over streaming omni-modal observations spanning screenshots, audio, video, interaction history, and device states. However, exposing this complete context uniformly at every step ignores that relevance is transient and sparse: the signal determining one action can become clutter at the next. Because executed actions feed back into the context, these errors compound into Sequential Distraction, driving the agent into irreversible trajectory drift. We formulate the underlying requirement as Focused Perception: a representation that is step-adaptive, action-sufficient, and non-intrusive. We realize this in FocusGUI, a lightweight adapter that dynamically prioritizes multimodal channels against a step-specific decision reference. Instead of uniformly processing all inputs, it compresses the most relevant evidence into a compact latent and subtly injects it into the frozen decoder's embeddings, guiding the model's attention without altering its native parameter space. FocusGUI trains from action supervision alone, requiring neither modality-relevance labels nor backbone fine-tuning. Across four omni-modal backbones, it raises overall Type Match by +33.2 and Exact Match by 3.0 times over the frozen backbone on average, and Type Match by +30.1 over the strongest existing enhancement method.
Figures & tables
Figure 1: Relevance is transient across steps and sparse within them. Across multi-step episodes, we systematically ablate individual modalities from the input of Qwen2.5-Omni and record the resulting change in the log-likelihood of the ground-truth action. A modality is useful if its ablation drops this likelihood, and interfering if its ablation increases it; the dominant modality is the one whose absence causes the steepest drop. (a) The dominant modality switches across 45.3% of adjacent steps, with 94.0% of episodes containing at least one switch. (b) 90.1% of steps contain at least one interfering modality. (c) Based on annotations, the target element occupies a median of a mere 0.87% of the screen area, and video content changes by a median of just 7.41% within a step.
Figure 2: FocusGUI. The frozen backbone’s original pathway is left intact (top). From the same observation, FocusGUI exposes perceptual channels, scores them against a decision reference recomputed at every step, compresses the result into a compact focus latent, and adds it as a zero-initialized residual at the K positions preceding the action boundary.
Overall
Localization
Semantic
Cross-modal
Temporal
Instant
Methods
TM ↑
EM ↑
GP ↑
TM ↑
EM ↑
GP ↑
TM ↑
EM ↑
GP ↑
TM ↑
EM ↑
GP ↑
TM ↑
EM ↑
GP ↑
TM ↑
EM ↑
GP ↑
Qwen2.5-Omni
49.61
12.60
5.50
57.32
12.20
4.94
48.65
17.12
6.75
42.20
9.17
3.17
45.97
12.10
7.40
58.54
12.20
5.07
+DRS-GUI
49.41
6.69
2.97
51.22
6.10
3.63
50.45
5.41
1.23
45.87
5.50
2.79
49.19
9.68
4.95
51.22
6.10
1.87
+OmniParser
48.43
7.68
3.87
51.22
7.32
4.51
49.55
7.21
1.87
44.95
5.50
2.41
46.77
10.48
5.83
51.22
7.32
4.48
+RegionFocus
48.23
7.68
4.07
52.44
4.88
3.82
49.55
7.21
2.65
44.04
6.42
3.46
46.77
11.29
5.21
50.00
7.32
5.13
+GUI-Actor
50.20
13.78
5.54
58.54
13.41
5.18
47.75
18.02
6.36
43.12
10.09
3.63
46.77
12.90
7.08
59.76
14.63
5.47
Table 1: Main results across omni-modal backbones and GUI task categories. FocusGUI rows are highlighted, and the best score in each column within each backbone block is bolded.
Components
Overall
Localization
Semantic
Cross-modal
Temporal
Instant
CD
SP
AP
RA
TM ↑
EM ↑
GP ↑
TM ↑
EM ↑
GP ↑
TM ↑
EM ↑
GP ↑
TM ↑
EM ↑
GP ↑
TM ↑
EM ↑
GP ↑
TM ↑
EM ↑
GP ↑
✗
✗
✗
✗
49.61
12.60
5.50
57.32
12.20
4.94
48.65
17.12
6.75
42.20
9.17
3.17
45.97
12.10
7.40
58.54
12.20
5.07
✗
✗
✓
✓
66.26
35.00
12.20
68.49
40.56
14.55
66.67
33.64
11.28
60.55
30.86
6.58
72.58
38.58
20.00
60.29
31.46
7.13
✓
✗
✓
✓
67.38
37.78
11.56
71.95
42.68
16.82
64.86
37.84
9.65
64.22
32.04
6.35
74.42
45.97
18.02
59.76
28.05
5.80
✓
✓
✗
✓
71.14
38.75
16.11
78.05
42.68
25.43
66.67
37.84
13.84
67.72
32.70
6.54
78.23
47.58
27.50
65.03
32.93
7.25
✓
✓
✓
✗
70.45
34.86
12.11
74.15
41.34
28.09
67.57
33.33
13.43
67.56
32.04
5.05
79.84
39.52
10.73
63.11
28.05
3.26
Table 2: Component ablation on Qwen2.5-Omni. CD: channel decomposition. SP: step-adaptive prioritization. AP: action-sufficient projection. RA: residual alignment. Each task category reports Type Match (TM), Exact Match (EM), and Goal Progress (GP).
Figure 3: FocusGUI holds accuracy as trajectories lengthen. Per-step Exact Match (left) and Type Match (right) on Qwen2.5-Omni, with steps binned by their index within the episode and 3+ collecting the fourth step onward. Shaded regions are 90% episode-clustered bootstrap intervals.
Figure 4: Learned channel priorities reorder across interaction phases. (a) Mean priority αˉ(m) per channel and phase on Qwen2.5-Omni; uniform allocation would be 14.3%. (b) Deviation of each cell from that channel’s all-step mean. FocusGUI trains on action annotations alone, with no modality-relevance supervision.
Overall TM
Overall EM
Overall GP
Temporal GP
Boundary window K ( dz=64 )
K=1
72.83
39.76
15.13
17.66
K=4
71.65
40.16
17.49
26.35
K=8
70.28
37.80
15.72
16.04
K=16
71.65
39.76
18.09
25.10
Action-sufficient dimension dz ( K=4 )
Table 3: Sensitivity to the boundary window K and the compact focus dimension dz on Qwen2.5-Omni. The default ( K=4 , dz=64 , shaded) is fixed across all backbones and never tuned per benchmark.
Figure 5: Case study: a decision that never appears on the screen. The instruction conditions the action on what is heard, so the screen looks the same at every step while the deciding evidence sits in audio. FocusGUI waits and taps once the engine sounds arrive; the frozen backbone acts early on visual salience and cannot recover, since its own action re-enters the context of every later step. Radar plots show the channel priorities αt(m) per step.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Symbol
Meaning
t
Interaction step index
Pm
Channel projector
Ot
Omni-modal observation
et(m)∈Rd
Channel embedding
g
User goal
d
Shared channel dim. (128)
It
Execution screenshot
et(all)
Integrated observation state
Vt
Video observation
qt
Decision reference
At
Audio observation
rm
Channel-identity embedding
Appendix
Table 4: Notation.
Figure 6: The capacity grid is flat around the default. Overall TM, EM, GP, and Temporal GP as a function of the boundary window K and the action-sufficient dimension dz on Qwen2.5-Omni. Circles mark measured configurations, and the surface is a thin-plate interpolation between them; the diamond marks the default K=4 , dz=64 used throughout the paper. The interpolated regions carry no measurement and are shown only to convey the shape of the grid.
Figure 7: MiniCPM-o 4.5 across interaction depth. Per-step Exact Match (left) and Type Match (right) with steps binned by their index within the episode and 3+ collecting the fourth step onward. Shaded regions are 90% episode-clustered bootstrap intervals. As on Qwen2.5-Omni, the frozen backbone declines with depth while FocusGUI holds, so the margin widens rather than staying constant.
Figure 8: VITA-1.5 across interaction depth. Per-step Exact Match (left) and Type Match (right). VITA-1.5 is the weakest of the four backbones at 13.98 Overall TM, and the depth-dependent separation appears at that level as well.
Figure 9: Baichuan-Omni-1.5 across interaction depth. Per-step Exact Match (left) and Type Match (right). The same widening margin holds on a third backbone family, indicating that the decay is a property of the interaction regime rather than of a particular model.
Metric (%)
Configuration
Params
TM
EM
GP
Parameter-matched concatenation
1,444,913
71.85
33.46
10.14
Dynamic prioritization
1,444,992
72.83
37.01
13.58
Δ (dyn. − concat.)
+79
+0.98
+3.54
+3.44
Appendix
Table 5: Parameter-matched control on Qwen2.5-Omni. The concatenation variant joins the perceptual channels with a fixed layout instead of scoring them against a decision reference, and is sized to the same trainable budget. Both configurations were trained under identical settings and evaluated on the same 508 test steps.
Figure 10: Paired prediction outcomes across task categories. Left: proportions of steps that both models predict correctly, FocusGUI corrects, FocusGUI regresses, or both predict incorrectly. Right: improvements in type match (TM) and exact match (EM), measured in percentage points.
Table 7: What the two configurations receive at inference. Every row above the rule is shared, so the columns are indistinguishable until the last two entries. Jt denotes the K embedding positions preceding the action boundary.
Figure 11: Case study on cross-modal temporal reasoning. The agent is instructed to pause a video based on specific auditory cues ("past/history/childhood"). The visual interface remains largely static, forcing the model to rely on streaming audio comprehension over multiple steps. FocusGUI successfully avoids premature visual actions, dynamically shifting priority to the audio channel and executing the spatial action (tapping pause) precisely when the semantic condition is met.
Figure 12: Case study on multi-step semantic planning and spatial localization. The agent executes a complex sequence of interface interactions based on a detailed text instruction. It successfully navigates through dynamic UI changes—locating a small filter icon, understanding the popup menu, and sequentially selecting "Like count", "Liked", and "This Week". This demonstrates FocusGUI’s capacity for precise spatial grounding and robust state tracking in visually dense environments.
Current benchmarks for graphical user interface (GUI) agents predominantly rely on static screenshots. However, real-world smartphone interaction routinely requires agents to process transient audio cues and temporal video dynamics that are tightly coupled with the moment of action. To bridge this gap, we introduce OmniGUI, the first step-level benchmark designed to evaluate GUI agents in omni-modal smartphone environments. OmniGUI provides continuous, interleaved multimodal inputs comprising static images, synchronous audio, and video clips at every action step. The dataset encompasses 709 expert-demonstrated episodes (2,579 action steps) across 29 applications, systematically annotated with objective multimodal dependency levels. Because dedicated omni-modal GUI agent frameworks are currently in their nascent stage, we select foundational omni-modal models capable of natively processing interleaved inputs to serve as agent proxies for our initial baselines. Our empirical evaluation reveals that while current models exhibit competency on visually static tasks, their action prediction performance degrades significantly in environments requiring synchronous temporal and auditory signals. Furthermore, ablation studies isolate specific operational bottlenecks, notably cross-modal interference when processing task-irrelevant environmental noise. The complete dataset, evaluation pipeline, and baseline prompts are provided in the supplementary material. Project page: https://omni-gui.github.io.
GUI agents built on vision-language models interact with continuously evolving interfaces, yet conventional inference repeatedly recomputes representations of largely overlapping observations and histories. This mismatch makes multi-step interaction costly, while changes in context, interface content, and action arguments prevent straightforward cache reuse. We introduce GUIFlow, a framework for full-pipeline incremental inference in multi-image GUI interaction. Our central insight is that advancing a decision need not require rebuilding all supporting representations: each interaction step can instead selectively update computational states maintained across the trajectory. To support new decisions as the context evolves, GUIFlow adapts how past computation is reused: historical KV states are position-aligned and injected into the current context; change signals guide selective updates of current-observation states in the vision encoder and language model; and earlier action responses serve as speculative drafts verified by the target model. Together, these mechanisms enable cross-step reuse throughout visual encoding, prefill, and decoding without additional training, a separate draft model, or changes to the model-visible inputs and action interface. Experiments demonstrate up to 3.39x geometric-mean speedup across offline multi-image benchmarks and up to 3.49x in online, closed-loop interaction on OSWorld, while maintaining task performance broadly comparable to the dense baseline.
Ye Li, Haoran Ding, Huanan Liu +10
Tsinghua University · The Chinese University of Hong Kong · Joy Future Academy, JD Group, Beijing, China
Recent advances in vision-language models (VLMs) have substantially improved the capabilities of GUI agents, while memory augmentation has emerged as a promising approach for reusing knowledge from past interactions. However, existing memory augmented GUI agents typically adopt a fixed representation strategy, providing retrieved experience as a static modality. It remains unclear whether a single memory modality is consistently optimal, or how modality preference changes across interaction steps. In this work, we conduct a systematic study of memory modality in VLM based GUI agents. Across GUI benchmarks, we find that the preferred memory representation is strongly step dependent. Motivated by the finding, we introduce SwitchMem, a routing framework that dynamically selects between textual and visual memory representations at each step. Across three GUI benchmarks, SwitchMem outperforms memory augmented baselines that rely on static memory representations. Our findings validate our claim that memory modality is a step dependent decision rather than a fixed design choice. Code and project resources are available at https://txxnrd.github.io/switchmem
Taeyun Roh, Suhyeong Park, Gijin Kim +2
Korea University · Myongji University · AIGEN Sciences