Focused Omni-Modal GUI Agents

Fuyao Zhang; Wenwen He; Jiaming Zhang; Yurong Hao; Xuan Wang; Guowei Guan; Che Wang; Longtao Huang

cs.CVOct 19, 2026

Focused Omni-Modal GUI Agents

Authors: Fuyao Zhang, Wenwen He, Jiaming Zhang, Yurong Hao, Xuan Wang, Guowei Guan, Che Wang, Longtao Huang,

Organizations: Nanyang Technological University · Alibaba Group

Abstract

Graphical User Interface (GUI) agents increasingly act over streaming omni-modal observations spanning screenshots, audio, video, interaction history, and device states. However, exposing this complete context uniformly at every step ignores that relevance is transient and sparse: the signal determining one action can become clutter at the next. Because executed actions feed back into the context, these errors compound into Sequential Distraction, driving the agent into irreversible trajectory drift. We formulate the underlying requirement as Focused Perception: a representation that is step-adaptive, action-sufficient, and non-intrusive. We realize this in FocusGUI, a lightweight adapter that dynamically prioritizes multimodal channels against a step-specific decision reference. Instead of uniformly processing all inputs, it compresses the most relevant evidence into a compact latent and subtly injects it into the frozen decoder's embeddings, guiding the model's attention without altering its native parameter space. FocusGUI trains from action supervision alone, requiring neither modality-relevance labels nor backbone fine-tuning. Across four omni-modal backbones, it raises overall Type Match by +33.2 and Exact Match by 3.0 times over the frozen backbone on average, and Type Match by +30.1 over the strongest existing enhancement method.

Figures & tables

Figure 1: Relevance is transient across steps and sparse within them. Across multi-step episodes, we systematically ablate individual modalities from the input of Qwen2.5-Omni and record the resulting change in the log-likelihood of the ground-truth action. A modality is useful if its ablation drops this likelihood, and interfering if its ablation increases it; the dominant modality is the one whose absence causes the steepest drop. (a) The dominant modality switches across 45.3% of adjacent steps, with 94.0% of episodes containing at least one switch. (b) 90.1% of steps contain at least one interfering modality. (c) Based on annotations, the target element occupies a median of a mere 0.87% of the screen area, and video content changes by a median of just 7.41% within a step.

Figure 2: FocusGUI. The frozen backbone’s original pathway is left intact (top). From the same observation, FocusGUI exposes perceptual channels, scores them against a decision reference recomputed at every step, compresses the result into a compact focus latent, and adds it as a zero-initialized residual at the $K$ positions preceding the action boundary.

	Overall			Localization			Semantic			Cross-modal			Temporal			Instant
Methods	TM $\uparrow$	EM $\uparrow$	GP $\uparrow$	TM $\uparrow$	EM $\uparrow$	GP $\uparrow$	TM $\uparrow$	EM $\uparrow$	GP $\uparrow$	TM $\uparrow$	EM $\uparrow$	GP $\uparrow$	TM $\uparrow$	EM $\uparrow$	GP $\uparrow$	TM $\uparrow$	EM $\uparrow$	GP $\uparrow$
Qwen2.5-Omni	49.61	12.60	5.50	57.32	12.20	4.94	48.65	17.12	6.75	42.20	9.17	3.17	45.97	12.10	7.40	58.54	12.20	5.07
+DRS-GUI	49.41	6.69	2.97	51.22	6.10	3.63	50.45	5.41	1.23	45.87	5.50	2.79	49.19	9.68	4.95	51.22	6.10	1.87
+OmniParser	48.43	7.68	3.87	51.22	7.32	4.51	49.55	7.21	1.87	44.95	5.50	2.41	46.77	10.48	5.83	51.22	7.32	4.48
+RegionFocus	48.23	7.68	4.07	52.44	4.88	3.82	49.55	7.21	2.65	44.04	6.42	3.46	46.77	11.29	5.21	50.00	7.32	5.13
+GUI-Actor	50.20	13.78	5.54	58.54	13.41	5.18	47.75	18.02	6.36	43.12	10.09	3.63	46.77	12.90	7.08	59.76	14.63	5.47

Table 1: Main results across omni-modal backbones and GUI task categories. FocusGUI rows are highlighted, and the best score in each column within each backbone block is bolded.

Components				Overall			Localization			Semantic			Cross-modal			Temporal			Instant
CD	SP	AP	RA	TM $\uparrow$	EM $\uparrow$	GP $\uparrow$	TM $\uparrow$	EM $\uparrow$	GP $\uparrow$	TM $\uparrow$	EM $\uparrow$	GP $\uparrow$	TM $\uparrow$	EM $\uparrow$	GP $\uparrow$	TM $\uparrow$	EM $\uparrow$	GP $\uparrow$	TM $\uparrow$	EM $\uparrow$	GP $\uparrow$
✗	✗	✗	✗	49.61	12.60	5.50	57.32	12.20	4.94	48.65	17.12	6.75	42.20	9.17	3.17	45.97	12.10	7.40	58.54	12.20	5.07
✗	✗	✓	✓	66.26	35.00	12.20	68.49	40.56	14.55	66.67	33.64	11.28	60.55	30.86	6.58	72.58	38.58	20.00	60.29	31.46	7.13
✓	✗	✓	✓	67.38	37.78	11.56	71.95	42.68	16.82	64.86	37.84	9.65	64.22	32.04	6.35	74.42	45.97	18.02	59.76	28.05	5.80
✓	✓	✗	✓	71.14	38.75	16.11	78.05	42.68	25.43	66.67	37.84	13.84	67.72	32.70	6.54	78.23	47.58	27.50	65.03	32.93	7.25
✓	✓	✓	✗	70.45	34.86	12.11	74.15	41.34	28.09	67.57	33.33	13.43	67.56	32.04	5.05	79.84	39.52	10.73	63.11	28.05	3.26

Table 2: Component ablation on Qwen2.5-Omni. CD: channel decomposition. SP: step-adaptive prioritization. AP: action-sufficient projection. RA: residual alignment. Each task category reports Type Match (TM), Exact Match (EM), and Goal Progress (GP).

Figure 3: FocusGUI holds accuracy as trajectories lengthen. Per-step Exact Match (left) and Type Match (right) on Qwen2.5-Omni, with steps binned by their index within the episode and

3{+}

collecting the fourth step onward. Shaded regions are

90\%

episode-clustered bootstrap intervals.

Figure 4: Learned channel priorities reorder across interaction phases. (a) Mean priority $\bar{\alpha}^{(m)}$ per channel and phase on Qwen2.5-Omni; uniform allocation would be 14.3%. (b) Deviation of each cell from that channel’s all-step mean. FocusGUI trains on action annotations alone, with no modality-relevance supervision.

	Overall TM	Overall EM	Overall GP	Temporal GP
Boundary window $K$ ( $d_{z}=64$ )
$K=1$	72.83	39.76	15.13	17.66
$K=4$	71.65	40.16	17.49	26.35
$K=8$	70.28	37.80	15.72	16.04
$K=16$	71.65	39.76	18.09	25.10
Action-sufficient dimension $d_{z}$ ( $K=4$ )

Table 3: Sensitivity to the boundary window

K

and the compact focus dimension

d_{z}

on Qwen2.5-Omni. The default (

K=4

d_{z}=64

, shaded) is fixed across all backbones and never tuned per benchmark.

Figure 5: Case study: a decision that never appears on the screen. The instruction conditions the action on what is heard, so the screen looks the same at every step while the deciding evidence sits in audio. FocusGUI waits and taps once the engine sounds arrive; the frozen backbone acts early on visual salience and cannot recover, since its own action re-enters the context of every later step. Radar plots show the channel priorities $\alpha_{t}^{(m)}$ per step.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Symbol	Meaning
$t$	Interaction step index	$P_{m}$	Channel projector
$\mathcal{O}_{t}$	Omni-modal observation	$e^{(m)}_{t}\!\in\!\mathbb{R}^{d}$	Channel embedding
$g$	User goal	$d$	Shared channel dim. (128)
$I_{t}$	Execution screenshot	$e^{(\mathrm{all})}_{t}$	Integrated observation state
$V_{t}$	Video observation	$q_{t}$	Decision reference
$A_{t}$	Audio observation	$r_{m}$	Channel-identity embedding

Appendix

Table 4: Notation.

Figure 6: The capacity grid is flat around the default. Overall TM, EM, GP, and Temporal GP as a function of the boundary window $K$ and the action-sufficient dimension $d_{z}$ on Qwen2.5-Omni. Circles mark measured configurations, and the surface is a thin-plate interpolation between them; the diamond marks the default $K=4$ , $d_{z}=64$ used throughout the paper. The interpolated regions carry no measurement and are shown only to convey the shape of the grid.

Figure 7: MiniCPM-o 4.5 across interaction depth. Per-step Exact Match (left) and Type Match (right) with steps binned by their index within the episode and

3{+}

collecting the fourth step onward. Shaded regions are

90\%

episode-clustered bootstrap intervals. As on Qwen2.5-Omni, the frozen backbone declines with depth while FocusGUI holds, so the margin widens rather than staying constant.

Figure 8: VITA-1.5 across interaction depth. Per-step Exact Match (left) and Type Match (right). VITA-1.5 is the weakest of the four backbones at 13.98 Overall TM, and the depth-dependent separation appears at that level as well.

Figure 9: Baichuan-Omni-1.5 across interaction depth. Per-step Exact Match (left) and Type Match (right). The same widening margin holds on a third backbone family, indicating that the decay is a property of the interaction regime rather than of a particular model.

		Metric (%)
Configuration	Params	TM	EM	GP
Parameter-matched concatenation	1,444,913	$71.85$	$33.46$	$10.14$
Dynamic prioritization	1,444,992	72.83	37.01	13.58
$\Delta$ (dyn. $-$ concat.)	$+79$	$+0.98$	$+3.54$	$+3.44$

Appendix

Table 5: Parameter-matched control on Qwen2.5-Omni. The concatenation variant joins the perceptual channels with a fixed layout instead of scoring them against a decision reference, and is sized to the same trainable budget. Both configurations were trained under identical settings and evaluated on the same 508 test steps.

Figure 10: Paired prediction outcomes across task categories. Left: proportions of steps that both models predict correctly, FocusGUI corrects, FocusGUI regresses, or both predict incorrectly. Right: improvements in type match (TM) and exact match (EM), measured in percentage points.

Configuration	Value
Shared channel dimension $d$	128
Action-sufficient dimension $d_{z}$	64
Residual slots $K$	4
Residual initialization	Zero
Residual scale $\gamma$	1.0
Channel temperature $\tau$	1.0

Appendix

Table 6: Default FocusGUI implementation configuration.

	Frozen backbone	$+$ FocusGUI
System prompt	identical
Multimodal observations	identical
Action history	identical
Action schema	identical
Decoding	greedy, beam size 1
Textual demonstrations	none

Appendix

Table 7: What the two configurations receive at inference. Every row above the rule is shared, so the columns are indistinguishable until the last two entries.

\mathcal{J}_{t}

denotes the

K

embedding positions preceding the action boundary.

Figure 11: Case study on cross-modal temporal reasoning. The agent is instructed to pause a video based on specific auditory cues ("past/history/childhood"). The visual interface remains largely static, forcing the model to rely on streaming audio comprehension over multiple steps. FocusGUI successfully avoids premature visual actions, dynamically shifting priority to the audio channel and executing the spatial action (tapping pause) precisely when the semantic condition is met.

GUI Agents

Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.

See all 178 papers

OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
Apr 3, 2026Felix Henry, Lihan Lin, Jiangyou Zhu +4Mobile GUI Automation GUI Agents
XPeng Motors
GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents
Oct 26, 2026Ye Li, Haoran Ding, Huanan Liu +10Efficient Inference GUI Agents
Tsinghua University · The Chinese University of Hong Kong · Joy Future Academy, JD Group, Beijing, China
Learning How to Remember: Memory Modality Selection for GUI Agents
Oct 13, 2026Taeyun Roh, Suhyeong Park, Gijin Kim +2Memory-Augmented Agents GUI Agents
Korea University · Myongji University · AIGEN Sciences

Apr 3, 2026J/K move · Enter open · S save

OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments

Felix Henry, Lihan Lin, Jiangyou Zhu, Yangfan+3

XPeng Motors

Mobile GUI Automation GUI Agents AI Agent Benchmarks Mobile GUI Agents Multimodal Agents Multimodal Model Evaluation

Open paper AskPDF arXiv

Current benchmarks for graphical user interface (GUI) agents predominantly rely on static screenshots. However, real-world smartphone interaction routinely requires agents to process transient audio cues and temporal video dynamics that are tightly coupled with the moment of action. To bridge this gap, we introduce OmniGUI, the first step-level benchmark designed to evaluate GUI agents in omni-modal smartphone environments. OmniGUI provides continuous, interleaved multimodal inputs comprising static images, synchronous audio, and video clips at every action step. The dataset encompasses 709 expert-demonstrated episodes (2,579 action steps) across 29 applications, systematically annotated with objective multimodal dependency levels. Because dedicated omni-modal GUI agent frameworks are currently in their nascent stage, we select foundational omni-modal models capable of natively processing interleaved inputs to serve as agent proxies for our initial baselines. Our empirical evaluation reveals that while current models exhibit competency on visually static tasks, their action prediction performance degrades significantly in environments requiring synchronous temporal and auditory signals. Furthermore, ablation studies isolate specific operational bottlenecks, notably cross-modal interference when processing task-irrelevant environmental noise. The complete dataset, evaluation pipeline, and baseline prompts are provided in the supplementary material. Project page: https://omni-gui.github.io.

Abstract

GUI Agents

Explore similar work

OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments

GUIFlow: Temporal Redundancy-Aware Incremental Inference for GUI Agents

Learning How to Remember: Memory Modality Selection for GUI Agents