cs.HCApr 3, 2026
SaveOmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
Organizations: XPeng Motors
Abstract
Current benchmarks for graphical user interface (GUI) agents predominantly rely on static screenshots. However, real-world smartphone interaction routinely requires agents to process transient audio cues and temporal video dynamics that are tightly coupled with the moment of action. To bridge this gap, we introduce OmniGUI, the first step-level benchmark designed to evaluate GUI agents in omni-modal smartphone environments. OmniGUI provides continuous, interleaved multimodal inputs comprising static images, synchronous audio, and video clips at every action step. The dataset encompasses 709 expert-demonstrated episodes (2,579 action steps) across 29 applications, systematically annotated with objective multimodal dependency levels. Because dedicated omni-modal GUI agent frameworks are currently in their nascent stage, we select foundational omni-modal models capable of natively processing interleaved inputs to serve as agent proxies for our initial baselines. Our empirical evaluation reveals that while current models exhibit competency on visually static tasks, their action prediction performance degrades significantly in environments requiring synchronous temporal and auditory signals. Furthermore, ablation studies isolate specific operational bottlenecks, notably cross-modal interference when processing task-irrelevant environmental noise. The complete dataset, evaluation pipeline, and baseline prompts are provided in the supplementary material. Project page: https://omni-gui.github.io.
Figures & tables
Figure 1 : Overview of the OmniGUI benchmark framework. Given a multimodal instruction (e.g., a spoken request), a GUI agent interacts with a smartphone interface across multiple steps. At each action step, the agent receives interleaved inputs comprising a static screenshot, real-time audio (e.g., user speech, application sounds), and a temporal video clip, along with the action history. Based on these synchronous multimodal signals in the environment, the agent predicts the subsequent action (e.g., TYPE , TAP ). Performance is quantitatively evaluated using Type Match (TM) and Exact Match (EM) metrics against ground-truth human demonstrations.
| Benchmark | Platform | #Tasks | #Steps | Input Modalities | Per-Step | Action Output | Manual | ||
| Image | Video | Audio | |||||||
| Vision-Only Benchmarks | |||||||||
| AITW [ 18 ] | Android | 30,378 | 715K+ | ✓ | ✗ | ✗ | ✗ | Coordinate | ✗ |
| GUI-Odyssey [ 16 ] | Android | 7,735 | 74K+ | ✓ | ✗ | ✗ | ✗ | Coordinate | ✗ |
| AndroidWorld [ 17 ] | Android | 116 | – | ✓ | ✗ | ✗ | ✗ | Coordinate | ✓ |
| Mind2Web [ 5 ] | Web | 2,350 | 12K+ | ✓ | ✗ | ✗ | ✗ | DOM Element | ✗ |
Table 1 : Comparison of OmniGUI with representative GUI agent benchmarks. Audio and Video indicate whether the benchmark provides auditory and video inputs beyond static screenshots. Per-Step denotes whether multimodal inputs are provided at every action step, rather than as pre-task or reference content. Action Output specifies the format of predicted actions. Manual indicates whether all tasks are manually designed.
| Action Primitive | Parameter Space | Semantics / Usage |
| Temporal / Idle | ||
| NONE (-1) | Wait or observe without taking action | |
| Positional Actions | ||
| TAP (0) | Click a UI element (button, link, icon) | |
| DOUBLE_TAP (1) | Trigger specific states (e.g., zoom, like) | |
| LONG_PRESS (2) | Open context menus or select items | |
Table 2 : Definition of the OmniGUI Action Space. The agent predicts an action primitive along with its parameters. Coordinates are normalized to the resolution-independent range .
Figure 2 : Dataset statistics of OmniGUI. (a) Application and language distribution, detailing the composition of 709 episodes and 2,579 fine-grained steps across 29 smartphone applications. (b) Distribution of episodes and steps across five core task dimensions, which are grounded in human-computer interaction and multimodal cognitive processes. (c) Proportion of episodes and steps categorized by multimodal dependency levels, derived objectively from GUI information availability.
| Model | Overall | Localization | Semantic Understand. | Cross-modal Discr. | Temporal Reason. | Instant Resp. | ||||||||||||||||||
| TM | EM | SR | GP | TM | EM | SR | GP | TM | EM | SR | GP | TM | EM | SR | GP | TM | EM | SR | GP | TM | EM | SR | GP | |
| Proprietary Models | ||||||||||||||||||||||||
| Gemini 3 Pro [ 9 ] | 80.0 | 63.6 | 33.4 | 43.6 | 86.3 | 76.2 | 55.9 | 62.6 | 77.4 | 61.1 | 31.4 | 42.0 | 76.6 | 59.1 | 30.1 | 41.3 | 78.9 | 61.0 | 22.7 | 36.9 | 81.8 | 62.7 | 27.6 | 35.6 |
| Gemini 3 Flash [ 9 ] | 78.3 | 61.3 | 30.3 | 43.5 | 85.0 | 75.6 | 53.1 | 63.1 | 75.3 | 58.5 | 25.5 | 41.1 | 72.8 | 56.0 | 23.5 | 38.7 | 80.0 | 60.3 | 25.3 | 39.4 | 79.2 | 57.9 | 22.8 | 34.2 |
| Gemini 2.5 Pro [ 4 ] | 75.7 | 44.1 | 15.5 | 26.3 | 86.1 | 58.1 | 31.7 | 41.5 | 72.8 | 37.7 | 11.7 | 22.4 | 70.6 | 40.1 | 13.2 | 25.1 | 73.8 | 44.3 | 9.7 | 22.5 | 76.6 | 42.1 | 11.0 | 19.5 |
| Gemini 2.5 Flash [ 4 ] | 69.5 | 37.8 | 12.4 | 24.5 | 75.1 | 50.9 | 29.0 | 42.6 | 70.4 | 34.3 | 8.0 | 18.2 | 64.9 | 35.7 | 11.8 | 25.3 | 67.7 | 35.1 | 9.1 | 21.8 | 71.0 | 34.5 | 3.9 | 13.7 |
Table 3 : Comprehensive evaluation results on OmniGUI. We report performance across four metrics: Type Match (TM), Exact Match (EM), Success Rate (SR), and Goal Progress (GP). The table presents both the Overall performance and a fine-grained breakdown across five specific task dimensions: Localization ( Local. ), Semantic Understanding ( Semantic Understand. ), Cross-modal Discrimination ( Cross-modal Discr. ), Temporal Reasoning ( Temporal Reason. ), and Instant Response ( Instant Resp. ). All metrics are reported as percentages (%). Bold indicates the best performance in each respective column.
| Model | Modality Input | AV-Critical (34.9%) | AV-Supportive (38.6%) | AV-Present (26.5%) | Overall (100%) | ||||||||||||
| TM | EM | SR | GP | TM | EM | SR | GP | TM | EM | SR | GP | TM | EM | SR | GP | ||
| Proprietary Models | |||||||||||||||||
| Gemini 3 Pro | Full (I+A+V) | 76.9 | 57.9 | 33.2 | 42.2 | 79.6 | 65.2 | 34.4 | 45.0 | 84.7 | 69.0 | 33.0 | 44.4 | 80.0 | 63.6 | 33.4 | 43.6 |
| No Audio (I+V) | 74.4 (-2.5) | 55.9 (-2.0) | 28.7 (-4.5) | 39.6 (-2.6) | 79.6 | 64.1 (-1.1) | 34.1 (-0.3) | 44.5 (-0.5) | 85.0 (+0.3) | 70.9 (+1.9) | 36.2 (+3.2) | 46.9 (+2.5) | 79.2 (-0.8) | 63.0 (-0.6) | 32.7 (-0.7) | 43.3 (-0.3) | |
| No Video (I+A) | 69.1 (-7.8) | 50.0 (-7.9) | 16.0 (-17.2) | 34.5 (-7.7) | 74.6 (-5.0) | 59.2 (-6.0) | 25.2 (-9.2) | 41.5 (-3.5) | 81.3 (-3.4) | 67.7 (-1.3) | 35.1 (+2.1) | 48.0 (+3.6) | 74.4 (-5.6) | 58.1 (-5.5) | 24.4 (-9.0) | 40.7 (-2.9) | |
| No AV (Img Only) | 67.3 (-9.6) | 48.9 (-9.0) | 17.2 (-16.0) | 34.5 (-7.7) | 75.3 (-4.3) | 59.0 (-6.2) | 26.3 (-8.1) | 41.0 (-4.0) | 81.8 (-2.9) | 68.9 (-0.1) | 37.8 (+4.8) | 48.8 (+4.4) | 74.2 (-5.8) | 58.0 (-5.6) | 26.1 (-7.3) | 40.8 (-2.8) | |
Table 4 : Modality ablation across dependency levels. We report all metrics (%) and show the performance gap inline ( ). To highlight the most impactful modalities, for each model and metric (column), the most severe performance drop among the three ablation settings is marked in bold red . Conversely, the most significant anomalous performance gain is marked in bold teal , revealing strong cross-modal interference when irrelevant modalities are provided in Present tasks.
| Model | Instruction | AV-Critical (35.4%) | AV-Supportive (38.4%) | AV-Present (26.2%) | Overall (100%) | ||||||||||||
| TM | EM | SR | GP | TM | EM | SR | GP | TM | EM | SR | GP | TM | EM | SR | GP | ||
| Gemini 3 Pro | Text (Baseline) | 76.9 | 57.9 | 33.2 | 42.2 | 79.6 | 65.2 | 34.4 | 45.0 | 84.7 | 69.0 | 33.0 | 44.4 | 80.0 | 63.6 | 33.4 | 43.6 |
| TTS Voice | 69.0 (-7.9) | 52.1 (-5.8) | 29.1 (-4.1) | 39.4 (-2.8) | 74.5 (-5.1) | 59.9 (-5.3) | 27.7 (-6.7) | 40.3 (-4.7) | 81.6 (-3.1) | 67.3 (-1.7) | 35.8 (+2.8) | 46.4 (+2.0) | 74.3 (-5.7) | 59.1 (-4.5) | 30.3 (-3.1) | 41.6 (-2.0) | |
| Qwen3-Omni | Text (Baseline) | 58.0 | 29.4 | 7.0 | 18.5 | 65.1 | 33.5 | 4.8 | 17.4 | 66.9 | 34.4 | 3.2 | 16.0 | 63.1 | 32.3 | 5.1 | 17.4 |
| TTS Voice | 55.7 (-2.3) | 26.6 (-2.8) | 6.0 (-1.0) | 16.8 (-1.7) | 58.8 (-6.3) | 27.2 (-6.3) | 1.8 (-3.0) | 14.7 (-2.7) | 62.6 (-4.3) | 33.9 (-1.5) | 3.7 (+0.5) | 17.7 (+1.7) | 58.7 (-4.4) | 28.5 (-3.8) | 3.8 (-1.3) | 16.2 (-1.2) | |
Table 5 : Impact of instruction modality (Text vs. TTS Voice). We report performance metrics (%) and show the gap inline ( ). To objectively highlight the cognitive load distribution, for each model and metric (across rows), the most severe performance drop among the three dependency levels is marked in bold red . Conversely, any performance maintenance or gain ( ) is marked in bold teal . This reveals that dual-audio stream processing (TTS + environmental audio) uniformly degrades performance on multimodal-dependent tasks, while static AV-Present tasks remain largely immune.
Figure 3 : Qualitative error analysis of Gemini 3.0 Pro. (Top) Auditory Neglect : The model fails to trigger an action in response to a transient acoustic state change (a pause in narration). (Bottom) Spatial Grounding Failure : The model correctly identifies the target action primitive ( TAP ) based on the multimodal context but fails to predict the precise spatial coordinates of the subtitle icon.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Temperature | Max Tokens | Top_p | Top_k | Do_Sample | Seed |
| Gemini 3.0 Pro | 0.0 | 4096 | - | - | - | - |
| Gemini 3.0 Flash | 0.0 | 4096 | - | - | - | - |
| Gemini 2.5 Pro | 0.0 | 4096 | - | - | - | - |
| Gemini 2.5 Flash | 0.0 | 4096 | - | - | - | - |
| Qwen3-Omni | 0.0 | 4096 | - | - | - | - |
| MiniCPM-o 4.5 | 0.0 | 4096 | - | - | False | - |
Table 6 : Hyperparameter configurations for all evaluated models.
| Chinese Applications (ZH) | English Applications (EN) | ||||||||||||||||||||
| App Name | Ep. | Stp. | Task Dimensions | Modality Dep. | App Name | Ep. | Stp. | Task Dimensions | Modality Dep. | ||||||||||||
| Loc | Sem | CrM | Tmp | Ins | Cri | Sup | Pre | Loc | Sem | CrM | Tmp | Ins | Cri | Sup | Pre | ||||||
| Bilibili | 29 | 91 | 7 | 5 | 6 | 6 | 5 | 10 | 15 | 4 | Duolingo | 35 | 133 | 5 | 5 | 5 | 15 | 5 | 16 | 11 | 8 |
| Douyin | 29 | 97 | 7 | 5 | 6 | 6 | 5 | 12 | 8 | 9 | Vimeo | 25 | 86 | 5 | 5 | 5 | 5 | 5 | 12 | 6 | 7 |
| Meituan | 26 | 103 | 6 | 5 | 6 | 5 | 4 | 4 | 5 | 17 | TED | 25 | 115 | 5 | 5 | 5 | 5 | 5 | 16 | 6 | 3 |
| QQ Music | 25 | 95 | 5 | 5 | 5 | 5 | 5 | 1 | 11 | 13 | Snapchat | 25 | 104 | 5 | 5 | 5 | 5 | 5 | 8 | 11 | 6 |
Table 7 : Comprehensive breakdown of the OmniGUI dataset per application. The table reports the volume of episodes (Ep.) and total steps (Stp.) alongside their exact distribution across the five Task Dimensions ( Loc. = Localization, Sem. = Semantic Understanding, CrM. = Cross-modal Discrimination, Tmp. = Temporal Reasoning, Ins. = Instant Response) and the three Multimodal Dependency levels ( Cri. = AV-Critical, Sup. = AV-Supportive, Pre. = AV-Present).
Figure 4 : Capability fingerprints of evaluated models. The radar charts map the Exact Match (EM) performance. (a) Across the five operational dimensions, models universally exhibit stronger capabilities on static Localization (Loc) compared to Temporal Reasoning (Tmp) and Cross-modal Discrimination (CrM). (b) Across the multimodal dependency levels, performance contracts monotonically as tasks transition from AV-Present (P) to AV-Critical (C), visually validating the necessity of multimodal perception mechanisms.
Figure 5 : Gemini 3.0 Pro performance disaggregated by application. The applications are sorted by Exact Match (EM) scores. The right-hand axis explicitly denotes the sample volume (episodes/steps) for each application. The vertical dotted lines represent the overall benchmark averages, providing a reference to identify application environments that present above-average or below-average difficulty.