Graphical User Interface (GUI) agents have the potential to assist users in interacting with complex software (e.g., PowerPoint, Photoshop). While prior research has primarily focused on automating user actions through clicks and keystrokes, this paradigm overlooks human intention, where users value the ability to explore, iterate, and refine their ideas while maintaining agency. To move beyond automation and toward collaboration, GUI agents must understand what users are doing and why. We introduce GUIDE (GUI User Intent Detection Evaluation), a benchmark that evaluates AI models on their ability to perceive user behavior, infer intent, and provide assistance in open-ended GUI tasks. GUIDE consists of 67.5 hours of screen recordings from 120 novice user demonstrations with think-aloud narrations, across 10 software. GUIDE defines three tasks - (i) Behavior State Detection, (ii) Intent Prediction, and (iii) Help Prediction that test a model's ability to recognize behavior state, reason about goals, and decide when and how to help. Evaluations across eight state-of-the-art multimodal models reveal that all models struggled, achieving only 44.6% and 55.0% accuracy on behavior state and help prediction. However, providing user context significantly improved the performance, raising help prediction by up to 50.2pp, highlighting the critical role of structured user understanding in effective assistance. Our dataset is available at https://guide-bench.github.io.
Figures & tables
Dataset
Domain #
Video #
Video Duration
Video Source
Primary Goal
Evaluation Focus
Behavior
Intent
Help
PsTuts [ 25 ]
1
-
71.4 h
Instructional Videos
Action Understanding
VideoWebArena [ 19 ]
6
74
3.8 h
Human-Recorded Tutorials
Task Automation
VideoGUI [ 27 ]
11
178
7.1 h
Instructional Videos
Task Automation
✓
UI-Vision [ 33 ]
83
450
4.8 h
Experts Performing Tasks
Task Automation
AssistGUI [ 14 ]
9
100
<8.3 h
Instructional Videos
Task Automation
✓
Table 1: Comparison of GUIDE with existing GUI video understanding datasets. GUIDE differs from existing benchmarks by (i) collecting screen recordings from novice users, (ii) capturing how they naturally behave in open-ended tasks with a focus on behavior understanding, and (iii) evaluating systems based on human user needs rather than task automation.
Figure 1: An example of the GUIDE benchmark, which jointly models three tasks: Behavior State Detection, Intent Prediction, and Help Prediction, to interpret what the user is doing, aiming to achieve, and whether and what they may need assistance with during open-ended software tasks.
Figure 2: Overview of the three core tasks in the GUIDE benchmark. (1) User Behavior State Detection identifies the user’s current behavioral mode (e.g., Exploration and Decision-Making ). (2) Intent Prediction infers what the user is trying to achieve (e.g., Create a progress bar ). (3) Help Prediction determines whether the user needs assistance and, if so, what kind of help is relevant (e.g., Get a guide on how to use text effects ). Together, these tasks enable a comprehensive understanding of user behavior and assistance needs in software GUI environments. We evaluate MLLMs on their ability to infer these solely from the visual input, without access to the demonstrator’s narration — a setting that closely reflects real-world use.
Figure 3: Our proposed taxonomy of user behavior states in GUI-based software tasks, organized into four main phases: Planning , Execution , Problem-Solving , and Evaluation . Each phase captures distinct patterns of user cognition and interaction, from initial goal formulation to iterative action, troubleshooting, and reflection.
Model
(1) Behavior Detection
(2) Intent Prediction
(3) Help Prediction
–
+ Prev.
–
+ Behavior
Help Need Detection
Help Content Prediction
–
+ Behv.
+Behv.+Intent
–
+ Behv.
+Behv.+Intent
Gemini-2.5-Flash [ 15 ]
36.91
38.19
65.40
66.77
53.64
76.33
78.07
49.53
53.75
78.59
Gemini-2.5-Pro [ 15 ]
42.44
43.79
67.80
70.16
69.82
84.73
82.38
52.74
57.03
79.69
GPT-4o-mini [ 35 ]
17.65
17.07
60.76
62.19
46.05
78.92
82.26
31.32
42.86
79.84
GPT-4o [ 35 ]
36.32
37.24
61.19
62.58
49.69
87.79
87.91
45.95
48.37
79.78
Table 2: Evaluation results on accuracy across (1) Behavior State Detection, (2) Intent Prediction, and (3) Help Prediction.
Model
Help Need Detection
–
+ Behavior State
+ Behavior State + Intent
Acc
Prec
Rec
F1
Acc
Prec
Rec
F1
Acc
Prec
Rec
F1
Gemini-2.5-Flash [ 15 ]
53.64
83.27
36.62
50.87
76.33
97.67
65.47
78.39
78.07
94.56
70.62
80.86
Gemini-2.5-Pro [ 15 ]
69.82
76.42
78.09
77.42
84.73
93.61
82.34
87.61
82.38
91.20
80.94
86.76
GPT-4o-mini [ 35 ]
46.05
83.03
22.31
35.17
76.73
97.61
66.23
78.92
79.71
97.20
71.29
82.26
GPT-4o [ 35 ]
49.69
74.41
35.14
47.73
87.79
95.39
85.53
90.19
87.91
95.12
85.95
90.30
Table 3: Results for Help Need Detection on accuracy, precision, recall, and F1-score across three conditions (default, with behavior state, with behavior state and intent).
Figure 4: Accuracy trends across the tasks in the online setting, where models are given progressively more of the video segment (25%, 50%, 75%, and 100%). Models show consistent improvement as they see more segments, with Gemini-2.5-Flash [ 15 ] and Qwen3-VL-8B [ 42 ] showing larger and more consistent gains across all four tasks compared to the smaller open-source models.
Model
Intent Prediction
Help Prediction
Acc
MBAcc
Acc
MBAcc
Gemini-2.5-Flash [ 15 ]
65.40
59.09
49.53
44.69
Gemini-2.5-Pro [ 15 ]
67.80
64.34
52.74
45.31
GPT-4o-mini [ 35 ]
60.76
50.24
31.32
28.59
GPT-4o [ 35 ]
61.19
56.58
45.95
41.25
Claude-4.5-Sonnet [ 1 ]
71.39
65.44
55.00
50.78
Table 4: Evaluation of Intent Prediction and Help Content Prediction , with Accuracy (Acc) and Multi-Binary Accuracy (MBAcc).
Variable
Value
# Videos
120
Total Duration
67.5 hours
Avg. Duration
33 min 44 sec
Max Duration
1 hour 23 min 50 sec
Min Duration
16 min 42 sec
Think-Aloud Narration Ratio
78%
Table B1 : Statistics of the GUIDE dataset.
Figure B1 : Distribution of screen recording video lengths in the dataset.
Figure B2 : Software categories represented in the dataset.
Category
Software
Tasks
Photo Editing
Photoshop, GIMP
1. Create a composite from two images. 2. Create a bakery logo with a warm, friendly identity. 3. Replace a photo’s background with a custom-designed pattern. 4. Design a movie poster.
Graphic Design
Figma, Canva
1. Design a mobile sign-up screen for a fictional app. 2. Design a custom 404 error page with a visual and animated element. 3. Design compact profile cards that display personal user details. 4. Design an event poster for a music festival.
Presentation Design
PowerPoint, Google Slides
1. Create a product pitch deck that highlights the MacBook’s key features. 2. Create an interactive timeline presenting a company’s history. 3. Create a 5-slide nature-themed shape-masked photo scrapbook. 4. Create a quiz deck with 3 multiple-choice questions.
Video Editing
Premiere Pro, CapCut
1. Edit a short interview to improve clarity and engagement. 2. Design a creative intro using animated text. 3. Edit a short instructional video to clearly guide a process. 4. Transform a long video into a highly engaging short-form clip.
Data Analysis
Microsoft Excel, Google Sheets
1. Design a Gantt chart for a mini project. 2. Summarize and visualize responses from a survey. 3. Visualize student performance across subjects. 4. Summarize and visualize product sales by category or region.
Table B2: Overview of open-ended tasks across software categories. Each category includes two software applications and four tasks designed to elicit natural and diverse user behaviors.
Behavior State
Description
Examples
Planning
Task Understanding and Preparation
The user is focused on the logistics of the task. This includes interpreting the task, gathering necessary digital assets, and configuring the software environment. Their goal is to set up the conditions needed to begin the work.
The user is engaged in high-level conceptual work. They are brainstorming ideas, outlining the structure of the outcome, or creating a plan for how to approach the task. This often involves creating preliminary, non-final content that serves as a guide.
Formulating high-level strategy, creating step lists, sketching rough layouts or wireframes. The output is a plan or outline, not the final polished product.
Execution
Exploration and Decision-Making
The user experiments with different options or features to understand their effects and decide which one to use. This exploratory phase involves deliberate trial and comparison, often pausing forward progress to evaluate alternatives.
Applying effects and undoing them, hovering over tools to see what they do, testing multiple font sizes to decide which fits best.
Performing Actions
The user is confidently using the software to make progress on the task. These actions are purposeful and executed with little hesitation.
Typing/deleting text, inserting and resizing images, applying formatting with clear intent, searching for functions to use.
Table C3: Taxonomy of user behavior states in open-ended GUI workflows.
Figure C3 : Distribution of user behavior states across Planning, Execution, Problem-Solving, and Evaluation phases across the videos in the dataset.
Dataset
Intent Prediction
Help Need Detection
Help Content Prediction
Behavior State
Count
(%)
Count
(%)
Count
(%)
Count
(%)
Planning
Task Understanding and Preparation
1054
8.51%
216
9.85%
103
5.65%
36
2.82%
Ideation and Planning
993
8.02%
282
12.86%
84
4.61%
41
3.21%
Execution
Exploration and Decision-Making
1847
14.92%
289
13.18%
162
8.89%
114
8.92%
Table C4: Distribution of behavior state labels across the full dataset and specific evaluation tasks. Note that annotated instances used in the evaluation tasks may involve two or more states (e.g., a single segment containing both Debugging and Seeking External Help ). Behavior State Detection uniformly sampled 200 instances from each class.
Figure C4 : Normalized confusion matrix for user behavior state classification. The most common errors occur when Frustration or Debugging is misclassified as Performing Actions or Exploration and Decision-Making.
Software : Canva, Task : Design a mobile sign-up screen for a fictional app
(4:11) “Nope, that’s not what I wanted to do. Try again. Alright, let’s do a text box.”
(4:40) “And we need to make this much smaller so it fits there at the top.”
(6:50) “Progress bar for… what? I don’t know. Just do progress bar. But you know what? Let’s try ChatGPT because maybe they can help us.”
(10:07) “Oh, that’s cool, okay. So I can create the progress bar using the free elements with the shapes. Alright, so let’s try and do that. Alright. Elements.”
(21:01) “Ooh, what is this? Ooh, I like that so much better. All right, we’re gonna keep this. Yes, we’re gonna keep this.”
(22:05) “Not necessarily sure how I can… isn’t there a way to like make it a certain size? How do I do that?”
Table D5: Example video illustrating the user’s on-screen actions accompanied by think-aloud narration.
Software : CapCut, Task : Design a creative intro using animated text.
(4:56)“but I want to make it a bit more dynamic so all right.”
(7:55)“oh I think here it should be only one word appearing at a time.”
(11:39) “there are so many effects that I get a little bit too overwhelmed to see so many.”
(17:55) “I would like to add additional different animation for this one so I like to keep this.”
(18:42) “I am satisfied with the results so I will export this.”
(20:14) “Maybe I would like to make it a bit slower.”
Table D6: Example video illustrating the user’s on-screen actions accompanied by think-aloud narration.
Figure F5 : Example outcomes of the assigned tasks. The diversity across outputs reflects the open-ended nature of the tasks.
Figure G6 : Annotation interface for validating and refining LLM-generated behavior-state labels. Annotators reviewed the predicted labels and the associated reasoning, correcting them if inaccurate. Each video segment was independently verified by two external annotators.
Figure G7 : Before participating in the annotation, annotators completed a quiz phase where they had to correctly classify example video segments. This process ensured that all annotators possessed a solid understanding of the behavior taxonomy and definitions.
Figure H8 : Prompt to generate a taxonomy of user behavior states given demonstration videos.
Figure H9 : Prompt to annotate a given video segment based on the taxonomy of user behavior states.
Figure H10 : Prompt used to annotate a given video segment with the user’s intent.
Figure H11 : Prompt used to annotate a given video segment with whether help is needed and, if so, what specific help is required.
Figure H12 : Prompt used to filter on-screen help-seeking behavior from segments previously marked as help needed.
Figure H13 : Prompt used to filter narration-based help-seeking behavior from segments previously marked as help needed.
Figure H14 : Prompt used to filter clear no-help-needed segments.
Figure H15 : Prompt used to evaluate the model on the (1) Behavior State Detection task.
Figure H16 : Prompt used to evaluate the model on the (2) Intent Prediction task.
Figure H17 : Prompt used to evaluate the model on the (3-1) Help Need Detection task.
Figure H18 : Prompt used to evaluate the model on the (3-2) Help Content Prediction task.
Autonomous agents capable of navigating Graphical User Interfaces (GUIs) hold the potential to revolutionize digital productivity. However, achieving true digital autonomy extends beyond reactive element matching; it necessitates a predictive mental model of interface dynamics and the ability to foresee the "digital world state" resulting from interactions. Despite the perceptual capabilities of modern Vision-Language Models (VLMs), existing benchmarks remain bifurcated (focusing either on black-box task completion or static, shallow grounding), thereby failing to assess whether agents truly comprehend the implicit functionality and transition logic of GUIs. To bridge this gap, we introduce AutoGUI-v2, a comprehensive benchmark designed to evaluate deep GUI functionality understanding and interaction outcome prediction. We construct the benchmark using a novel VLM-human collaborative pipeline that recursively parses multi-platform screenshots into hierarchical functional regions to generate diverse evaluation tasks. Providing 2,753 tasks across six operating systems, AutoGUI-v2 rigorously tests agents on region and element-level semantics, grounding, and dynamic state prediction. Our evaluation reveals a striking dichotomy in VLMs: while open-source models fine-tuned on agent data (e.g., Qwen3-VL) excel at functional grounding, commercial models (e.g., Gemini-2.5-Pro-Thinking) dominate in functionality captioning. Crucially, all models struggle with complex interaction logic of uncommon actions, highlighting that deep functional understanding remains a significant hurdle. By systematically measuring these foundational capabilities, AutoGUI-v2 offers a new lens for advancing the next generation of GUI agents.
Hongxin Li, Xiping Wang, Jingran Su +4
University of Chinese Academy of Sciences (UCAS) · 2New Laboratory of Pattern Recognition (NLPR), CASIA · 3State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA +3
Current Graphical User Interface (GUI) agents operate primarily under a reactive paradigm: a user must provide an explicit instruction for the agent to execute a task. However, an intelligent AI assistant should be proactive, which is capable of anticipating user intentions directly from continuous visual inputs, such as mobile or desktop screenshots, and offering timely recommendations without explicit user prompting. Transitioning to this proactive paradigm presents significant challenges. Real-world screen activity is rarely linear; it consists of long-horizon trajectories fraught with noisy browsing, meaningless actions, and multithreaded task-switching. To address this gap, we introduce PIRA-Bench (Proactive Intent Recommendation Agent Benchmark), a novel benchmark for evaluating multimodal large language models (MLLMs) on continuous, weakly-supervised visual inputs. Unlike reactive datasets, PIRA-Bench features complex trajectories with multiple interleaved intents and noisy segments with various user profile contexts, challenging agents to detect actionable events while fitting to user preferences. Furthermore, we propose the PIRF baseline, a memory-aware, state-tracking framework that empowers general MLLMs to manage multiple task threads and handle misleading visual inputs. PIRA-Bench serves as an initial step toward robust and proactive GUI-based personal assistants.
Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions is still poor, limiting user trust. Our analysis of failure cases from advanced models suggests a long-tail pattern in GUI operations, where a relatively small fraction of complex and diverse interactions accounts for a disproportionate share of task failures. We hypothesize that this issue largely stems from the scarcity of data for complex interactions. To address this problem, we propose a new benchmark CUActSpot for evaluating models' capabilities on complex interactions across five modalities: GUI, text, table, canvas, and natural image, as well as a variety of actions (click, drag, draw, etc.), covering a broader range of interaction types than prior click-centric benchmarks that focus mainly on GUI widgets. We also design a renderer-based data-synthesis pipeline: scenes are automatically generated for each modality, screenshots and element coordinates are recorded, and an LLM produces matching instructions and action traces. After training on this corpus, our Phi-Ground-Any-4B outperforms open-source models with fewer than 32B parameters. We will release our benchmark, data, code, and models at https://github.com/microsoft/Phi-Ground.git
Miaosen Zhang, Xiaohan Zhao, Zhihong Tan +14
1Southeast University · 2Mohamed bin Zayed University of Artificial Intelligence · 3Wuhan University +2