cs.CVMar 26, 2026
SaveGUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks
Organizations: KAIST · Carnegie Mellon University · University of Oxford · Konkuk University · Google Inc. · SkillBench
Abstract
Graphical User Interface (GUI) agents have the potential to assist users in interacting with complex software (e.g., PowerPoint, Photoshop). While prior research has primarily focused on automating user actions through clicks and keystrokes, this paradigm overlooks human intention, where users value the ability to explore, iterate, and refine their ideas while maintaining agency. To move beyond automation and toward collaboration, GUI agents must understand what users are doing and why. We introduce GUIDE (GUI User Intent Detection Evaluation), a benchmark that evaluates AI models on their ability to perceive user behavior, infer intent, and provide assistance in open-ended GUI tasks. GUIDE consists of 67.5 hours of screen recordings from 120 novice user demonstrations with think-aloud narrations, across 10 software. GUIDE defines three tasks - (i) Behavior State Detection, (ii) Intent Prediction, and (iii) Help Prediction that test a model's ability to recognize behavior state, reason about goals, and decide when and how to help. Evaluations across eight state-of-the-art multimodal models reveal that all models struggled, achieving only 44.6% and 55.0% accuracy on behavior state and help prediction. However, providing user context significantly improved the performance, raising help prediction by up to 50.2pp, highlighting the critical role of structured user understanding in effective assistance. Our dataset is available at https://guide-bench.github.io.
Figures & tables
| Dataset | Domain # | Video # | Video Duration | Video Source | Primary Goal | Evaluation Focus | ||
|---|---|---|---|---|---|---|---|---|
| Behavior | Intent | Help | ||||||
| PsTuts [ 25 ] | 1 | - | 71.4 h | Instructional Videos | Action Understanding | |||
| VideoWebArena [ 19 ] | 6 | 74 | 3.8 h | Human-Recorded Tutorials | Task Automation | |||
| VideoGUI [ 27 ] | 11 | 178 | 7.1 h | Instructional Videos | Task Automation | ✓ | ||
| UI-Vision [ 33 ] | 83 | 450 | 4.8 h | Experts Performing Tasks | Task Automation | |||
| AssistGUI [ 14 ] | 9 | 100 | <8.3 h | Instructional Videos | Task Automation | ✓ | ||
Table 1: Comparison of GUIDE with existing GUI video understanding datasets. GUIDE differs from existing benchmarks by collecting screen recordings from novice users, capturing how they naturally behave in open-ended tasks with a focus on behavior understanding, and evaluating systems based on human user needs rather than task automation.
Figure 1: An example of the GUIDE benchmark, which jointly models three tasks: Behavior State Detection, Intent Prediction, and Help Prediction, to interpret what the user is doing, aiming to achieve, and whether and what they may need assistance with during open-ended software tasks.
Figure 2: Overview of the three core tasks in the GUIDE benchmark. (1) User Behavior State Detection identifies the user’s current behavioral mode (e.g., Exploration and Decision-Making ). (2) Intent Prediction infers what the user is trying to achieve (e.g., Create a progress bar ). (3) Help Prediction determines whether the user needs assistance and, if so, what kind of help is relevant (e.g., Get a guide on how to use text effects ). Together, these tasks enable a comprehensive understanding of user behavior and assistance needs in software GUI environments. We evaluate MLLMs on their ability to infer these solely from the visual input, without access to the demonstrator’s narration — a setting that closely reflects real-world use.
Figure 3: Our proposed taxonomy of user behavior states in GUI-based software tasks, organized into four main phases: Planning , Execution , Problem-Solving , and Evaluation . Each phase captures distinct patterns of user cognition and interaction, from initial goal formulation to iterative action, troubleshooting, and reflection.
| Model | (1) Behavior Detection | (2) Intent Prediction | (3) Help Prediction | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| – | + Prev. | – | + Behavior | Help Need Detection | Help Content Prediction | |||||
| – | + Behv. | +Behv.+Intent | – | + Behv. | +Behv.+Intent | |||||
| Gemini-2.5-Flash [ 15 ] | 36.91 | 38.19 | 65.40 | 66.77 | 53.64 | 76.33 | 78.07 | 49.53 | 53.75 | 78.59 |
| Gemini-2.5-Pro [ 15 ] | 42.44 | 43.79 | 67.80 | 70.16 | 69.82 | 84.73 | 82.38 | 52.74 | 57.03 | 79.69 |
| GPT-4o-mini [ 35 ] | 17.65 | 17.07 | 60.76 | 62.19 | 46.05 | 78.92 | 82.26 | 31.32 | 42.86 | 79.84 |
| GPT-4o [ 35 ] | 36.32 | 37.24 | 61.19 | 62.58 | 49.69 | 87.79 | 87.91 | 45.95 | 48.37 | 79.78 |
Table 2: Evaluation results on accuracy across (1) Behavior State Detection, (2) Intent Prediction, and (3) Help Prediction.
| Model | Help Need Detection | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| – | + Behavior State | + Behavior State + Intent | ||||||||||
| Acc | Prec | Rec | F1 | Acc | Prec | Rec | F1 | Acc | Prec | Rec | F1 | |
| Gemini-2.5-Flash [ 15 ] | 53.64 | 83.27 | 36.62 | 50.87 | 76.33 | 97.67 | 65.47 | 78.39 | 78.07 | 94.56 | 70.62 | 80.86 |
| Gemini-2.5-Pro [ 15 ] | 69.82 | 76.42 | 78.09 | 77.42 | 84.73 | 93.61 | 82.34 | 87.61 | 82.38 | 91.20 | 80.94 | 86.76 |
| GPT-4o-mini [ 35 ] | 46.05 | 83.03 | 22.31 | 35.17 | 76.73 | 97.61 | 66.23 | 78.92 | 79.71 | 97.20 | 71.29 | 82.26 |
| GPT-4o [ 35 ] | 49.69 | 74.41 | 35.14 | 47.73 | 87.79 | 95.39 | 85.53 | 90.19 | 87.91 | 95.12 | 85.95 | 90.30 |
Table 3: Results for Help Need Detection on accuracy, precision, recall, and F1-score across three conditions (default, with behavior state, with behavior state and intent).
Figure 4: Accuracy trends across the tasks in the online setting, where models are given progressively more of the video segment (25%, 50%, 75%, and 100%). Models show consistent improvement as they see more segments, with Gemini-2.5-Flash [ 15 ] and Qwen3-VL-8B [ 42 ] showing larger and more consistent gains across all four tasks compared to the smaller open-source models.
| Model | Intent Prediction | Help Prediction | ||
|---|---|---|---|---|
| Acc | MBAcc | Acc | MBAcc | |
| Gemini-2.5-Flash [ 15 ] | 65.40 | 59.09 | 49.53 | 44.69 |
| Gemini-2.5-Pro [ 15 ] | 67.80 | 64.34 | 52.74 | 45.31 |
| GPT-4o-mini [ 35 ] | 60.76 | 50.24 | 31.32 | 28.59 |
| GPT-4o [ 35 ] | 61.19 | 56.58 | 45.95 | 41.25 |
| Claude-4.5-Sonnet [ 1 ] | 71.39 | 65.44 | 55.00 | 50.78 |
Table 4: Evaluation of Intent Prediction and Help Content Prediction , with Accuracy (Acc) and Multi-Binary Accuracy (MBAcc).
| Variable | Value |
|---|---|
| # Videos | 120 |
| Total Duration | 67.5 hours |
| Avg. Duration | 33 min 44 sec |
| Max Duration | 1 hour 23 min 50 sec |
| Min Duration | 16 min 42 sec |
| Think-Aloud Narration Ratio | 78% |
Table B1 : Statistics of the GUIDE dataset.
Figure B1 : Distribution of screen recording video lengths in the dataset.
Figure B2 : Software categories represented in the dataset.
| Category | Software | Tasks |
|---|---|---|
| Photo Editing | Photoshop, GIMP | 1. Create a composite from two images. 2. Create a bakery logo with a warm, friendly identity. 3. Replace a photo’s background with a custom-designed pattern. 4. Design a movie poster. |
| Graphic Design | Figma, Canva | 1. Design a mobile sign-up screen for a fictional app. 2. Design a custom 404 error page with a visual and animated element. 3. Design compact profile cards that display personal user details. 4. Design an event poster for a music festival. |
| Presentation Design | PowerPoint, Google Slides | 1. Create a product pitch deck that highlights the MacBook’s key features. 2. Create an interactive timeline presenting a company’s history. 3. Create a 5-slide nature-themed shape-masked photo scrapbook. 4. Create a quiz deck with 3 multiple-choice questions. |
| Video Editing | Premiere Pro, CapCut | 1. Edit a short interview to improve clarity and engagement. 2. Design a creative intro using animated text. 3. Edit a short instructional video to clearly guide a process. 4. Transform a long video into a highly engaging short-form clip. |
| Data Analysis | Microsoft Excel, Google Sheets | 1. Design a Gantt chart for a mini project. 2. Summarize and visualize responses from a survey. 3. Visualize student performance across subjects. 4. Summarize and visualize product sales by category or region. |
Table B2: Overview of open-ended tasks across software categories. Each category includes two software applications and four tasks designed to elicit natural and diverse user behaviors.
| Behavior State | Description | Examples |
| Planning | ||
| Task Understanding and Preparation | The user is focused on the logistics of the task. This includes interpreting the task, gathering necessary digital assets, and configuring the software environment. Their goal is to set up the conditions needed to begin the work. | Reading task instructions, opening required software/files/templates, arranging workspace (resizing windows, organizing directories), downloading images for photo editing. |
| Ideation and Planning | The user is engaged in high-level conceptual work. They are brainstorming ideas, outlining the structure of the outcome, or creating a plan for how to approach the task. This often involves creating preliminary, non-final content that serves as a guide. | Formulating high-level strategy, creating step lists, sketching rough layouts or wireframes. The output is a plan or outline, not the final polished product. |
| Execution | ||
| Exploration and Decision-Making | The user experiments with different options or features to understand their effects and decide which one to use. This exploratory phase involves deliberate trial and comparison, often pausing forward progress to evaluate alternatives. | Applying effects and undoing them, hovering over tools to see what they do, testing multiple font sizes to decide which fits best. |
| Performing Actions | The user is confidently using the software to make progress on the task. These actions are purposeful and executed with little hesitation. | Typing/deleting text, inserting and resizing images, applying formatting with clear intent, searching for functions to use. |
Table C3: Taxonomy of user behavior states in open-ended GUI workflows.
Figure C3 : Distribution of user behavior states across Planning, Execution, Problem-Solving, and Evaluation phases across the videos in the dataset.
| Dataset | Intent Prediction | Help Need Detection | Help Content Prediction | |||||
| Behavior State | Count | (%) | Count | (%) | Count | (%) | Count | (%) |
| Planning | ||||||||
| Task Understanding and Preparation | 1054 | 8.51% | 216 | 9.85% | 103 | 5.65% | 36 | 2.82% |
| Ideation and Planning | 993 | 8.02% | 282 | 12.86% | 84 | 4.61% | 41 | 3.21% |
| Execution | ||||||||
| Exploration and Decision-Making | 1847 | 14.92% | 289 | 13.18% | 162 | 8.89% | 114 | 8.92% |
Table C4: Distribution of behavior state labels across the full dataset and specific evaluation tasks. Note that annotated instances used in the evaluation tasks may involve two or more states (e.g., a single segment containing both Debugging and Seeking External Help ). Behavior State Detection uniformly sampled 200 instances from each class.
Figure C4 : Normalized confusion matrix for user behavior state classification. The most common errors occur when Frustration or Debugging is misclassified as Performing Actions or Exploration and Decision-Making.
| Software : Canva, Task : Design a mobile sign-up screen for a fictional app | |
|---|---|
| (4:11) “Nope, that’s not what I wanted to do. Try again. Alright, let’s do a text box.” | (4:40) “And we need to make this much smaller so it fits there at the top.” |
| (6:50) “Progress bar for… what? I don’t know. Just do progress bar. But you know what? Let’s try ChatGPT because maybe they can help us.” | (10:07) “Oh, that’s cool, okay. So I can create the progress bar using the free elements with the shapes. Alright, so let’s try and do that. Alright. Elements.” |
| (21:01) “Ooh, what is this? Ooh, I like that so much better. All right, we’re gonna keep this. Yes, we’re gonna keep this.” | (22:05) “Not necessarily sure how I can… isn’t there a way to like make it a certain size? How do I do that?” |
Table D5: Example video illustrating the user’s on-screen actions accompanied by think-aloud narration.
| Software : CapCut, Task : Design a creative intro using animated text. | |
|---|---|
| (4:56)“but I want to make it a bit more dynamic so all right.” | (7:55)“oh I think here it should be only one word appearing at a time.” |
| (11:39) “there are so many effects that I get a little bit too overwhelmed to see so many.” | (17:55) “I would like to add additional different animation for this one so I like to keep this.” |
| (18:42) “I am satisfied with the results so I will export this.” | (20:14) “Maybe I would like to make it a bit slower.” |
Table D6: Example video illustrating the user’s on-screen actions accompanied by think-aloud narration.
Figure F5 : Example outcomes of the assigned tasks. The diversity across outputs reflects the open-ended nature of the tasks.
Figure G6 : Annotation interface for validating and refining LLM-generated behavior-state labels. Annotators reviewed the predicted labels and the associated reasoning, correcting them if inaccurate. Each video segment was independently verified by two external annotators.
Figure G7 : Before participating in the annotation, annotators completed a quiz phase where they had to correctly classify example video segments. This process ensured that all annotators possessed a solid understanding of the behavior taxonomy and definitions.
Figure H8 : Prompt to generate a taxonomy of user behavior states given demonstration videos.
Figure H9 : Prompt to annotate a given video segment based on the taxonomy of user behavior states.
Figure H10 : Prompt used to annotate a given video segment with the user’s intent.
Figure H11 : Prompt used to annotate a given video segment with whether help is needed and, if so, what specific help is required.
Figure H12 : Prompt used to filter on-screen help-seeking behavior from segments previously marked as help needed.
Figure H13 : Prompt used to filter narration-based help-seeking behavior from segments previously marked as help needed.
Figure H14 : Prompt used to filter clear no-help-needed segments.
Figure H15 : Prompt used to evaluate the model on the (1) Behavior State Detection task.
Figure H16 : Prompt used to evaluate the model on the (2) Intent Prediction task.
Figure H17 : Prompt used to evaluate the model on the (3-1) Help Need Detection task.
Figure H18 : Prompt used to evaluate the model on the (3-2) Help Content Prediction task.