Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search Systems

Maria Movin; Claudia Hauff; Aron Henriksson; Panagiotis Papapetrou

cs.IRApr 9, 2026

Same Outcomes, Different Journeys: A Trace-Level Framework for Comparing Human and GUI-Agent Behavior in Production Search Systems

Authors: Maria Movin, Claudia Hauff, Aron Henriksson, Panagiotis Papapetrou

Organizations: Spotify, Sweden · Stockholm University, Sweden · Spotify, Netherlands

Abstract

LLM-driven GUI agents are increasingly used in production systems to automate workflows and simulate users for evaluation and optimization. Yet most GUI-agent evaluations emphasize task success and provide limited evidence on whether agents interact in human-like ways. We present a trace-level evaluation framework that compares human and agent behavior across (i) task outcome and effort, (ii) query formulation, and (iii) navigation across interface states. We instantiate the framework in a controlled study in a production audio-streaming search application, where 39 participants and a state-of-the-art GUI agent perform ten multi-hop search tasks. The agent achieves task success comparable to participants and generates broadly aligned queries, but follows systematically different navigation strategies: participants exhibit content-centric, exploratory behavior, while the agent is more search-centric and low-branching. These results show that outcome and query alignment do not imply behavioral alignment, motivating trace-level diagnostics when deploying GUI agents as proxies for users in production search systems.

Figures & tables

Figure 1: Overview of the trace-level evaluation framework: task design, human–agent trace collection in a production search application, and behavioral comparison across outcomes, query formulation, and navigation metrics.

Linear
ID	Task Instruction
T1	Find the artist who was guest at the podcast “Sommar i P1” 30 June 2023, and play one of his tracks
T2	Find the artist who has covered the song with the lyrics ‘Jag vill känna att jag lever’, originally performed by Helen Sjöholm, and play a track from their latest album
T3	Play the first song of an album released 2023 of one of the singers who is currently twice in the top-10 of the UK singles charts
T4	Play the most streamed song of the artist interviewed in the New York Times podcast about pop music from 22 August
T5	Play the first track of the newest album of the composer writing the soundtrack for a Netflix series where chess features heavily

Table 1: Multi-hop search tasks by reasoning pattern (T1–T10)

Panel of Figure 2 — Figure 2: Prompt instructions added to the AppAgent system prompt to encourage in-application reasoning rather than answering from prior knowledge.

ID	S/ $n_{c}$	Queries	S/n	Fam	TD	t (s)	Queries
	Agent		Participant
T1	2/5	sommar i p1 sommar i p1s sommar i p1 30 june 2023	11/18	11	2.4	$230\pm 182$	sommar i p1 sommar i p1 2023 guest in sommar i p1
$\ddagger$ T2	3/5	jag vill känna att jag jag vill känna att j jag vill känna	12/18	12	2.4	$108\pm 86$	jag vill känna att jag lever gabriellas sång cover på jag vill känna att jag lever
T3	4/5	top 50 uk uk singles chart uk top 50	10/14	6	2.9	$171\pm 180$	top 10 uk uk top 10 singles uk singles
T4	2/4	new york times pop music podcast new york times pop music $\dagger$ popcast	8/15	8	3.0	$218\pm 137$	new york times new york times podcast which artist was interviewed in new york times podcast
$\ddagger$ T5	3/5	chess netflix netflix chess soundtrack netflix chess series	7/12	6	2.6	$110\pm 103$	queens gambit netflix netflix series chws

Table 2: Multi-hop tasks and execution statistics for agent runs and participants.

Cohort	Success s/n	Success % [95% CI]	t (s)	#Actions
Agents	$22/39$	$56.4\ [41.0,70.7]$	$1076\pm 760$	$11\ [9,19]$
Participants	$80/150$	$53.3\ [45.4,61.1]$	$209\pm 175$	$18\ [12,30]$
Participants’ Cohorts
Expert	$39/62$	$62.9\ [50.5,73.8]$	$150\pm 107$	$18\ [12,25]$
Regular	$41/88$	$46.6\ [36.5,56.9]$	$251\pm 200$	$21\ [15,38]$
Familiar	$45/75$	$60.0\ [48.7,70.3]$	$184\pm 172$	$19\ [13,32]$

Table 3: Task success and effort by cohort (CS Set; all tasks). Success is reported as completed tasks (s/n), percentage, and 95% CI. Time is mean

\pm

SD (s); actions are median [Q1, Q3]. Statistical comparisons are reported in the text.

Panel of Figure 3 — Figure 3: Coverage of agent queries within the participants’ query space. Curves compare agent (green), a size-matched random subset of unique participant queries (pink; mean with 95% bootstrap CI shaded), and a top- $N$ frequency oracle (gold). (a) $N$ fixed to the total number of unique agent queries ( $N=40$ ). (b) Each curve adds $k$ queries, with $k$ varying up to $N_{\max}=40$ . Both panels use the full query set (TE).

Panel of Figure 4 — Figure 4: Macro-averaged transition graphs by cohort (TE set). Node size: visitation share; edge width: transition frequency. For readability, the participant-only track node is omitted from the plots but included in all metrics. Shown are transitions with probability $\geq 0.05$ ; transitions with $<0.1$ probability are in gray .

GUI Agents

Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.

See all 178 papers

VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
Apr 23, 2026Qijun Han, Haoqin Tu, Zijun Wang +11GUI Agents GUI Automation
UC Santa Cruz · CMU · UNC-Chapel Hill +2
GUIDE: Interpretable GUI Agent Evaluation via Hierarchical Diagnosis
Apr 6, 2026Yuwen Zhai, Runze Li, Liang Wang +6AI Agent Evaluation GUI Agents
Alibaba Group · School of Computer Science and Technology, East China Normal University · Shanghai Innovation Institute
Diagnosing Personalization Failures in Mobile GUI Agents
Oct 22, 2026Zhixin Lin, Dongliang Xu, Jungang Li +6GUI Agents Mobile GUI Agents
Shandong University · VIVO AI Lab · The Hong Kong University of Science and Technology (GZ) +5

Apr 23, 2026J/K move · Enter open · S save

VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation

Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai+10

UC Santa Cruz · CMU · UNC-Chapel Hill+2

GUI Agents GUI Automation

Open paper AskPDF arXiv

Autonomous GUI agents face two fundamental challenges: early stopping, where agents prematurely declare success without verifiable evidence, and repetitive loops, where agents cycle through the same failing actions without recovery. We present VLAA-GUI, a modular GUI agentic framework built around three integrated components that guide the system on when to Stop, Recover, and Search. First, a mandatory Completeness Verifier enforces UI-observable success criteria and verification at every finish step -- with an agent-level verifier that cross-examines completion claims with decision rules, rejecting those lacking direct visual evidence. Second, a mandatory Loop Breaker provides multi-tier filtering: switching interaction mode after repeated failures, forcing strategy changes after persistent screen-state recurrence, and binding reflection signals to strategy shifts. Third, an on-demand Search Agent searches online for unfamiliar workflows by directly querying a capable LLM with search ability, returning results as plain text. We additionally integrate a Coding Agent for code-intensive actions and a Grounding Agent for precise action grounding, both invoked on demand when required. We evaluate VLAA-GUI across five top-tier backbones, including Opus 4.5, 4.6 and Gemini 3.1 Pro, on two benchmarks with Linux and Windows tasks, achieving top performance on both (77.5% on OSWorld and 61.0% on WindowsAgentArena). Notably, three of the five backbones surpass human performance (72.4%) on OSWorld in a single pass. Ablation studies show that all three proposed components consistently improve a strong backbone, while a weaker backbone benefits more from these tools when the step budget is sufficient. Further analysis also shows that the Loop Breaker nearly halves wasted steps for loop-prone models.

Abstract

GUI Agents

Explore similar work

VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation

GUIDE: Interpretable GUI Agent Evaluation via Hierarchical Diagnosis

Diagnosing Personalization Failures in Mobile GUI Agents