cs.CVApr 2, 2026

GPA: Learning GUI Process Automation from Demonstrations

Authors: Zirui Zhao, Jun Hao Liew, Yan Yang, Wenzhuo Yang, Ziyang Luo, Doyen Sahoo, Silvio Savarese, Junnan Li

Organizations: Salesforce AI Research

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

GUI Process Automation (GPA) is a lightweight but general vision-based Robotic Process Automation (RPA), which enables fast and stable process replay with only a single demo. Addressing the fragility of traditional RPA and the non-deterministic risks of current vision language model-based GUI agents, GPA introduces three core benefits: (1) Robustness via Sequential Monte Carlo-based localization to handle rescaling and detection uncertainty; (2) Deterministic and Reliability safeguarded by readiness calibration; and (3) Privacy through fast, fully local execution. This approach delivers the adaptability, robustness, and security required for enterprise workflows. It can also be used as an MCP/CLI tool by other agents with coding capabilities so that the agent only reasons and orchestrates while GPA handles the GUI execution. We conducted a pilot experiment to compare GPA with Gemini 3 Pro (with CUA tools) and found that GPA achieves higher success rate with 10 times faster execution speed in finishing long-horizon GUI tasks.

Figures & tables

 GPAGUI AgentsTraditional RPA
SetupLearn from a single demonstrationPrompt engineeringManual scripting and selector labelling
Runtime AITiny local models onlyLLM call per actionNo AI
ReliabilityDeterministic, reliable replayNon-deterministicFragile under UI drift
PrivacyFully localScreenshots sent to external providersLocal
VariablesAuto-extracted at build timeAd hoc via promptingHard-coded
Table 1: Comparison of GPA with VLM-based GUI agents and traditional RPA . GPA combines demonstration-based workflow construction with deterministic local execution.
Figure 1: Two phases of GPA: demonstration phase (a) and execution phase (b) . In the demonstration phase, a user interacts with the local desktop environment while the recorder captures screenshots and actions, parses them into step subgraphs, and applies LLM-based post-processing for cleanup and variable extraction. In the execution phase, GPA receives the latest observation from the local computer, loads the current recorded step, parses the screenshot into a UI graph, and combines the parsed graph, recorded workflow graph, and action in SMC localization. A finite state machine orchestrates all the components, checking readiness of each step, control execution, and handle errors.
Figure 2: Example UI graph with detected elements and edges . The highlighted node is the target, and nearby nodes helps localizing the target.
Figure 3: SMC likelihood visualization . The target to click is the checkbox next to “Edit”. (a) shows the likelihood of directly matching the target checkbox. The pointwise maximum over candidates forms the likelihood surface, showing high ambiguity since matching candidates are all identical. (b) shows the likelihood of matching the neighbor nodes of target, which helps us to differentiate the checkbox. (c) shows the joint likelihood, combining the results of (a) and (b).
Figure 4: Mixture prior over scale ss : one mode favors no resizing ( s=1s=1 ), and the other favors proportional rescaling.
Figure 5: Process of SMC . The blue points are particles that gradually converge to the target prediction.
Figure 6: Step-execution FSM and the Decide state detail . The top-level FSM is shared across all steps, while the decision logic adds a small scroll-specific branch when needed.
  GPAGemini 3 pro
Task levelAvg demo stepsSuccess (%)Time (s)Success (%)Time (s)
Simple10.8010017.8493.2210.66
Hard27.2710040.9687.64383.24
Avg22.1310033.7489.38329.31
Table 2: Pilot results: GPA vs. Gemini GUI agent. Success rate and runtime aggregated by task difficulty (defined by demonstration length).
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Precheck pipeline sequence diagram. While the environment executes the current action (gray bar), the pipeline speculatively processes upcoming steps in a background thread. The runner retrieves cached results via try_get , using them directly if confidence is sufficient.
More in this topic

GUI Automation

GUI automation uses AI agents to understand and operate graphical interfaces across desktop, mobile, and web applications.

See all 39 papers

Explore similar work

CardsList

  1. GUI-PRA: Process Reward Agent for GUI Tasks

    Sep 27, 2025Tao Xiong, Xavier Hu, Yurun Chen +6Process Reward ModelsGUI Agents

    Zhejiang University · MiLM Plus, Xiaomi Inc.

  2. AutoRPA: Efficient GUI Automation through LLM-Driven Code Synthesis from Interactions

    May 20, 2026Minghao Chen, Xinyi Hu, Zhou Yu +1GUI AgentsLLM-Based Program Synthesis

    Zhejiang Key Laboratory of Space Information Sensing and Transmission, School of Computer Science, Hangzhou Dianzi University, China.

  3. Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction

    Apr 7, 2026Yuzhe Zhang, Xianwei Xue, Xingyong Wu +8GUI AgentsGUI Automation

    Beijing University of Technology, Beijing, China · Baidu Inc., Beijing, China