Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction

Yuzhe Zhang; Xianwei Xue; Xingyong Wu; Mengke Chen; Chen Liu; Xinran He; Run Shao; Feiran Liu

cs.CLApr 7, 2026

Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correction

Authors: Yuzhe Zhang, Xianwei Xue, Xingyong Wu, Mengke Chen, Chen Liu, Xinran He, Run Shao, Feiran Liu,

Organizations: Beijing University of Technology, Beijing, China · Baidu Inc., Beijing, China

Abstract

Autonomous GUI agents based on vision-language models (VLMs) often assume deterministic environment responses, generating actions without verifying whether previous operations succeeded. In real-world settings with network latency, rendering delays, and system interruptions, this assumption leads to undetected action failures, repetitive ineffective behaviors, and catastrophic error accumulation. Moreover, learning robust recovery strategies is challenging due to the high cost of online interaction and the lack of real-time feedback in offline datasets.We propose VeriGUI (Verification-driven GUI Agent), which explicitly models action outcomes and recovery under noisy environments. VeriGUI introduces a Thinking--Verification--Action--Expectation (TVAE) framework to detect failures and guide corrective reasoning, and a two-stage training pipeline that combines Robust SFT with synthetic failure trajectories and GRPO with asymmetric verification rewards. We further construct a Robustness Benchmark based on AndroidControl to evaluate failure recognition and correction. Experiments show that VeriGUI significantly reduces failure loops and improves recovery success while maintaining competitive standard task performance.

Figures & tables

Figure 1: Overview of VeriGUI’s closed-loop framework. Unlike blind action agents (above) that repeat failed actions indefinitely, VeriGUI (below) verifies each action’s outcome against predicted effects, enabling failure detection and reasoned recovery.

Figure 2: Overview of VeriGUI’s architecture and training pipeline. (A) The TVAE inference cycle implements a closed-loop verification mechanism where structured thinking precedes verification. (B) The two-stage training pipeline: Stage 1 establishes basic verification capabilities via Robust SFT on mixed success/failure trajectories; Stage 2 refines self-correction via GRPO, using offline data to simulate online feedback through composite rewards.

Category	Model	AndroidControl-High							AITW-Gen		GUI Odyssey
Category	Model	TM	GR	SR	TSR	PG	Sim-TSR	ASO $\downarrow$	PG	Sim-TSR	TM	GR	SR
Closed- source	GPT-5.1	70.1	30.0	23.1	–	–	–	–	10.4	2.5	68.1	34.0	27.9
Closed- source	Gemini-3-flash	83.3	36.9	42.2	–	–	–	–	17.8	4.5	83.2	48.8	46.9
Open-src 3B	Qwen2.5-VL-3B	60.1	52.8	31.2	0	2.8	0	$\infty$	1.5	0.1	62.0	47.4	39.9
Open-src 3B	UI-R1-3B	62.4	50.4	30.3	0	4.9	0	$\infty$	8.8	2.4	62.5	52.6	32.0
Open-src 7B/8B	Qwen2.5-VL-7B	68.5	61.7	42.8	6.2	12.8	10.6	1.15	11.7	3.0	72.3	55.8	48.5

Table 1: Offline benchmark results. VeriGUI is trained at both 3B and 7B scales. Bold indicates best among all open-source models.

\infty

ASO indicates no tasks completed.

Category	Model	LR $\downarrow$	RSR $\uparrow$
Closed-source	GPT-5.1	11.0	21.9
Closed-source	Gemini-3-flash	15.0	37.0
Open-source 3B	Qwen2.5-VL-3B	30.0	35.0
Open-source 3B	UI-R1-3B	29.5	29.0
Open-source 7B/8B	Qwen2.5-VL-7B	23.5	34.1
	OS-Atlas-7B	43.3	10.7

Table 2: Robustness Benchmark results. Bold indicates best among all open-source models; VeriGUI achieves the highest Recovery Success Rate at both scales.

Category	Model	MiniWoB++	AW
Open-source 3B	Qwen2.5-VL-3B	21.2	6.1
Open-source 3B	UI-R1-3B	33.4	10.9
Open-source 7B	Qwen2.5-VL-7B	47.0	14.9
Open-source 7B	UI-S1-7B	56.6	22.7
Ours	VeriGUI-3B	35.6	12.6
Ours	VeriGUI-7B	59.7	25.1

Table 3: Online benchmark results. Bold indicates best among all open-source models; VeriGUI-7B outperforms all open-source baselines; VeriGUI-3B surpasses same-scale models and approaches 7B-scale performance.

Figure 3: Qualitative comparison across training stages. The base model repeatedly attempts the same “open app” action without recognizing failure, entering an infinite loop. After SFT, the model learns to verify and reflect on past actions, though occasional incorrect reflections still cause short loops. After GRPO, the model correctly identifies the failure after one attempt and successfully recovers with an alternative action.

Configuration	Step-level (AndroidControl)			Task-level (AndroidControl)				Robustness
Configuration	TM	GR	SR	TSR	PG	Sim-TSR	ASO $\downarrow$	LR $\downarrow$	RSR
Base (Qwen2.5-VL-3B)	60.1	52.8	31.2	0	2.8	0	$\infty$	30.0	35.0
+ Standard SFT	67.2	56.6	42.6	6.7	16.4	10.0	3.33	30.4	29.7
+ Robust SFT	68.7	52.4	41.7	10.0	20.8	13.3	2.25	26.5	45.5
+ GRPO (VeriGUI-3B)	72.2	56.4	46.6	13.3	23.3	16.7	1.25	24.3	51.1

Table 4: Ablation on training stages (VeriGUI-3B, AndroidControl-High). Standard SFT uses only Type A (Success Trajectories) samples, while Robust SFT leverages both Type A and Type B (Failure Recovery Trajectories) samples.

Type A:B Ratio	SR	Sim-TSR	LR $\downarrow$	RSR
100:0 (no negatives)	42.6	10.0	30.4	29.7
90:10	42.1	10.0	29.6	30.3
70:30 (default)	41.7	13.3	26.5	45.5
50:50	39.2	12.4	27.1	45.4

Table 5: Effect of negative sample ratio (VeriGUI-3B, AndroidControl-High). The 70:30 balance maximizes RSR while maintaining standard accuracy.

Reward Components	SR	Sim-TSR	LR $\downarrow$	RSR
$R_{\text{act}}$ only	42.9	13.6	25.5	44.9
$R_{\text{act}}+R_{\text{ver}}$	44.2	15.1	25.4	48.1
$R_{\text{act}}+R_{\text{eff}}+R_{\text{ver}}$	46.6	16.7	24.3	51.1

Table 6: Ablation on reward components (VeriGUI-3B, AndroidControl-High). All three components contribute to the final performance.

Model	Tok/Step	Tok/Traj	Time/Step (s)	Time/Traj (s)
Qwen2.5-VL-3B	76.3	847	1.06	12.1
Qwen2.5-VL-7B	81.5	724	1.29	11.3
VeriGUI-3B	110.8	896	1.85	15.3
VeriGUI-7B	117.2	712	2.40	14.6

Table 7: Trajectory-level efficiency on AndroidControl-High (50 trajectories). Per-step token and time costs increase with TVAE, but recovery reduces total trajectory length, keeping trajectory-level overhead modest.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Click	Scroll	Input_text	Navigate_back	Open_app	Wait	Overall SR
Qwen2.5-VL-3B	27.4	68.2	18.3	35.2	1.8	32.1	31.2
UI-R1-3B	26.8	70.5	16.9	33.8	2.1	30.6	30.3
OS-Genesis-7B	32.5	74.3	21.4	38.6	1.5	35.8	35.7
OS-Atlas-7B	32.1	72.8	22.7	40.1	1.2	36.4	35.5
Qwen2.5-VL-7B	40.2	78.6	31.5	45.3	2.4	42.7	42.8
AgentCPM-GUI-8B	30.8	71.2	24.6	36.9	1.9	34.5	34.1

Appendix

Table 8: Step Accuracy (%) breakdown by action type on AndroidControl-High (VeriGUI-3B). Click actions dominate the test set (approximately 62%) and largely determine overall performance. Scroll actions exhibit uniformly high accuracy across models due to their tolerance for coordinate imprecision. The open_app action shows consistently low accuracy because most models tend to predict click instead. Bold indicates best among all open-source models at 3B scale.

Model	Click		Input_text
Model	TM	GR\|TM	TM	GR\|TM
Qwen2.5-VL-3B	78.5	34.9	28.4	64.4
UI-R1-3B	80.2	33.4	26.8	63.1
OS-Genesis-7B	82.4	39.4	32.5	65.8
OS-Atlas-7B	83.1	38.6	34.2	66.4
Qwen2.5-VL-7B	86.3	46.6	40.8	77.2

Appendix

Table 9: Type Match (TM) and Grounding accuracy given correct Type Match (GR|TM) for click and input_text actions (%) on AndroidControl-High (VeriGUI-3B). Bold indicates best among all open-source models at 3B scale.

Model	Short	Medium	Long
Sim-TSR (%)
Qwen2.5-VL-7B	21.4	8.3	2.1
UI-TARS-7B	28.6	10.2	3.4
UI-S1-7B	31.2	13.6	4.8
VeriGUI-3B	33.3	14.5	5.6
PG (%)

Appendix

Table 10: Task-level metrics stratified by trajectory length on AndroidControl-High (VeriGUI-3B). Only models achieving non-zero Sim-TSR in aggregate are included. Performance degrades substantially for longer tasks, consistent with the multiplicative nature of sequential step-level errors. Bold indicates best among included models.

Model	Type Match (%)		Step Acc. (%)
Model	First	All	First	All
Qwen2.5-VL-3B	52.4	60.1	24.8	31.2
UI-R1-3B	54.2	62.4	23.5	30.3
Qwen2.5-VL-7B	62.8	68.5	35.6	42.8
UI-TARS-7B	66.4	71.9	40.2	47.7
UI-S1-7B	64.8	70.1	37.8	44.5

Appendix

Table 11: Comparison of first-step and all-steps performance on AndroidControl-High (VeriGUI-3B). First-step metrics are consistently lower, indicating that task initialization without history context is more challenging. Bold indicates best among all open-source models at 3B scale.

Model	TM ${}_{\text{rec}}$	GR ${}_{\text{rec}}$	RSR
Qwen2.5-VL-3B	62.4	56.1	35.0
UI-R1-3B	58.6	49.5	29.0
OS-Atlas-7B	42.8	25.0	10.7
Qwen2.5-VL-7B	65.2	52.3	34.1
UI-TARS-7B	72.5	62.8	45.5
UI-S1-7B	70.8	61.6	43.6

Appendix

Table 12: Detailed robustness metrics on AndroidControl-High (VeriGUI-3B). TM

{}_{\text{rec}}

denotes type match rate for recovery actions; GR

{}_{\text{rec}}

denotes grounding accuracy for correctly-typed recovery actions. RSR requires both correct type and correct parameters. Bold indicates best among all open-source models at 3B scale.

Category	Metric	Formula	Description
Step-level [-0.3em] Metrics [0.5em] Single-step [-0.3em] predictions [-0.3em] without [-0.3em] simulation	Type Match (TM)	$\displaystyle\text{TM}=\frac{1}{\|\mathcal{D}\|}\sum_{i=1}^{\|\mathcal{D}\|}\mathbb{I}\left[\text{type}(\hat{A}_{i})=\text{type}(A_{i}^{*})\right]$	Accuracy of action type prediction (click, scroll, input, etc.).
	Grounding Rate (GR)	$\displaystyle\text{GR}=\frac{1}{\|\mathcal{D}\|}\sum_{i}\begin{cases}\mathbb{I}[\hat{c}_{i}\in\mathcal{B}_{i}]&\text{spatial}\\ \mathbb{I}[\hat{t}_{i}=t_{i}^{*}]&\text{text}\end{cases}$	Coordinate-in-bounding-box rate for spatial actions.
	Step Success Rate (SR)	$\displaystyle\text{SR}=\frac{1}{\|\mathcal{D}\|}\sum_{i=1}^{\|\mathcal{D}\|}\mathbb{I}\left[d(\hat{A}_{i},A_{i}^{*})\leq\delta\right]$	Joint correctness of type and parameters.
Task-level [-0.3em] Metrics [0.5em] End-to-end [-0.3em] performance [-0.3em] under pseudo- [-0.3em] online simulation	Task Success Rate (TSR)	$\displaystyle\text{TSR}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}\left[\forall t:\hat{A}_{i,t}=A_{i,t}^{*}\right]$	Zero-error task success rate. All steps must be correct on first attempt.
	Progress (PG)	$\displaystyle\text{PG}=\frac{1}{N}\sum_{i=1}^{N}\frac{\|\{t:\hat{A}_{i,t}=A_{i,t}^{*},\forall t^{\prime}<t\}\|}{T_{i}}$	Average task progress before first error.
	Simulated Task Success Rate (Sim-TSR)	$\displaystyle\text{Sim-TSR}=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}\left[\text{task }i\text{ done in }\leq 2T_{i}\right]$	Task success rate allowing error recovery. Key metric for self-correction capability.

Appendix

Table 13: Summary of evaluation metrics. Notation :

\mathcal{D}

= test set;

N

= number of task trajectories;

\hat{A}

= predicted action;

A^{*}

= ground-truth action;

T_{i}

= ground-truth trajectory length;

\mathcal{D}_{\text{succ}}

= successfully completed tasks;

\mathcal{D}_{\text{err}}

= error injection test set.

Parameter	Stage 1 (SFT)	Stage 2 (GRPO)
Learning rate	$1\times 10^{-5}$	$5\times 10^{-6}$
Batch size	32	16
Epochs	2	15
Max sequence length	8192	8192
Image resolution	smart_resize	smart_resize
Warmup ratio	0.03	0.1

Appendix

Table 14: Training hyperparameters for both stages. The same configuration is used for both VeriGUI-3B and VeriGUI-7B.

GUI Agents

Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.

See all 178 papers

VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
Apr 23, 2026Qijun Han, Haoqin Tu, Zijun Wang +11GUI Agents GUI Automation
UC Santa Cruz · CMU · UNC-Chapel Hill +2
Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents
May 28, 2026Tianpeng Bu, Xin Liu, Qihua Chen +7GUI Agents Long-Horizon Agent Evaluation
Alibaba Cloud Computing.
GUI-PRA: Process Reward Agent for GUI Tasks
Sep 27, 2025Tao Xiong, Xavier Hu, Yurun Chen +6Process Reward Models GUI Agents
Zhejiang University · MiLM Plus, Xiaomi Inc.

Apr 23, 2026J/K move · Enter open · S save

VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation

Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai+10

UC Santa Cruz · CMU · UNC-Chapel Hill+2

GUI Agents GUI Automation

Open paper AskPDF arXiv

Autonomous GUI agents face two fundamental challenges: early stopping, where agents prematurely declare success without verifiable evidence, and repetitive loops, where agents cycle through the same failing actions without recovery. We present VLAA-GUI, a modular GUI agentic framework built around three integrated components that guide the system on when to Stop, Recover, and Search. First, a mandatory Completeness Verifier enforces UI-observable success criteria and verification at every finish step -- with an agent-level verifier that cross-examines completion claims with decision rules, rejecting those lacking direct visual evidence. Second, a mandatory Loop Breaker provides multi-tier filtering: switching interaction mode after repeated failures, forcing strategy changes after persistent screen-state recurrence, and binding reflection signals to strategy shifts. Third, an on-demand Search Agent searches online for unfamiliar workflows by directly querying a capable LLM with search ability, returning results as plain text. We additionally integrate a Coding Agent for code-intensive actions and a Grounding Agent for precise action grounding, both invoked on demand when required. We evaluate VLAA-GUI across five top-tier backbones, including Opus 4.5, 4.6 and Gemini 3.1 Pro, on two benchmarks with Linux and Windows tasks, achieving top performance on both (77.5% on OSWorld and 61.0% on WindowsAgentArena). Notably, three of the five backbones surpass human performance (72.4%) on OSWorld in a single pass. Ablation studies show that all three proposed components consistently improve a strong backbone, while a weaker backbone benefits more from these tools when the step budget is sufficient. Further analysis also shows that the Loop Breaker nearly halves wasted steps for loop-prone models.

Abstract

GUI Agents

Explore similar work

VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation

Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents

GUI-PRA: Process Reward Agent for GUI Tasks