Graphical user interface (GUI) agents powered by vision language models (VLMs) are rapidly moving from passive assistance to autonomous operation. However, this unrestricted action space exposes users to severe and irreversible financial, privacy or social harm. Existing safeguards rely on prompt engineering, brittle heuristics and VLM-as-critic lack formal verification and user-tunable guarantees. We propose CORA (COnformal Risk-controlled GUI Agent), a post-policy, pre-action safeguarding framework that provides statistical guarantees on harmful executed actions. CORA reformulates safety as selective action execution: we train a Guardian model to estimate action-conditional risk for each proposed step. Rather than thresholding raw scores, we leverage Conformal Risk Control to calibrate an execute/abstain boundary that satisfies a user-specified risk budget and route rejected actions to a trainable Diagnostician model, which performs multimodal reasoning over rejected actions to recommend interventions (e.g., confirm, reflect, or abort) to minimize user burden. A Goal-Lock mechanism anchors assessment to a clarified, frozen user intent to resist visual injection attacks. To rigorously evaluate this paradigm, we introduce Phone-Harm, a new benchmark of mobile safety violations with step-level harm labels under real-world settings. Experiments on Phone-Harm and public benchmarks against diverse baselines validate that CORA improves the safety--helpfulness--interruption Pareto frontier, offering a practical, statistically grounded safety paradigm for autonomous GUI execution. Code and benchmark are available at cora-agent.github.io.
Figures & tables
Figure 1: Overview of our proposed CORA framework . CORA is a safety shield for mobile GUI agents that turns open-ended action proposal into selective execution. Given a locked, clarified user intent, the base agent proposes a low-level GUI action (tap/type/swipe) from the current screenshot and UI tree. An action-conditional Guardian scores the risk of executing the specific proposed action in the current screen under the locked intent. Conformal risk calibration maps these scores to a user-tunable execute/abstain threshold, enabling a principled safety–autonomy trade-off without modifying the base policy. Low-risk actions are executed automatically, while high-risk actions are rejected and routed to a Diagnostician that produces an interpretable risk type, a UI-grounded rationale, and a minimal intervention (Reflect/Abort/Ask to Confirm).
Phone-Harm (Harm-150)
MobileRisk
Method
HR ↓
GAR ↑
IF1 ↑
mHR ↓
OIR ↓
Acc ↑
Prec ↑
Rec ↑
F1 ↑
FPR ↓
GPT-5 (Base)
0.40
37.50
4.26
0.34
0.80
68.92
9.52
33.33
14.81
27.94
Gemini-3 (Base)
0.84
43.33
10.53
0.58
7.69
78.69
10.53
18.18
13.33
15.32
UI-TARS-1.5 (Base)
5.73
54.55
57.14
11.82
7.69
50.00
50.00
6.86
12.07
6.86
AutoGLM-VLM (Base)
3.58
44.44
39.56
6.42
1.58
58.82
62.50
44.12
51.72
26.47
AutoGLM + Prompt
4.30
45.45
42.55
7.51
3.85
74.86
13.33
17.39
15.09
16.67
Table 1: Comparison of different methods on the Harm-150 subset of our proposed Phone-Harm benchmark and the public MobileRisk benchmark introduced in OS-Sentinel ( Sun et al., 2025a ) . All values are reported in percentages (%), and the desired direction of each metric is indicated by ↑ or ↓ .
Figure 2: Safety–Helpfulness–Interruption Pareto frontier on Phone-Harm. Methods are plotted in the HR–GAR plane; marker size encodes the Over-Intervention Rate, and the ideal operating point lies in the top-left corner with a small marker. (a) Harm-150 subset, covering all eight methods. (b) Full benchmark under the Harm-150 + Normal-150 mixture defined in Table 2 , restricted to the five methods with runs on both subsets. The legend in panel (a) applies to both subfigures. Quantitative discussion is given in Section 5.2 .
Method
HR ↓
GAR ↑
IF1 ↑
mHR ↓
OIR ↓
GPT-5 (Base)
0.20
68.75
4.26
0.29
0.40
Gemini-3 (Base)
0.42
71.67
10.53
0.59
3.85
UI-TARS-1.5 (Base)
2.87
77.28
57.14
3.71
3.85
AutoGLM-VLM (Base)
1.79
72.22
39.56
2.48
0.79
CORA (Ours)
2.19
89.69
85.29
2.44
9.83
Table 2: Unified evaluation on the merged Phone-Harm + Normal-150 split (300 tasks). Values are in %; arrows indicate the preferred direction.
Setting
Succ. (x/20)
Succ. Rate
Ret. vs GPT-5
GPT-5 (Base)
11/20
55.0%
100.0%
UI-TARS-1.5 (Base, avg of 3 runs)
4/20
20.0%
36.4%
AutoGLM (Prev setting)
6/20
30.0%
54.5%
CORA (Ours)
8/20
40.0%
72.7%
Table 3: Capability retention under safety control on AndroidWorld core20 safe/general tasks. Values are reported as success count, success rate, and retention relative to GPT-5.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Harm-150 distribution overview. (a) App distribution over the full Harm-150 subset, highlighting a long-tail coverage. (b–d) Sub-category distributions for Misuse , Misbehavior , and Injection , respectively, demonstrating concentrated risk modes within each harm type.
Figure 4: Qualitative cases from Harm-150 demonstrating CORA’s safety behaviour: prompt-injection, misbehavior mitigation, and deliberate misuse prevention.
Table 7: Runtime and latency analysis of CORA compared to the base policy (AutoGLM-Phone-9B). All measurements are conducted on a single node with 8 NVIDIA A100 (80GB) GPUs.
Mobile graphical user interface (GUI) agents have demonstrated remarkable capabilities in automating complex tasks, yet they introduce critical safety risks because a single erroneous action can lead to irreversible consequences. Existing safety mechanisms are primarily reactive, lacking the ability to assess risks before execution. In this paper, we introduce SeerGuard, a consequence-aware safety framework designed to mitigate these risks through pre-execution instruction-level screening and action-level risk assessment. Specifically, the action-level assessment analyzes agent-proposed actions within current GUI states, anticipating likely outcomes to identify risks before they are executed. To enable these capabilities, we construct a unified safety-augmented world model (SAWM) via multi-task learning, integrating semantic next-state prediction with safety risk assessment. Extensive experiments demonstrate that SeerGuard generalizes effectively across diverse mobile GUI agents. On Qwen3-VL-8B-Instruct, it increases the safety-utility score from 0.191 to 0.596 at ω=0.8 and reduces the risk-cost score from 0.347 to 0.135 at α=0.8. Further analyses on our SAWM validate the effectiveness of the instruction-level screening, alongside the capability of action risk assessment and next-state prediction.
Autonomous GUI agents based on vision-language models (VLMs) often assume deterministic environment responses, generating actions without verifying whether previous operations succeeded. In real-world settings with network latency, rendering delays, and system interruptions, this assumption leads to undetected action failures, repetitive ineffective behaviors, and catastrophic error accumulation. Moreover, learning robust recovery strategies is challenging due to the high cost of online interaction and the lack of real-time feedback in offline datasets.We propose VeriGUI (Verification-driven GUI Agent), which explicitly models action outcomes and recovery under noisy environments. VeriGUI introduces a Thinking--Verification--Action--Expectation (TVAE) framework to detect failures and guide corrective reasoning, and a two-stage training pipeline that combines Robust SFT with synthetic failure trajectories and GRPO with asymmetric verification rewards. We further construct a Robustness Benchmark based on AndroidControl to evaluate failure recognition and correction. Experiments show that VeriGUI significantly reduces failure loops and improves recovery success while maintaining competitive standard task performance.
Yuzhe Zhang, Xianwei Xue, Xingyong Wu +8
Beijing University of Technology, Beijing, China · Baidu Inc., Beijing, China
Multimodal GUI agents have emerged as a promising paradigm for digital task automation, yet transitioning from benchmark-oriented models to dependable real-world applications remains challenging due to limited environment coverage, brittle task construction, and unreliable reward verification. In this work, we present UI-Venus-2, a general-purpose foundation GUI agent designed to operate across mobile, web, and desktop environments through a unified closed-loop reasoning-action framework. To bridge the gap toward practical deployment, we jointly scale three critical dimensions: (1) Environments, expanding coverage to more than 170 multilingual mobile apps and native desktop operating systems; (2) Tasks, employing a deep-research pipeline for function-grounded instruction generation; and (3) Verification, adopting trace-level and sample-level evaluators with visual keypoints and multi-model voting to ensure reliable RL signals for training. Furthermore, we integrate safety-aware mechanisms to ensure controlled execution of consequential actions. By offering a capable, efficient, and open-source foundation, UI-Venus-2 advances the field toward more generalizable, verifiable, and self-reflective agents for real-world applications.