The Planner-Operator-Reflector (POR) framework is widely used in GUI agents to maintain objective alignment in complex tasks through modular collaboration. However, desktop GUIs introduce a key challenge: large, dense interfaces often exhibit subtle or scattered state changes, placing most of the burden on the reflector, which must compare pre- and post-action screens, while the planner and operator reason over a single state. Existing reflectors collapse change detection and outcome verification into one step, leaving evidence implicit and yielding weakly grounded decisions. To address this limitation, we propose Evidence-First Reflection (EFR), a two-stage reflector that explicitly decouples action-induced visual differences extraction from outcome verification. EFR identifies the action location and candidate changed regions with Set-of-Marks annotations, describes and filters action-relevant changes, and makes the final judgment from the cleaned evidence. This evidence-reasoning decoupled design makes reflection better grounded in screen transitions, while reducing both visual search complexity and reasoning burden. Experiments on OSWorld-Verified and WindowsAgentArena demonstrate that EFR improves reflector accuracy by 7.11%, yielding average end-to-end task success gains of 5.94% and 4.95% on the two benchmarks, respectively.
Figures & tables
Figure 1: Two representative examples of interface changes in desktop environments. Fig. 1(a) : the action outcome is indicated by a subtle cursor movement after clicking the position of the letter “a”, making the change easy to overlook. Fig. 1(b) : switching the year triggers extensive updates across the interface, making it challenging to organize all relevant evidence to assess the action’s effect.
Figure 2: The conventional reflector performs one-step reasoning to judge action success directly, whereas EFR first describes page changes from SoM annotations as explicit evidence and then verifies the action based on the extracted evidence.
Figure 3: Overview of EFR in POR framework. Given the pre-action screenshot Spre , post-action screenshot Spost , grounded action description Aground , and intended action description Aintend , EFR first annotates candidate visual differences and the action location with SoM, then filters action-relevant evidence, and finally judges action success based on the cleaned evidence.
Figure 4: A concrete example of SoM annotation. Fig. 4(a) shows the pre-action marked screenshot, where the executed action location and difference regions are annotated. Fig. 4(b) shows the post-action marked screenshot, where difference regions are annotated.
Figure 5: From total changes to clean evidence. The reflector first compares the annotated pre-action and post-action screenshots and outputs change descriptions together with relevant/noise tags. In this example, the bottom right Area 5 is filtered out as environment noise. The remaining relevant regions are then redrawn to form the clean evidence pair.
Task Type
GUI-Owl-32B [ 36 ]
Qwen3-VL-32B [ 3 ]
Kimi-K2.5 [ 10 ]
Seed-1.8 [ 5 ]
POR
EFR
Δ
POR
EFR
Δ
POR
EFR
Δ
POR
EFR
Δ
OS
61.11
62.50
+1.39
56.94
55.56
-1.38
59.72
68.06
+8.34
59.72
70.83
+11.11
Office
30.16
33.62
+3.46
23.02
26.72
+3.70
50.97
61.20
+10.23
47.23
51.25
+4.02
Daily
50.24
58.02
+7.78
42.03
48.67
+6.64
57.88
64.50
+6.62
49.00
58.36
+9.36
Professional
64.63
80.95
+16.32
66.67
73.47
+6.80
80.95
78.91
-2.04
62.59
63.27
+0.68
Workflow
16.92
24.60
+7.68
15.48
21.51
+6.03
32.50
41.66
+9.16
25.32
25.78
+0.46
Table 1: Mean success rates (%) over three runs on the OSWorld-Verified benchmark. Rows correspond to task types. For each model backbone, the table reports the mean POR result, the mean EFR result, and their absolute difference Δ in percentage points. The best mean accuracy in each row is bolded; ties are bolded together.
Task Type
GUI-Owl-32B [ 36 ]
Qwen3-VL-32B [ 3 ]
Kimi-K2.5 [ 10 ]
Seed-1.8 [ 5 ]
POR
EFR
Δ
POR
EFR
Δ
POR
EFR
Δ
POR
EFR
Δ
Office
5.43
8.66
+3.23
6.98
6.98
+0.00
16.28
18.60
+2.32
36.43
37.02
+0.59
Web Browsing
21.33
32.76
+11.43
30.00
43.22
+13.22
44.11
56.44
+12.33
47.44
49.78
+2.34
Windows System
48.61
44.44
-4.17
47.22
51.39
+4.17
73.61
79.17
+5.56
83.14
80.56
-2.58
Coding
58.33
75.00
+16.67
66.67
65.28
-1.39
73.47
81.81
+8.34
58.33
67.92
+9.59
Media & Video
39.63
34.92
-4.71
34.92
48.52
+13.60
59.35
61.30
+1.95
24.79
26.40
+1.61
Table 2: Mean success rates (%) over three runs on the WindowsAgentArena benchmark. For each model backbone, Δ denotes the mean EFR result minus the mean POR result in percentage points. The best mean accuracy in each row is bolded; ties are bolded together.
Figure 6: Per-app success-rate comparison between POR and POR-EFR on OSWorld-Verified and WindowsAgentArena. Each subplot corresponds to one backbone model.
Setting
Overall (%)
Drop (%)
POR baseline (w/o both)
37.86
-6.97
Only Two-stage Judging (w/o SoM)
42.11
-2.72
Only SoM (w/o Two-stage Judging)
38.13
-6.70
Diff. Identification w/ Intended Action
43.45
-1.38
Outcome Verification w/o Screenshots
34.25
-10.58
Full method (SoM + Two-stage Judging)
44.83
–
Table 3: Ablation results of GUI-Owl-32B on OSWorld-Verified.
Figure 7: Average per-task cost comparison between POR and POR-EFR on OSWorld-Verified and WindowsAgentArena. The first row reports OSWorld-Verified and the second row reports WindowsAgentArena. From left to right, each row shows latency, token usage, and executed steps.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Task success rate vs. reflector success rate in GUI-Owl-32B trajectories on the OSWorld-Verified benchmark.
Application
Reflector
Diff. Error
Judge. Error
Total
Rate (%)
POR
EFR
POR
EFR
POR
EFR
POR
EFR
POR
EFR
Δ
chrome
3
0
2
0
1
0
26
19
11.54
0.00
-11.54
gimp
0
0
0
0
0
0
11
5
0.00
0.00
+0.00
libreoffice_calc
23
11
16
7
7
4
37
41
62.16
26.83
-35.33
libreoffice_impress
12
2
8
0
4
2
31
30
38.71
6.67
-32.04
libreoffice_writer
4
1
4
1
0
0
14
7
28.57
14.29
-14.28
Appendix
Table 4: Manual failure attribution for GUI-Owl-32B under the conventional POR framework and EFR on OSWorld-Verified, using the same model and mutually exclusive error categories. “Diff. Error” denotes page-difference identification errors, “Judge. Error” denotes success-judgment errors given correctly identified page differences, and “Total” denotes all failed trajectories. For the reflector-error rate, Δ denotes EFR minus POR in percentage points. The lower rate in each row is bolded; ties are bolded together.
Figure 9: A one-step reflection example for clicking the “By date” button in Chrome History. The clean evidence keeps Areas 1–4 and removes Area 5, which only reflects the taskbar clock update.
Table 5: Correspondence between the original app-level partition and the merged task categories used in the main results.
Action Type
Action Details
Description
Executable Code Template
click
Click at a grounded screen location.
pyautogui.click(x=x, y=y)
double_click
Double-click at a grounded screen location.
pyautogui.doubleClick(x=x, y=y)
right_click
Right-click at a grounded screen location.
pyautogui.rightClick(x=x, y=y)
type
Focus an input region and type text, optionally clearing the field or pressing Enter.
Table 6: Action space directly adopted from Mobile-Agent-v3 [ 36 ] . The code column shows the executable template after grounding. Shared import prefixes are omitted for brevity.
Figure 10: Average per-step cost comparison between POR and POR-EFR on OSWorld-Verified and WindowsAgentArena. The first row reports OSWorld-Verified and the second row reports WindowsAgentArena. From left to right, each row shows latency and token usage averaged over executed steps.
Figure 11: Outcome transition distributions on OSWorld-Verified and WindowsAgentArena. Each pie chart shows the proportion of trajectories that remain failed, are corrected from failure to success, or regress from success to failure after applying our method.
Figure 12: Case 1: EFR succeeds by recognizing subtle cursor and formatting evidence, while the conventional reflector repeatedly misses the small change and keeps retrying the same click.
Figure 13: Case 2: EFR rejects a target-cell update that also damages other cells, triggers recovery, and succeeds on retry; the conventional reflector accepts the same corrupted intermediate state as success.
Figure 14: Case 3: Both reflectors fail to complete the slide-editing task. EFR identifies an unsuccessful drag action, but the agent still does not roll back or re-plan correctly, and grounding errors continue to affect later steps.
Figure 15: Case 4: The conventional reflector exports the image as export.jpg , while EFR observes the duplicated suffix export.jpg.jpg but incorrectly judges it as harmless.
Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution. Existing approaches rely on open-ended multimodal reasoning after each action, which is costly and poorly matched to the structured nature of GUI state transitions. We propose StepReflect, which formulates per-step GUI reflection as supervised structured prediction conditioned on explicit transition specifications and paired visual evidence. StepReflect is trained through a staged pipeline combining supervised fine-tuning, teacher-student distillation, and preference- and reward-based refinement. Offline, the resulting 8B model achieves 82.16% transition-level accuracy on AndroidWorld, exceeding zero-shot GPT-5.2 by 11.83 percentage points under the same structured input. Online, across M3A, Agent-SAMA, MAI-UI-8B, and Seed-2.0-Pro, StepReflect achieves higher task success in three of four agent configurations and remains within one successful task of the GPT-5.2 Reflection Agent in the fourth. It also reduces paid API charges relative to GPT-based reflection in all four configurations. These results establish StepReflect as a practical, locally deployable alternative to repeated frontier-model reflection for long-horizon mobile GUI agents.
Linqiang Guo, Wei Liu, Li Gu +3
Peter · Concordia University Montreal, Quebec, Canada
Recent graphical user interface (GUI) grounders have significantly advanced single-shot accuracy on standard benchmarks, yet their performance degrades sharply on small targets, densely packed controls and out-of-distribution interfaces. We attribute this gap to a paradigmatic limitation shared by existing approaches: none of them treats a produced coordinate as a hypothesis to be reflected upon and revised under new visual evidence. This manifests as three coupled issues: 1) Lack of post-hoc reflection. The prediction is frozen at the moment of emission, leaving no internal mechanism to challenge or refine it. 2) Visual evidence decoupled from the prediction. The auxiliary visual evidence is gathered to support the upcoming coordinate rather than to scrutinise the one already committed to. 3) Refinement over views, not over predictions. The iterative zoom-in refines the inspected region instead of inheriting a previous coordinate as a spatial prior to be corrected. In this paper, we propose LookAgain, a closed-loop GUI grounder driven by post-prediction visual reflection. LookAgain reformulates grounding as a multi-turn predict-look-again-refine process with two primitives: "locate" posts a coordinate hypothesis, renders a marker on the image and appends a local patch of the predicted region. It anchors the next reasoning step to the previous prediction as a spatial prior; "confirm" accepts or reject the hypothesis and terminates the procedure. We train the LookAgain grounder with SFT on constructed reflective trajectories as a cold start, followed by GRPO with terminal grounding correctness as the sole reward. Extensive experiments show that LookAgain consistently improves performance on both refusal-aware and general GUI grounding benchmarks, achieving state-of-the-art results. Comprehensive ablations further verify the effectiveness of the proposed framework.
Renshan Zhang, Haoyang Meng, Yixiao He +3
Harbin Institute of Technology, Shenzhen · Beijing University of Posts and Telecommunications · Shenzhen Loop Area Institute +1
Long-horizon GUI automation remains challenging due to error accumulation over extended interaction sequences. Process Reward Models (PRMs) provide dense step-level supervision for mitigating error accumulation, yet standard PRMs are poorly suited to GUI verification. Standard PRM judgments often rely on superficial visual alignment rather than functional correctness, reflecting an evaluative knowledge gap caused by missing domain-specific adjudication logic. Standard PRMs also perform passive, single-pass visual assessment, which creates Visual Ambiguity when reliable judgment requires actively locating, parsing, or inspecting task-relevant UI evidence. We introduce GUI-PRA, a Process Reward Agent that transforms GUI process evaluation from passive scoring into active investigation. GUI-PRA couples Experience-Injected Criterion Synthesis, which distills generalized verification principles into state-specific criteria, with Criterion-Guided Autoregressive Perception, which uses these criteria to navigate multi-granularity visual tools and gather grounded evidence. On AndroidWorld and Mobile-MiniWoB++, GUI-PRA achieves improvements of 5.0% and 6.5% over standard PRMs on the Qwen-VL series, with Qwen3-VL attaining 54.74% success rate on AndroidWorld. On the offline OS-Critic Bench, GUI-PRA demonstrates strong competitiveness against fully trained critic models.