cs.AIAug 25, 2026
SaveReflection with Action-Induced Visual Differences for Desktop GUI Agents
Organizations: Shanghai Jiao Tong University
Abstract
The Planner-Operator-Reflector (POR) framework is widely used in GUI agents to maintain objective alignment in complex tasks through modular collaboration. However, desktop GUIs introduce a key challenge: large, dense interfaces often exhibit subtle or scattered state changes, placing most of the burden on the reflector, which must compare pre- and post-action screens, while the planner and operator reason over a single state. Existing reflectors collapse change detection and outcome verification into one step, leaving evidence implicit and yielding weakly grounded decisions. To address this limitation, we propose Evidence-First Reflection (EFR), a two-stage reflector that explicitly decouples action-induced visual differences extraction from outcome verification. EFR identifies the action location and candidate changed regions with Set-of-Marks annotations, describes and filters action-relevant changes, and makes the final judgment from the cleaned evidence. This evidence-reasoning decoupled design makes reflection better grounded in screen transitions, while reducing both visual search complexity and reasoning burden. Experiments on OSWorld-Verified and WindowsAgentArena demonstrate that EFR improves reflector accuracy by 7.11%, yielding average end-to-end task success gains of 5.94% and 4.95% on the two benchmarks, respectively.
Figures & tables
Figure 1: Two representative examples of interface changes in desktop environments. Fig. 1(a) : the action outcome is indicated by a subtle cursor movement after clicking the position of the letter “a”, making the change easy to overlook. Fig. 1(b) : switching the year triggers extensive updates across the interface, making it challenging to organize all relevant evidence to assess the action’s effect.
Figure 2: The conventional reflector performs one-step reasoning to judge action success directly, whereas EFR first describes page changes from SoM annotations as explicit evidence and then verifies the action based on the extracted evidence.
Figure 3: Overview of EFR in POR framework. Given the pre-action screenshot , post-action screenshot , grounded action description , and intended action description , EFR first annotates candidate visual differences and the action location with SoM, then filters action-relevant evidence, and finally judges action success based on the cleaned evidence.
Figure 4: A concrete example of SoM annotation. Fig. 4(a) shows the pre-action marked screenshot, where the executed action location and difference regions are annotated. Fig. 4(b) shows the post-action marked screenshot, where difference regions are annotated.
Figure 5: From total changes to clean evidence. The reflector first compares the annotated pre-action and post-action screenshots and outputs change descriptions together with relevant/noise tags. In this example, the bottom right Area 5 is filtered out as environment noise. The remaining relevant regions are then redrawn to form the clean evidence pair.
| Task Type | GUI-Owl-32B [ 36 ] | Qwen3-VL-32B [ 3 ] | Kimi-K2.5 [ 10 ] | Seed-1.8 [ 5 ] | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| POR | EFR | POR | EFR | POR | EFR | POR | EFR | |||||
| OS | 61.11 | 62.50 | +1.39 | 56.94 | 55.56 | -1.38 | 59.72 | 68.06 | +8.34 | 59.72 | 70.83 | +11.11 |
| Office | 30.16 | 33.62 | +3.46 | 23.02 | 26.72 | +3.70 | 50.97 | 61.20 | +10.23 | 47.23 | 51.25 | +4.02 |
| Daily | 50.24 | 58.02 | +7.78 | 42.03 | 48.67 | +6.64 | 57.88 | 64.50 | +6.62 | 49.00 | 58.36 | +9.36 |
| Professional | 64.63 | 80.95 | +16.32 | 66.67 | 73.47 | +6.80 | 80.95 | 78.91 | -2.04 | 62.59 | 63.27 | +0.68 |
| Workflow | 16.92 | 24.60 | +7.68 | 15.48 | 21.51 | +6.03 | 32.50 | 41.66 | +9.16 | 25.32 | 25.78 | +0.46 |
Table 1: Mean success rates (%) over three runs on the OSWorld-Verified benchmark. Rows correspond to task types. For each model backbone, the table reports the mean POR result, the mean EFR result, and their absolute difference in percentage points. The best mean accuracy in each row is bolded; ties are bolded together.
| Task Type | GUI-Owl-32B [ 36 ] | Qwen3-VL-32B [ 3 ] | Kimi-K2.5 [ 10 ] | Seed-1.8 [ 5 ] | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| POR | EFR | POR | EFR | POR | EFR | POR | EFR | |||||
| Office | 5.43 | 8.66 | +3.23 | 6.98 | 6.98 | +0.00 | 16.28 | 18.60 | +2.32 | 36.43 | 37.02 | +0.59 |
| Web Browsing | 21.33 | 32.76 | +11.43 | 30.00 | 43.22 | +13.22 | 44.11 | 56.44 | +12.33 | 47.44 | 49.78 | +2.34 |
| Windows System | 48.61 | 44.44 | -4.17 | 47.22 | 51.39 | +4.17 | 73.61 | 79.17 | +5.56 | 83.14 | 80.56 | -2.58 |
| Coding | 58.33 | 75.00 | +16.67 | 66.67 | 65.28 | -1.39 | 73.47 | 81.81 | +8.34 | 58.33 | 67.92 | +9.59 |
| Media & Video | 39.63 | 34.92 | -4.71 | 34.92 | 48.52 | +13.60 | 59.35 | 61.30 | +1.95 | 24.79 | 26.40 | +1.61 |
Table 2: Mean success rates (%) over three runs on the WindowsAgentArena benchmark. For each model backbone, denotes the mean EFR result minus the mean POR result in percentage points. The best mean accuracy in each row is bolded; ties are bolded together.
Figure 6: Per-app success-rate comparison between POR and POR-EFR on OSWorld-Verified and WindowsAgentArena. Each subplot corresponds to one backbone model.
| Setting | Overall (%) | Drop (%) |
|---|---|---|
| POR baseline (w/o both) | 37.86 | -6.97 |
| Only Two-stage Judging (w/o SoM) | 42.11 | -2.72 |
| Only SoM (w/o Two-stage Judging) | 38.13 | -6.70 |
| Diff. Identification w/ Intended Action | 43.45 | -1.38 |
| Outcome Verification w/o Screenshots | 34.25 | -10.58 |
| Full method (SoM + Two-stage Judging) | 44.83 | – |
Table 3: Ablation results of GUI-Owl-32B on OSWorld-Verified.
Figure 7: Average per-task cost comparison between POR and POR-EFR on OSWorld-Verified and WindowsAgentArena. The first row reports OSWorld-Verified and the second row reports WindowsAgentArena. From left to right, each row shows latency, token usage, and executed steps.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Task success rate vs. reflector success rate in GUI-Owl-32B trajectories on the OSWorld-Verified benchmark.
| Application | Reflector | Diff. Error | Judge. Error | Total | Rate (%) | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| POR | EFR | POR | EFR | POR | EFR | POR | EFR | POR | EFR | ||
| chrome | 3 | 0 | 2 | 0 | 1 | 0 | 26 | 19 | 11.54 | 0.00 | -11.54 |
| gimp | 0 | 0 | 0 | 0 | 0 | 0 | 11 | 5 | 0.00 | 0.00 | +0.00 |
| libreoffice_calc | 23 | 11 | 16 | 7 | 7 | 4 | 37 | 41 | 62.16 | 26.83 | -35.33 |
| libreoffice_impress | 12 | 2 | 8 | 0 | 4 | 2 | 31 | 30 | 38.71 | 6.67 | -32.04 |
| libreoffice_writer | 4 | 1 | 4 | 1 | 0 | 0 | 14 | 7 | 28.57 | 14.29 | -14.28 |
Table 4: Manual failure attribution for GUI-Owl-32B under the conventional POR framework and EFR on OSWorld-Verified, using the same model and mutually exclusive error categories. “Diff. Error” denotes page-difference identification errors, “Judge. Error” denotes success-judgment errors given correctly identified page differences, and “Total” denotes all failed trajectories. For the reflector-error rate, denotes EFR minus POR in percentage points. The lower rate in each row is bolded; ties are bolded together.
Figure 9: A one-step reflection example for clicking the “By date” button in Chrome History. The clean evidence keeps Areas 1–4 and removes Area 5, which only reflects the taskbar clock update.
Table 5: Correspondence between the original app-level partition and the merged task categories used in the main results.
| Action Type | Action Details | |
|---|---|---|
| Description | Executable Code Template | |
| click | Click at a grounded screen location. | pyautogui.click(x=x, y=y) |
| double_click | Double-click at a grounded screen location. | pyautogui.doubleClick(x=x, y=y) |
| right_click | Right-click at a grounded screen location. | pyautogui.rightClick(x=x, y=y) |
| type | Focus an input region and type text, optionally clearing the field or pressing Enter. | pyautogui.doubleClick(x=x, y=y); hotkey(‘‘ctrl’’, ‘‘a’’); press(‘‘delete’’); pyautogui.typewrite(text, interval=1.0); pyautogui.press(‘‘enter’’) |
| hotkey | Execute a keyboard shortcut combination. | pyautogui.hotkey(key1, ..., keyn) |
Table 6: Action space directly adopted from Mobile-Agent-v3 [ 36 ] . The code column shows the executable template after grounding. Shared import prefixes are omitted for brevity.
Figure 10: Average per-step cost comparison between POR and POR-EFR on OSWorld-Verified and WindowsAgentArena. The first row reports OSWorld-Verified and the second row reports WindowsAgentArena. From left to right, each row shows latency and token usage averaged over executed steps.
Figure 11: Outcome transition distributions on OSWorld-Verified and WindowsAgentArena. Each pie chart shows the proportion of trajectories that remain failed, are corrected from failure to success, or regress from success to failure after applying our method.
Figure 12: Case 1: EFR succeeds by recognizing subtle cursor and formatting evidence, while the conventional reflector repeatedly misses the small change and keeps retrying the same click.
Figure 13: Case 2: EFR rejects a target-cell update that also damages other cells, triggers recovery, and succeeds on retry; the conventional reflector accepts the same corrupted intermediate state as success.
Figure 14: Case 3: Both reflectors fail to complete the slide-editing task. EFR identifies an unsuccessful drag action, but the agent still does not roll back or re-plan correctly, and grounding errors continue to affect later steps.
Figure 15: Case 4: The conventional reflector exports the image as export.jpg , while EFR observes the duplicated suffix export.jpg.jpg but incorrectly judges it as harmless.
More in this topic
GUI Agents
Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.
See all 178 papers