cs.AIMar 19, 2026
SaveOS-Themis: A Scalable Critic Framework for Generalist GUI Rewards
Organizations: University of Science and Technology of China · Shanghai AI Laboratory · CUHK MMLab · The Hong Kong University of Science and Technology ·
Abstract
Reinforcement Learning (RL) has the potential to improve the robustness of GUI agents in stochastic environments, yet training is highly sensitive to the quality of the reward function. Existing reward approaches struggle to achieve both scalability and performance. To address this, we propose OS-Themis, a scalable and accurate multi-agent critic framework. Unlike a single judge, OS-Themis decomposes trajectories into verifiable milestones to isolate critical evidence for decision making and employs a review mechanism to strictly audit the evidence chain before making the final verdict. To facilitate evaluation, we further introduce OmniGUIRewardBench (OGRBench), a holistic cross-platform benchmark for GUI outcome rewards, where all evaluated models achieve their best performance under OS-Themis. Extensive experiments on AndroidWorld show that OS-Themis yields a 10.3% improvement when used to support online RL training, and a 6.9% gain when used for trajectory validation and filtering in the self-training loop, highlighting its potential to drive agent evolution.
Figures & tables
Figure 1: Limitations of existing approaches for reward modeling in GUI environments.
Figure 2: Overview of OS-Themis Framework. The framework primarily consists of two modules: the Milestone Verification Module (MVM) and the Verdict Calibration Module (VCM) . For a GUI trajectory containing several <screenshot,think,action> steps, the Selector Agent in the MVM extracts key steps as milestones, which are then assigned binary scores by the Verifier Agent . Subsequently, the Reviewer Agent in the VCM continuously interacts with the MVM to ensure the rationality and completeness of the milestones, while the Judge Agent conducts the final scoring of the trajectory based on all information exchanged between the modules.
| Model | Ubuntu | Mobile | Windows | macOS | Web | Overall | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | F1 | Acc | Prec | Rec | F1 | |
| DigiRL | ||||||||||||||
| Qwen3-VL-8B | 62.1 | 64.9 | 69.2 | 76.6 | 66.2 | 63.6 | 72.7 | 40.0 | 63.2 | 69.6 | 64.4 | 62.2 | 72.1 | 66.8 |
| Qwen3-VL-30B-A3B | 61.0 | 67.6 | 67.0 | 75.6 | 69.0 | 71.1 | 67.5 | 49.0 | 54.2 | 63.0 | 62.5 | 58.9 | 80.6 | 68.1 |
| Qwen3-VL-32B | 65.7 | 66.5 | 72.3 | 77.8 | 71.8 | 71.2 | 75.3 | 55.8 | 62.1 | 67.9 | 67.6 | 65.8 | 72.1 | 68.8 |
| Qwen3-VL-235B | 65.5 | 67.0 | 70.2 | 75.2 | 72.3 | 70.9 | 76.6 | 43.8 | 63.2 | 62.8 | 67.4 | 66.6 | 69.0 | 67.8 |
Table 1: Performance comparison of different models under the DigiRL, ZeroGUI, and OS-Themis frameworks on OGRBench. Each framework reports Accuracy (Acc) and F1-score (F1), while the Overall performance includes Acc, Precision (Prec), Recall (Rec), and F1. Higher values indicate better performance. Bold and underlined denote the best and second-best results, respectively.
| Backbone | Reward used for RL training | Acc |
|---|---|---|
| Qwen3-VL-4B | – | 45.3 |
| SEAgent | 47.8 | |
| ZeroGUI (Qwen3-VL-235B) | 46.1 | |
| OS-Themis (Qwen3-VL-8B) | 50.9 | |
| OS-Themis (Qwen3-VL-235B) | 51.3 | |
| Qwen3-VL-8B | – | 47.6 |
Table 2: RL performance on AndroidWorld with different reward sources. Each row is trained independently from the same initialization for the given backbone.
Figure 3: The performance of Qwen3-VL-4B under online RL scaling with OS-Themis, including mean reward growth and corresponding AndroidWorld accuracy across different training scales.
Figure 4: The performance of Qwen3-VL-4B and Qwen3-VL-8B on AndroidWorld after SFT using filtered data (the parentheses indicate the filtering method; All Data means the data is unfiltered).
| Variant | Agents | Overall | |||||
|---|---|---|---|---|---|---|---|
| Selector | Verifier | Reviewer | Judge | Acc | Prec | Rec | |
| OS-Themis (Full) | ✓ | ✓ | ✓ | ✓ | 88.0 | 92.8 | 82.3 |
| OS-Themis w/o Selector | ✗ | ✓ | ✓ | ✓ | 83.3 | 79.7 | 88.9 |
| OS-Themis w/o Verifier | ✓ | ✗ | ✓ | ✓ | 81.9 | 77.2 | 90.1 |
| OS-Themis w/o Reviewer | ✓ | ✓ | ✗ | ✓ | 86.9 | 85.7 | 88.4 |
| OS-Themis w/o Judge | ✓ | ✓ | ✓ | ✗ | 52.5 | 89.7 | 5.0 |
Table 3: Ablation results of OS-Themis by removing individual agents. Metrics include Overall Acc, Prec, and Rec (%). Bold indicates the best result.
| Framework | Overall | ||
|---|---|---|---|
| Acc | Prec | Rec | |
| OS-Themis w/o Reviewer | 86.9 | 85.7 | 88.4 |
| OS-Themis w/ Reviewer (Advisor) | 87.8 | 88.7 | 86.4 |
| OS-Themis w/ Reviewer (Critic) | 88.0 | 92.8 | 82.3 |
Table 4: Ablation study on the Reviewer Agent within the OS-Themis framework, comparing versions without a Reviewer and with Reviewers in different roles (Advisor and Critic). Metrics include Overall Acc, Prec, and Rec (%). Bold indicates the best result.
| Framework | Overall | ||
|---|---|---|---|
| Acc | Prec | Rec | |
| OS-Themis w/o Assignment Goal | 86.9 | 84.6 | 90.0 |
| OS-Themis w/ Assignment Goal | 88.0 | 92.8 | 82.3 |
Table 5: Ablation study on the Assignment Goal within the OS-Themis framework reporting Overall Acc, Prec, and Rec (%). Bold indicates the best result.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Ubuntu | Mobile | Windows | macOS | WebA | Overall | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Acc | Prec | Rec | Acc | Prec | Rec | Acc | Prec | Rec | Acc | Prec | Rec | Acc | Prec | Rec | Acc | Prec | Rec | |
| DigiRL | ||||||||||||||||||
| Qwen3-VL-8B | 62.1 | 63.7 | 66.2 | 69.2 | 63.3 | 96.9 | 66.2 | 60.6 | 67.0 | 72.7 | 36.8 | 43.8 | 63.2 | 61.1 | 80.8 | 64.4 | 62.2 | 72.1 |
| Qwen3-VL-30B-A3B | 61.0 | 60.4 | 76.6 | 67.0 | 61.5 | 98.0 | 69.0 | 60.5 | 86.2 | 67.5 | 36.4 | 75.0 | 54.2 | 54.4 | 74.8 | 62.5 | 58.9 | 80.6 |
| Qwen3-VL-32B | 65.7 | 69.0 | 64.1 | 72.3 | 66.9 | 92.9 | 71.8 | 64.9 | 78.7 | 75.3 | 44.4 | 75.0 | 62.1 | 60.8 | 76.8 | 67.6 | 65.8 | 72.1 |
| Qwen3-VL-235B | 65.5 | 67.9 | 66.2 | 70.2 | 66.4 | 86.7 | 72.3 | 66.1 | 76.6 | 76.6 | 43.8 | 43.8 | 63.2 | 66.3 | 59.6 | 67.4 | 66.6 | 69.0 |
Table 6: Comparison of Accuracy (Acc), Precision (Prec), and Recall (Rec) across various platforms in OmniGUIRewardBench, evaluated using DigiRL, ZeroGUI, OS-Themis, and other GUI critic models (e.g., SEAgent).
Figure 5: Data Distribution of OGRBench
| Category | Method | Overall | |
|---|---|---|---|
| Prec | Recall | ||
| Official | Rule-based | 83.8 | 55.9 |
| Critic Model | SEAgent | 71.6 | - |
| LLM-as-a-Judge | Claude-3.7-Sonnet | 64.3 | 89.8 |
| GPT-4o | 68.1 | 80.3 | |
| GPT-4o Mini | 64.5 | 78.3 | |
Table 7: Performance of different models on AgentRewardBench within the OS-Themis framework.
| Benchmark | Platform | #Positive | #Negative |
|---|---|---|---|
| OSWorld | Ubuntu | 393 | 348 |
| AndroidWorld | Android | 98 | 90 |
| WindowsAgentArena | Windows | 94 | 119 |
| MacOSArena | MacOS | 16 | 61 |
| WebArena | Web | 99 | 91 |
| Total | 700 | 709 | |
Table 8: Statistical Overview of the OmniGUIRewardBench Data.
| Benchmark | Platform | Samples | ||||
|---|---|---|---|---|---|---|
| Ubuntu | Android | Windows | macOS | Web | ||
| AgentRewardBench | ✗ | ✗ | ✗ | ✗ | ✓ | 1106 |
| CUARewardBench | ✓ | ✗ | ✗ | ✗ | ✗ | 272 |
| OmniGUIRewardBench | ✓ | ✓ | ✓ | ✓ | ✓ | 1409 |
Table 9: Statistics of Existing GUI ORM Benchmarks
Figure 6: Evolution of validation mean reward for Qwen3-VL models trained via the OS-Themis framework with different base models.
| Variant | Overall (%) | |||
|---|---|---|---|---|
| Acc | Prec | Rec | F1 | |
| Base (all 8B) | 79.4 | 86.3 | 69.4 | 77.0 |
| Selector | 80.6 | 86.1 | 72.7 | 78.9 |
| Reviewer | 79.3 | 89.1 | 66.4 | 76.1 |
| Judge | 82.4 | 84.5 | 79.1 | 81.7 |
| Verifier | 82.5 | 88.9 | 74.1 | 80.8 |
Table 10: Single-agent scaling study in OS-Themis. “ ” upgrades the corresponding agent to Qwen3-VL-235B; all others remain Qwen3-VL-8B. Bold : best.
Figure 7: Scaling performance comparison of Qwen3-VL-235B under the OS-Themis framework on OGRBench using three voting strategies: Majority, All, and Any.
Figure 8: The performance of Qwen3-VL-4B and Qwen3-VL-8B on AndroidWorld after Test-Time Scaling (TTS).
Figure 9: Distribution of Milestone Percentage
Figure 10: Illustration of trajectory steps selected for correctness evaluation under different methods. Colored points represent selected steps.
| Statistic | Value | Statistic | Value |
|---|---|---|---|
| Milestone Percentage | |||
| Task-Level Mean | 58.47% | Step-Level Overall | 35.57% |
| Median | 66.67% | Std Deviation | 30.83% |
| Milestone Count per Task | |||
| Mean | 7.04 | Median | 6.00 |
| Std Deviation | 4.16 | – | – |
Table 11: Comprehensive milestone statistics under Qwen3-VL-235B. Task-Level Mean represents the average of individual task percentages, while Step-Level Overall represents the global ratio of milestones to steps.
| Metric | Value |
|---|---|
| Latency | 117.6 s |
| Completion tokens | 6416.8 |
| Prompt tokens | 164624.0 |
| Calls | 14.1 |
Table 12: Per-trajectory average latency and token/call statistics on OmniGUIRewardBench.
Figure 11: Timeline of parallel trajectory rollout across environment workers. Each worker samples a trajectory in two rollout groups (green and blue). After a rollout finishes, the trajectory is evaluated (hatched, 117.6 s on average) and this evaluation is decoupled from the online environments, allowing it to overlap with ongoing sampling in other workers; thus the evaluation introduces negligible blocking to subsequent rollouts.
Figure 12: Execution trajectory of the GUI Agent for the task of editing a file in Markor.
Figure 13: The Selector Agent and Verifier Agent within the Milestone Verification Module identify milestones and mine trajectory information, while the lower part presents a counter-example of a direct judgment made through the module.
Figure 14: The Reviewer Agent within the Verdict Calibration Module correctly identifies potential issues in the interaction information and feeds them back to the Milestone Verification Module. After verification, the Judge Agent correctly determines that the trajectory failed to complete the task.
| Selector Agent Prompts |
|---|
| Part 1: Initial System Prompt for the Selector Agent |
| You are Selector Model — Initial Key-Step Selector for GUI task verification. Goal: |
| - From the agent’s historical trace, identify a nuanced, sufficiently rich set of “key steps” whose success or failure determines whether the task is complete. |
| - Err on the side of high coverage: prefer a longer, fine-grained list that spans every necessary sub-step rather than a minimal shortlist. |
| - For every chosen step you MUST craft a concrete assessment_goal that tells the verifier the exact observable outcome to look for (e.g., “After this step, the note titled ‘Trip’ is deleted from the list”). |
Table 13: Selector Agent Prompts (System & User).
| Verifier Agent Prompts |
|---|
| Part 1: System Prompt for the Verifier Agent |
| You are Verifier Model — a vision-first verifier of a single GUI step per message. Mission: |
| - For each message, judge whether the selector’s assessment_goal for this step was achieved by comparing BEFORE vs AFTER screenshots in light of the step’s purpose. |
| - For termination actions ( ACTION=‘‘terminate’’ ), you may only have a single final screenshot; determine whether the assessment_goal and overall completion are satisfied from that one image. |
| - Ensure the step advances the overall TASK_GOAL ; a visually successful action that contradicts or harms the goal counts as failure. |
Table 14: Verifier Agent Prompts (System & User).
| Reviewer Agent Prompts |
|---|
| Part 1: System Prompt for the Reviewer Agent |
| You are Reviewer Model — a per-step risk auditor for GUI task execution. Mission: |
| - Scan each step for concrete, verifiable risks that could break the task goal. |
| - Check for missing required actions (e.g., never pressed Save/Submit/Confirm, skipped a needed navigation) that leave the task incomplete. |
| - Check for redundant or excessive actions (e.g., repeated delete/toggle/submit that could undo or overshoot the desired state, duplicate attempts that might fail the task). |
Table 15: Reviewer Agent Prompts (System & User).
| Judge Agent Prompts |
|---|
| Part 1: System Prompt for the Judge Agent |
| You are Judge Model — Final Task Completeness Judge. Goal: |
| - Decide whether the GUI task is completed, using: (1) the task goal and (2) verified results from the verifier Model as packaged by the Selector Model. |
| - Treat the task as completed when, and only when, the goal-focused evidence (preferably verified, but also including a clear, contradiction-free trajectory) allows you to be confident that all required outcomes of the goal have been achieved; do not require redundant checks of non-critical steps once the end state is proven. |
| - Anchor every judgment to the task goal: missing or unverified minor intermediate steps should not block a “completed” decision if the final result demonstrably satisfies the goal. |
Table 16: Judge Agent Prompts (System & User).
More in this topic
Reward Modeling
Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.
See all 325 papers