Adaptive-Group Process Rewards for GUI Agents

Zirui Shang; Yang Liu; Xin Shu; Zhi Gao; Xinxiao Wu; Lifeng Fan

cs.AIOct 13, 2026

Adaptive-Group Process Rewards for GUI Agents

Authors: Zirui Shang, Yang Liu, Xin Shu, Zhi Gao, Xinxiao Wu, Lifeng Fan

Organizations: Beijing Key Laboratory of Intelligent Information Technology, School of Computer Science & Technology, Beijing Institute of Technology · State Key Laboratory of General Artificial Intelligence, BIGAI · Wuhan University · Guangdong Laboratory of Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University

Abstract

Training graphical user interface (GUI) agents with reinforcement learning suffers from sparse outcome rewards, leading to inefficient learning. Process rewards offer a promising direction for credit assignment to intermediate actions; however, accurately evaluating an action requires identifying evidence across long-horizon GUI trajectories, as its value may depend on information beyond its immediate state and observation. In this paper, we propose an adaptive-group process reward framework that dynamically partitions a GUI trajectory into semantically coherent subgoal groups, enabling process reward estimation at both the action and group levels. At the action level, a critic evaluates each action based on the visual changes and subsequent actions within the group for action correctness and contribution to the subgoal. At the group level, the framework aggregates action-level evaluations to assess subgoal completion and contribution to the overall task, and further refines these assessments by leveraging the context of all groups in the trajectory. The process reward of each action is obtained by combining its action-level evaluation with the refined group-level evaluation. We further augment GRPO's outcome advantage with the proposed process rewards to provide finer-grained guidance for GUI agent training. Experiments on process reward benchmarks and downstream RL training demonstrate the effectiveness of the proposed framework.

Figures & tables

Figure 1: Fixed-window step rewards, milestone rewards, and adaptive-group process rewards on an executed GUI trajectory. (a) A fixed local window can include irrelevant context and miss later evidence. (b) Predefined milestones can leave a valid executed path uncredited. (c) Adaptive groups follow the executed subgoals and score actions within each group.

Figure 2: Overview of the proposed framework. (a) A grouping critic partitions the trajectory from the instruction and responses into subgoal groups. (b) Two-level scoring produces an action-level score and a refined group-level score, which are used to generate the process rewards. (c) The process rewards shape the GRPO advantage.

Reward model	P	NPV	R	S	OA
Qwen2.5-VL-7B	$54.4$	$49.4$	$53.8$	$50.0$	$52.0$
Qwen2.5-VL-32B	$60.3$	$64.8$	$79.8$	$41.5$	$61.7$
Qwen2.5-VL-72B	$58.5$	$64.8$	$83.0$	$34.8$	$60.1$
GLM-4.5V-106B	$64.0$	$78.5$	$89.0$	$44.5$	$67.9$
Qwen3-VL-32B-T	$68.3$	$77.3$	$85.2$	$56.1$	$71.4$
Qwen3-VL-235B-T	$66.5$	$75.0$	$84.1$	$53.0$	$69.4$

Table 1: Process reward evaluation on CUARewardBench, compared with models under the OpenCUA-Reflector template. P, NPV, R, S, and OA denote precision, negative predictive value, recall, specificity, and overall accuracy. Best results are in bold; second-best are underlined.

Reward model	P	NPV	R	S	OA
Qwen2.5-VL-7B	$56.7$	$54.2$	$67.0$	$43.3$	$55.8$
Qwen2.5-VL-32B	$65.3$	$62.4$	$67.8$	$59.8$	$64.0$
Qwen2.5-VL-72B	$58.7$	$67.1$	$85.2$	$33.5$	$60.7$
GLM-4.5V-106B	$69.5$	$64.2$	$66.1$	$67.7$	$66.9$
Qwen3-VL-32B-T	$78.2$	$66.3$	$63.2$	$80.5$	$71.4$
Qwen3-VL-235B-T	$74.5$	$63.2$	$59.3$	$77.4$	$67.9$

Table 2: Process reward evaluation on CUARewardBench, compared with models under the SE-WSM template. Best results are in bold; second-best are underlined.

Framework	Model	P	R	F1
CUAJudge	GPT-5-mini	$49.7$	$30.8$	$38.0$
CUAJudge	Qwen3-VL-8B	$\mathbf{67.7}$	$12.7$	$21.3$
OSThemis	Qwen3-VL-8B	$17.0$	$14.3$	$15.6$
SeekJudge	Qwen3-VL-8B	$41.4$	$20.4$	$27.3$
SeekJudge	SeekJudge-9B	$44.5$	$33.3$	$38.1$
SeekJudge	Qwen3.6-27B	$44.0$	$30.8$	$36.3$

Table 3: Step-level evaluation on CUAStepBench. Best results are in bold.

	Chrome		Impress		Calc		OA
Method	1 run	3-run mean	1 run	3-run mean	1 run	3-run mean	1 run	3-run mean
UI-TARS	$32.6$	$28.2$	$33.8$	$32.1$	$10.6$	$7.6$	$25.6$	$22.6$
Outcome GRPO	$43.5$	$37.0$	$44.7$	$41.6$	$19.1$	$12.8$	$35.7$	$30.4$
SeekJudge	$45.7$	$38.3$	$46.8$	$41.1$	$19.1$	$14.9$	$37.1$	$31.4$
Ours	$\mathbf{49.9}$	$\mathbf{43.4}$	$\mathbf{48.7}$	$\mathbf{43.3}$	$\mathbf{23.4}$	$\mathbf{17.0}$	$\mathbf{40.6}$	$\mathbf{34.5}$

Table 4: Task success rate (%) on OSWorld. All methods use UI-TARS-1.5-7B as the base model. Impress and Calc denote LibreOffice Impress and LibreOffice Calc. OA is the overall success rate over the three applications’ tasks.

Grouping	P	NPV	R	S	OA
Single-group	$69.1$	$\mathbf{96.3}$	$\mathbf{98.4}$	$49.7$	$75.7$
Fixed-window	$67.4$	$93.8$	$97.2$	$47.2$	$73.7$
Grouping w/ screenshots	$\mathbf{79.4}$	$81.8$	$85.1$	$\mathbf{75.2}$	$80.4$
Ours	$\mathbf{79.4}$	$84.1$	$87.8$	$73.9$	$\mathbf{81.3}$

Table 5: Grouping ablation on CUARewardBench. Single-group places all actions in one group. Fixed-window denotes the fixed-window step-reward methods, consecutive windows of 5 screenshots. Grouping w/ screenshots additionally uses screenshots when partitioning.

Scoring	P	NPV	R	S	OA
w/o reference	$74.7$	$83.7$	$89.0$	$65.2$	$77.9$
w/o action scoring	$68.6$	$85.3$	$92.2$	$51.6$	$73.3$
w/o group scoring	$70.8$	$91.5$	$95.6$	$54.8$	$76.6$
w/o refinement	$76.0$	$86.2$	$90.7$	$67.1$	$79.7$
Sum aggregation	$70.6$	$\mathbf{94.4}$	$\mathbf{97.2}$	$53.5$	$76.9$
Ours	$\mathbf{79.4}$	$84.1$	$87.8$	$\mathbf{73.9}$	$\mathbf{81.3}$

Table 6: Scoring and aggregation ablation on CUARewardBench. w/o reference omits

u_{k}

. w/o refinement omits

F_{\mathrm{rev}}

. w/o group scoring uses only

r_{t}^{\mathrm{act}}

. w/o action scoring uses only

\tilde{r}_{k(t)}^{\mathrm{group}}

. Sum aggregation replaces the product with an equal sum of the two scores.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Contribution	Completed	Partial	Failed	Uncertain
Milestone	$1.0$	$0.4$	$0.0$	$0.4$
Useful-efficient	$0.8$	$0.4$	$0.0$	$0.4$
Useful-inefficient	$0.6$	$0.4$	$0.0$	$0.4$
Exploratory	$0.4$	$0.4$	$0.0$	$0.2$
Redundant	$0.2$	$0.2$	$0.0$	$0.2$
Harmful	$0.0$	$0.0$	$0.0$	$0.0$

Appendix

Table 7: Lookup table

M

mapping the two labels to a scalar. The same table is used at both the action level and the group level.

Setting	Sync	Async
w/o Ours	$\mathbf{92.3}$	$\mathbf{92.3}$
Ours	$60.3$	$87.9$
$\Delta$	$-32.0$	$-4.4$

Appendix

Table 8: Sampling throughput (trajectories per hour). Sync starts the next trajectory after scoring finishes; Async starts the next immediately and scores the completed trajectory in the background.

\Delta

is the change from adding Ours.

Cause	Count	Share
Insufficient application knowledge	$26$	$41.3\%$
Visual or interaction misread	$14$	$22.2\%$
Missed instruction constraint	$12$	$19.0\%$
Overestimated group contribution	$11$	$17.5\%$

Appendix

Table 9: Underlying causes of critic–human disagreements on CUARewardBench key steps.

Figure 3: A Chrome trajectory for enabling Do Not Track.

Figure 4: A Calc trajectory that fills the Seq No. column with “No. #”.

Figure 5: A failure-case trajectory that opens the first link in the latest Bills email in a new Chrome tab. The dashed outline marks the incorrectly scored step.

RL for GUI Agents

Reinforcement learning for GUI agents trains screen-based AI to complete multi-step tasks across mobile, web, and desktop interfaces.

See all 50 papers

StainFlow: Entity-Stain Tracking and Evidence Linking for Process Rewards in GUI Agents
Jun 5, 2026Haojie Hao, Longkun Hao, Yihang Lou +8Reinforcement Learning RL for GUI Agents
Beihang University · Peking University · Renmin University of China +4
GUI-PRA: Process Reward Agent for GUI Tasks
Sep 27, 2025Tao Xiong, Xavier Hu, Yurun Chen +6Process Reward Models GUI Agents
Zhejiang University · MiLM Plus, Xiaomi Inc.
GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis
Aug 24, 2026Long Zhang, Yuhan Chen, Chaoran Zhang +7RL for GUI Agents Reward Design for RL
Wuhan University · Xiaomi Inc.

Jun 5, 2026J/K move · Enter open · S save

StainFlow: Entity-Stain Tracking and Evidence Linking for Process Rewards in GUI Agents

Haojie Hao, Longkun Hao, Yihang Lou, Yan Bai+7

Beihang University · Peking University · Renmin University of China+4

Reinforcement Learning RL for GUI Agents Process Reward Models Credit Assignment in RL Sparse-Reward RL

Open paper AskPDF arXiv

Reinforcement Learning (RL) has become a promising approach for improving GUI Agents in long-horizon, stochastic digital environments, but trajectory-level success feedback is too sparse to provide reliable credit assignment for intermediate exploration steps. To mitigate this issue, recent studies introduce Process Reward Models (PRMs), which provide finer-grained training feedback through global milestone verification or local step-level evaluation. However, these methods still suffer from two level-specific limitations: global milestone decomposition is subjective and singular, making it difficult to accommodate the multiple valid execution paths in real GUI tasks, while fixed local judging windows may miss long-range key evidence or dilute the decision signal with irrelevant frames. Inspired by stain-tracing mechanisms in network flow analysis, we propose StainFlow, an entity-stain-flow process reward model for GUI Agents. To reduce the subjectivity of global partitioning, we introduce the Global Entity Stain Tracking module, which extracts visually verifiable task entities and tracks how their stain concentrations and states evolve along the trajectory, allowing task phases to be objectively separated by changes in the entity evidence flow. To improve the accuracy of local verification, we introduce the Local Stain Evidence Linking module. Centered on the triggering entities of each candidate key node, it retrieves relevant steps based on their stain concentrations and state changes, and dynamically constructs high-density evidence windows for verifying true key nodes. Extensive experiments on AndroidWorld and OGRBench show that StainFlow relatively improves online RL success by 3.2% and trajectory completion judgment accuracy by 1.8%.

Abstract

RL for GUI Agents

Explore similar work

StainFlow: Entity-Stain Tracking and Evidence Linking for Process Rewards in GUI Agents

GUI-PRA: Process Reward Agent for GUI Tasks

GSAR: Goal-State-Anchor Rewards for Mobile GUI Agents with Self-Evolving Data Synthesis