GUI agents can formulate a correct immediate plan in their reasoning yet emit an action that fails to execute it, creating an execution gap. This recurring mismatch undermines reliable task completion, as a single execution error can derail a multi-step workflow. In outcome-supervised reinforcement learning, broadcasting a negative response-level advantage across all generated tokens risks penalizing valid planning together with erroneous execution. To address this credit-assignment conflict, we introduce PACT (Plan-Action Consistency Training), a training framework that combines reference-aligned plan protection and plan-aligned action repair without altering the original scalar action reward. Using annotated reference actions to identify execution gaps, PACT protects the verified plan by masking negative policy-gradient contributions across its token span. Keeping the generated reasoning unchanged as context, it then favors the reference action over the emitted erroneous action through a preference surrogate with stop-gradient pair weights. Compared with the outcome-only baseline, PACT raises AndroidWorld task success from 60.34% to 68.97% (+8.63 pp), while improving rule-based plan-action consistency on AndroidControl High from 83.71% to 89.74% (+6.03 pp). Adapting PACT to desktop agents also improves task success from 40.92% to 44.17% (+3.25 pp) on the full 369-task OSWorld benchmark. Together, these results establish PACT as an effective credit-assignment framework for bridging execution gaps and improving task completion across mobile and desktop GUI environments.
Figures & tables
Figure 1: Execution gaps and targeted correction. Top: Both models continue the same supplied Pinterest reasoning; O launches the app, while PACT clicks the intended + control. Bottom: With independently generated reasoning and actions on Moon, O states a downward swipe but emits upward finger motion, while PACT executes the downward swipe. On-screen circles mark the click or gesture start; swipe arrows show finger motion. The separate training schematic illustrates reference-aligned plan protection of selected plans and additional action repair under a negative response advantage.
Metric
Success criterion
T
The current plan is correct for the task.
A
The action is valid for the current task and state.
C
The action implements the model’s stated plan.
Joint repair
T=A=C=1 for the same plan–action pair.
Table 1: Step-level judgments and training implications in PACT.
Figure 2: PACT training framework. Reference-verified execution-gap selection identifies reference-supported plans paired with incorrect, inconsistent actions. Reference-aligned plan protection zeros negative policy-gradient coefficients on the plan span. Plan-aligned action repair favors reference-derived over erroneous action suffixes under the same input and reasoning trace. Both feed the GRPO actor update.
Configuration
Action Repair
Reference-Verified Execution-Gap Selection
Reference-Aligned Plan Protection
O
×
×
×
OA
✓
×
×
OAR
✓
✓
×
PACT
✓
✓
✓
Table 2: Cumulative training configurations.
Model
Size
AndroidWorld SR (%)
OSWorld SR (%)
ScreenSpot-v2 Mobile (%)
UI-TARS-7B-SFT [ 13 ]
7B
33.0
17.7
93.61 [ 13 ]
UI-TARS-1.5-7B [ 1 ]
7B
30.0 [ 25 ]
27.5
91.23 [ 26 ]
UI-Venus-Navi-7B [ 7 ]
7B
49.1
–
–
GUI-Owl-7B [ 23 ]
7B
66.4
29.5 [ 3 ]
–
Qwen3-VL-8B-Instruct [ 25 ]
8B
47.6
–
92.38 [ 25 ]
Step-GUI-8B [ 22 ]
8B
67.7
40.2
–
Table 3: Main benchmark results. (A) Published GUI-agent context. (B) Matched RL comparisons.
Setting
Model
Ret. (%)
W → R ↑
R → W ↓
Net ΔE↓
AC High
OA
87.89
669
638
−31
OAR
86.34
591
720
+129
PACT
89.51
631
553
−78
AC Low
OA
90.03
436
656
+220
OAR
89.07
380
719
+339
PACT
92.37
476
502
+26
Table 4: Reference-action retention on aligned AndroidControl states. ∣K0∣ : High 5,270; Low 6,580. Non-initial states use each model’s own generated history. W/R: wrong/right. Net error change vs. O: ΔE=(R→W)−(W→R) .
Figure 3: Retention and repair on aligned states (higher is better). Top: O-correct action retention. Bottom: net correct-action gains over OAR, computed as extra repairs plus avoided regressions. PACT leads retention across settings and net gains on AndroidControl.
Metric (%; out of 23 states)
O
OA
OAR
PACT
Fixed plan
Reference action accuracy
56.52
95.65
100.0
100.0
Free generation
Reference action accuracy
0.0*
95.65
95.65
95.65
Plan–action consistency ( C )
0.0*
100.0
100.0
100.0
Semantic joint repair
0.0*
100.0
95.65
95.65
Table 5: Execution diagnostics on the 23-state execution-gap set. Asterisks (*) mark O’s selection criteria. Joint repair is defined in Table 1 (a). Full semantic results are in the supplementary material.
Subset
Model
T↑
A↑
C↑
Joint repair ↑
All ( N=30 )
OAR
19/30
13/30
25/30
13/30
PACT
23/30
19/30
28/30
19/30
Both models consistent C=1 , N=24
OAR
13/24
12/24
24/24
12/24
PACT
18/24
16/24
24/24
16/24
Table 6: Free-generation results on all 30 states and the 24 with COAR=CPACT=1 , using Table 1 (a) metrics.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Mobile
Desktop
Initialization and fixed reference
Qwen3.5-9B SFT
Qwen3.5-9B SFT
RL training examples
10,000
18,000
Training iterations / epochs
1,250 / 2
562 / 1
Prompts per iteration / responses per prompt ( K )
16 / 8
32 / 8
Original rollout batch size ( B0 )
128
256
Optimizer
AdamW
Distributed Adam
Appendix
Table S1: Training hyperparameters for the mobile and desktop studies. Repair parameters apply to OA, OAR, and PACT.
Action
Required parameters
Reference support / plan compatibility
Positive suffix construction
Click , DoubleClick , RightClick
One box coordinate
T : distance <49.95 ( 0.05×999 ). C : distance ≤1 .
Copy reference box .
Drag , Swipe
start and end coordinates
T : each endpoint distance <139.86 ( 0.14×999 ). C : each endpoint distance ≤1 .
Copy both reference endpoints.
Type
One quoted content string
T,C : exact string equality, preserving whitespace and escapes.
Copy reference content .
Hotkey
One nonempty keys list or tuple
T,C : equality of normalized, ordered key tuples.
Copy reference keys .
Wait
No arguments
T,C : matching Wait action.
Wait() .
CallUser , Finished
Optional content string
Terminal actions receive native scoring with exact content matching.
Excluded from auxiliary selection.
Appendix
Table S2 : Desktop training-action verification and reference-derived suffix construction. Spatial distances are Euclidean distances on the normalized 0–999 grid.
Setting
Model
Consistent / judgeable
Type ↓
Coord. ↓
Text ↓
Crule (%) ↑
AC High
O
6879 / 8218
1279
19
41
83.71
OA
6440 / 8046
1559
25
22
80.04
OAR
6997 / 8096
1032
21
46
86.43
PACT
7046 / 7852
753
16
37
89.74
AC Low
O
7182 / 8313
1081
19
31
86.39
OA
6534 / 8327
1753
24
16
78.47
Appendix
Table S3: AndroidControl consistency and mismatch counts over 8,444 states per setting. Type, coordinate, and text columns partition the inconsistent judgeable outputs. Bold marks the best values within each setting, including ties.
Configuration
All targets (501)
Text targets (290)
Icon targets (211)
O
460 (91.82%)
271 (93.45%)
189 (89.57%)
OA
481 (96.01%)
286 (98.62%)
195 (92.42%)
OAR
478 (95.41%)
286 (98.62%)
192 (91.00%)
PACT
481 (96.01%)
285 (98.28%)
196 (92.89%)
Appendix
Table S4: ScreenSpot-v2 Mobile point-in-target accuracy across all 501 examples, 290 text targets, and 211 icon targets.
Configuration
Retained / 460
Retention (%)
Wrong to right ( ↑ )
Right to wrong ( ↓ )
OA
458 / 460
99.57
23
2
OAR
456 / 460
99.13
22
4
PACT
459 / 460
99.78
22
1
Appendix
Table S5: Target-level retention and repair on ScreenSpot-v2 Mobile (460 O-correct targets).
Condition / metric
O
OA
OAR
PACT
Fixed plan: reference action accuracy
56.52 (13/23)
95.65 (22/23)
100.0 (23/23)
100.0 (23/23)
Free generation: reference action accuracy
0.0 (0/23)*
95.65 (22/23)
95.65 (22/23)
95.65 (22/23)
Free generation: plan correctness ( T )
100.0 (23/23) *
100.0 (23/23)
95.65 (22/23)
95.65 (22/23)
Free generation: action validity ( A )
0.0 (0/23)*
100.0 (23/23)
95.65 (22/23)
95.65 (22/23)
Free generation: plan–action consistency ( C )
0.0 (0/23)*
100.0 (23/23)
100.0 (23/23)
100.0 (23/23)
Free generation: semantic joint repair
0.0 (0/23)*
100.0 (23/23)
95.65 (22/23)
95.65 (22/23)
Appendix
Table S6: Complete fixed-plan continuation and free-generation results on the 23-state execution-gap diagnostic set (%, with exact counts out of 23 in parentheses). Joint repair requires T=A=C=1 . Asterisks (*) mark O’s selection criteria. Bold indicates row-wise maxima, including ties.
Candidate category
States
Click-versus-launch conflict
8
Scroll-direction error
2
Premature text entry / skipped selection
17
Click-versus-wait conflict
1
Swipe-versus-click/selection conflict
1
Ambiguous-reference boundary case
1
Appendix
Table S7: Natural execution-gap composition and free-generation results on all 30 states, the 24 with COAR=CPACT=1 , and the remaining six. Bold marks the better value within each comparison, including ties.
Computer-Use Agents (CUAs) leverage large language models to execute GUI operations on desktop environments, yet they generate actions without evaluating action quality, leading to irreversible errors that cascade through subsequent steps. We propose IntentScore, a plan-aware reward model that learns to score candidate actions from 398K offline GUI interaction steps spanning three operating systems. IntentScore trains with two complementary objectives: contrastive alignment for state-action relevance and margin ranking for action correctness. Architecturally, it embeds each candidate's planning intent in the action encoder, enabling discrimination between candidates with similar actions but different rationales. IntentScore achieves 97.5% pairwise discrimination accuracy on held-out evaluation. Deployed as a re-ranker for Agent S3 on OSWorld, an environment entirely unseen during training, IntentScore improves task success rate by 6.9 points, demonstrating that reward estimation learned from heterogeneous offline trajectories generalizes to unseen agents and task distributions.
Rongqian Chen, Yu Li, Zeyu Fang +3
George Washington University, Washington D.C., USA
Large language model (LLM)-based agents frequently generate seemingly coherent plans that fail upon execution due to infeasible actions, constraint violations, and compounding errors over extended horizons. PIVOT (Plan-Inspect-eVOlve Trajectories) addresses this plan-execution misalignment through a self-supervised framework that treats trajectories as optimizable objects iteratively refined via environment interaction. The framework comprises four stages: PLAN generates candidate trajectories; INSPECT executes them and computes structured losses with textual gradients encoding plan-execution discrepancies; EVOLVE applies these signals to produce improved trajectories; and VERIFY performs a final global check against task constraints. A monotonic acceptance process ensures a non-decreasing solution quality. Empirical evaluations on DeepPlanning and GAIA demonstrate state-of-the-art performance: with human-in-the-loop (HITL) feedback, PIVOT establishes a strong upper bound up to 94% relative improvement in constraint satisfaction, while its fully autonomous variant retains substantial gains, showing that the core trajectory-refinement mechanism remains effective without external supervision. At the same time, PIVOT remains computationally efficient, requiring up to 3x to 5x fewer tokens than competing refinement methods. These findings establish that (self- or human-supervised) feedback-based trajectory optimization is a principled methodology for mitigating plan-execution gaps in autonomous agent systems.
Long-horizon agentic tasks demand strong reasoning and efficient execution across successive interactions with dynamic environments. A common approach decouples high-level planning from low-level execution through separate planner and actor roles. To investigate coordination failures in these tasks, we prompt both agents for structured state assertions and compare their reports programmatically to detect explicit contradictions. Our analyses reveal systematic disagreement about the same task-relevant state facts, a phenomenon we term planner-actor state mismatch. We further find that providing agents with task-relevant state information reduces mismatch and improves coordination and task performance. Based on the systematic analysis of the state mismatch, we propose Consistent Plan-Act (ConPAct), which feeds detected contradictions back to both agents to form consistent state interpretations and fine-tunes them on curated consistent interactions for better coordination. ConPAct improves performance across various environments and model configurations, e.g., increasing MiniGrid success rate from 38.6% to 54.4% with GPT-5.6-sol/terra as planner and actor respectively, demonstrating that state consistency can guide both inference-time correction and coordination training.
Heng-Zhuang Li, Yi-Kai Zhang, Yu Wang +4
School of Artificial Intelligence, Nanjing University · National Key Laboratory for Novel Software Technology, Nanjing University · LongCat Team, Meituan +1