cs.AIApr 6, 2026
SaveIntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents
Organizations: George Washington University, Washington D.C., USA
Abstract
Computer-Use Agents (CUAs) leverage large language models to execute GUI operations on desktop environments, yet they generate actions without evaluating action quality, leading to irreversible errors that cascade through subsequent steps. We propose IntentScore, a plan-aware reward model that learns to score candidate actions from 398K offline GUI interaction steps spanning three operating systems. IntentScore trains with two complementary objectives: contrastive alignment for state-action relevance and margin ranking for action correctness. Architecturally, it embeds each candidate's planning intent in the action encoder, enabling discrimination between candidates with similar actions but different rationales. IntentScore achieves 97.5% pairwise discrimination accuracy on held-out evaluation. Deployed as a re-ranker for Agent S3 on OSWorld, an environment entirely unseen during training, IntentScore improves task success rate by 6.9 points, demonstrating that reward estimation learned from heterogeneous offline trajectories generalizes to unseen agents and task distributions.
Figures & tables
Figure 1: Architecture of IntentScore . The state encoder is computed once per step; the intention-aware action encoder is computed per candidate. Reward estimation is temperature-scaled cosine similarity. Training uses a dual objective: state-action alignment (InfoNCE) plus reward learning (margin ranking on hard negatives).
Figure 2: Deployment inference pipeline. The CUA generates multiple candidate actions for the current state. The state and action encoders map inputs into a shared latent space, where cosine similarity determines action quality.
| Model | Hard | Real Inc. | Gap1 mean | Gap1 0.10 |
|---|---|---|---|---|
| Alignment loss only (baseline) | 0.866 | 0.917 | 0.202 | 69.5% |
| + Margin ranking loss( , ) | 0.879 | 0.931 | 0.213 | 71.9% |
| + Aggressive margin ( , ) | 0.886 | 0.935 | 0.251 | 76.8% |
| + MPNet + SigLIP2 + larger model | 0.902 | 0.928 | 0.309 | 79.5% |
| + Intention-aware action encoder | 0.949 | 0.962 | 0.362 | 87.8% |
| + Cross-OS pretraining ( IntentScore ) | 0.975 | 0.997 | 0.373 | 93.2% |
Table 1: Ablation study on training methodology. Hard = adjacent-step discrimination accuracy; Real Inc. = detection of labeled-incorrect actions; Gap1 mean = mean pairwise score gap on Hard test; Gap 0.10 = fraction of Hard test pairs exceeding the deployment override threshold. All evaluations use the held-out Ubuntu test set with task-level splits.
| Model | Chrome | GIMP | Calc | Impress | Writer | Multi | OS | TDB | VLC | VSCode | All |
|---|---|---|---|---|---|---|---|---|---|---|---|
| S3 (baseline) | 53.2 | 53.8 | 42.6 | 46.8 | 52.2 | 29.0 | 58.3 | 66.7 | 29.4 | 60.9 | 45.2 |
| S3 + IntentScore | 65.2 | 53.8 | 51.1 | 44.7 | 56.5 | 38.7 | 70.8 | 66.7 | 47.1 | 65.2 | 52.1 |
| +12.0 | = | +8.5 | -2.1 | +4.3 | +9.7 | +12.5 | = | +17.7 | +4.3 | +6.9 |
Table 4: Per-domain success rate (%) on OSWorld. TDB = Thunderbird. Green indicates improvement over baseline; red indicates regression. IntentScore improves Agent S3 on 7 of 10 domains under the constrained deployment configuration.
| System | SR | OR |
|---|---|---|
| S3 w/ N=3(baseline) | 45.2% | — |
| S3 + random selection | 44.2% (-1.0) | 56.9% |
| S3 + LLM self-evaluation | 45.8% (+0.6) | 52.1% |
| S3 + Alignment only | 48.0% (+2.8) | 40.1% |
| S3 + IntentScore | 52.1% (+6.9) | 46.6% |
Table 5: End-to-end task success rate on OSWorld (361 tasks). SR = success rate, OR = override rate.
| Step-level decision distribution | |
|---|---|
| Total steps | 6,826 |
| Dedup (all candidates identical) | 2,288 (33.5%) |
| Scored by model | 4,538 (66.5%) |
| Among scored steps | |
| Agree (scorer confirms #1) | 2,118 (46.7%) |
| Defer (low confidence) | 304 (6.7%) |
Table 6: Scorer behavior across 361 OSWorld tasks.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Specification |
|---|---|
| In-context history encoder | |
| Input per step | |
| Step projection | Linear(3074, 512) GELU LayerNorm(512) |
| Sequence model | 2-layer GRU, hidden size 384, dropout 0.1 |
| Output | Last hidden state |
| Padding | Zero-padding when fewer than previous steps exist |
Table 7: Detailed architecture of IntentScore . All text fields use frozen all-mpnet-base-v2 embeddings (768d); screenshots use frozen SigLIP-SO400M embeddings (1152d). The shared output dimension is 384.
| Ubuntu | Windows + Mac | Total | |
|---|---|---|---|
| Tasks | 5,000 | 17,625 | 22,625 |
| Steps | 77,448 | 321,380 | 398,828 |
| Thought present | 59% | varies | — |
Table 8: AgentNet dataset statistics by operating system.
| Parameter | Stage 1 (Pretrain) | Stage 2 (Finetune) |
|---|---|---|
| Data | 398K steps (Ubuntu+Win+Mac) | 77K steps (Ubuntu) |
| Epochs | 30 | 40 |
| Learning rate | ||
| Margin weight ( ) | 2.0 | 3.0 |
| Margin ( ) | 0.20 | 0.20 |
| Batch size | 1024 | 1024 |
Table 9: Complete training hyperparameters for IntentScore .
| Scorer field | Agent S3 source | Parsing |
|---|---|---|
| observation | Plan (Screenshot Analysis) | Regex extraction |
| thought | Plan (Next Action) | Per-candidate extraction |
| action | NL description from plan_code | Comment extraction |
| code | exec_code | Remove import prefix |
| xy | Pixel coords from exec_code | Normalize by resolution |
| reflection | reflection field | Direct (null empty) |
Table 10: Field mapping from Agent S3’s output format to IntentScore ’s input format. The planning intention is extracted from each candidate’s plan, enabling per-candidate reward estimation.
| Model | Features | AUC | Top-1/5 |
|---|---|---|---|
| V1 baseline | vision + obs + act + inst | 0.770 | 56.4% |
| V2 | + thought + reflection | 0.847 | 66.5% |
| V3 | + code | 0.887 | 69.3% |
| V4 (Transformer) | same as V3 | 0.510 | 29.0% |
Table 11: Binary classification baselines (step-level split). Accuracy improves with additional features but is evaluated on an easier split; on the task-level split used for IntentScore , binary classification collapses to 7.3% (Table 2 ).
Figure 3: Decision timeline for a complete OSWorld trajectory (28 steps, task: “write grammar test answers”). Each bar shows the score gap between the top-ranked and second-ranked candidate, colored by IntentScore ’s decision: override (select a different candidate), agree (confirm Agent S3’s default), or dedup (all candidates identical). Red stars mark the four detailed case studies below.
Figure 4: Case study 1 (step 7): IntentScore overrides Alt+Tab in favor of the “Window” menu for reliable document switching. The score gap of 0.236 reflects the intent-aware encoder’s ability to distinguish navigation strategies.
Figure 5: Case study 2 (step 10): IntentScore again overrides a navigation hotkey ( Ctrl+F6 ) in favor of the “Window” menu. The screenshot shows Answer.docx with the Window menu open, listing both documents.
Figure 6: Case study 3 (step 16): Three click candidates target the same line at slightly different positions. IntentScore selects the one that places the cursor directly after “Grammar test 2:” for precise text insertion.