Computer-use agents operate over long horizons under noisy perception, multi-window contexts, evolving environment states. Existing approaches, from RL-based planners to trajectory retrieval, often drift from user intent and repeatedly solve routine subproblems, leading to error accumulation and inefficiency. We present IntentCUA, a multi-agent computer-use framework designed to stabilize long-horizon execution through intent-aligned plan memory. A Planner, Plan-Optimizer, and Critic coordinate over shared memory that abstracts raw interaction traces into multi-view intent representations and reusable skills. At runtime, intent prototypes retrieve subgroup-aligned skills and inject them into partial plans, reducing redundant re-planning and mitigating error propagation across desktop applications. In end-to-end evaluations, IntentCUA achieved a 74.83% task success rate with a Step Efficiency Ratio of 0.91, outperforming RL-based and trajectory-centric baselines. Ablations show that multi-view intent abstraction and shared plan memory jointly improve execution stability, with the cooperative multi-agent loop providing the largest gains on long-horizon tasks. These results highlight that system-level intent abstraction and memory-grounded coordination are key to reliable and efficient desktop automation in large, dynamic environments.
Figures & tables
Figure 1. Overview of IntentCUA . Offline: raw user traces are multi-view labeled, embedded into a shared intent space, and clustered into intent groups (IG) and subgroups (SG); SG action patterns are converted into parameterized skill schemas (“skill hints”) and stored with their SG in the IG/SG index, while plan memory stores only user-approved global plans (G). Online: the Planner/Plan-Optimizer/Critic query and reuse skills; cache-first reuse and template-based gap filling reduce re-planning on long-horizon desktop tasks.
Figure 2. Multi-view intent representation. Control traces use [E,A,D], browsing traces [E,K,D]. A multi-view encoder aligns views into a shared space, inducing environment-centric IG and finer SG. SG centroids enable retrieval, and SG action patterns are converted into parameterized skill schemas (“skill hints”) with verb–argument structure for planning.
Figure 3. Cache-first planning with plan memory. A query intent is gated by IG and ranked over SG. Case 1 (miss): synthesize a plan from retrieved skill templates. Case 2 (hit): reuse the stored plan. Case 3 (partial): align to the nearest plan and fill gaps with SG-derived skill hints, reducing retries.
Method
Success (%) ↑
Completion (%) ↑
B
22.73
33.78
B+Tg
46.43
57.41
B+TSG+Z
54.64
77.56
B+TSG+SSG+PM
53.85
81.23
B+TSG+Z+PM
62.51
85.00
B+TSG+Z+SSG+PM
74.83
91.14
Table 1. Component-wise ablation on planning. We report task success (%) and averaged plan completion (%).
Figure 4. IntentCUA in action: the system recalls intent units from memory and decomposes a multi-application command into intent-level plan units, each executed through learned skills and recomposed into an end-to-end automation plan.
Method
WebVoyager
ScreenAgent
Ours
Total(%)
UI-TARS-1.5 ( Seed, 2025 )
35.9
42.9
46.0
38.8
UFO 2 ( Zhang et al., 2025b )
69.0
41.4
38.0
51.2
IntentCUA (ours)
71.6
77.1
78.0
74.8
Table 2. End-to-end success rate comparison across datasets (%). Columns show WebVoyager, ScreenAgent, our in-house suite, and overall average.
Figure 5. Success rate by step length (bin size = 5 steps). The x-axis shows step-length bins and the y-axis shows task success rate (%).
Figure 6. Performance by task length (bin size = 5 steps). Left: Step Efficiency Ratio (SER). Right: Average latency per task (minutes)
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7. Planner–Plan-Optimizer–Critic interaction. The Planner decomposes a command into structured Plan Units and retrieves or synthesizes a global plan. The Plan-Optimizer grounds each unit into executable GUI actions conditioned on the current state, while the Critic validates the post-state Safter and triggers local re-optimization if needed.
Loss Variant
Separation ↑
Baseline Embedding
5.60
InfoNCE only
5.64
InfoNCE + Prediction
6.92
InfoNCE + Reconstruction
23.17
Full (Con + Pred + Rec)
7.74
Appendix
Table 3. Representation loss ablation on intent embedding quality. We report size-weighted density separation (inter/intra) over all HDBSCAN subgroups.
Loss Variant
ENV purity
ACT purity
Baseline Embedding
0.83(0.19)
0.48(0.25)
InfoNCE only
0.82 (0.22)
0.37 (0.22)
InfoNCE + Prediction
0.83 (0.21)
0.42 (0.26)
InfoNCE + Reconstruction
0.86 (0.19)
0.42 (0.26)
Full (Con + Pred + Rec)
0.84 (0.20)
0.42 (0.23)
Appendix
Table 4. ENV/ACT purity under different representation loss variants. We report mean purity with standard deviation in parentheses.
Figure 8. Domain distribution of collected trace data. Each slice indicates a domain category and its proportion within the trace corpus (%).
Figure 9. Domain distribution of the 286 evaluation testcases. Each slice shows a domain category and the success rate achieved within that domain (%).
WebVoyager
ScreenAgent
Ours
Total
Mean (%)
70.3
81.1
86.2
78.5
STD (%)
8.2
5.3
7.9
7.8
Appendix
Table 5. Step consistency (%) across five repeated planning runs per testcase (cosine threshold = 0.93).