Computer Use Agents (CUAs) fundamentally rely on graphical user interface (GUI) grounding to translate language instructions into executable screen actions, but editing-level grounding in dense coding interfaces (such as VS Code and Cursor), where sub-pixel accuracy is required to interact with dense IDE elements, remains underexplored. Existing approaches typically rely on single-shot coordinate prediction, which lacks a mechanism for error correction and often fails in high-density interfaces. In this technical report, we conduct an empirical study of pixel-precise cursor localization in coding environments. Instead of a single-step execution, our agent engages in an iterative refinement process, utilizing visual feedback from previous attempts to reach the target element. This closed-loop grounding mechanism allows the agent to self-correct displacement errors and adapt to dynamic UI changes. We evaluate our approach across Claude, Qwen, and GPT on a suite of complex coding benchmarks, demonstrating that multi-turn refinement significantly outperforms state-of-the-art single-shot models in both click precision and overall task success rate. Our results suggest that iterative visual reasoning is a critical component for the next generation of reliable software engineering agents. Code: https://github.com/microsoft/precision-cua-bench/tree/main.
Figures & tables
Figure 1: Data collection system overview. A process-separated architecture connects the editor extension host (VS Code or Cursor) with the renderer DOM through a localhost WebSocket channel: the extension iterates over symbolic cursor stops and the renderer-side payload measures the corresponding pixel geometry via getBoundingClientRect() inside requestAnimationFrame . The synchronized records are written to JSONL with screen-absolute coordinates and editor metadata. Because both editors share the Electron+Monaco stack, the same pipeline produces directly comparable records across them.
Accuracy (%) ↑
D bb (px) ↓
D c (px) ↓
Prompt
Model
T1
T2
T3
T4
T5
T1
T2
T3
T4
T5
T1
T2
T3
T4
T5
Baseline
Qwen 3.5 9B
1.5
1.6
1.7
1.7
1.8
272.0
266.6
266.3
265.2
264.9
284.0
278.0
277.5
276.5
276.1
GPT-5.4-Pro
13.5
25.3
32.2
37.0
41.0
185.2
216.4
220.2
235.2
255.8
195.3
224.4
225.5
240.8
261.4
Claude Opus 4.7
21.0
31.0
37.5
42.0
45.4
292.4
327.6
360.1
388.1
411.8
301.0
336.7
368.4
396.4
420.3
Baseline CoT
Qwen 3.5 9B
1.4
1.9
2.1
2.3
2.6
450.3
437.4
451.0
445.5
445.1
459.5
444.0
457.2
451.4
450.2
GPT-5.4-Pro
11.1
25.8
33.0
37.5
41.2
181.5
230.2
234.6
252.0
257.5
191.6
237.5
240.6
258.2
262.8
Table 1: Multi-turn grounding results. Accuracy (%), point-to-bounding-box distance (D bb , px) and point-to-center distance (D c , px) across five refinement turns (T1–T5) for each of the four system-prompt variants and the three evaluated models (Qwen-3.5-9B, Claude Opus 4.7, and GPT-5.4-Pro). Each cell is averaged over the full 5,390-sample benchmark, spanning both editors (VS Code, Cursor) and both themes (Dark, Light). Higher is better for accuracy; lower is better for distances. Distances are reported under cumulative semantics and therefore aggregate over the residual unresolved subset at each turn (Section 4.2 ).
Character Acc. (%) ↑
Word Acc. (%) ↑
Line Acc. (%) ↑
Prompt
Model
T1
T2
T3
T4
T5
T1
T2
T3
T4
T5
T1
T2
T3
T4
T5
Baseline
Qwen 3.5 9B
0.0
0.1
0.1
0.1
0.1
0.4
0.5
0.5
0.6
0.8
9.5
10.0
10.4
10.5
10.9
GPT-5.4-Pro
3.5
9.3
15.8
21.9
27.2
28.5
49.8
59.0
62.1
64.3
36.9
62.4
67.7
69.1
70.4
Claude Opus 4.7
6.9
16.9
26.1
32.6
37.4
41.0
56.1
57.6
58.4
59.0
56.9
59.3
60.7
61.7
62.3
Baseline CoT
Qwen 3.5 9B
0.0
0.0
0.1
0.2
0.3
0.0
0.0
0.0
0.0
0.4
9.7
13.3
14.4
14.9
16.4
GPT-5.4-Pro
3.5
11.0
18.3
23.7
28.1
20.9
48.3
56.7
59.8
62.5
31.7
60.6
65.9
68.3
70.2
Table 2: Multi-turn element-wise grounding accuracy. Accuracy (%) decomposed by target granularity (character, word, and line) across five refinement turns (T1–T5) for the four system-prompt variants and the three evaluated models (Qwen-3.5-9B, Claude Opus 4.7, and GPT-5.4-Pro). Computed on the full 5,390-sample benchmark (3,518 character, 1,092 word, 780 line), matching Table 1 . Higher is better.
Overall localization (Accuracy ↑ , D bb↓ , D c↓ )
Accuracy (%) ↑
D bb (px) ↓
D c (px) ↓
Prompt
IDE
Theme
T1
T2
T3
T4
T5
T1
T2
T3
T4
T5
T1
T2
T3
T4
T5
Baseline
VS Code
Dark
34.6
51.1
61.4
67.9
73.1
23.0
11.7
12.3
13.9
16.3
25.4
13.5
14.2
16.1
18.4
Light
35.4
49.3
59.5
66.2
71.1
23.0
13.1
12.5
14.4
16.4
25.8
15.3
14.7
16.5
18.7
Cursor
Dark
6.9
11.0
14.3
17.1
19.0
475.3
495.8
513.5
531.4
543.5
487.5
509.2
524.0
540.8
553.0
Light
8.3
13.8
16.2
18.7
20.2
475.5
505.7
520.0
534.6
545.1
488.6
518.3
531.9
546.7
556.6
Table 3: Cross-editor and cross-theme analysis for Claude Opus 4.7. We stratify the evaluation set by editor (VS Code, Cursor) and color theme (Dark, Light) and report multi-turn results (T1–T5) for each of the four system-prompt variants. The top block reports overall localization (accuracy, point-to-bounding-box distance over incorrect predictions, point-to-center distance over incorrect predictions); the bottom block reports element-wise accuracy at character, word, and line granularities. All accuracies are cumulative over turns.
Table 4: Comparison against specialized GUI grounding baselines ( GUI-Actor Wu et al. (2025) and GUI-G2 Tang et al. (2025) , evaluated single-turn) and against general-purpose vision-language models (Qwen-3.5-9B and Claude Opus 4.7, reported at the end of the five-turn refinement protocol). The lightweight in-domain finetune of GTA1 Yang et al. (2025) attains the strongest overall accuracy and the lowest residual distance, with the largest gain at the character granularity, indicating that targeted task-specific supervision is an effective lever for fine-grained text-cursor grounding.