What Does "Now" Mean to Your Computer-Use Agent? Stale Screenshots, Page Loads, and Which Check to Run Before a Click
Organizations: Principal AI Engineer, Author—Data Engineering for Multimodal AI (O’Reilly), Stanford School of Engineering
Abstract
Computer-use agents plan on a screenshot for several seconds and then click at a position on the screen; if the page has changed in the meantime, the click lands on an unintended element. Prior work treats the planning interval as the source of this risk. Measuring planned clicks on 31 public web pages, we find that the risk lies in page loading: 0.4% of clicks planned on fully loaded pages became stale within 30 seconds, against 17% of clicks planned in the first second after a page opened. On these pages, waiting about one second for network activity to stop removes most of this risk, yet the wait rules of eight of ten widely used open-source agents leave more than 1% of clicks stale. Commercial pages keep loading for longer: in the HTTP Archive's public crawl, 15% of 65 desktop booking and buying home pages went 500 ms without network requests within 3 seconds of their HTML being parsed, against 63% of 3,028 .gov pages, and 52% shifted their layout within 6.5 seconds of a network-idle wait capped at 3 seconds, against 8% (61% against 26% among pages recorded in full). Where a check before the click is needed, as before irreversible actions and on pages that are still loading, a pixel comparison verifies the target's position as accurately as a frontier vision-language model at one eighth of the latency, and a 4B model matches the frontier model in verifying whether the previous action succeeded. We release a benchmark generator, a break-even formula for when a check pays, and three rules for agent developers: https://github.com/multimodal-ai-engineering/preflight-bench
Figures & tables
| Wait rule | Wait | Stale (95% CI) | Agent (wait rule) | Stale (95% CI) | Wait |
|---|---|---|---|---|---|
| no wait | 0 s | 14.4% (12.6–16.5) | Anthropic CU demo (2 s delay) | 2.0% (1.3–2.9) | 2.02 s |
| 0.5 s delay | 0.50 | 2.5% (1.7–3.5) | browser-use (load or idle) | 4.0% (3.1–5.2) | 0.94 |
| load event | 0.43 | 3.0% (2.2–4.2) | OpenAI CUA sample (no wait) | 15.4% (13.5–17.4) | 0.35 |
| layout stable 0.5 s | 0.65 | 2.8% (2.0–3.8) | Stagehand v4 (no wait) | 12.5% (10.8–14.5) | 0.34 |
| boxes stable 0.5 s | 0.81 | 1.8% (1.2–2.7) | Skyvern (load, 1 s, animations) | 0.46% (0.21–1.0) | 2.03 |
| 1 s delay | 1.00 | 3.1% (2.3–4.2) | Agent S3 (1 s delay) | 11.7% (10.1–13.6) | 1.02 |
| Home pages | Pages | Quiet within 3 s | Time to quiet | Last shift | Shifted after the wait |
|---|---|---|---|---|---|
| booking and buying | 65 | 15% | 7.4 s | 5.1 s | 52% (40–65) [61%] |
| main host only † | 57 | 11% | 7.8 s | 5.8 s | 60% (47–72) [64%] |
| e-commerce sample | 4,972 | 19% | 6.4 s | 2.2 s | 22% (21–23) [30%] |
| .gov | 3,028 | 63% | 2.1 s | 0.65 s | 7.8% (6.8–8.7) [26%] |
| study sites | 21 | 71% | 1.9 s | 0.45 s | 4.8% (0–14) |
| Question | Check | Detected | False alarms | Latency (ms) |
| Target still in place? (1,064 stale clicks) | pixel check | 82% | 4% | 312 |
| PUSV, tuned to 2% false alarms | 24% | 0.3% | 5 | |
| Qwen3-VL 2B | 0.2% | 0% | 1,450 | |
| Qwen3-VL 4B | 66% | 37% | 2,425 | |
| Claude Haiku 5.5 | 98% | 95% | 1,276 | |
| Claude Opus 5.5 | 83% | 12% | 2,366 |
Computer-Use Agents
Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.
See all 162 papers