ScaleDiag: Diagnosing the Limits of Test-Time Scaling in GUI Grounding
Organizations: University of Technology Sydney, Australia · Adelaide University, Australia · CSIRO Technology, Australia
Abstract
GUI grounding maps a screenshot and a natural language instruction to the coordinates of the intended interface element. Existing approaches improve grounding through visual scaling, instruction scaling, and sampling scaling, yet it remains unclear which grounding bottlenecks these strategies address and where their gains saturate. To address this gap, we introduce ScaleDiag, a controlled diagnostic framework that isolates the effects of visual input, instruction information, and prediction sampling on GUI grounding by independently varying target scale and surrounding context, target-specific information and auxiliary descriptions, and candidate coverage and selection, respectively. We evaluate ScaleDiag across five benchmarks and up to eight vision-language models and find that each scaling strategy operates within a much narrower regime than commonly assumed. (1) Visual scaling is governed by target occupancy rather than a fixed crop ratio, with full-screen errors typically rescued once the target occupies only 0.32% of the cropped view, while further zooming eventually becomes harmful. (2) Instruction scaling is driven by information that identifies the target rather than description richness itself, as even ground-truth-derived attributes such as color, size, and shape are often neutral or harmful, while appearance and spatial descriptions account for over 90% of the achievable gain. (3) Repeated sampling exposes a different limitation, as correct coordinates are frequently generated but remain difficult to select, leaving 5.1--11.0 points of oracle headroom while geometric consensus provides almost no improvement. Together, these results show that test-time scaling is constrained by distinct operating regimes rather than benefiting monotonically from more visual focus, richer instructions, or additional samples. Our code is available at https://github.com/YenanLiu/ScaleDiag.
Figures & tables
| Method | SS-Pro | UI-Vision | MMBench | OSW-G | SS-v2 | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| All | T/I | Gap | All | T/I | Gap | All | T/I | Gap | All | All | T/I | Gap | |
| Qwen3-VL-8B [ 35 ] | 50.8 | 69.8/20.0 | 49.8 | 24.0 | 59.7/16.2 | 43.5 | 81.7 | 88.6/76.2 | 12.3 | 54.4 | 93.4 | 98.1/87.4 | 10.7 |
| MAI-UI-8B [ 36 ] | 62.9 | 78.3/37.9 | 40.4 | 39.6 | 76.5/31.6 | 44.9 | 88.5 | 94.7/83.5 | 11.2 | 62.1 | 95.4 | 98.2/91.7 | 6.5 |
| Qwen3-VL-32B [ 35 ] | 57.8 | 76.0/28.3 | 47.7 | 35.0 | 67.6/27.9 | 39.6 | 87.3 | 91.8/83.7 | 8.1 | 65.6 | 95.0 | 97.8/91.3 | 6.4 |
| Qwen3.5-4B [ 37 ] | 49.7 | 64.8/25.3 | 39.5 | 33.2 | 64.5/26.3 | 38.2 | 81.7 | 87.0/77.4 | 9.6 | 56.2 | 91.8 | 96.2/86.1 | 10.1 |
| Qwen3.5-9B [ 37 ] | 60.7 | 74.5/38.2 | 36.3 | 38.6 | 63.8/33.1 | 30.7 | 84.0 | 87.8/80.9 | 6.8 | 62.4 | 93.5 | 97.2/88.6 | 8.6 |
| Setting | App | Spa | Fun | Goa | Qwen3-VL-8B | MAI-UI-8B | Qwen2.5-VL-7B | GTA1-7B | Qwen3.5-4B | Qwen3.5-9B | Qwen3.5-27B |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Base | 60.8 | 66.0 | 32.8 | 37.9 | 62.5 | 67.8 | 75.3 | ||||
| Single | ✓ | 70.8 | 70.6 | 37.4 | 49.8 | 74.0 | 76.1 | 78.9 | |||
| Single | ✓ | 66.4 | 68.3 | 34.8 | 50.6 | 69.4 | 71.9 | 76.5 | |||
| Single | ✓ | 62.1 | 64.3 | 32.7 | 40.2 | 63.1 | 67.4 | 71.7 | |||
| Single | ✓ | 57.8 | 65.4 | 31.6 | 33.7 | 60.9 | 64.8 | 71.7 | |||
| Pair | ✓ | ✓ | 74.2 | 74.4 | 38.8 | 55.9 | 77.9 | 79.6 | 82.9 |
GUI Grounding
Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.
See all 61 papers