GroundCascade: Selective Second Opinions for GUI Grounding
Organizations: University of Connecticut Storrs, CT, USA · Embry-Riddle Aeronautical University Daytona Beach, FL, USA
Abstract
Graphical user interface (GUI) grounding maps a screenshot and a natural-language instruction to the screen location where an agent should act. Zoomed views can reveal small targets but may lose the context needed to identify them. We introduce GroundCascade, a training-free controller that uses agreement between a grounder's full-screen and zoomed predictions to decide when to consult a complementary model. It selects between zoomed predictions while preserving corroborated full-screen locations. An optional Finalizer predicts from an unmarked joint crop when disagreement remains (Full); omitting it yields the two-stage Selective variant. GroundCascade reaches 85.33% on ScreenSpot-Pro with Indeed-UI-32B/8B/32B and 88.43% on the visible-target OSWorld-G-Refined subset with Qwen3.6-27B/3.8-27B/3.8-27B, gains of 2.66 and 2.35 percentage points over the stronger standalone two-pass grounder. On UI-Vision, GroundCascade reaches 72.20% category-balanced accuracy, with Selective slightly higher at 72.28%. Across these three evaluations, GroundCascade in Full mode uses 2.42-2.73 calls per request, 31-39% fewer than unconditionally acquiring both models' two-pass predictions. At matched second-model call budgets, selective acquisition gains 1.52-2.56 points over standalone Two-pass, compared with 0.22-0.69 points from stratified random acquisition. GroundCascade leads the official ScreenSpot-Pro leaderboard and exceeds reported GUI-specialist comparisons on OSWorld-G-Refined and UI-Vision, subject to differences in external evaluation protocols.
Figures & tables
| Standalone accuracy (%) | GroundCascade accuracy (%) | Inference cost | ||||||
| Benchmark | Context | Two-pass | Selective | Full | Selective | Full | Saved (%) | |
| I32/I8/I32 | SSP | 73.69 | 82.67 | 84.19 | 85.33 | 2.49 | 2.56 | 36.11 |
| Q36/Q38/Q38 | OSG-R | 84.90 | 86.08 | 88.04 | 88.43 | 2.37 | 2.42 | 39.28 |
| UIV | 66.02 | 69.72 | 72.28 | 72.20 | 2.62 | 2.73 | 31.20 | |
| I32 first: I32/8B/I32 | 8B first: 8B/I32/I32 | |||||
| 8B model | Acc. | Calls | I32 | Acc. | Calls | I32 |
| I8 | 85.33 | 2.56 | 2.06 | 83.93 | 2.54 | 0.54 |
| K8 | 85.01 | 2.55 | 2.06 | 83.68 | 2.56 | 0.56 |
| Benchmark | requests | Gate gain | Random mean | Random 95% |
|---|---|---|---|---|
| SSP | 390 | +1.52 | +0.28 | [-0.32, +0.89] |
| OSG-R | 97 | +1.96 | +0.22 | [-0.39, +0.78] |
| UIV | 1728 | +2.56 | +0.69 | [+0.33, +1.04] |
| UIV category | Context | Two-pass | Selective | Full | calls |
|---|---|---|---|---|---|
| Basic | 71.16 | 72.63 | 75.45 | 75.68 | 138 |
| Functional | 66.48 | 68.74 | 71.67 | 71.78 | 149 |
| Spatial | 60.41 | 67.80 | 69.72 | 69.15 | 308 |
| Mean | 66.02 | 69.72 | 72.28 | 72.20 | 595 |
| Benchmark | req. | Rep. | Reg. | pp | |
|---|---|---|---|---|---|
| SSP | I32/I8/I32 | 99 | 27 | 9 | 1.14 |
| OSG-R | Q36/Q38/Q38 | 21 | 3 | 1 | +0.39 |
| UIV | Q36/Q38/Q38 | 595 | 70 | 75 | -0.08 |
| Metric | SSP | OSG-R | UIV |
|---|---|---|---|
| precision | 157/390 | 42/97 | 1038/1728 |
| recall | 157/274 | 42/71 | 1038/1662 |
| precision | 69/99 | 12/21 | 405/595 |
| recall | 69/133 | 12/32 | 405/899 |
| requests | 99 | 21 | 595 |
| Before : correct (%) | 30.3 | 42.9 | 31.9 |
GUI Grounding
Graphical user interface (GUI) grounding maps instructions to the screen elements or locations an agent should interact with.
See all 60 papers