GUI Grounding

GUI grounding studies how models identify the buttons, fields, and other screen regions that match a user’s instruction, enabling agents to operate websites and desktop or mobile applications. Common approaches train vision-language models to predict coordinates or boxes, or use reinforcement learning, distillation, and element-matching objectives; inference-time methods add zooming, visual search, or verification to refine predictions. Benchmarks such as ScreenSpot-Pro, ScreenSpot-v2, OSWorld-G, and UI-Vision test grounding across interface types and levels of complexity. A central challenge is making predictions reliable under clutter, occlusion, and changes in layout while controlling the extra computation used for refinement.