cs.CLNov 15, 2026

Rank, Don't Regress: A 279M Text Ranker Beats 2B and 7B VLMs on Referential Urdu GUI Grounding

Authors: Hamza Raziq Khan, Aimal Khan

Organizations: FAST NUCES, Peshawar, Pakistan

Abstract

UI grounding asks a model to find the screen element that a user instruction refers to. Most current systems use a vision-language model (VLM) that reads a screenshot and outputs pixel coordinates. We study this task on Urdu mobile web pages, a setting with no published UI grounding results. Instead of regressing coordinates, we take the candidate elements from the page DOM and train a 279M-parameter cross-encoder (mDeBERTa-v3-base) to pick one. The click point is the centre of the picked element's box, so it is exact whenever the right element is picked. We built a frozen test set of 304 instructions from six Urdu news sites; 149 of them (49%) are referential and never quote the full target text. Test domains were banned from training data before collection started. The ranker scores 73.4% point-in-box accuracy, against 29.6% for UI-TARS-2B and 23.7% for Qwen2.5-VL-7B, both zero-shot. On the referential slice the ranker scores 65.8%, against 30.2% for a string-matching baseline that needs no training. In a controlled comparison at small scale, with training data drawn from the same pages, a coordinate-regression model of similar size scores 1.6%. Adding element geometry as input tokens gives the best variant, at 75.7%. The ranker's errors concentrate in two places: elements that share identical text, and ordinal instructions such as "the fourth link".

Figures & tables

Explore similar work

CardsList
  1. RankGround: Efficient High-Resolution GUI Grounding via Lightweight Reranker-Guided Crop Selection

    Sep 16, 2026Liyang Fan, Xinping Bi, Yitai Li +3Efficient Multimodal InferenceMultimodal Reranking

  2. PointerBench: A Diagnostic Benchmark for Fine-Grained GUI Grounding

    Oct 22, 2026Valentino Sacco, Saurabh Pandey, Alexander Florian Jercher +1GUI GroundingMultimodal Model Evaluation