Autoregressive (AR) vision-language models (VLMs) have long dominated multimodal understanding, reasoning, and graphical user interface (GUI) grounding. Recently, discrete diffusion vision-language models (DVLMs) have shown strong performance in multimodal reasoning, offering bidirectional attention, parallel token generation, and iterative refinement. However, their potential for GUI grounding remains unexplored. In this work, we evaluate whether discrete DVLMs can serve as a viable alternative to AR models for GUI grounding. We adapt LLaDA-V for single-turn action and bounding-box prediction, framing the task as text generation from multimodal input. To better capture the hierarchical structure of bounding-box geometry, we propose a hybrid masking schedule that combines linear and deterministic masking, improving grounding accuracy by up to 6.1 points in Step Success Rate (SSR) over the GUI-adapted LLaDA-V trained with linear masking. Evaluations on four datasets spanning web, desktop, and mobile interfaces show that the adapted diffusion model with hybrid masking consistently outperforms the linear-masked variant and performs competitively with autoregressive counterparts despite limited pretraining. Systematic ablations reveal that increasing diffusion steps, generation length, and block length improves accuracy but also increases latency, with accuracy plateauing beyond a certain number of diffusion steps. Expanding the training data with diverse GUI domains further reduces latency by about 1.3 seconds and improves grounding accuracy by an average of 20 points across benchmarks. These results demonstrate that discrete DVLMs are a promising modeling framework for GUI grounding and represent an important step toward diffusion-based GUI agents.
Figures & tables
Figure 1: Overview of Hybrid Masking Adaptation of LLaDA-V for GUI Grounding. (a) The adapted framework takes a natural-language instruction and a GUI screenshot (either from web, desktop, or mobile interfaces) as input. LLaDA-V trained with the linear masking predicts the action type, optional type_in text, and anchor coordinates (x1,y1) . LLaDA-V model trained with the full deterministic masking predicts the remaining bounding-box coordinates (x2,y2) conditioned on the image, instruction, and anchor. (b) Linear Masking Phase: action and anchor tokens in the response are randomly masked according to the linear masking during the forward corruption process during training for coarse grounding during training. (c) Deterministic Masking Phase: all response tokens are fully masked in the forward corruption process during training, and LLaDA-V predicts the bounding-box extent during denoising.
Figure 2: The above figure shows an instance where LLaDA-V 8B trained with linear and full deterministic masking provides a more accurate target bounding box and action prediction compared to the model trained with linear masking. The green bounding box is the ground truth and the red one is the predicted. Both models receive image and natural language instruction and produce action type and target bounding as text, visualised on the GUI image.
UI Domain
Dataset
Percentage
Samples
Web
Mind2Web
16.67
20K
WebLinX
16.67
20K
OS-Atlas
16.67
20K
Total
50.00
60K
Mobile
OS-Atlas
16.67
20K
Rico Widget Caption
16.67
20K
Table 1: Training data composition for data scaling experiments. The dataset spans web, mobile, and desktop domains, totaling 120K samples.
Inference Parameters
Accuracy
Avg Lat
Diff. Steps
Gen Len
Block Len
Conv Steps
SSR (%)
F1 (%)
sec
32
32
32
13
78.15
99.00
2.56
64
64
64
25
80.67
99.00
4.84
128
128
128
25
80.63
99.87
5.01
Table 2: LLaDA-V 8B fine-tuned only on the Mind2Web training set (7k samples) without cropping and OCR-based target annotation, trained for 10 epochs. The fine-tuned model was evaluated on the Mind2Web test set.
Figure 3: GUI Data Scaling behavior of LLaDA-V 8B trained with Linear Masking: Comparison between LLaDA-V 8B trained on 7k web GUI samples from Mind2Web and 120k mobile, web and desktop GUI samples across four GUI grounding datasets. M2W: Mind2Web, SWT: ScreenSpot-Web-Text, SWI: ScreenSpot-Web-Icon, VWA: Visual Web Arena. Left plot shows Step Sucess Rate (SSR), the center plot shows the number of Converged Steps, and right shows average Inference Latency measured in seconds. Training with large-scale GUI multi-domain data improves SSR, reduces the number of Converged Steps required to produce a highly confident output while reducing Inference Latency, demonstrating better generalization and efficiency across GUI domains.
Inference Parameters
Accuracy
Avg Lat
Diff Steps
Gen Len
Block Len
Conv Steps
SSR (%)
F1 (%)
sec
64
64
64
25
80.67
99
4.84
64
64
64
25
83.31
99
4.46
Table 3: LLaDA-V 8B fine-tuned only on the Mind2Web training set (7K samples) without and with cropping and OCR-based target annotation, trained for 10 epochs. The highlighted row shows the model trained with random cropping and OCR-based target annotation. Both models were tested on the Mind2Web test set.
Dataset
Metric
Phi (3B)
Qwen 2.5 VL (3B)
Qwen 2.5 VL (7B)
LLaDA-V 8B (Lin)
LLaDA-V 8B (Ours)
M2W
Conv Steps
21.00
21.00
21.00
16.00
23.00
SSR (%)
56.80
79.30
81.90
82.40
83.90
F1 (%)
94.40
99.60
99.90
98.50
100.00
Lat. (s)
–
–
1.10
3.02
5.44
SWI
Conv Steps
23.00
23.00
23.00
18.00
23.00
SSR (%)
62.60
79.10
85.40
57.80
63.10
Table 4: AR vs. NAR GUI grounding comparison. The table reports GUI grounding performance on four benchmarks across the listed evaluation metrics. The rightmost column (in grey) highlights our proposed LLaDA-V variant trained with the hybrid linear and deterministic masking schedule.
Inference Parameters
Accuracy
Diff. Steps
Gen Len
Block Len
SSR (%)
F1 (%)
32
32
32
0.00
0.10
64
64
64
0.00
0.12
128
128
128
0.00
0.10
256
64
64
0.00
0.10
Table 5: Zero Shot performance of LLaDA-V 8B on the Mind2Web test set
Inference Parameters
Accuracy
Avg Lat
Diff. Steps
Gen Len
Block Len
Conv Steps
SSR (%)
F1 (%)
sec
32
32
32
13
78.15
99.00
2.56
64
64
64
25
80.67
99.00
4.84
128
128
128
25
80.63
99.87
5.01
256
64
64
25
80.69
99.87
4.84
Table 6: LLaDA-V 8B fine-tuned only on the Mind2Web training set (7k samples) without cropping and OCR based annotation. The table shows the effect of inference parameters on Avg Latency and Accuracy.
Dataset
Metric
LLaDA-V 8B (Lin)
LLaDA-V 8B (Ours)
M2W
Conv Steps
16.00
23.00
11.00
SSR (%)
82.40
83.90
81.00
F1 (%)
98.50
100.00
96.66
Lat. (s)
3.02
5.44
2.74
SWI
Conv Steps
18.00
23.00
11.00
SSR (%)
57.80
63.10
59.60
Table 7: Accuracy–Latency Trade-Off with Hybrid Masking. The table compares LLaDA-V trained with default linear masking (third column) and our hybrid masking (fourth and fifth columns) across four benchmarks. The hybrid model achieves higher SSR but with slightly higher latency, while reducing diffusion steps lowers latency with minimal loss in accuracy.
Inference Parameters
Accuracy
Latency (seconds)
Diff. Steps
Gen Len
Block Len
Conv Steps
SSR (%)
F1 (%)
Lowest
Highest
Avg
8
64
32
4
71.65
99.00
0.99
2.14
1.27
16
64
32
8
74.82
99.00
1.14
2.76
1.47
32
32
32
13
82.50
99.00
2.24
3.43
2.56
64
64
64
25
83.31
99.00
3.72
10.35
4.46
128
128
128
25
82.92
99.00
4.01
28.13
5.02
Table 8: LLaDA-V 8B inference performance across different inference settings. The model was fine-tuned on the Mind2Web training set (7k samples) with cropping and OCR-based target annotation, trained for 10 epochs.
Figure 4: Effect of data annotation quality on grounding accuracy. The figure compares predictions from LLaDA-V trained with default linear masking using the Mind2Web train split with and without cropped images and OCR-based annotations. The top example, trained without OCR text-based annotations and cropping, produces an inaccurate bounding box due to inconsistent icon-level targets and high-resolution inputs. The bottom example, trained with cropped images and OCR-guided text annotations, provides more stable supervision, allowing the model to correctly localize the target element. The green bounding box is the ground truth and the red one is the prediction.
Figure 5: Effect of hybrid masking on bounding-box accuracy. The figure compares predictions from LLaDA-V trained with default linear masking (top) and with the proposed hybrid masking that combines linear and deterministic full masking (bottom). The linear-masked model correctly predicts the action type but generates an inaccurate bounding box, missing the target region. In contrast, the hybrid-masked model, guided by conditional refinement between anchor and extent coordinates, produces a precise bounding box that accurately localizes the target element. The green bounding box is the ground truth and the red one is the prediction.
Diffusion large language models (dLLMs) achieve high decoding efficiency through block-parallel, arbitrary-order generation, making them attractive for latency-sensitive applications. GUI agents represent a natural testbed for this paradigm, as they must repeatedly perceive screen states and emit structured, spatially grounded actions in real time. However, whether dLLMs can be extended into capable multimodal GUI agents while preserving their parallel decoding advantage remains an open question. We present LLaDA-UI, a 16.7B-parameter MoE-based, block-wise diffusion vision-language GUI agent. LLaDA-UI follows a two-stage training pipeline: general multimodal pre-training aligns a native-resolution vision encoder with the LLaDA2.0-mini-base diffusion language backbone, followed by GUI-agent supervised fine-tuning on diverse mobile, desktop, web, and grounding data. Across widely adopted grounding benchmarks and navigation benchmarks spanning multiple platforms, LLaDA-UI substantially outperforms Qwen2.5-VL-7B and surpasses Qwen3-VL-8B on four of six reported GUI benchmarks. These results establish block-wise diffusion as a practical generative paradigm for multimodal GUI agents.
Zhangxuan Gu, Haoxing Chen, Qi Qin +17
AGI Research Center, Inclusion AI · Venus Team · Westlake University
GUI grounding maps natural-language instructions to click locations and is essential for reliable GUI agents. The task remains difficult on high-resolution, densely populated interfaces because a vision-language model (VLM) may recognize a requested control without locating it precisely enough for interaction. Most existing methods provide various forms of localization assistance, but still rely on a direct click prediction, allowing visual ambiguity or an inaccurate initial estimate to propagate to the final result. In this paper, we introduce GUI-Lens, a coarse-to-fine grounding framework that allows a general-purpose VLM to determine the target through active visual observations. Specifically, GUI-Lens extracts OCR text and detected UI components from the screenshot and presents their positions as coordinate references. Using the instruction, the current view, and these references, the VLM selects the region and scale of the next view, which is cropped and enlarged to provide finer visual details. This process continues over successively focused views until the target is determined. Proposed crops and clicks are checked against the instruction throughout the process, and the final local position is mapped back to the original screen coordinates. Experiments on four GUI grounding benchmarks and three general-purpose VLM backends show that GUI-Lens improves overall grounding accuracy by up to 24.9 percentage points and achieves state-of-the-art performance with GPT-5.5.
Zichuan Fu, Shirong Wang, Wenlin Zhang +10
City University of Hong Kong · Westlake University · Tencent Jarvis Lab
Graphical user interface (GUI) grounding is a key capability for computer-use agents, mapping natural-language instructions to actionable regions on the screen. Existing Multimodal Large Language Model (MLLM) approaches typically formulate GUI grounding as a text-based coordinate generation task. However, directly generating precise coordinates from visual inputs is challenging and often data-intensive. A more intuitive strategy is to first identify instruction-relevant visual patches and then determine the exact click location within them. Motivated by recent observations that general MLLMs exhibit native grounding ability embedded in their attention maps, we propose GUI-AIMA, an attention-based and coordinate-free supervised fine-tuning framework for efficient GUI grounding. GUI-AIMA aligns the intrinsic multimodal attention of MLLMs with patch-wise grounding signals. These signals are calculated adaptively for diverse user instructions by multi-head aggregation on simplified query-visual attention matrices. Besides, its coordinate-free manner can easily integrate a plug-and-play zoom-in stage. GUI-AIMA-3B was trained with only 509k samples (around 101k screenshots), demonstrating exceptional data efficiency and verifying that light training can trigger the native grounding capability of MLLMs. It achieves state-of-the-art performance among 3B models, attaining an average accuracy of 61.5% on ScreenSpot-Pro, 92.1% on ScreenSpot-v2, 68.1% on OSWorld-G, 79.1% on MMBench-GUI-L2, and 60.0% on UI-Vision. Project page: https://github.com/sjz5202/GUI-AIMA .