LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents

Zhangxuan Gu; Haoxing Chen; Qi Qin; Yi Xin; Kai Gan; Lin Liu; Long Cui; Xiaomei Wang

cs.CVSep 9, 2026

LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agents

Authors: Zhangxuan Gu, Haoxing Chen, Qi Qin, Yi Xin, Kai Gan, Lin Liu, Long Cui, Xiaomei Wang,

Organizations: AGI Research Center, Inclusion AI · Venus Team · Westlake University

Abstract

Diffusion large language models (dLLMs) achieve high decoding efficiency through block-parallel, arbitrary-order generation, making them attractive for latency-sensitive applications. GUI agents represent a natural testbed for this paradigm, as they must repeatedly perceive screen states and emit structured, spatially grounded actions in real time. However, whether dLLMs can be extended into capable multimodal GUI agents while preserving their parallel decoding advantage remains an open question. We present LLaDA-UI, a 16.7B-parameter MoE-based, block-wise diffusion vision-language GUI agent. LLaDA-UI follows a two-stage training pipeline: general multimodal pre-training aligns a native-resolution vision encoder with the LLaDA2.0-mini-base diffusion language backbone, followed by GUI-agent supervised fine-tuning on diverse mobile, desktop, web, and grounding data. Across widely adopted grounding benchmarks and navigation benchmarks spanning multiple platforms, LLaDA-UI substantially outperforms Qwen2.5-VL-7B and surpasses Qwen3-VL-8B on four of six reported GUI benchmarks. These results establish block-wise diffusion as a practical generative paradigm for multimodal GUI agents.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of LLaDA-UI. Left: performance of LLaDA-UI and baselines on GUI-agent benchmarks. Right: a real AndroidWorld prediction case showing how the complete model output is progressively denoised into the final response. Best viewed in Adobe Acrobat Reader.

Figure 2: The framework of LLaDA-UI. GUI observations from web, mobile, and desktop environments are encoded at native resolution and interleaved with task and interaction-history tokens. The LLaDA2.0 block-wise diffusion decoder progressively denoises masked output tokens into structured reasoning and executable actions.

Stages&Objective	S0: Vision-Language Alignment	S1: Perception Enhancement	S2: Multi-task Pre-training
Data	Image Caption	+ OCR, Grounding Counting, Pure text Interleaved data	+ Multimodal Reasoning Multimodal VQA
Tokens	5B	70B	70B
Sequence length	16K	16K	16K
Trainable Params	Projector	All	All

Table 1: Overview of the foundational pretraining stages, including data composition, token scale, sequence length, and trainable components.

Figure 3: Overview of the GUI data-generation pipeline. Task posing is followed by sub-capability decomposition, capability composition, environment execution, and validation into training trajectories.

	Autoregressive Models				Diffusion Models
	Qwen2.5-VL	Qwen2.5-VL	Qwen3-VL	Qwen3-VL	LLaDA-V	SDAR-VL	LLaDA-UI
Architecture	Dense	Dense	Dense	Dense	Dense	Dense	MoE
Base LLM	Qwen2.5	Qwen2.5	Qwen3	Qwen3	LLaDA	Qwen3	LLaDA2.0
Pre-train tokens	4.1T	4.1T	2.2T	2.2T	–	–	145B
Reasoning
$\mathrm{MMMU}_{\mathrm{val}}$	46.4	51.3	53.4	69.6 69.6	48.6	53.0	56.2

Table 2: General multimodal and text results before GUI-agent SFT. AR and diffusion models are shown in blue and orange , respectively.

	Grounding		Mobile		Desktop	Web
Model	SS-V2	SS-Pro	AndroidWorld	MobileWorld	OSWorld-V	WebVoyager
General-purpose vision-language models
GPT-4o	20.1	0.8	30.6	-	-	-
Qwen2.5-VL-7B	86.0	26.8	25.5	7.0	3.0	11.0
Qwen3-VL-8B	93.0	52.7	47.9	9.4	33.9	45.2
Qwen3.5-9B	94.0	65.2 65.2	57.8	17.9	41.8 41.8	46.6

Table 3: GUI-agent results (%). ScreenSpot-V2 and ScreenSpot-Pro evaluate grounding; AndroidWorld and MobileWorld evaluate mobile navigation; OSWorld-Verified evaluates desktop navigation; and WebVoyager evaluates web navigation. LLaDA-UI’s WebVoyager score uses gpt-4o-2024-11-20 as the judge. Bold and underlined values indicate the best and second-best reported results in each column, respectively.

Decode setting	EOS policy	Block	Steps	Success	Avg. actions
Baseline	Early stop	32	32	42.7% (50/116)	14.9
Two-stage decoding	Early stop	32	32	50.9% (58/115)	13.7
No-EOS	Disabled	32	32	52.6% (61/116) 52.6% (61/116)	15.8
Larger block/fewer steps	Default	64	16	33.0% (38/115)	15.1

Table 4: AndroidWorld decoding ablation on an intermediate LLaDA-UI checkpoint.

Domain	Model	Mean	Median	Range	Tokens	Mean speedup
Web	LLaDA-UI	4.764	4.801	4.474–5.156	144	3.579 $\times$
	Qwen3-VL-8B	17.050	16.458	15.873–20.044	55.2	–
OSWorld	LLaDA-UI	6.379	6.313	5.911–6.741	129	6.826 $\times$
	Qwen3-VL-8B	43.545	42.787	42.083–47.330	70.0	–
MobileWorld	LLaDA-UI	5.921	5.944	5.434–6.299	67	8.951 $\times$
	Qwen3-VL-8B	53.000	52.394	51.882–54.298	65.2	–

Table 5: Model API latency over five non-cache-hit calls per domain. Each cell reports mean / median / range in seconds. These fixed-input measurements characterize per-call latency rather than throughput.

Metric	Rate	Count
Task success	51.72%	60/116 tasks
Parse rate	98.65%	1,824/1,849 steps
Schema-valid rate	98.59%	1,823/1,849 steps
Coordinate-valid rate	100.00%	1,457/1,457 actions
Exact-repetition rate	18.82%	348/1,849 steps
Multiple-action rate	1.30%	24/1,849 steps

Table 6: Structured-action validity on AndroidWorld for an intermediate LLaDA-UI checkpoint. These diagnostic values are not the final benchmark result.

Optimal actions	Tasks	Successes	Success rate
1–5	54	34	62.96%
6–10	35	17	48.57%
11–20	20	8	40.00%
20+	7	1	14.29%

Table 7: AndroidWorld success for the same intermediate LLaDA-UI checkpoint, stratified by annotated optimal task length.

GUI Agents

Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.

See all 172 papers

Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding
Mar 27, 2026Shrinidhi Kumbhar, Haofu Liao, Srikar Appalaraju +1Diffusion VLMs GUI Grounding
Arizona State University · AWS Agentic AI
LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model
Apr 22, 2026Inclusion AI, Tiwei Bie, Haoxing Chen +15Unified Multimodal Models Multimodal Generation
AGI Research Center, Inclusion AI
LaViDa: A Large Diffusion Language Model for Multimodal Understanding
May 22, 2025Shufan Li, Konstantinos Kallidromitis, Hritik Bansal +7Multimodal Understanding Vision-Language Models
1UCLA · 2Panasonic AI Research · 4Salesforce Research +1

Mar 27, 2026J/K move · Enter open · S save

Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding

Shrinidhi Kumbhar, Haofu Liao, Srikar Appalaraju, Kunwar Yashraj Singh

Arizona State University · AWS Agentic AI

Diffusion VLMs GUI Grounding Vision-Language Models GUI Agents Multimodal Diffusion Models

Open paper AskPDF arXiv

Autoregressive (AR) vision-language models (VLMs) have long dominated multimodal understanding, reasoning, and graphical user interface (GUI) grounding. Recently, discrete diffusion vision-language models (DVLMs) have shown strong performance in multimodal reasoning, offering bidirectional attention, parallel token generation, and iterative refinement. However, their potential for GUI grounding remains unexplored. In this work, we evaluate whether discrete DVLMs can serve as a viable alternative to AR models for GUI grounding. We adapt LLaDA-V for single-turn action and bounding-box prediction, framing the task as text generation from multimodal input. To better capture the hierarchical structure of bounding-box geometry, we propose a hybrid masking schedule that combines linear and deterministic masking, improving grounding accuracy by up to 6.1 points in Step Success Rate (SSR) over the GUI-adapted LLaDA-V trained with linear masking. Evaluations on four datasets spanning web, desktop, and mobile interfaces show that the adapted diffusion model with hybrid masking consistently outperforms the linear-masked variant and performs competitively with autoregressive counterparts despite limited pretraining. Systematic ablations reveal that increasing diffusion steps, generation length, and block length improves accuracy but also increases latency, with accuracy plateauing beyond a certain number of diffusion steps. Expanding the training data with diverse GUI domains further reduces latency by about 1.3 seconds and improves grounding accuracy by an average of 20 points across benchmarks. These results demonstrate that discrete DVLMs are a promising modeling framework for GUI grounding and represent an important step toward diffusion-based GUI agents.

Abstract

GUI Agents

Explore similar work

Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding

LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model

LaViDa: A Large Diffusion Language Model for Multimodal Understanding