Organizations: Zhejiang Key Laboratory of Accessible Perception and Intelligent Systems, Zhejiang University · College of Computer Science and Technology, Zhejiang University · AntGroup
Autonomous Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) enable digital automation on end-user devices. While scaling both parameters and data has yielded substantial gains, advanced methods still suffer from prohibitive deployment costs on resource-constrained devices. When facing complex in-the-wild scenarios, lightweight GUI agents are bottlenecked by limited capacity and poor task scalability under end-to-end episodic learning, impeding adaptation to multi-agent systems (MAS), while training multiple skill-specific experts remains costly. Can we strike an effective trade-off in this cost-scalability dilemma, enabling lightweight MLLMs to participate in realistic GUI workflows? To address these challenges, we propose the LAMO framework, which endows a lightweight MLLM with GUI-specific knowledge and task scalability, allowing multi-role orchestration to expand its capability boundary for GUI automation. LAMO combines role-oriented data synthesis with a two-stage training recipe: (i) supervised fine-tuning with Perplexity-Weighted Cross-Entropy optimization for knowledge distillation and visual perception enhancement, and (ii) reinforcement learning for role-oriented cooperative exploration. With LAMO, we develop a task-scalable native GUI agent, LAMO-3B, supporting monolithic execution and MAS-style orchestration. When paired with advanced planners as a plug-and-play policy executor, LAMO-3B can continuously benefit from planner advances, enabling a higher performance ceiling. Extensive static and online evaluations validate the effectiveness of our design.
Figures & tables
Figure 1: Overview of the Lightweight Agent Multi - role Orchestration (LAMO) framework. LAMO integrates role-oriented data synthesis with a two-round training recipe to enhance screen perception, long-horizon reasoning, and multi-role orchestration. It enables versatile inference modes, allowing a lightweight MLLM to function as end-to-end monolithic agent, coordinated MAS, or plug-and-play executor paired with advanced planners. Such scalability expands the capability boundary of lightweight MLLMs in GUI automation via MAS adaptation.
Method
Development
Creative
CAD
Scientific
Office
MacOS
Avg.
Text
Icon
Avg.
Text
Icon
Avg.
Text
Icon
Avg.
Text
Icon
Avg.
Text
Icon
Avg.
Text
Icon
Avg.
Text
Icon
Avg.
Generic Models
Claude Computer Use Anthropic (2024)
22.0
3.9
12.6
25.9
3.4
16.8
14.5
3.7
11.9
33.9
15.8
25.8
30.1
16.3
26.9
11.0
4.5
8.1
23.4
7.1
17.1
Qwen2.5-VL-3B Bai et al. (2025b)
38.3
3.4
21.4
40.9
4.9
25.8
22.3
6.3
18.4
44.4
10.0
29.5
48.0
17.0
40.9
33.6
4.5
20.4
37.8
6.6
25.9
Qwen2.5-VL-7B Bai et al. (2025b)
51.9
4.8
29.1
36.9
8.4
24.9
17.8
1.6
13.8
48.6
8.2
31.1
53.7
18.9
45.7
34.6
7.9
22.4
39.9
7.6
26.8
Kimi-VL-A3B-Instruct Team et al. (2025)
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
34.5
Table 1: Grounding accuracy on ScreenSpot-pro. Bold represents the best results, underlined is the second best.
Method
AC–Low
AC–High
Type
Ground.
SR
Type
Ground.
SR
GPT-4o Hurst et al. (2024)
74.3
38.7
28.4
63.1
30.9
21.2
OS-Genesis-7B Sun et al. (2025)
90.7
–
74.2
66.2
–
44.5
GUI-R1-3B Luo et al. (2025)
–
–
–
58.0
56.2
46.6
GUI-R1-7B Luo et al. (2025)
–
–
–
71.6
65.6
51.7
SeeClick Cheng et al. (2024)
93.0
73.4
75.0
82.9
62.9
59.1
Table 2: Performance comparison on AndroidControl-Low (AC-Low) and AndroidControl-High (AC-High).
Method
Success Rate
Qwen2.5-VL-3B Bai et al. (2025b)
34.6
OS-Atlas-7B Wu et al. (2024)
35.2
AgentCPM-GUI-8B Zhang et al. (2025b)
37.8
Qwen2.5-VL-7B Bai et al. (2025b)
54.0
UI-TARS-7B Qin et al. (2025)
58.7
UI-S1-7B Lu et al. (2025b)
60.9
Table 3: Performance on MiniWob++.
Method
Success Rate
Aguvis-72B Xu et al. (2024)
26.1
Claude Computer Use Anthropic (2024)
27.9
Gemini-2.5-pro Gemini Team (2025)
31.0
Qwen2.5-VL-32B Bai et al. (2025b)
31.5
GPT-4o + Aria-UI Yang et al. (2025b)
44.8
UI-TARS-72B Qin et al. (2025)
46.6
Table 4: Performance on AndroidWorld.
Planner
Policy Executor
Success Rate
Gemini-2.5-pro
Generic Models
Qwen2.5-VL-32B Bai et al. (2025b)
43.6
Qwen3-VL-4B Bai et al. (2025a)
33.3
GUI-specific Models
UI-TARS-1.5-7B Seed (2025)
28.2
InfiGUI-R1-3B Liu et al. (2025)
10.3
Table 5: Comparison of different executors in OSWorld. We adopt the official split of 39 tasks (see App. A.2 ).
Method
SP
SP-v2
SP-pro
MW
LAMO-3B
84.3
86.4
36.1
50.0
— w/o ILG
82.6 (–2.1%)
83.2 (–3.8%)
26.8 (–34.7%)
48.7 (–2.7%)
SFT stage
83.4 (–1.1%)
83.9 (–3.0%)
27.2 (–32.7%)
40.8 (–22.5%)
— w/o PWCE
82.9 (–1.7%)
83.5 (–3.5%)
26.1 (–38.3%)
39.4 (–26.9%)
Qwen2.5-VL-3B
78.3 (–7.7%)
81.3 (–6.3%)
23.9 (–51.0%)
34.6 (–44.5%)
Table 6: Ablation results on ScreenSpot (SP), ScreenSpot-v2 (SP-v2), ScreenSpot-pro (SP-pro), and MiniWob++ (MW). Numbers in parentheses indicate the relative performance drop (%) vs. LAMO-3B.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
PC/Web Environment
Atomic Action
Description
pyautogui.click(x=x1, y=y1)
Click/Tap the UI element at screen coordinate (x1,y1) .
pyautogui.doubleClick(x=x1, y=y1)
Double-click at (x1,y1) .
pyautogui.rightClick(x=x1, y=y1)
Right-click at (x1,y1) .
pyautogui.moveTo(x=x1, y=y1)
Move the cursor to (x1,y1) .
pyautogui.dragTo(x=x1, y=y1)
Drag the cursor to (x1,y1) .
Appendix
Table 7: Action space for LAMO-3B.
Domain
Tasks
chrome
4
gimp
2
libreoffice_calc
3
libreoffice_impress
2
libreoffice_writer
2
multi_apps
17
Appendix
Table 8: Statistics of the OSWorld official split small-scale test tasks.
Figure 2: A toy example of the ILG data augmentation workflow.
Method
Mobile
Desktop
Web
Avg.
Text
Icon
Text
Icon
Text
Icon
GPT-4o Hurst et al. (2024)
30.5
23.2
20.6
19.4
11.1
7.8
18.8
CogAgent Hong et al. (2024)
67.0
24.0
74.2
20.0
70.4
28.6
47.4
SeeClick Cheng et al. (2024)
78.4
50.7
70.1
29.3
55.2
32.5
55.1
R-VLM Park et al. (2025b)
85.0
61.1
81.4
52.8
66.5
51.4
66.3
MP-GUI Wang et al. (2025b)
86.8
65.9
70.8
56.4
58.3
46.6
64.1
Appendix
Table 9: Performance comparison on ScreenSpot.
Method
Mobile
Desktop
Web
Avg.
Text
Icon
Text
Icon
Text
Icon
SeeClick Cheng et al. (2024)
78.4
50.7
70.1
29.3
55.2
32.5
55.1
GPT-4o + SeeClick Cheng et al. (2024)
85.2
58.8
79.9
37.1
72.7
30.1
63.6
OS-Atlas-4B Wu et al. (2024)
87.2
59.7
72.7
46.4
85.9
63.1
71.9
GPT-4o + OS-Atlas-4B Wu et al. (2024)
95.5
75.8
79.4
49.3
90.2
66.5
79.1
InternVL3-8B Zhu et al. (2025)
–
–
–
–
–
–
81.4
Appendix
Table 10: Performance comparison on ScreenSpot-v2.
Figure 3: Illustrative bad case when using Gemini-2.5-pro as the planner and LAMO-3B as the policy executor in AndroidWorld. The goal of the task is "Create a playlist titled ’Ultimate Fails Series’ with the following files in VLC (located in Internal Memory/VLCVideos), in order: highlight_41_4K_2023_03_30.mp4, scene_68_4K_copy.mp4" . The results indicate that LAMO-3B precisely aligns and executes the planner’s instruction. At this stage, the overall performance of the framework is primarily constrained by the planner’s decision-making quality: an erroneous plan can directly precipitate task failure. Specifically, at Step 10, the planner attempted to enter the playlist name without first focusing the text input field; it then failed to recognize the missing/incorrect entry and proceeded to trigger the save action, ultimately causing task failure.
Figure 4: A case in AndroidWorld corresponding to the same task as in Figure 3 , but using GPT-5 as the planner. Compared with the previously shown episode, GPT-5 exhibits markedly enhanced planning capabilities, benefiting from richer world knowledge that enables more accurate interpretation of end-user devices usage and provides more reasonable high-level planning, while our LAMO-3B act as a plug-and-play policy executor for low-level GUI execution. In particular, at Step 9 of this episode, GPT-5 encountered the same scenario as Gemini-2.5-pro but correctly inferred that the input field on the screen had not been activated and therefore first clicked on the input box before entering the text. As the planner improves, the overall performance of the framework correspondingly advances.
Figure 5: Case Study: LAMO-3B achieves GUI automation on the MiniWob++ benchmark through end-to-end reasoning. The results demonstrate LAMO-3B’s capabilities in screen perception, long-horizon reasoning, and history-awareness, as well as precise pixel-level GUI interaction.
Figure 6: Case Study: LAMO-3B orchestrates a multi-agent system (MAS) to produce GUI automation on the MiniWob++ benchmark. These results demonstrate the multi-role orchestration capabilities of LAMO-3B: the Observer facilitates fine-grained perception of screen details, while the Planner provides dynamic task planning alongside actionable tips to mitigate execution errors. Furthermore, the Allocator analyzes contextual information to predict the optimal actions for the current screen state, and the Executor precisely aligns with the Allocator’s instructions to perform low-level GUI interactions.
Figure 7: Example visualizations from ScreenSpot-pro, illustrating LAMO-3B’s ability to perceive on-screen visual clues and spatial context. Moreover, our semantic enhancement strategy for screen grounding (Sec. 4.1 ), addressing the first challenge in SG via data distillation, fosters a profound multimodal understanding of UI environments and element descriptions. This empowers LAMO-3B to precisely ground the target element while simultaneously capturing spatial context and intricate layout details.
Figure 8: Example visualizations from ScreenSpot-pro, demonstrating LAMO-3B’s ability to comprehend multilingual on-screen information.
Figure 9: A bad case on OSWorld. In this instance, LAMO-3B suffered from an over-reliance on textual semantic matching—mistakenly attending to the "Total Earned" header while overlooking the spatial constraints for cell E3. Tabular interfaces, which often feature extensive sparse regions, amplify this susceptibility, leading to grounding misalignment in long-horizon spreadsheet manipulation.
Figure 10: A bad case on OSWorld. In this instance, the Planner’s instruction is to ’Click the "Replace All" button, which is the icon to the right of the "Replace" input field.’ However, the ground truth target is a purely graphical element devoid of local textual clues, necessitating that the policy executor possesses prior functional knowledge of the software’s interface. Although LAMO-3B successfully identifies the target’s Region-of-Interest, its lack of such software-specific priors ultimately leads to a grounding failure.
Figure 11: A complete episode on OSWorld. The goal of the task: "Please help me set the current user’s line length for code wrapping to 50 characters in VS Code." In this episode, LAMO-3B precisely aligns the planner’s instructions and accurately executes a diverse set of atomic actions in pixel-level, demonstrating both the robustness of LAMO-3B as a policy executor and the richness of its action space (Tab. 7 ), which together enable scalable execution of high - level plans produced by the planner across a broad spectrum of long-horizon GUI tasks.
We present LiteGUI, a new framework for building lightweight GUI agents. GUI interaction poses unique challenges due to the long-horizon nature of complex tasks and the existence of multiple valid interaction paths, which are difficult for lightweight GUI agents to handle effectively. To address these challenges, LiteGUI introduces a two-stage post-training framework operating on logged GUI states. First, we propose Guided On-Policy Distillation, which provides training-time privileged teacher guidance at each GUI state by selecting the most-matched valid action from human-verified multi-solution action annotations, while leaving the student's on-policy rollout unchanged. Second, we develop Multi-Solution Dual-Level GRPO, which combines action-level supervision with history-conditioned, per-state planning-quality supervision while accounting for multiple valid actions at each logged GUI state. Together, these stages support multi-step GUI tasks without requiring the student to reproduce fixed demonstration trajectories. We further develop a scalable data generation pipeline and a corresponding multi-solution dataset, Lite-Dataset, to support the training and evaluation of GUI agents, addressing a gap in the existing literature. Extensive experiments across multiple GUI benchmarks demonstrate that LiteGUI substantially improves the performance of lightweight GUI agents over existing training paradigms, including SFT, on-policy distillation and GRPO, and achieves competitive or superior performance compared with substantially larger models.
Modern GUI agents typically rely on a model-centric and step-wise interaction paradigm, where LLMs must re-interpret the UI and re-decide actions at every screen, which is fragile in long-horizon tasks. In this paper, we propose Executable Agentic Memory (EAM), a structured Knowledge Graph (KG) that shifts GUI planning from free-form generation to a robust retrieval-and-execution process. Our approach includes a sample-efficient memory construction pipeline using state-aware DFS and action-group mining to compress multi-step routines. To ensure efficient planning, we introduce a value-guided graph search where a lightweight Q-function model steers Monte Carlo Tree Search (MCTS) over the KG. We theoretically establish bias-consistency for the Q-model and derive sample complexity bounds for path recovery. Empirically, EAM outperforms state-of-the-art baselines like UI-TARS-7B by up to 19.6% on AndroidWorld, while reducing token costs 6× relative to GPT-4o. With a 2.8s average latency, EAM enables reliable, quick, and long-horizon GUI automation.
Zerui Qin, Sheng Yue, Xingyuan Hua +2
1Tsinghua University, China · 2Sun Yat-sen University, China
Multimodal Large Language Model (MLLM)-based Graphical User Interface (GUI) agents develop rapidly, with visual grounding that maps natural language instructions to target UI elements serving as the core capability. Existing GUI agents typically fine-tune MLLM on massive datasets to handle challenges in understanding instructions and UI interfaces, which not only incurs high data annotation costs but also makes performance dependent on data quality and distribution. To avoid such cumbersome yet ineffective training, we notice that complex UI interfaces can be decomposed into basic visual elements directly understandable by common MLLMs. Consequently, we propose ZoomUI that leverages inference scaling to guide common MLLMs in progressively anchor instruction elements to increasingly detailed interface elements. Specifically, ZoomUI first optimizes the latent thinking to transform original instruction into element visual features description, and subsequently leverages internal attention to iteratively zoom in target element interface region. Evaluations on extensive benchmarks demonstrate that ZoomUI reaches or even surpasses SOTA baselines.
Ziwei Liu, Tao Feng, Borui Kang +2
Department of Computer Science and Technology, Sichuan University, China · Department of Computer Science and Technology, Tsinghua University, China · School of Computer Science, Nanjing University, China +1