cs.AIApr 15, 2026
SaveTowards Scalable Lightweight GUI Agents via Multi-role Orchestration
Organizations: Zhejiang Key Laboratory of Accessible Perception and Intelligent Systems, Zhejiang University · College of Computer Science and Technology, Zhejiang University · AntGroup
Abstract
Autonomous Graphical User Interface (GUI) agents powered by Multimodal Large Language Models (MLLMs) enable digital automation on end-user devices. While scaling both parameters and data has yielded substantial gains, advanced methods still suffer from prohibitive deployment costs on resource-constrained devices. When facing complex in-the-wild scenarios, lightweight GUI agents are bottlenecked by limited capacity and poor task scalability under end-to-end episodic learning, impeding adaptation to multi-agent systems (MAS), while training multiple skill-specific experts remains costly. Can we strike an effective trade-off in this cost-scalability dilemma, enabling lightweight MLLMs to participate in realistic GUI workflows? To address these challenges, we propose the LAMO framework, which endows a lightweight MLLM with GUI-specific knowledge and task scalability, allowing multi-role orchestration to expand its capability boundary for GUI automation. LAMO combines role-oriented data synthesis with a two-stage training recipe: (i) supervised fine-tuning with Perplexity-Weighted Cross-Entropy optimization for knowledge distillation and visual perception enhancement, and (ii) reinforcement learning for role-oriented cooperative exploration. With LAMO, we develop a task-scalable native GUI agent, LAMO-3B, supporting monolithic execution and MAS-style orchestration. When paired with advanced planners as a plug-and-play policy executor, LAMO-3B can continuously benefit from planner advances, enabling a higher performance ceiling. Extensive static and online evaluations validate the effectiveness of our design.
Figures & tables
Figure 1: Overview of the Lightweight Agent Multi - role Orchestration (LAMO) framework. LAMO integrates role-oriented data synthesis with a two-round training recipe to enhance screen perception, long-horizon reasoning, and multi-role orchestration. It enables versatile inference modes, allowing a lightweight MLLM to function as end-to-end monolithic agent, coordinated MAS, or plug-and-play executor paired with advanced planners. Such scalability expands the capability boundary of lightweight MLLMs in GUI automation via MAS adaptation.
| Method | Development | Creative | CAD | Scientific | Office | MacOS | Avg. | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Text | Icon | Avg. | Text | Icon | Avg. | Text | Icon | Avg. | Text | Icon | Avg. | Text | Icon | Avg. | Text | Icon | Avg. | Text | Icon | Avg. | |
| Generic Models | |||||||||||||||||||||
| Claude Computer Use Anthropic (2024) | 22.0 | 3.9 | 12.6 | 25.9 | 3.4 | 16.8 | 14.5 | 3.7 | 11.9 | 33.9 | 15.8 | 25.8 | 30.1 | 16.3 | 26.9 | 11.0 | 4.5 | 8.1 | 23.4 | 7.1 | 17.1 |
| Qwen2.5-VL-3B Bai et al. (2025b) | 38.3 | 3.4 | 21.4 | 40.9 | 4.9 | 25.8 | 22.3 | 6.3 | 18.4 | 44.4 | 10.0 | 29.5 | 48.0 | 17.0 | 40.9 | 33.6 | 4.5 | 20.4 | 37.8 | 6.6 | 25.9 |
| Qwen2.5-VL-7B Bai et al. (2025b) | 51.9 | 4.8 | 29.1 | 36.9 | 8.4 | 24.9 | 17.8 | 1.6 | 13.8 | 48.6 | 8.2 | 31.1 | 53.7 | 18.9 | 45.7 | 34.6 | 7.9 | 22.4 | 39.9 | 7.6 | 26.8 |
| Kimi-VL-A3B-Instruct Team et al. (2025) | – | – | – | – | – | – | – | – | – | – | – | – | – | – | – | – | – | – | – | – | 34.5 |
Table 1: Grounding accuracy on ScreenSpot-pro. Bold represents the best results, underlined is the second best.
| Method | AC–Low | AC–High | ||||
|---|---|---|---|---|---|---|
| Type | Ground. | SR | Type | Ground. | SR | |
| GPT-4o Hurst et al. (2024) | 74.3 | 38.7 | 28.4 | 63.1 | 30.9 | 21.2 |
| OS-Genesis-7B Sun et al. (2025) | 90.7 | – | 74.2 | 66.2 | – | 44.5 |
| GUI-R1-3B Luo et al. (2025) | – | – | – | 58.0 | 56.2 | 46.6 |
| GUI-R1-7B Luo et al. (2025) | – | – | – | 71.6 | 65.6 | 51.7 |
| SeeClick Cheng et al. (2024) | 93.0 | 73.4 | 75.0 | 82.9 | 62.9 | 59.1 |
Table 2: Performance comparison on AndroidControl-Low (AC-Low) and AndroidControl-High (AC-High).
| Method | Success Rate |
|---|---|
| Qwen2.5-VL-3B Bai et al. (2025b) | 34.6 |
| OS-Atlas-7B Wu et al. (2024) | 35.2 |
| AgentCPM-GUI-8B Zhang et al. (2025b) | 37.8 |
| Qwen2.5-VL-7B Bai et al. (2025b) | 54.0 |
| UI-TARS-7B Qin et al. (2025) | 58.7 |
| UI-S1-7B Lu et al. (2025b) | 60.9 |
Table 3: Performance on MiniWob++.
| Method | Success Rate |
|---|---|
| Aguvis-72B Xu et al. (2024) | 26.1 |
| Claude Computer Use Anthropic (2024) | 27.9 |
| Gemini-2.5-pro Gemini Team (2025) | 31.0 |
| Qwen2.5-VL-32B Bai et al. (2025b) | 31.5 |
| GPT-4o + Aria-UI Yang et al. (2025b) | 44.8 |
| UI-TARS-72B Qin et al. (2025) | 46.6 |
Table 4: Performance on AndroidWorld.
| Planner | Policy Executor | Success Rate |
|---|---|---|
| Gemini-2.5-pro | Generic Models | |
| Qwen2.5-VL-32B Bai et al. (2025b) | 43.6 | |
| Qwen3-VL-4B Bai et al. (2025a) | 33.3 | |
| GUI-specific Models | ||
| UI-TARS-1.5-7B Seed (2025) | 28.2 | |
| InfiGUI-R1-3B Liu et al. (2025) | 10.3 | |
Table 5: Comparison of different executors in OSWorld. We adopt the official split of 39 tasks (see App. A.2 ).
| Method | SP | SP-v2 | SP-pro | MW |
|---|---|---|---|---|
| LAMO-3B | 84.3 | 86.4 | 36.1 | 50.0 |
| — w/o ILG | 82.6 (–2.1%) | 83.2 (–3.8%) | 26.8 (–34.7%) | 48.7 (–2.7%) |
| SFT stage | 83.4 (–1.1%) | 83.9 (–3.0%) | 27.2 (–32.7%) | 40.8 (–22.5%) |
| — w/o PWCE | 82.9 (–1.7%) | 83.5 (–3.5%) | 26.1 (–38.3%) | 39.4 (–26.9%) |
| Qwen2.5-VL-3B | 78.3 (–7.7%) | 81.3 (–6.3%) | 23.9 (–51.0%) | 34.6 (–44.5%) |
Table 6: Ablation results on ScreenSpot (SP), ScreenSpot-v2 (SP-v2), ScreenSpot-pro (SP-pro), and MiniWob++ (MW). Numbers in parentheses indicate the relative performance drop (%) vs. LAMO-3B.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| PC/Web Environment | |
|---|---|
| Atomic Action | Description |
| pyautogui.click(x=x1, y=y1) | Click/Tap the UI element at screen coordinate . |
| pyautogui.doubleClick(x=x1, y=y1) | Double-click at . |
| pyautogui.rightClick(x=x1, y=y1) | Right-click at . |
| pyautogui.moveTo(x=x1, y=y1) | Move the cursor to . |
| pyautogui.dragTo(x=x1, y=y1) | Drag the cursor to . |
Table 7: Action space for LAMO-3B.
| Domain | Tasks |
|---|---|
| chrome | 4 |
| gimp | 2 |
| libreoffice_calc | 3 |
| libreoffice_impress | 2 |
| libreoffice_writer | 2 |
| multi_apps | 17 |
Table 8: Statistics of the OSWorld official split small-scale test tasks.
Figure 2: A toy example of the ILG data augmentation workflow.
| Method | Mobile | Desktop | Web | Avg. | |||
|---|---|---|---|---|---|---|---|
| Text | Icon | Text | Icon | Text | Icon | ||
| GPT-4o Hurst et al. (2024) | 30.5 | 23.2 | 20.6 | 19.4 | 11.1 | 7.8 | 18.8 |
| CogAgent Hong et al. (2024) | 67.0 | 24.0 | 74.2 | 20.0 | 70.4 | 28.6 | 47.4 |
| SeeClick Cheng et al. (2024) | 78.4 | 50.7 | 70.1 | 29.3 | 55.2 | 32.5 | 55.1 |
| R-VLM Park et al. (2025b) | 85.0 | 61.1 | 81.4 | 52.8 | 66.5 | 51.4 | 66.3 |
| MP-GUI Wang et al. (2025b) | 86.8 | 65.9 | 70.8 | 56.4 | 58.3 | 46.6 | 64.1 |
Table 9: Performance comparison on ScreenSpot.
| Method | Mobile | Desktop | Web | Avg. | |||
|---|---|---|---|---|---|---|---|
| Text | Icon | Text | Icon | Text | Icon | ||
| SeeClick Cheng et al. (2024) | 78.4 | 50.7 | 70.1 | 29.3 | 55.2 | 32.5 | 55.1 |
| GPT-4o + SeeClick Cheng et al. (2024) | 85.2 | 58.8 | 79.9 | 37.1 | 72.7 | 30.1 | 63.6 |
| OS-Atlas-4B Wu et al. (2024) | 87.2 | 59.7 | 72.7 | 46.4 | 85.9 | 63.1 | 71.9 |
| GPT-4o + OS-Atlas-4B Wu et al. (2024) | 95.5 | 75.8 | 79.4 | 49.3 | 90.2 | 66.5 | 79.1 |
| InternVL3-8B Zhu et al. (2025) | – | – | – | – | – | – | 81.4 |
Table 10: Performance comparison on ScreenSpot-v2.
Figure 3: Illustrative bad case when using Gemini-2.5-pro as the planner and LAMO-3B as the policy executor in AndroidWorld. The goal of the task is "Create a playlist titled ’Ultimate Fails Series’ with the following files in VLC (located in Internal Memory/VLCVideos), in order: highlight_41_4K_2023_03_30.mp4, scene_68_4K_copy.mp4" . The results indicate that LAMO-3B precisely aligns and executes the planner’s instruction. At this stage, the overall performance of the framework is primarily constrained by the planner’s decision-making quality: an erroneous plan can directly precipitate task failure. Specifically, at Step 10, the planner attempted to enter the playlist name without first focusing the text input field; it then failed to recognize the missing/incorrect entry and proceeded to trigger the save action, ultimately causing task failure.
Figure 4: A case in AndroidWorld corresponding to the same task as in Figure 3 , but using GPT-5 as the planner. Compared with the previously shown episode, GPT-5 exhibits markedly enhanced planning capabilities, benefiting from richer world knowledge that enables more accurate interpretation of end-user devices usage and provides more reasonable high-level planning, while our LAMO-3B act as a plug-and-play policy executor for low-level GUI execution. In particular, at Step 9 of this episode, GPT-5 encountered the same scenario as Gemini-2.5-pro but correctly inferred that the input field on the screen had not been activated and therefore first clicked on the input box before entering the text. As the planner improves, the overall performance of the framework correspondingly advances.
Figure 5: Case Study: LAMO-3B achieves GUI automation on the MiniWob++ benchmark through end-to-end reasoning. The results demonstrate LAMO-3B’s capabilities in screen perception, long-horizon reasoning, and history-awareness, as well as precise pixel-level GUI interaction.
Figure 6: Case Study: LAMO-3B orchestrates a multi-agent system (MAS) to produce GUI automation on the MiniWob++ benchmark. These results demonstrate the multi-role orchestration capabilities of LAMO-3B: the Observer facilitates fine-grained perception of screen details, while the Planner provides dynamic task planning alongside actionable tips to mitigate execution errors. Furthermore, the Allocator analyzes contextual information to predict the optimal actions for the current screen state, and the Executor precisely aligns with the Allocator’s instructions to perform low-level GUI interactions.
Figure 7: Example visualizations from ScreenSpot-pro, illustrating LAMO-3B’s ability to perceive on-screen visual clues and spatial context. Moreover, our semantic enhancement strategy for screen grounding (Sec. 4.1 ), addressing the first challenge in SG via data distillation, fosters a profound multimodal understanding of UI environments and element descriptions. This empowers LAMO-3B to precisely ground the target element while simultaneously capturing spatial context and intricate layout details.
Figure 8: Example visualizations from ScreenSpot-pro, demonstrating LAMO-3B’s ability to comprehend multilingual on-screen information.
Figure 9: A bad case on OSWorld. In this instance, LAMO-3B suffered from an over-reliance on textual semantic matching—mistakenly attending to the "Total Earned" header while overlooking the spatial constraints for cell E3. Tabular interfaces, which often feature extensive sparse regions, amplify this susceptibility, leading to grounding misalignment in long-horizon spreadsheet manipulation.
Figure 10: A bad case on OSWorld. In this instance, the Planner’s instruction is to ’Click the "Replace All" button, which is the icon to the right of the "Replace" input field.’ However, the ground truth target is a purely graphical element devoid of local textual clues, necessitating that the policy executor possesses prior functional knowledge of the software’s interface. Although LAMO-3B successfully identifies the target’s Region-of-Interest, its lack of such software-specific priors ultimately leads to a grounding failure.
Figure 11: A complete episode on OSWorld. The goal of the task: "Please help me set the current user’s line length for code wrapping to 50 characters in VS Code." In this episode, LAMO-3B precisely aligns the planner’s instructions and accurately executes a diverse set of atomic actions in pixel-level, demonstrating both the robustness of LAMO-3B as a policy executor and the richness of its action space (Tab. 7 ), which together enable scalable execution of high - level plans produced by the planner across a broad spectrum of long-horizon GUI tasks.
More in this topic
GUI Agents
Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.
See all 178 papers