EE-MCP: Self-Evolving MCP-GUI Agents via Automated Environment Generation and Experience Learning

Tiantian He; Yihang Chen; Keyue Jiang; Ka Yiu Lee; Kaiwen Zhou; Kun Shao; Shuai Wang

cs.AIApr 10, 2026

EE-MCP: Self-Evolving MCP-GUI Agents via Automated Environment Generation and Experience Learning

Authors: Tiantian He, Yihang Chen, Keyue Jiang, Ka Yiu Lee, Kaiwen Zhou, Kun Shao, Shuai Wang

Organizations: University College London · Huawei Noah’s Ark Lab · The Chinese University of Hong Kong · Independent Researcher ·

Abstract

Computer-use agents that combine GUI interaction with structured API calls via the Model Context Protocol (MCP) show promise for automating software tasks. However, existing approaches lack a principled understanding of how agents should balance these two modalities and how to enable iterative self-improvement across diverse applications. We formulate MCP-GUI interplay as a unified hybrid policy learning problem where the agent learns when each modality provides complementary advantages, and show that distillation and experience augmentation target fundamentally different failure modes - requiring application-aware mechanism selection. Built on this formulation, we propose a self-evolving framework with a fully automatic pipeline that orchestrates automatic environment generation and validation, trajectory collection, gap-driven task synthesis, and quality-filtered training - all without manual intervention. A key innovation is our experience bank, which accumulates LLM-learned rules from trajectory comparison, enabling inference-time improvement without fine-tuning. Systematic cross-application analysis across three desktop applications reveals that the optimal strategy depends on MCP-GUI composition: distillation achieves 77.8% pass rate on MCP-dominant tasks (+17.8pp), while the experience bank excels on GUI-intensive tasks (+10.0pp).

Figures & tables

Panel of Figure 1 — Figure 1: Self-Evolving Policy Learning Framework. Left : Expert trajectories are collected from Claude on seed tasks $\mathcal{G}_{0}$ . Center : The self-evolution loop—the agent is evaluated via LLM judge to produce a multi-dimensional performance profile $\mathcal{P}_{k}$ (tracking MCP/GUI ratio, difficulty, and skills), which drives targeted task and environment generation. New trajectories are accumulated with existing data for SFT training. Right : Two complementary outputs: (1) a fine-tuned agent $\pi_{K}$ (effective for MCP-dominant tasks), and (2) an experience bank $\mathcal{E}_{K}$ that enables inference-time improvement (effective for GUI-intensive tasks).

Panel of Figure 2 — Figure 2: Expert Trajectory Distillation. Claude executes tasks step-by-step, producing trajectories with screenshots $s_{t}$ , reasoning $r_{t}$ , actions $a_{t}$ (MCP or GUI), and observations $o_{t}$ . These serve as training data for the student model.

Dimension	Metrics
1. Modality Balance	$\rho_{\text{mcp}}$ , $\rho_{\text{gui}}$ , $\rho_{\text{hybrid}}$
2. Difficulty Level	$\sigma_{\text{easy}}$ , $\sigma_{\text{medium}}$ , $\sigma_{\text{hard}}$
3. Skill Categories	$\sigma_{\text{retrieve}}$ , $\sigma_{\text{manip}}$ , $\sigma_{\text{search}}$ , $\sigma_{\text{exec}}$ , $\sigma_{\text{nav}}$ , $\sigma_{\text{config}}$
4. Format Quality	$\phi_{\text{format}}$ , $\phi_{\text{parse}}$ , $\phi_{\text{args}}$
5. Efficiency	$\bar{n}_{\text{steps}}$ , $\phi_{\text{complete}}$ , $\phi_{\text{timeout}}$
Output: Gap Analysis $\rightarrow$ Task Generation

Table 1: Multi-dimensional performance profile

\mathcal{P}_{k}

tracks agent capabilities across five dimensions. These metrics feed into gap analysis for targeted task generation.

Application	Seed Tasks	MCP Tools
VS Code	30	9
LibreOffice Calc	45	16
Chrome	45	11
Total	120	36

Table 2: Application benchmark overview. Each application has a distinct set of MCP tools, requiring the agent to adapt its modality selection strategy.

Strategy	Pass	Score	Eff.	MCP%
Baseline	60.0%	0.654	0.662	43.7%
Experience Bank (inference-time only)
+ Exp. Bank	62.2%	0.640	0.646	45.0%
Trajectory Distillation
+ Distill. + Exp.	77.8%	0.770	0.724	36.3%

Table 3: Chrome results after one iteration (Qwen3-VL-8B, 45 tasks). Multi-iteration trends in Table 4 .

	Iter 0		Best Iter
Strategy	Pass	Score	Pass	Score
Exp. Bank Only	60.0	0.654	64.4 (iter 2)	0.674
Distill. + Exp.	60.0	0.654	77.8 (iter 1)	0.770

Table 4: Self-evolution results across 3 iterations on Chrome (Qwen3-VL-8B, 45 tasks). Pass rate (%) and LLM Score shown per iteration. Distillation achieves 77.8% in a single iteration; the experience-only variant improves steadily to 64.4% without fine-tuning.

	Chrome (45)	VS Code (30)	Calc (45)
Baseline	60.0	53.3	44.4
+ Exp. Bank	62.2	63.3 (+10.0)	46.7 (+2.3)
+ Distill. + Exp.	77.8 (+17.8)	43.3	42.2

Table 5: Cross-application comparison (pass rate %). The optimal strategy depends on task characteristics: distillation excels for Chrome (+17.8pp), while the experience bank is more effective for GUI-intensive VS Code (+10.0pp) and Calc (+2.3pp). Task counts: VS Code 30, Chrome/Calc 45.

Panel of Figure 3 — Figure 3: Self-evolution improvement on Chrome (Qwen3-VL-8B, Distillation + Experience Bank). Left : Baseline profile with identified weaknesses. Center : Evolution mechanisms—gap analysis, experience bank, and training pool growth. Right : Best metrics achieved, separated by source iteration: peak pass rate at iter 1 (+17.8pp), efficiency gains by iter 3 (MCP +12.1pp, actions $-$ 33%). Full profile comparison in Appendix L .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Skill	Example Tasks
Data Retrieval	Read file (VS Code), Get cell (Calc), Get page (Chrome)
Data Manip.	Write code (VS Code), Edit cell (Calc), Fill form (Chrome)
Search	Find text (VS Code), Search cells (Calc), Search page (Chrome)
Execution	Run tests (VS Code), Calculate (Calc), Navigate URL (Chrome)
Navigation	Go to file (VS Code), Switch sheet (Calc), Switch tab (Chrome)
Configuration	Change theme (VS Code), Set format (Calc), Preferences (Chrome)

Appendix

Table 6: Application-agnostic skill categories with examples.

Parameter	Value	Description
Attempts/task	3	Max attempts
Success threshold	0.5	Action success rate
Learning rate	2e-5	SFT learning rate
LoRA rank	8	LoRA rank
Image pixels	50,176	Per-image resolution
Max images	30	Per training sample

Appendix

Table 7: Pipeline and SFT hyperparameters.

Strategy	Easy	Med.	Hard	Score
Baseline	12/15	11/15	4/15	0.654
+ Exp. Bank	14/15	10/15	4/15	0.640
+ Distill. + Exp.	15/15	12/15	8/15	0.770

Appendix

Table 8: Chrome results by difficulty level (Qwen3-VL-8B, 45 tasks: 15 easy, 15 medium, 15 hard).

Strategy	Baseline	Best
Exp. Bank	44.4%	46.7%
Distill. + Exp.	44.4%	44.4%

Appendix

Table 9: Calc self-evolution results (Qwen3-VL-8B, 45 tasks). The experience bank provides a marginal gain (+2.3pp); distillation does not improve over baseline. Hard tasks (0/15) remain unsolved across all strategies.

Strategy	Qwen3-8B	Qwen2.5-7B
Baseline	60.0%	48.9%
+ Exp. Bank	62.2%	40.0%
+ Distill. + Exp.	77.8%	—
Native <tool_call>	✓	✗
Instruction following	Strong	Weak

Appendix

Table 10: Cross-model comparison on Chrome (45 tasks). Self-evolution mechanisms only improve Qwen3-VL-8B; Qwen2.5-VL-7B shows no improvement from any intervention.

Computer-Use Agents

Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.

See all 160 papers

Xiaomi-GUI-0 Technical Report
Jun 30, 2026Wanxia Cao, Chengzhen Duan, Pei Fu +29Mobile GUI Automation RL for GUI Agents
UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization
Apr 15, 2026Zhengxi Lu, Fei Tang, Guangyi Liu +8Long-Horizon LLM Agents Mobile GUI Automation
Zhejiang University · Apple · Tencent
Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents
Aug 4, 2026Siqi Fan, Minghao Li, Xiaoqian Ma +6Computer-Use Agents RL for LLM Agents
University of Electronic Science and Technology of China · AI Platform, Xiaohongshu Inc. · Gaoling School of Artificial Intelligence, Renmin University of China +1

Jun 30, 2026J/K move · Enter open · S save

Xiaomi-GUI-0 Technical Report

Wanxia Cao, Chengzhen Duan, Pei Fu, Pengzhi Gao+28

Mobile GUI Automation RL for GUI Agents GUI Agents AI Agent Benchmarks Mobile GUI Agents Multimodal Agents

Open paper AskPDF arXiv

Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions such as tapping, swiping, text entry, and navigation. However, existing GUI agents are trained and evaluated largely on offline trajectories, simulated environments, and standardized benchmarks. These differ substantially from real applications in interface layout, interaction logic, and abnormal-state distribution, and cannot faithfully characterize execution stability in real-world use, where account states, permission dialogs, payment authentication, and risk control continually reshape the state distribution and open a persistent gap between benchmark scores and real usability. To close this gap, we propose Xiaomi-GUI-0, a native multimodal GUI agent for real mobile environments, trained and evaluated within a real-device closed loop. At its core is a real-device-dominant hybrid infrastructure, where physical devices are the primary execution environment and sandboxes provide auxiliary support, so that data collection, training, rollout, and evaluation share an execution distribution close to real deployment. We construct multi-source training data spanning high-frequency head tasks, high-generalization data for long-tail intents, and capability-enhancement data for reflection and memory, and introduce an error-driven data flywheel that turns failure trajectories into corrected actions, reflective explanations, and recovery demonstrations. The model is trained through a progressive three-stage pipeline of supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning. Evaluated on public benchmarks and our in-house RealMobile, Xiaomi-GUI-0 achieves 72.0% success on RealMobile and 78.9% on AndroidWorld, while substantially improving execution stability and abnormal-state recognition in real-world tasks.

Abstract

Computer-Use Agents

Explore similar work

Xiaomi-GUI-0 Technical Report

UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization

Screenshots or Tools? Eliciting Tool Use and Managing Multimodal Context in Hybrid GUI-MCP Computer-Use Agents