Hybrid Self-evolving Structured Memory for GUI Agents

Sibo Zhu; Wenyi Wu; Kun Zhou; Stephen Wang; Biwei Huang

cs.AIMar 11, 2026

Hybrid Self-evolving Structured Memory for GUI Agents

Authors: Sibo Zhu, Wenyi Wu, Kun Zhou, Stephen Wang, Biwei Huang

Organizations: University of California, San Diego

Abstract

The remarkable progress of vision-language models (VLMs) has enabled GUI agents to interact with computers in a human-like manner. Yet real-world computer-use tasks remain difficult due to long-horizon workflows, diverse interfaces, and frequent intermediate errors. Prior work equips agents with external memory built from large collections of trajectories, but relies on flat retrieval over discrete summaries or continuous embeddings, falling short of the structured organization and self-evolving characteristics of human memory. Inspired by the brain, we propose Hybrid Self-evolving Structured Memory (HyMEM), a graph-based memory that couples discrete high-level symbolic nodes with continuous trajectory embeddings. HyMEM maintains a graph structure to support multi-hop retrieval, self-evolution via node update operations, and on-the-fly working-memory refreshing during inference. Extensive experiments show that HyMEM consistently improves open-source GUI agents, enabling 7B/8B backbones to match or surpass strong closed-source models; notably, it boosts Qwen2.5-VL-7B by +22.5% and outperforms Gemini2.5-Pro-Vision and GPT-4o.

Figures & tables

Method	Data Form	MM	Struct.	Global Up.	Local Up.
MemGPT	Discrete	✗	✗	✗	✓
ExpeL	Discrete	✗	✗	✓	✗
ReasoningBank	Discrete	✗	✗	✓	✗
AWM	Discrete	✗	✗	✓	✗
A-MEM	Discrete	✗	✓	✓	✗
G-Memory	Discrete	✗	✓	✓	✗

Table 1: Comparison of Memory Paradigms. Abbreviations: MM : Multimodal support; Struct. : Structured organization; Up. : Update. Methods are compared by data form, MM, and structure, as well as Global (self-evolution) and Local (working memory refresh) updates.

Figure 1: Overview of the Hybrid Self-Evolving Memory ( HyMEM ) system. Top: memory construction via graph evolution, where new trajectories are integrated using retrieval and VLM-based redundancy checks. Bottom: four inference phases—(1) Query & Retrieve via structured graph expansion, (2) Working Memory Initialization using hybrid encoding, (3) Agent Execution , and (4) On-the-fly Working Memory Refresh based on new observations.

		WebVoyager				Mind2Web				MMInA	Overall
Backbone	Model/Method	Amz	Cour	Recp	Map	Info	Svc	Ent	Trav	Wiki	Overall
Closed-Source	GPT-4o	24.4%	7.1%	13.3%	36.6%	7.8%	14.1%	3.4%	19.4%	51.0%	19.7%
	Gemini-Pro-Vision	41.7%	42.9%	20.0%	53.7%	16.7%	22.4%	0.0%	19.4%	50.0%	29.6%
	Claude-4	63.4%	28.6%	33.3%	70.0%	25.6%	40.0%	6.9%	9.7%	52.0%	36.6%
Open-Source	Qwen2.5-VL-32B	46.3%	26.2%	6.7%	29.3%	14.1%	20.0%	6.9%	9.7%	43.0%	22.5%
	CogAgent	12.2%	9.5%	26.7%	9.8%	24.4%	8.2%	13.8%	16.1%	21.0%	15.7%

Table 2: Task success rates (%) across WebVoyager, Mind2Web, and MMInA. Bolded entries indicate best-performing open-source models in each domain. Overall is averaged across available domains.

Figure 2: Success rate comparison between static memory and self-evolving HyMEM . (a): Global evolution. (b): Local evolution via working memory refresh.

Memory Size	500	1000	2000	5000	8000
Amazon	19.5%	31.7%	63.4%	65.9%	63.4%
Google Map	31.7%	46.3%	51.2%	51.2%	56.1%
AllRecipes	11.1%	13.3%	13.3%	15.6%	20.0%

Table 3: Impact of Memory Bank Size on Task Success across different domains. Increasing the number of stored trajectories significantly improves the retrieval quality and overall agent success rate.

Figure 3: (a) Scaling Behavior of Memory Size in Coursera domain and (b) Graph Compression Analysis.

Strategy	Amazon	Coursera	GMap
$\|\widetilde{\mathcal{N}}_{\text{initial}}\|=5$	63.4%	54.8%	53.7%
$\|\widetilde{\mathcal{N}}_{\text{initial}}\|=10$	53.7%	31.0%	46.3%
$\|\widetilde{\mathcal{N}}_{\text{initial}}\|=1$	46.3%	31.0%	48.8%

Table 4: Task success rate under different retrieval strategies, showing the similarity and diversity trade-off.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Visualization of Discrete Memory Update Flow.

Figure 5: t-SNE visualization of continuous memory embeddings across domains. Each point represents a stored trajectory embedding, color-coded by domain.

Figure 6: Successful case by Qwen2.5-VL-7B + Hybrid Memory in the Amazon task. The model accurately understands the goal, retrieves relevant memory, and selects the correct product by filtering.

Figure 7: Failure case by Qwen2.5-VL-7B base model in the Amazon task. The model does not know how to filter products by the requirement, and loses track of the task state.

Figure 8: Successful case by Qwen2.5-VL-7B + Hybrid Memory in the Amazon task. The model accurately understands the goal, retrieves relevant memory, and selects the correct product by filtering.

Figure 9: Failure case by Qwen2.5-VL-7B base model in the Amazon task. The model does not know how to filter products by the requirement, and loses track of the task state.

Panel of Figure 10 — Figure 10: Guidance Digestion Prompt used to synthesize retrieved experience summaries into a compact Guidance Block for inference-time navigation.

Panel of Figure 11 — Figure 11: Local self-evolution prompt for conservative working-memory refresh: it updates only on genuine phase shifts and preserves reusable takeaways.

Panel of Figure 12 — Figure 12: Global evolution prompt for strategy deduplication and graph maintenance. The judge selects ADD / UPDATE / REPLACE using multimodal context and returns a strict JSON decision for deterministic memory updates.

GUI Agents

Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.

See all 178 papers

Learning How to Remember: Memory Modality Selection for GUI Agents
Oct 13, 2026Taeyun Roh, Suhyeong Park, Gijin Kim +2Memory-Augmented Agents GUI Agents
Korea University · Myongji University · AIGEN Sciences
Mem-W: Latent Memory-Native GUI Agents
May 10, 2026Guibin Zhang, Yaohui Ling, Fanci Meng +2Agent Memory GUI Agents
LV-NUS Lab, NUS & NTU
MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents
May 18, 2026Ziyun Zeng, Hang Hua, Bocheng Zou +3Multimodal Memory GUI Agents
University of Rochester · MIT-IBM Watson AI Lab · University of Wisconsin-Madison

Oct 13, 2026J/K move · Enter open · S save

Learning How to Remember: Memory Modality Selection for GUI Agents

Taeyun Roh, Suhyeong Park, Gijin Kim, Wonjune Jang+1

Korea University · Myongji University · AIGEN Sciences

Memory-Augmented Agents GUI Agents Agent Memory Management Multimodal Memory Vision-Language Agents

Open paper AskPDF arXiv

Recent advances in vision-language models (VLMs) have substantially improved the capabilities of GUI agents, while memory augmentation has emerged as a promising approach for reusing knowledge from past interactions. However, existing memory augmented GUI agents typically adopt a fixed representation strategy, providing retrieved experience as a static modality. It remains unclear whether a single memory modality is consistently optimal, or how modality preference changes across interaction steps. In this work, we conduct a systematic study of memory modality in VLM based GUI agents. Across GUI benchmarks, we find that the preferred memory representation is strongly step dependent. Motivated by the finding, we introduce SwitchMem, a routing framework that dynamically selects between textual and visual memory representations at each step. Across three GUI benchmarks, SwitchMem outperforms memory augmented baselines that rely on static memory representations. Our findings validate our claim that memory modality is a step dependent decision rather than a fixed design choice. Code and project resources are available at https://txxnrd.github.io/switchmem

Abstract

GUI Agents

Explore similar work

Learning How to Remember: Memory Modality Selection for GUI Agents

Mem-W: Latent Memory-Native GUI Agents

MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents