cs.AIOct 13, 2026

Learning How to Remember: Memory Modality Selection for GUI Agents

Authors: Taeyun Roh, Suhyeong Park, Gijin Kim, Wonjune Jang, Jaewoo Kang

Organizations: Korea University · Myongji University · AIGEN Sciences

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent advances in vision-language models (VLMs) have substantially improved the capabilities of GUI agents, while memory augmentation has emerged as a promising approach for reusing knowledge from past interactions. However, existing memory augmented GUI agents typically adopt a fixed representation strategy, providing retrieved experience as a static modality. It remains unclear whether a single memory modality is consistently optimal, or how modality preference changes across interaction steps. In this work, we conduct a systematic study of memory modality in VLM based GUI agents. Across GUI benchmarks, we find that the preferred memory representation is strongly step dependent. Motivated by the finding, we introduce SwitchMem, a routing framework that dynamically selects between textual and visual memory representations at each step. Across three GUI benchmarks, SwitchMem outperforms memory augmented baselines that rely on static memory representations. Our findings validate our claim that memory modality is a step dependent decision rather than a fixed design choice. Code and project resources are available at https://txxnrd.github.io/switchmem

Figures & tables

Figure 1: The same retrieved memory can lead to different outcome for a GUI agent depending on modality.
Figure 2: Relative frequency of Text and Image wins among interaction steps. Neither modality consistently dominates across benchmarks.
Figure 3: Performance gap between fixed modality and step-wise oracle modality memory selection.
Figure 4: layer-wise modality-preference probing with D-CoE on AITW. Using examples with at least five-of-seven agreement in sampled preference, AUROC increases in deeper layers, indicating a stronger modality-preference signal in later hidden representations.
Figure 5: Overview of SwitchMem. During offline training, the difference between the actor’s hidden states obtained with Image and Text memory is used to construct a continuous modality preference target. A pairwise router scores both memory candidates and is trained to match their score difference to this target. At inference, the router selects one memory modality for the frozen actor to generate the next action.
MethodAugmentationAITW (%)LearnGUI (%)MobileMiniWoB (%)
General-purpose
GPT-5.4-mini–34.00024.91251.087
Qwen3.5-9B–41.92023.48558.696
GUI specialized
Aguvis-7B–33.24024.73629.348
UI-TARS-2B-SFT–23.2609.3658.696
Table 1: Performance comparison on AITW, LearnGUI, and MobileMiniWoB++. AITW and LearnGUI use the official benchmark-provided step level action scores, whereas MobileMiniWoB++ reports task success rate. The memory-augmented group compares different augmentation strategies using the same frozen Qwen3.5-9B actor and evaluation protocol.
BenchmarkBothImageSwitchesRun
AITW83.836.51.722.42
LearnGUI90.881.31.794.95
MobileMiniWoB57.876.31.564.50
Table 2: Within-trajectory routing behavior of SwitchMem. Both and Image denote the percentages of dual-modality trajectories and Image-routed steps, respectively. Switches and Run measure the average modality transition count and contiguous run length per trajectory.
Figure 6: Changes in Image over Text utility (left axis) and SwitchMem’s Image-selection rate (right axis).
Figure 7: Empirical analysis on LearnGUI. (a) Comparison with simpler routing baselines. Utility and Binary use direct preference supervision, while Basic routing excludes actor hidden states. (b) SwitchMem versus ratio-matched random routing, where the dashed line denotes the analytic expected score and the star marks the deployed threshold. (c) Efficiency Analysis of SwitchMem. RAG-GUI is omitted in figure due to its high latency of 11.118 s/step.
MethodAITWLearnGUIMobileMiniWoB++
No memory29.98025.70523.913
Text32.64038.23032.609
Image30.44040.99733.696
SwitchMem33.68041.05536.957
Table 3: Generalization to the GUI specialized UI-TARS-7B-SFT actor. SwitchMem achieves the best performance across all three benchmarks, improving over the strongest fixed memory condition.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: layer-wise modality-preference probing on Learn-GUI. AUROC is computed by trajectory-grouped prediction on the 5-of-7 stable-preference subset. The dashed line denotes chance.
Figure 9: Comparison with simpler routing baselines on AITW. Binary is closely competitive with SwitchMem. The y-axis is truncated.
Figure 10: SwitchMem versus random routing on AITW. Random routing controls for aggregate Image usage but does not condition its selections on the interaction state.
MethodAITWLearnGUIMobileMini
No memory1.2971.2720.906
Text1.1541.3770.889
Image1.2791.6581.558
Both1.3091.6801.441
EchoTrail1.2051.3641.189
AGMem1.3721.8691.388
Appendix
Table 4: Mean inference latency in seconds per interaction step. Method-specific inference stages are included, while retrieval and input preprocessing are excluded.
Figure 11: Paired actor prompts used by SwitchMem. Both variants contain the same retrieved step from a successfully completed trajectory and expose its demonstrated action. Text memory renders the past GUI state as a textual observation, whereas Image memory replaces only that observation with the corresponding full screenshot. The image is cropped only for visualization; the actor receives the full screenshot. The current task, current screenshot, action schema, actor, and decoding configuration are held fixed. For readability, the common system prompt and current screenshot are collapsed in the figure.
DatasetBest fixedSwitchMemΔ\Delta95% CI
AITW49.30051.420+2.120[1.270, 2.945]
LearnGUI37.44138.131+0.690[0.149, 1.263]
MobileMiniWoB++60.87063.043+2.174[-3.261, 7.609]
Appendix
Table 5: Paired performance differences between SwitchMem and the strongest fixed memory baseline. Confidence intervals are reported in percentage points.
Figure 12: Text-favoring AITW case. (a) A Sparse initial state requiring a Spatial action. (b,c) Text and Image representations of the same retrieved demonstration, which also shows a Costco sign-in page. Text memory yields a correct tap on the current browser address bar, whereas Image memory yields a typing action before selecting the bar. SwitchMem selects Text. T/I denotes Text/Image action-matching scores; memory coordinates are normalized and rounded.
Figure 13: Image-favoring AITW case. (a) A Dense post-initial state requiring a Function action: the requested Walmart search results are already displayed. (b,c) Text and Image representations of the same retrieved Amazon demonstration at a query-entry state. Image memory yields correct task completion, whereas Text memory yields an unnecessary Enter action. SwitchMem selects Image. T/I denotes Text/Image action-matching scores; memory coordinates are normalized and rounded.
More in this topic

Memory-Augmented Agents

Memory-augmented agents use stored experience and knowledge to act coherently across long tasks, changing environments, and repeated interactions.

See all 347 papers

Explore similar work

CardsList

  1. MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents

    May 18, 2026Ziyun Zeng, Hang Hua, Bocheng Zou +3Multimodal MemoryGUI Agents

    University of Rochester · MIT-IBM Watson AI Lab · University of Wisconsin-Madison

  2. Hybrid Self-evolving Structured Memory for GUI Agents

    Mar 11, 2026Sibo Zhu, Wenyi Wu, Kun Zhou +2GUI AgentsAgent Memory Management

    University of California, San Diego

  3. Mem-W: Latent Memory-Native GUI Agents

    May 10, 2026Guibin Zhang, Yaohui Ling, Fanci Meng +2Agent MemoryGUI Agents

    LV-NUS Lab, NUS & NTU