ScopeMem: Latent Memory That Learns Where to Revise Frozen GUI Agents

Zhicheng Zheng; Cheng Zhang; Shengping Yu; Tianxiang Zhao

cs.AIOct 19, 2026

ScopeMem: Latent Memory That Learns Where to Revise Frozen GUI Agents

Authors: Zhicheng Zheng, Cheng Zhang, Shengping Yu, Tianxiang Zhao

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Imperial College London

Abstract

Latent memory is gaining ground as a lightweight history supplement for GUI agents. It supplies history to a frozen vision-language model via prefix vectors trained with step-wise cross-entropy (CE). While resulting gains are naturally attributed to encoded history, our controlled experiments show they primarily stem from action revision. The prefix learns to alter the model's default actions. However, this revision is unscoped because step-wise CE pushes outputs toward reference actions at every step without explicitly preserving correct decisions. Consequently, it repairs errors but also breaks 8-13% of initially correct answers. Overall accuracy metrics hide these losses. We therefore propose ScopeMem, a latent-memory method for scoped revision. Its encoder conditions prefix generation on the model's initial answer and is trained to correct errors while preserving correct decisions. On the controlled ablation set, ScopeMem reduces the share of broken correct decisions to 5.6% while repairing more errors than any evaluated CE-trained prefix. On AndroidControl, GUI-Odyssey, and zero-shot AITW, ScopeMem outperforms all evaluated prompting and latent-memory baselines, enabling a frozen Qwen3-VL-8B to match or surpass GUI-specialized agents and Qwen3-VL-32B.

Figures & tables

Figure 1: Revision cuts both ways. Of the frozen actor’s 1,463 decisions (centre), a fixed-query latent-memory prefix (left) repairs 196 errors but harms 115 correct decisions. Its 5.5-point accuracy gain (pp: percentage points) hides these harms. ScopeMem (right) learns to focus revisions on the actor’s errors, repairing 254 and harming 49.

Figure 2: ScopeMem gives the frozen actor a second opinion on its first. The actor 1 observes the context and 2 answers alone in a first pass. The encoder $E_{\phi}$ 3 reads the first opinion, its confidence, the task and textual history to generate a prefix. This prefix is 4 prepended to the same context, and the actor 5 answers again. Gold parts exist only in training: 6 the preservation term is active when the first opinion matches the reference action, and 7 gradients update $E_{\phi}$ only.

Panel of Table 1 — Table 1: Only ScopeMem combines learned revision, access to the first opinion, and an objective for preserving correct decisions. Reads : sees the first opinion; learned : trained; prevented : preservation is a training objective; reported : repairs and harms reported; : partially.

Figure 3: Prefixes trained with cross-entropy alone also change correct decisions, and confidence barely separates repairs from harms. (a) Ablation set: gain over the plain actor (pp), share of correct actions changed, and harm rate. Shading marks ScopeMem . (b) AUC of confidence signals and expanded features for detecting the plain actor’s errors and separating ScopeMem ’s repairs from harms.

Method	AndroidControl		GUI-Odyssey		AITW (zero-shot)
	EM	TM	EM	TM	Gen.	Inst.	Google	Single	Shop	Macro
General-purpose models
Qwen3-VL-4B	60.64	76.42	57.18	78.06	54.19	68.05	57.40	66.67	53.44	59.95
Qwen3-VL-32B	61.88	78.37	61.84	80.30	57.54	69.82	62.13	63.16	56.87	61.90
InternVL3.5-8B	35.28	57.80	29.52	60.41	40.22	39.64	28.99	47.37	34.35	38.12
GLM-4.1V-9B	40.96	70.74	35.80	67.56	37.43	44.97	39.05	40.35	35.88	39.54

Table 2: On the frozen actor, only ScopeMem matches or beats GUI-specialised models on all three benchmarks. Step accuracy (%) on the main test sets; EM: full action; TM: action type; AITW subsets General, Install, GoogleApps, Single, WebShopping and their mean (Macro).

\hookrightarrow

: on the frozen plain actor ( † : on Qwen2.5-VL-7B); last row: gain in pp.

Benchmark	Repairs / harms		Only one system		Advantage (95% CI)
Benchmark	ScopeMem	Mem-W	keeps	repairs	Advantage (95% CI)
AndroidControl	57/27	55/36	16/7	13/11	+1.95 [ $-$ 0.56, 4.62]
GUI-Odyssey	264/36	218/59	33/10	71/25	+4.29 [2.76, 5.84]
AITW	87/32	82/44	17/5	15/10	+2.03 [0.36, 3.77]

Table 3: Mem-W breaks more correct decisions than ScopeMem . Counts on the main test sets are relative to the plain actor. “Only one system” counts correct steps only one preserves or errors only one repairs ( ScopeMem /Mem-W). Advantage: ScopeMem ’s accuracy minus Mem-W’s (pp, 95% CI).

Second epoch	R	H	Rate (%)		Acc	Gain
	R		repair	harm	Acc	Gain
Fixed-query prefix
cross-entropy alone	196	115	33.6	13.1	65.62	+5.54
Task-only encoder
cross-entropy alone	246	71	42.1	8.1	72.04	+11.96
scoped	242	58	41.4	6.6	72.66	+12.58

Table 4: Scoped preservation lowers harms for every encoder. Matched variants on the ablation set ( first epoch only : checkpoint before the second epoch). Rates: R over 584 errors, H over 879 correct steps; Acc: accuracy (%); Gain: over the plain actor (pp).

Figure 4: On the ablation set, arrows compare the same encoder trained with cross-entropy alone and with scoped (solid) or unscoped (dashed) preservation. Only ScopeMem lowers harms and raises repairs together, reaching the shaded region. The fixed-query prefix (harm rate 13.1%, repair rate 33.6%) lies outside the plotted range.

Figure 5: Case study: one recalled episode repairs step 7 and breaks step 9 for Mem-W, whereas ScopeMem revises step 7 and keeps step 9.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Variant	Main	Second	Third
Task only, CE	246/71	250/70	241/70
Task only, scoped	242/58	248/57	234/58
History, CE	240/68	243/66	242/67
History, scoped	235/53	236/50	234/54
First opinion, CE	243/67	246/64	249/64
First opinion, unscoped	212/47	217/45	217/48

Appendix

Table 9: Scoped preservation reduces harms for every encoder in all three runs. Repairs/harms on the ablation set in each training run.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 159 papers

Learning How to Remember: Memory Modality Selection for GUI Agents
Oct 13, 2026Taeyun Roh, Suhyeong Park, Gijin Kim +2Memory-Augmented Agents GUI Agents
Korea University · Myongji University · AIGEN Sciences
Mem-W: Latent Memory-Native GUI Agents
May 10, 2026Guibin Zhang, Yaohui Ling, Fanci Meng +2Agent Memory GUI Agents
LV-NUS Lab, NUS & NTU
Hybrid Self-evolving Structured Memory for GUI Agents
Mar 11, 2026Sibo Zhu, Wenyi Wu, Kun Zhou +2GUI Agents Agent Memory Management
University of California, San Diego

Oct 13, 2026J/K move · Enter open · S save

Learning How to Remember: Memory Modality Selection for GUI Agents

Taeyun Roh, Suhyeong Park, Gijin Kim, Wonjune Jang+1

Korea University · Myongji University · AIGEN Sciences

Memory-Augmented Agents GUI Agents Agent Memory Management Multimodal Memory Vision-Language Agents

Open paper AskPDF arXiv

Recent advances in vision-language models (VLMs) have substantially improved the capabilities of GUI agents, while memory augmentation has emerged as a promising approach for reusing knowledge from past interactions. However, existing memory augmented GUI agents typically adopt a fixed representation strategy, providing retrieved experience as a static modality. It remains unclear whether a single memory modality is consistently optimal, or how modality preference changes across interaction steps. In this work, we conduct a systematic study of memory modality in VLM based GUI agents. Across GUI benchmarks, we find that the preferred memory representation is strongly step dependent. Motivated by the finding, we introduce SwitchMem, a routing framework that dynamically selects between textual and visual memory representations at each step. Across three GUI benchmarks, SwitchMem outperforms memory augmented baselines that rely on static memory representations. Our findings validate our claim that memory modality is a step dependent decision rather than a fixed design choice. Code and project resources are available at https://txxnrd.github.io/switchmem

Abstract

Memory-Augmented VLMs

Explore similar work

Learning How to Remember: Memory Modality Selection for GUI Agents

Mem-W: Latent Memory-Native GUI Agents

Hybrid Self-evolving Structured Memory for GUI Agents