DeltaReplay: Task-Relative Memory Reuse for Mobile GUI Agents

Yudong Bai; Yihong Chen; Quanming Yao; Yaqing Wang

cs.AIOct 8, 2026

DeltaReplay: Task-Relative Memory Reuse for Mobile GUI Agents

Authors: Yudong Bai, Yihong Chen, Quanming Yao, Yaqing Wang

Organizations: Tsinghua University

Abstract

Memory-augmented mobile GUI agents store successful execution trajectories and reuse them in later tasks, but a stored trajectory rarely matches a new task exactly. The new task may use different parameters, share only some of its steps with a stored trajectory, or have no relevant record in memory. Forcing the agent to use irrelevant memory can mislead it, whereas discarding memory that may still be useful deprives it of guidance from past experience. To address this dilemma, we propose DeltaReplay, a step-level memory reuse framework that decides how to use existing memory without modifying it. We observe that the reusable part of a stored record is determined not by the record itself but by its relation to the new task, mainly through two factors: page-level consistency and action-level generality. We therefore store execution trajectories as paths in a transition graph, whose nodes (pages) and edges (actions between pages) capture these two factors. At reuse time, the action on each edge is split into a task-independent operation and task-specific parameters. DeltaReplay then compares each recorded step with the new task and the current screen, and decides whether to follow it, execute it after replacing its parameters, or leave it to the base agent. On AndroidWorld and SPA-Bench, DeltaReplay improves the task success rate over a base agent with the same backbone by up to 10.3 and 25.0 percentage points, respectively. These results indicate that deciding at each step how to use retrieved memory lets agents benefit even from partially matching trajectories.

Figures & tables

Figure 1: Reusing partly matching memory step by step. (Top left) The old and new tasks share their first steps but differ in one value and in the later steps. (Top right) Existing methods use or discard a stored record as a whole and fail here. (Bottom) DeltaReplay aligns the two task descriptions and decides at each step whether to replay the step, re-ground it with the new value, or discard it.

Uses supported	direct	partial	inapplicable	Representative work
Prompt only	$\times$ reasons anyway	$\checkmark$ guides reasoning	$\times$ misleads	GUI-explorer ( Xie et al., 2025 ) ;
				MAGNET ( Sun et al., 2026 )
Execute or discard	$\checkmark$ executes, fast	$\triangle$ predefined arguments only	$\checkmark$ falls back	AppAgentX ( Jiang et al., 2025 ) ;
				Darwinian Memory ( Mi et al., 2026 )
Re-ground or discard	$\times$ re-grounds anyway	$\triangle$ on known screens	$\checkmark$ falls back	UI-KOBE ( Chai et al., 2026 )
DeltaReplay (ours)	$\checkmark$ Route A: replay	$\checkmark$ Route B: re-ground (parameters)	$\checkmark$ Route C: discard	This work

Table 1: Behavior of agents with different uses of memory in each regime of Eq. ( 1 ).

Figure 2: Overview of DeltaReplay. Memory holds recorded runs (top), retrieved by task identity; the current run (bottom) is a new task. Structural matching compares the current screen with the recorded screen at the tracked position ( $=$ or $\neq$ ). Route A (blue) replays a matched step on the corresponding current element; Route B (orange) re-grounds a matched step that carries a parameter of the old task (WiFi off $\rightarrow$ on). When the screen differs, the selector checks whether any recorded step still applies; if none does, Route C lets the base agent act.

Method	Backbone	AndroidWorld-116		SPA-Bench-60
Method	Backbone	SR (%) $\uparrow$	Lat. (s) $\downarrow$	SR (%) $\uparrow$	Lat. (s) $\downarrow$
Base agent	Qwen3-VL-30B	43.1	9.3	23.3	10.3
Base agent	Qwen3.8-Flash	49.1	24.8	41.7	21.0
GUI-Owl	GUI-Owl-1.5	50.9	9.6	–	–
RAG-prompting	Qwen3.8-Flash	41.4	34.8	–	–
GUI-explorer	Qwen3.8-Flash	47.4	19.0	–	–

Table 2: Overall results (SR: task success rate; Lat.: latency per step). Best in bold, second-best underlined; Qwen3-VL-30B is Qwen3-VL-30B-A3B-Instruct; –: not run.

AndroidWorld-116		Same task (20)		Same template (22)		Same app (48)		None (26)
Method	Backbone	SR	Lat.	SR	Lat.	SR	Lat.	SR	Lat.
Base agent	Qwen3-VL-30B	70.0	8.8	63.6	7.8	31.3	10.1	26.9	9.2
Base agent	Qwen3.8-Flash	80.0	21.7	63.6	19.5	39.6	27.9	30.8	23.1
GUI-Owl	GUI-Owl-1.5	75.0	7.1	81.8	10.8	37.5	9.9	30.8	9.5
RAG-prompting	Qwen3.8-Flash	80.0	32.2	77.3	31.8	25.0	38.0	11.5	32.8
GUI-explorer	Qwen3.8-Flash	80.0	19.0	86.4	15.9	29.2	20.7	23.1	18.2

Table 3: Results by the closest record in memory; best per column in bold, second-best underlined. On SPA-Bench-60, memory includes the base-agent runs shown here, so its same-family and no-family-record groups consist mostly of tasks both base agents failed (Section 5.1 ).

Same template ( $n{=}22$ )
Variant	SR (%)	Calls	A/B/C (%)
Base agent	63.6	2.11	0/0/100
DeltaReplay (full)	86.4	1.00	41/15/44
Naive replay	9.1	0.00	100/0/0
w/o Route C (no discard)	45.5	0.17	68/32/0
Same app ( $n{=}30$ of 48)

Table 4: Ablations (Qwen3.8-Flash, AndroidWorld) on the tasks each ablation affects; variants in Appendix F . Calls: model calls per step; A/B/C: share of executed steps per route.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Tasks
Same task (20)	BrowserMaze, CameraTakePhoto, CameraTakeVideo, ClockStopWatchPausedVerify, ClockStopWatchRunning, ExpenseDeleteDuplicates, ExpenseDeleteDuplicates2, SystemBluetoothTurnOff, SystemBluetoothTurnOffVerify, SystemBluetoothTurnOn, SystemBluetoothTurnOnVerify, SystemBrightnessMaxVerify, SystemBrightnessMinVerify, SystemWifiTurnOff, SystemWifiTurnOffVerify, SystemWifiTurnOnVerify, TasksCompletedTasksForDate, TasksHighPriorityTasks, TasksIncompleteTasksOnDate, TurnOffWifiAndTurnOnBluetooth
Same template (22)	ContactsAddContact, ContactsNewContactDraft, ExpenseAddSingle, ExpenseDeleteMultiple, ExpenseDeleteSingle, MarkorCreateNote, MarkorCreateNoteFromClipboard, NotesIsTodo, NotesRecipeIngredientCount, NotesTodoItemCount, OsmAndFavorite, OsmAndMarker, SimpleCalendarAddOneEvent, SimpleCalendarAddOneEventInTwoWeeks, SimpleCalendarAddRepeatingEvent, SimpleCalendarDeleteOneEvent, SimpleCalendarLocationOfEvent, SimpleSmsReply, SimpleSmsReplyMostRecent, SimpleSmsSend, SportsTrackerActivitiesCountForWeek, TasksHighPriorityTasksDueOnDate
Same app (48)	BrowserDraw, BrowserMultiply, ClockTimerEntry, ExpenseAddMultiple, ExpenseAddMultipleFromGallery, ExpenseAddMultipleFromMarkor, ExpenseDeleteMultiple2, MarkorAddNoteHeader, MarkorChangeNoteContent, MarkorCreateFolder, MarkorCreateNoteAndSms, MarkorDeleteAllNotes, MarkorDeleteNewestNote, MarkorDeleteNote, MarkorEditNote, MarkorMergeNotes, MarkorMoveNote, MarkorTranscribeReceipt, MarkorTranscribeVideo, NotesMeetingAttendeeCount, OsmAndTrack, SimpleCalendarAddOneEventRelativeDay, SimpleCalendarAddOneEventTomorrow, SimpleCalendarAnyEventsOnDate, SimpleCalendarDeleteEvents, SimpleCalendarDeleteEventsOnRelativeDay, SimpleCalendarEventOnDateAtTime, SimpleCalendarEventsInNextWeek, SimpleCalendarEventsInTimeRange, SimpleCalendarEventsOnDate, SimpleCalendarFirstEventAfterStartTime, SimpleCalendarNextEvent, SimpleCalendarNextMeetingWithPerson, SimpleSmsResend, SimpleSmsSendClipboardContent, SimpleSmsSendReceivedAddress, SportsTrackerActivitiesOnDate, SportsTrackerActivityDuration, SportsTrackerLongestDistanceActivity, SportsTrackerTotalDistanceForCategoryOverInterval, SportsTrackerTotalDurationForCategoryThisWeek, SystemBrightnessMax, SystemBrightnessMin, SystemCopyToClipboard, SystemWifiTurnOn, TasksDueNextWeek, TasksDueOnDate, TurnOnWifiAndOpenApp
None (26)	AudioRecorderRecordAudio, AudioRecorderRecordAudioWithFileName, FilesDeleteFile, FilesMoveFile, OpenAppTaskEval, RecipeAddMultipleRecipes, RecipeAddMultipleRecipesFromImage, RecipeAddMultipleRecipesFromMarkor, RecipeAddMultipleRecipesFromMarkor2, RecipeAddSingleRecipe, RecipeDeleteDuplicateRecipes, RecipeDeleteDuplicateRecipes2, RecipeDeleteDuplicateRecipes3, RecipeDeleteMultipleRecipes, RecipeDeleteMultipleRecipesWithConstraint, RecipeDeleteMultipleRecipesWithNoise, RecipeDeleteSingleRecipe, RecipeDeleteSingleWithRecipeWithNoise, RetroCreatePlaylist, RetroPlayingQueue, RetroPlaylistDuration, RetroSavePlaylist, SaveCopyOfReceiptTaskEval, SimpleDrawProCreateDrawing, VlcCreatePlaylist, VlcCreateTwoPlaylists

Appendix

Table 5: AndroidWorld-116 tasks by coverage group.

Group	A (%)	B (%)	C (%)
Same task	56.1	0.0	43.9
Same template	41.5	14.8	43.7
Same app	0.0	10.1	89.9
None	0.0	0.0	100.0
All	13.7	8.1	78.2

Appendix

Table 7: Share of executed steps taken by each route (DeltaReplay, Qwen3.8-Flash, AndroidWorld-116), by coverage group.

Route	Steps	Corr.	No-op	Harm.	Uncl.	Prec. (%)
A: replay	50	41	6	2	1	83.7
B: re-ground	50	39	8	3	0	78.0
C: discard	50	37	6	6	1	75.5
All	150	117	20	11	2	79.1

Appendix

Table 8: Blind step-level annotation of 150 executed steps of DeltaReplay (Qwen3.8-Flash, AndroidWorld; 50 per route); precision is correct steps over judged steps.

Explore similar work

What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States
Jun 30, 2026Chen Liu, Ling Chen, Hanzhang Zhou +7RL for GUI Agents GUI Agents
University of Technology Sydney · Tongyi Lab, Alibaba Group · Adelaide University
STAMP: Training Explicit Memory for Mobile GUI Agents in Controllable and Scalable Virtual Environments
May 28, 2026Junyang Wang, Haiyang Xu, Xi Zhang +4GUI Agents AI Agent Benchmarks
Beijing Jiaotong University · Tongyi AI Lab, Alibaba Group
PreAct: Computer-Using Agents that Get Faster on Repeated Tasks
Jun 16, 2026Bojie LiComputer-Use Agents LLM Agent Skill Learning
Pine AI

Jun 30, 2026, cs.CVJ/K move · Enter open · S save

What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States

Chen Liu, Ling Chen, Hanzhang Zhou, Xu Zhang+6

University of Technology Sydney · Tongyi Lab, Alibaba Group · Adelaide University

RL for GUI Agents GUI Agents Mobile GUI Agents Long-Horizon Agent Tasks Agent Memory Management RL for LLM Agents

Open paper AskPDF arXiv

Mobile GUI agents increasingly face long-horizon tasks that require reading, updating, and reusing task-relevant data across pages and applications. Existing methods treat memory largely as passive storage, where past observations are accumulated and retrieved when needed. Yet retrieving a value does not reveal its current role in the workflow. The agent must still infer from accumulated records whether the value should be used now, has already been used, or must wait for a later dependency. This implicit reconstruction becomes unreliable in long trajectories with repeated values, distractors, and outdated states, causing repeated or missed operations. To address this, we propose Active Task Driving Memory (ATMem), which shifts GUI-agent memory from passive storage to an actively maintained execution state. ATMem maintains task-relevant information as a continually updated execution state that links each value to its role and current status, enabling action selection based on the current workflow state. While supervised fine-tuning enables the agent to construct ATMem, it does not teach when ATMem is beneficial. We therefore introduce STR-GRPO, an online reinforcement learning method that encourages selective use of ATMem based on its contribution to task completion. STR-GRPO contrasts memory-on and memory-off rollouts to estimate when memory use improves execution, while memory-cost-aware reward discourages costly memory usage that does not improve execution. To evaluate whether agents can complete all in-scope work while avoiding out-of-scope actions, we build a challenging mobile benchmark. From a list of near identical entries, agents must act on every entry that satisfies the instruction and reject entries that violate its constraints. We further introduce App-Level Progress and Scope-Aware F1 to measure these two dimensions separately.

Abstract

Explore similar work

What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States

STAMP: Training Explicit Memory for Mobile GUI Agents in Controllable and Scalable Virtual Environments

PreAct: Computer-Using Agents that Get Faster on Repeated Tasks