From Interaction to Capability: Task Advancement as State and Supervision for GUI Agents

Ziyang Chen; Qian Li; Chengwei Qin

cs.AIOct 27, 2026

From Interaction to Capability: Task Advancement as State and Supervision for GUI Agents

Authors: Ziyang Chen, Qian Li, Chengwei Qin

Organizations: The Hong Kong University of Science and Technology (Guangzhou) · Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University

Abstract

GUI memory should preserve the results of past actions and learn reusable operations. This is difficult in long-horizon workflows, where agents must use information across applications and useful operations are interleaved with failed attempts. We introduce Task Advancement Memory (TAM), which defines task advancement as verified task progress, and supports ongoing task execution and learns reusable operations. During execution, Task Advancement State represents how verified task progress changes the remaining goals, allowing the agent to build on earlier achievements across applications. After each task, advancement supervision identifies and abstracts the interactions that produced a verified local result. The learned operations preserve the relation between required inputs, expected outcomes, and verification conditions. This retains locally successful operations from both successful and failed tasks. On MobileWorld GUI-only tasks, Advancement State alone raises success from 12.0% to 28.2%. Full TAM achieves a 17.6% relative gain in task success over the strongest evaluated baseline and reduces lifecycle tokens per solved task by 38.7%. On SPA-Bench, TAM achieves state-of-the-art task success in the single-app English Level 3 setting.

Figures & tables

Figure 1: Task advancement connects execution and learning. (a) To create the calendar event, the agent must retain the meeting time after leaving the email. (b–c) Recording a save action leaves its outcome uncertain. TAM retains the confirmed event and its details, making participant notification the remaining goal. (d) Even if notification fails, the verified event-creation operation can be reused with new meeting details.

Figure 2: Overview of TAM. Task Advancement State guides cross-application execution through task reduction. After the route query, the remaining task is to notify Susan, with the arrival time already specified. Advancement supervision learns a reusable operator from the successful query, even if the overall task fails. Later tasks reuse the operator’s goal and verification condition with new locations, adding a recorded GUI route when compatible.

Method	Success (%) $\uparrow$	Steps $\downarrow$	EQA@50 (%) $\uparrow$	LTS (M) $\downarrow$
Qwen3-VL-32B-Instruct ( Bai et al., 2025 )	12.0	–	–	–
AWM ( Wang et al., 2025b )	13.7	25.3	11.9	0.720
Mobile-Agent-E ( Wang et al., 2025a )	15.4	22.9	12.2	4.472
StructAgent ( Wu et al., 2026 )	24.8	27.8	19.1	$\geq 1.115^{\ast}$
GUI-explorer ( Xie et al., 2025 )	25.6	27.4	20.6	$\geq 1.478^{\ast}$
MemGUI-Agent (ConAct) ( Liu et al., 2026b )	29.1	22.4	23.7	1.257

Table 1: Results on MobileWorld GUI-only tasks. All methods use Qwen3-VL-32B-Instruct and a limit of 50 controller rounds. EQA@50 jointly rewards success and earlier completion. LTS measures the total model-token expenditure required to obtain a solution.

Method	Input	Base Model	Success (%) $\uparrow$
CogAgent ( Hong et al., 2024 )	screen	CogAgent	0.0
DigiRL ( Bai et al., 2024 )	screen	DigiRL	0.0
AutoDroid ( Wen et al., 2024 )	a11y tree	GPT-4o	12.0
SeeAct ( Zheng et al., 2024a )	SoM	GPT-4o	12.0
AppAgent ( Zhang et al., 2025 )	SoM	GPT-4o	14.0
Mobile-Agent-v2 ( Wang et al., 2024 )	SoM	GPT-4o	20.0

Table 2: Task success on SPA-Bench single-app English Level 3. Results for the first seven baselines and GUI-explorer are taken from Xie et al. (2025) .

Table 5

Figure 3: Mechanism analysis. (a) Task-averaged information identification and target use over 52 values from 21 tasks. (b) State availability and target use on 18 field instances with source inputs observed in both conditions. (c) Candidate and retained step counts, normalized by source-trajectory length.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Call	Inputs	Outputs
Compiler	Instruction and requirement schema	Requirements, types, dependencies, and conditions
Actor	Instruction, live observation, recent context, and TAM readout	Observation updates, execution summary, and one primitive action
Outcome extraction	Relevant unresolved outcome and observed action result	Proposed relation, bound values, and supporting evidence
Task-end review	Candidate segments, entry snapshots, result evidence, and nearby operators	Local support judgment, retained steps, citations, and operator description

Appendix

Table 5: Inputs and outputs of TAM’s model calls. Outcome extraction is triggered for relevant unresolved outcomes; task-end review processes recorded candidates.

Setting	MobileWorld	SPA-Bench
Temperature	0	0
Actor output limit (tokens)	8,192	1,024
Recent action records	6	5
Operator context (tokens)	900	900
Persistent operator capacity	512	512

Appendix

Table 6: TAM reference implementation defaults. Baseline controllers retain their own history management and role-specific output limits.

Configuration	State	Cache	Graph
Base agent	–	–	–
Advancement State	$\checkmark$	–	–
State + Evidence Cache	$\checkmark$	$\checkmark$	–
State + Operator Graph	$\checkmark$	–	$\checkmark$
Full TAM	$\checkmark$	$\checkmark$	$\checkmark$

Appendix

Table 7: Component ablations. State includes task compilation and advancement updates. Graph includes both retrieval and task-end learning.

Figure 4: Information requirements in the MobileWorld GUI-only inventory. Bars show within-group percentages and labels give task counts. Categories describe each task’s principal requirement. Application counts follow benchmark metadata.

Figure 5: Availability after acquisition. Tasks contribute when they have qualifying later decisions: 32 overall, 24 in a different application from the source, and 22 outside current XML and recent text. Points show task rates and diamonds their means. Horizontal jitter separates overlapping points.

Figure 6: Ten-decision follow-up of every binding with a complete observation window, yielding 106 bindings from 24 tasks. Both curves use these same values throughout, with equal task weights, and compare text locations within TAM executions.

Figure 7: Operation preparation and use in (a) the full diagnostic panel and (b) the shared-source subset of ten instances. The 20 full-panel instances comprise ten operations evaluated in two saved revisions. Each line joins TAM and flat-state percentages on the same set of operations. A criterion requires every audited parameter to satisfy it.

Context	Task mean	Query mean	Tokens
TAM state	96.2	95.0	4,040
Ordinary notes	80.8	80.0	4,003
Full XML/actions	92.3	90.0	6,439
XML + sampled images	88.5	85.0	19,347
Recent context	26.9	25.0	3,883

Appendix

Table 8: Readout accuracy (%) on the same 20 queries and mean input tokens per readout. Task means give equal weight to each of the 13 tasks. Query means give equal weight to each query.

Figure 8: Readout accuracy by context. Each point compares an alternative context with the same TAM state condition. Axes give task-averaged accuracy on identical questions. Points above the dashed diagonal favor TAM.

Context	Incomplete	Time	Amount
TAM state	0	0	1
Ordinary notes	3	1	1
Full XML/actions	0	1	1
XML + sampled images	0	2	1
Recent context	22	0	1

Appendix

Table 9: Errors among the same 28 requested fields per context. Incomplete fields contain missing or partial answers. Time and amount errors concern derived values. All observed field errors are included.

Figure 9: Correctness and coverage of retained learning material. Correctness is the supported fraction of retained items. Coverage is the retained fraction of all supported items. Policies share the same pool within each panel. Source-value weights give tasks equal representation before selection.

Weighting	Selection	Correctness	Coverage
Task	All	88.0	100.0
	Success-only	99.4	64.3
	TAM storage	93.8	88.3
Reference	All	84.2	100.0
	Success-only	99.0	63.6
	TAM storage	92.3	81.6

Appendix

Table 10: Correctness and coverage (%) under fixed weights on the same 234 complete fields. Weights are the inverse of each group’s size in the full candidate pool. Supported values remain in the coverage denominator when their group retains none.

Figure 10: Timing within failed workflows containing a source-checked correct candidate, with tasks weighted equally. Both curves use the original recording time. The dashed curve tracks records subsequently retained for learning, with the same execution denominator.

Information type	Correct material retained	Executions yielding retained material	Correct record by midpoint	Median first-record position
Amounts and quantities	100.0	100.0	50.0	34.6
Contact and address details	66.7	66.7	62.5	33.3
Dates and times	79.4	77.9	87.5	20.0
All (deduplicated)	74.7	74.8	72.9	28.6

Appendix

Table 11: Retention and timing in failed workflows containing correct candidates (%). Rates give equal weight to tasks within each information type. Positions are fractions of executed decisions, and the median uses the task-weighted timing distribution. Types may share executions, and the overall row uses their deduplicated union.

Figure 11: A recorded source-to-target chain. The failed source workflow correctly obtains a phone number. Its contact-acquisition operator is later recalled without the old number or GUI route. The target actor reads a different number from a resume and enters it as the SMS recipient. Panels use cropped recorded screenshots. Enlarged details in (b) show the phone number wrapped across two lines.

Figure 12: Two diagnostic subsets of the full 86-field target audit. (a) The 24 newly acquired inputs with a changed reference and comparable source and target fields. (b) The 29 fields matched to executed text entry. The two orange cases in (b) comprise one incorrect value and one correct value entered only in an unrelated field. Unresolved cases remain visible.

Figure 13: Resource allocation for all 96 tasks with valid archived executions using the shared actor prompt format. Shares refer to tokens used by additional calls and actor text devoted to state and operator guidance. All task values are shown. Boxes span the interquartile range, orange lines mark medians, and whiskers extend to observations within 1.5 interquartile ranges of the box.

Analysis	Source	Inclusion rule and purpose	Evaluated units
Diagnostic scope	B	Every official GUI-only task. Task definitions identify the acquired-content transfers to which field scoring applies.	117 tasks; 52 transfer tasks
Information use	H	Common references with a source and a downstream destination. The same fields are scored for all five methods.	21 tasks; 52 values
Value persistence	H	Source-verifiable acquisitions followed at every later decision. The temporal comparison requires ten subsequent decisions to keep the same values throughout.	202 bindings in 32 tasks; ten-step subset: 106 in 24 tasks
Operation preparation	P	Structurally selected diagnostic tasks. All catalogued operations enter the full panel. Shared-source cases test organization when both runs encounter the required inputs.	20 operation instances; 10 shared-source instances containing 18 fields
Readout	Q	Cross-app access to captured facts, with at most two claims per task, then all source-resolvable queries across the same five contexts.	20 queries; 13 tasks; 100 responses
Supervision scope	H	Supported candidates with confirmed persistent writes, paired before and after review; equal task weights.	335 segments; 215 episodes; 36 tasks

Appendix

Table 12: Sources and statistical units of the diagnostic analyses. B: benchmark metadata; H: historical executions; P: paired state control; Q: fixed-checkpoint readout.

Selection	Supported	Contradicted	Unassessed
All admitted	189	16	167
Success-only	148	2	87
TAM storage	150	7	108

Appendix

Table 13: Accounting for all 372 admitted resource candidates. A candidate is supported when all its specified reference slots agree, contradicted when any disagrees, and otherwise unassessed. This candidate-level table complements the value-level quality comparison.

GUI Agents

Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.

See all 178 papers

What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States
Jun 30, 2026Chen Liu, Ling Chen, Hanzhang Zhou +7RL for GUI Agents GUI Agents
University of Technology Sydney · Tongyi Lab, Alibaba Group · Adelaide University
AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents
Mar 19, 2026Yibo Shi, Jungang Li, Linghao Zhang +25Agent Memory Memory-Augmented Agents
XJTU · HKUST(GZ) · HKUST +8
Executable Agentic Memory for GUI Agent
May 12, 2026Zerui Qin, Sheng Yue, Xingyuan Hua +2GUI Agents LLM Agent Memory
1Tsinghua University, China · 2Sun Yat-sen University, China

Jun 30, 2026J/K move · Enter open · S save

What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States

Chen Liu, Ling Chen, Hanzhang Zhou, Xu Zhang+6

University of Technology Sydney · Tongyi Lab, Alibaba Group · Adelaide University

RL for GUI Agents GUI Agents Mobile GUI Agents Long-Horizon Agent Tasks Agent Memory Management RL for LLM Agents

Open paper AskPDF arXiv

Mobile GUI agents increasingly face long-horizon tasks that require reading, updating, and reusing task-relevant data across pages and applications. Existing methods treat memory largely as passive storage, where past observations are accumulated and retrieved when needed. Yet retrieving a value does not reveal its current role in the workflow. The agent must still infer from accumulated records whether the value should be used now, has already been used, or must wait for a later dependency. This implicit reconstruction becomes unreliable in long trajectories with repeated values, distractors, and outdated states, causing repeated or missed operations. To address this, we propose Active Task Driving Memory (ATMem), which shifts GUI-agent memory from passive storage to an actively maintained execution state. ATMem maintains task-relevant information as a continually updated execution state that links each value to its role and current status, enabling action selection based on the current workflow state. While supervised fine-tuning enables the agent to construct ATMem, it does not teach when ATMem is beneficial. We therefore introduce STR-GRPO, an online reinforcement learning method that encourages selective use of ATMem based on its contribution to task completion. STR-GRPO contrasts memory-on and memory-off rollouts to estimate when memory use improves execution, while memory-cost-aware reward discourages costly memory usage that does not improve execution. To evaluate whether agents can complete all in-scope work while avoiding out-of-scope actions, we build a challenging mobile benchmark. From a list of near identical entries, agents must act on every entry that satisfies the instruction and reject entries that violate its constraints. We further introduce App-Level Progress and Scope-Aware F1 to measure these two dimensions separately.

Abstract

GUI Agents

Explore similar work

What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States

AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agents

Executable Agentic Memory for GUI Agent