Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution

Wenkui Yang; Yiming Hao; Kai Zhou; Chengyu Pu; Yuzhe Zhang; Jie Cao; Jin Huang; Shuai Ma

cs.HCOct 14, 2026

Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution

Authors: Wenkui Yang, Yiming Hao, Kai Zhou, Chengyu Pu, Yuzhe Zhang, Jie Cao, Jin Huang, Shuai Ma,

Organizations: SAIS, University of Chinese Academy of Sciences, Beijing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Meituan, Beijing, China · Peking University, Beijing, China ·

Abstract

Graphical user interface (GUI) agents now execute workflows across real applications, but when execution drifts, users must determine what changed, what should have remained untouched, and what can still be recovered. Plan-centric interfaces leave users supervising procedure, but users live with changes to application state. We introduce task-state governance, an interaction paradigm that represents a task as an editable contract linking desired outcomes, facts that must be preserved, and observed application state. We instantiate this paradigm in TaskVM, which projects live state into editable task controls, executes user edits as genuine interface actions, verifies outcomes through independent observation, and marks each action as reversible, compensatable, or irreversible. Across an automated contract benchmark, a controlled user study, and an open live-model exploration, the paradigm strengthened contract compliance and users' recovery control and trust, while execution outcomes were held constant. Task contracts open a design space for governable agent interfaces.

Figures & tables

Figure 1. Task-state governance on a live multi-application workflow. Rather than supervising an agent’s low-level execution trajectory, the user governs a unified Task-state VM. Observed application state (bottom left) is projected into interactive task variables (desired, preserve, observed); edits are translated into real GUI gestures by the GUI Encoder and Executor, while an Independent Verifier checks fresh visible observations against non-interference constraints. Image is a semantically translated replica of the production interface.

Figure 2. The TaskVM governance interface (a faithful English rendering of the production UI, redrawn for language accessibility). A Task Intent & Global Governance: the current mission and task contract, lifecycle controls (start, pause, resume, checkpoint, rollback), and a global rollback dialog that previews which committed steps would be replayed or restored and flags effects that are not auto-compensable. B Execution Progress & Local Governance: live status of the four model-invoking roles (state compiler, task architect, CUA executor, verifier), task-topology progress, local checkpoint options, and the model call chain. C Editable Task State & Live World: the desired–preserve–observed triad rendered beside the live GUI, with patch and conflict controls and a runtime-state card linking every variable to visible evidence.

Figure 3. TaskVM data flow as a hub-and-spoke architecture. The kernel is the sole owner of task state; six spokes exchange typed objects with it, and the substrate port is the only door to the real GUI world ( observe() in, act(GuiAction) out; real gestures only, with no backdoor writes). Every arrow is labeled with the actual data object that moves. The label prefix F1 to F8 indexes the eight semantic stages of one governed round trip (top band), so a number denotes stage membership , not size or importance; a prime ( ′ ) marks the paired/return process of the same stage rather than a different stage (for example, F7 fresh observation and F7 ′ verdict are the question and the answer of Verify ). Amber cards host model-invoking roles (architect, state compiler, CUA model); violet equals governance, teal equals fresh evidence, blue equals real GUI.

Panel of Figure 4 — Figure 4. Stage 1 automated technical evaluation results on the frozen suite of 36 task instances spanning 12 structural task families. (a) Overall bars come from the skills-OFF main wave; zone bars come from the skills-ON comparison wave with unmatched denominators, as documented in Table 2 .

Condition	COV strict rate	COV atomic mean	Model calls	Wall time (s)
TaskVM (ours)	0.583 (21/36)	0.870	1,126	19,960
Direct-CUA	0.500 (18/36)	0.761	1,361	9,010
Planner-CUA	0.472 (17/36)	0.737	1,031	6,695
Template Control †	0.083 (3/36)	0.712	144	8,527
TaskVM advantage over baselines
vs Direct-CUA	+0.083 (+16.6%)	+0.109 (+14.3%)	$\cdot$	$\cdot$

Table 1. Stage 1 automated bench under skills injection OFF, 36 task instances per condition. TaskVM achieves the highest Contract-Outcome Verification (COV) strict pass rate and the highest atomic predicate mean; the lower block reports its advantage over each baseline.

Zone-S: sequential / simple families ( sqv , fav , lpv )
Zone / Condition	TaskVM	Direct-CUA	Planner-CUA
COV strict rate	0.556 (5/9)	0.467 (7/15)	0.533 (8/15)
COV atomic mean	0.852	0.733	0.767
Zone-G: governance-dense families
COV strict rate	0.714 (10/14)	0.571 (12/21)	0.476 (10/21)
COV atomic mean	0.922	0.822	0.797

Table 2. COV results split by task zone (skills injection ON). Zone-G families exercise intent landing, supersession, pause boundaries, and reversible restore. Subsets and denominators differ across conditions; rates describe the tested populations and are not matched contrasts.

CFE non-inferiority, TaskVM vs Direct-CUA (skills-ON wave)
Paired family flip	TaskVM	Direct-CUA	Planner-CUA
fail $\rightarrow$ pass (skills helped)	4	0	0
pass $\rightarrow$ fail (skills hurt)	3	0	0
unchanged	14	18	18
missing (no OFF data)	0	3	3
Net gain	+1	0	0

Table 3. Effects of procedural skills injection. Upper block: paired family-level outcome flips between the skills-OFF and skills-ON waves. Lower block: contract-faithful execution (CFE) non-inferiority of TaskVM against its strongest baseline within the skills-ON wave, cluster-bootstrapped (10,000 resamples, 35 paired clusters).

Measure	TaskVM	Conv CUA	$\Delta$	Paired counts	$p$	$d_{z}$
Subjective measures (1–7 unless noted; higher is better except NASA-TLX)
Trust (2-item TIA composite $\uparrow$ )	5.54	4.36	+1.18	21 / 7	.0030	+0.55
Recovery control (4 study-specific items $\uparrow$ )	5.56	4.63	+0.93	22 / 13	.0050	+0.51
Stop and redirect ( rc_stop_redirect )	5.68	4.43	+1.24	23 / 7	.0035	+0.58
Correct errors ( rc_correct )	5.49	4.59	+0.89	20 / 10	.0160	+0.42
Overall control ( rc_overall )	5.35	4.35	+1.00	19 / 9	.0099	+0.44

Table 4. Stage 2 within-subject user study evaluation (

N=37

). We report means, paired differences (

\Delta

), directional participant counts (in favor of TaskVM / in favor of Conversational CUA), Wilcoxon signed-rank

p

-values, and standardized effect sizes (

d_{z}

). Bold entries are statistically significant at

p<.05

. Write-probe margins are a manipulation check of the replay protocol: the environment and the replayed trajectory were identical across arms, so identical margins indicate that neither arm offered systematically easier intervention opportunities, not that the interfaces would execute with equivalent competence outside the rig.

Figure 5. Expanded governance interactions in TaskVM, shown as explanatory English redraws. (E1) Planned checkpoint modes. (E2) A rollback preview distinguishing restorable effects from non-compensatable residuals. (E3) An evidence drawer presenting execution and verification records. (E4) A command palette for governance actions. These panels illustrate interface functions; they are not a quantitative summary of Stage 3 use.

Paired contrast	mean $\Delta$	$d_{z}$	lower / higher	Wilcoxon $p$
Stage 3 vs Stage 2 TaskVM arm	$-9.70$	$-0.83$	12 / 2	.011
Stage 3 vs Stage 2 Conversational arm	$-34.40$	$-1.43$	14 / 0	$<$ .001
Stage 3 vs Stage 2 two-arm mean	$-22.05$	$-1.80$	14 / 0	$<$ .001

Table 5. Stage 3 exploratory follow-up with a live model on participant-authored tasks (

N=14

returning participants). NASA-TLX total workload is paired against each participant’s own Stage 2 scores. All 14 participants scored lower in Stage 3 than in their earlier conversational-arm sessions. Comparisons are reported descriptively, since task selection, practice, study procedures, and execution mode changed together across stages.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Predicate	Meaning (what it certifies)
required_final_state	The target fields specified by the task end in the desired state.
protected_non_interference	No observable field in the protected set was modified.
required_witness_trajectory	The execution visited the witness checkpoints that prove the right path was taken.
latest_intent_landed	The most recent user intent is the one reflected in the final world.
superseded_intent_suppressed	A superseded intent did not leave residue effects behind.
pause_stop_boundary	A pause or stop was honored at its transactional boundary.

Appendix

Table 6. COV predicates and what each certifies.

System	Governing object (what a task is)	Obligation created by an edit	Evidence of completion	Recovery model	Scope
Plan- and execution-centric oversight
Plover ( Venkatesan et al., 2026 )	A structured plan over executable steps, locally revisable	Replanning of upcoming steps; executed history stays immutable	Executor step outcomes plus terminal screenshot check	Retry of pending steps within plan scope	Desktop GUI environments
Sidekick ( Chang et al., 2026 )	Concurrent agent tracks and their status	Steering of which track continues and how	Agent-reported status indicators	Resumption of suspended tracks	Multi-task workflow context
AgentLens ( Kim et al., 2026 )	An adaptive visual stream of the mobile UI	Which view renders and which touches are forwarded	Native UI layout inspection	Not part of the reported model	Mobile OS application layer
ALLOY ( Li et al., 2025 )	A reusable workflow graph	Which workflow is replayed and with what bindings	Per-step execution assertions	Workflow modification; state rollback not in the reported model	Web browser DOM tasks
Malleable and generative views

Appendix

Table 7. TaskVM in the design space of interactive agent and malleable interface systems, organized by the primary object the user governs. The remaining columns trace what follows from that object: the obligation an edit creates, the evidence that counts as completion, and the meaning of recovery. The final row states the state-contract stance examined in this paper.

Figure 6. Stage 2 baseline Conversational CUA interface: an application screencast overlaid with a natural language chat drawer. Due to language localization requirements, the interface has been semantically redrawn in the target language.

Figure 8. Per-family strict COV pass rates across conditions on the frozen suite (36 instances in 12 structural families), grouped into Zone-S (structural and execution families) and Zone-G (governance-dense families).

Family / Instance	Zn	TM	DC	PC	Family / Instance	Zn	TM	DC	PC
Zone-S (structural / execution)					Zone-G (governance-dense), continued
Loop blv-01	S	✓	✓	✓	UI drift drv-02	G	–	–	–
Loop blv-02	S	–	✓	✓	UI drift drv-03	G	✓	–	–
Loop blv-03	S	✓	✓	–	Interrupt itv-01	G	✓	✓	✓
Fan-out fav-01	S	✓	✓	✓	Interrupt itv-02	G	✓	✓	✓
Fan-out fav-02	S	–	–	–	Interrupt itv-03	G	–	–	✓

Appendix

Table 8. Per-instance strict COV outcomes in the frozen skills-OFF wave (36 instances in 12 families), using the same family-to-zone dictionary as Figure 8 . Zone-S (structural/execution): Loop blv , Fan-out fav , Partial-failure pfv , Cross-app cav , Sequence sqv . Zone-G (governance-dense): Conflict cfv , UI-drift drv , Interrupt itv , Local-patch lpv , Goal-patch gpv , Irreversible irv , Rollback rbv . Totals: TaskVM 21/36, Direct-CUA 18/36, Planner-CUA 17/36.

Task	Archetype	Participant goal [translated]	Mid-task event (trigger)	Budget
r8-t1a	pause–retarget	Social feed: like the post mentioning the “overtime mobilization meeting”.	Experimenter announces the target changed to the “core CPI surprise drop” post; a similar decoy post exists (bound, pre-first-effect).	4 min
r8-t1b	pause–retarget	Task board: mark “organize the smoke-test checklist” as done.	Target changes to “update dependency list” (same trigger).	4 min
r8-t2a	mixed rollback	Social feed: like and bookmark the overtime-meeting post.	After the first effect lands, the experimenter requests an undo; the interface must display the reversibility boundary.	5 min
r8-t2b	mixed rollback	Chat: send a contact the message “weekly meeting moved to Thursday 2 pm”.	After send confirmation, undo is requested; a sent message is irreversible and must be honestly reported as such.	5 min
r8-t3a	external drift	Social feed: like the “team-building photo collection” post.	World-side drift is injected after target binding, before the effect lands; the participant must reconcile the conflict.	4 min
r8-t3b	external drift	Calendar: change the “daily standup” RSVP from accept to tentative .	Same drift injection (bound, pre-effect).	4 min

Appendix

Table 9. Stage 2 task battery (eight tasks, four pairs). Trigger points were determined by normalized execution events, not fixed wall-clock delays. Goal texts translated from the Chinese study script.

PID	Episodes	Model calls	#Apps	#Form.
P01	4	74	2	4
P02	3	52	2	2
P05	3	9	2	2
P06	4	42	1	3
P10	3	93	3	3
P13	3	149	3	2

Appendix

Table 10. Stage 3 usage per returning participant, as retained in the anonymized archive. #Form. = distinct goal formulations issued.

Theme	Content	Participants	Representative quote [translated]
F1	Faithful execution and precise intent understanding	P21, P37, P27, P25 (4)	“The system recognized my intent and made the keyword changes, and it did not over-guess what I wanted to do.” (P27)
F2	Governance features used and valued mid-flight	P35, P30 (2)	“Mid-execution I added an instruction to ‘ask Situ Nian whether she can work overtime’; the agent finished the original task, then correctly found the contact and surfaced the prompt. I found this delightful.” (P30)
F3	Participants spontaneously request stronger governance automation	P30, P27 (2)	“Automatically set a checkpoint at each node, so I can roll back at any time.” (P30); “I hope the system can autonomously identify the critical points where problems may occur.” (P27)
L1	Offline simulator has finite data coverage	P35, P06, P13 (3)	“I wanted to search for Valorant, cats, and the ‘Blue-and-White Porcelain’ MV, but the world model had none of them, which was quite frustrating.” (P13)
L2	Executor (model) reliability, orthogonal to the interface	P10, P20, P13, P22 (4)	A complex task “fell into a loop of repeated comments” (P20); repeated retries of the same send action (P13).
L3	UI flow and tutorial coverage	P06, P01, P05, P20, P13, P25, P02 (7)	Progress not visible during retries (P06); features discovered too late, i.e., tutorial coverage rather than missing capability (P01, P05).

Appendix

Table 11. Stage 3 open-ended response evidence map. F = favourable theme, L = limitation theme. [translated] from Chinese originals.

Figure 9. Stage 3 cockpit screenshot voluntarily submitted by participant P30 immediately after a self-directed session. The task was an ordered four-message chain in the simulated messaging application: rescheduling a quarterly budget review to two designated contacts, followed by a mid-chain correction. After the chain, the participant appended a self-authored final instruction asking one contact whether she could work overtime that week. The left cockpit shows all four plan nodes passed and the goal status done (73 model calls, matching the session ledger); the right panel shows the delivered messages inside the simulated app, including the appended overtime question, illustrating that a mid-flight user instruction was honored only after the original ordered chain had completed. Every personal name visible in this screenshot (e.g., the contact “Situ Nian”) is a programmatically injected fictional identity of the offline simulator’s seeded corpus; it does not denote any real person, any resemblance to a real individual is purely coincidental, and its appearance does not constitute a failure of anonymization.

Figure 10. Stage 3 screenshot voluntarily submitted by participant P06 at the moment the system reported task completion. The participant authored the task “search Bilibili for an Arena of Valor video and like it”; the image captures the verification-pass state (banner “Verification passed, task completed”; all six plan nodes verified, 19 model calls, zero governance interventions used), while the right panel shows the liked video inside the simulated video application. Any personal name that may appear in this screenshot is a programmatically injected fictional identity of the offline simulator’s seeded corpus; it does not denote any real person, any resemblance to a real individual is purely coincidental, and its appearance does not constitute a failure of anonymization.

Module	Item	Text	Scale
NASA-TLX	tlx_mental	How mentally demanding was the task?	0–100
	tlx_physical	How physically demanding was the task?	0–100
	tlx_temporal	How hurried or rushed was the pace of the task?	0–100
	tlx_performance	How successful were you in accomplishing what you were asked to do?	0–100 (Perfect $\to$ Failure)
	tlx_effort	How hard did you have to work to accomplish your level of performance?	0–100
	tlx_frustration	How insecure, discouraged, irritated, stressed, and annoyed were you?	0–100

Appendix

Table 12. Condition-end questionnaire: all administered items (English strings verbatim).

Stage 1 (automated benchmark, frozen waves)
Condition	Model roles	Notes
TaskVM	GPT-5.6-sol $\times$ 4 roles: state compiler, task architect, CUA executor, independent model verifier	Verifier reads fresh screenshots only; per-trial role call ledgers retained
Direct-CUA	GPT-5.6-sol, single agentic loop	Same seeds, same substrate
Planner-CUA	GPT-5.6-sol, planner then executor	Same seeds, same substrate
Template Control	none (scripted gesture driver)	Assembly sanity check; zero LLM calls
Stage 2 (user study)

Appendix

Table 13. Model and runtime configuration by stage and condition. All LLM roles used the same production model; differences are architectural, not model-level.

Dimension	Category	$n$
Gender (self-reported)	Female / Male / Undisclosed	15 / 21 / 1
Age band	18–24 / 25–34 / 35–44	19 / 15 / 2
Affiliation sector	Academic / Industry	32 / 4
Field of study ( $n{=}35$ )	CS/AI disciplines	20
	EE / basic sciences / other STEM	9
	Business / humanities / other	6

Appendix

Table 14. Aggregate demographics and background of the Stage 2 sample (

N{=}37

). Gender, age, and affiliation from the compensation record (

n{=}36

); field and agent-related items from the background questionnaire (

n{=}35

Item / code	Original option wording (abridged)
Q1 ai_ml	Artificial Intelligence / Machine Learning / Deep Learning
Q1 cv	Computer Vision / Image Processing
Q1 nlp	Natural Language Processing / Large Language Models
Q1 robotics	Robotics / Embodied Intelligence / Automation / Control
Q1 hci	Human–Computer Interaction / UX / Interaction Design
Q1 systems	Computer Systems / Operating Systems / Distributed Systems

Appendix

Table 15. Mapping from dataset option codes to the original questionnaire wording (abridged English transcription; the full five-item instrument, including its warning boxes distinguishing GUI agents from other AI tools, is in the supplementary material).

PID	Gender	Age	Sector	Field of study (specialization)	AgentBG	GUI-use	S3
P01	F	25–34	Ind.	AI / ML / DL (computer vision)	2	Daily	✓
P02	M	25–34	Acad.	AI / ML / DL (computer vision)	3	Multi-daily	✓
P03	M	18–24	Acad.	NLP / LLMs (opponent modeling)	4	Multi-daily
P04	F	18–24	Acad.	HCI / UX (AI–human interaction)	4	Tried
P05	M	25–34	Acad.	Computer Systems (computer architecture)	3	Tried	✓
P06	M	18–24	Acad.	Math. / Physics / Basic Sci. (systems analysis & integration; data fusion)	2	Never	✓

Appendix

Table 16. Per-participant demographics and background roster (

N{=}37

). Age = participant’s band from the exact self-reported age; Sector: Acad. = university or public research institute, Ind. = industry employer (institution names withheld against re-identification); Field = questionnaire category with the participant’s free-text specialization in parentheses [translated]; AgentBG = agent professional background level (Table 15 ); GUI-use = self-reported computer-use/GUI-agent usage frequency over the past 6 months; S3 = returned for Stage 3.

GUI Agents

Graphical user interface (GUI) agents perceive and operate apps through screens, controls, and other software interfaces.

See all 178 papers

From Interaction to Capability: Task Advancement as State and Supervision for GUI Agents
Oct 27, 2026Ziyang Chen, Qian Li, Chengwei QinGUI Agents Procedural Memory for LLM Agents
The Hong Kong University of Science and Technology (Guangzhou) · Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University
StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents
Jul 24, 2026Yan Yang, Xiangru Jian, Ziyang Luo +7Computer-Use Agents Tool-Using Agents
Salesforce AI Research
Stateful Governance for Concurrent Agentic Systems
Aug 3, 2026Yuxiang Peng, Xiaodi WuAgentic Workflows AI Agent Governance
Purdue University · West Lafayette, Indiana, USA · University of Maryland, College Park +1

Oct 27, 2026J/K move · Enter open · S save

From Interaction to Capability: Task Advancement as State and Supervision for GUI Agents

Ziyang Chen, Qian Li, Chengwei Qin

The Hong Kong University of Science and Technology (Guangzhou) · Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University

GUI Agents Procedural Memory for LLM Agents Long-Horizon Agent Tasks

Open paper AskPDF arXiv

GUI memory should preserve the results of past actions and learn reusable operations. This is difficult in long-horizon workflows, where agents must use information across applications and useful operations are interleaved with failed attempts. We introduce Task Advancement Memory (TAM), which defines task advancement as verified task progress, and supports ongoing task execution and learns reusable operations. During execution, Task Advancement State represents how verified task progress changes the remaining goals, allowing the agent to build on earlier achievements across applications. After each task, advancement supervision identifies and abstracts the interactions that produced a verified local result. The learned operations preserve the relation between required inputs, expected outcomes, and verification conditions. This retains locally successful operations from both successful and failed tasks. On MobileWorld GUI-only tasks, Advancement State alone raises success from 12.0% to 28.2%. Full TAM achieves a 17.6% relative gain in task success over the strongest evaluated baseline and reduces lifecycle tokens per solved task by 38.7%. On SPA-Bench, TAM achieves state-of-the-art task success in the single-app English Level 3 setting.

Abstract

GUI Agents

Explore similar work

From Interaction to Capability: Task Advancement as State and Supervision for GUI Agents

StateAct: Program State, before Pixels, for Long-Horizon Computer-Use Agents

Stateful Governance for Concurrent Agentic Systems