Task-State Governance: A Virtual Machine for Verifiable and Recoverable GUI Agent Execution
Authors: Wenkui Yang, Yiming Hao, Kai Zhou, Chengyu Pu, Yuzhe Zhang, Jie Cao, Jin Huang, Shuai Ma, Feng Tian
Organizations: SAIS, University of Chinese Academy of Sciences, Beijing, China · Institute of Automation, Chinese Academy of Sciences, Beijing, China · Meituan, Beijing, China · Peking University, Beijing, China · Institute of Software, Chinese Academy of Sciences, Beijing, China
Graphical user interface (GUI) agents now execute workflows across real applications, but when execution drifts, users must determine what changed, what should have remained untouched, and what can still be recovered. Plan-centric interfaces leave users supervising procedure, but users live with changes to application state. We introduce task-state governance, an interaction paradigm that represents a task as an editable contract linking desired outcomes, facts that must be preserved, and observed application state. We instantiate this paradigm in TaskVM, which projects live state into editable task controls, executes user edits as genuine interface actions, verifies outcomes through independent observation, and marks each action as reversible, compensatable, or irreversible. Across an automated contract benchmark, a controlled user study, and an open live-model exploration, the paradigm strengthened contract compliance and users' recovery control and trust, while execution outcomes were held constant. Task contracts open a design space for governable agent interfaces.
Figures & tables
Figure 1. Task-state governance on a live multi-application workflow. Rather than supervising an agent’s low-level execution trajectory, the user governs a unified Task-state VM. Observed application state (bottom left) is projected into interactive task variables (desired, preserve, observed); edits are translated into real GUI gestures by the GUI Encoder and Executor, while an Independent Verifier checks fresh visible observations against non-interference constraints. Image is a semantically translated replica of the production interface.
Figure 2. The TaskVM governance interface (a faithful English rendering of the production UI, redrawn for language accessibility). A Task Intent & Global Governance: the current mission and task contract, lifecycle controls (start, pause, resume, checkpoint, rollback), and a global rollback dialog that previews which committed steps would be replayed or restored and flags effects that are not auto-compensable. B Execution Progress & Local Governance: live status of the four model-invoking roles (state compiler, task architect, CUA executor, verifier), task-topology progress, local checkpoint options, and the model call chain. C Editable Task State & Live World: the desired–preserve–observed triad rendered beside the live GUI, with patch and conflict controls and a runtime-state card linking every variable to visible evidence.
Figure 3. TaskVM data flow as a hub-and-spoke architecture. The kernel is the sole owner of task state; six spokes exchange typed objects with it, and the substrate port is the only door to the real GUI world ( observe() in, act(GuiAction) out; real gestures only, with no backdoor writes). Every arrow is labeled with the actual data object that moves. The label prefix F1 to F8 indexes the eight semantic stages of one governed round trip (top band), so a number denotes stage membership , not size or importance; a prime ( ′ ) marks the paired/return process of the same stage rather than a different stage (for example, F7 fresh observation and F7 ′ verdict are the question and the answer of Verify ). Amber cards host model-invoking roles (architect, state compiler, CUA model); violet equals governance, teal equals fresh evidence, blue equals real GUI.
Figure 4. Stage 1 automated technical evaluation results on the frozen suite of 36 task instances spanning 12 structural task families. (a) Overall bars come from the skills-OFF main wave; zone bars come from the skills-ON comparison wave with unmatched denominators, as documented in Table 2 .
Condition
COV strict rate
COV atomic mean
Model calls
Wall time (s)
TaskVM (ours)
0.583 (21/36)
0.870
1,126
19,960
Direct-CUA
0.500 (18/36)
0.761
1,361
9,010
Planner-CUA
0.472 (17/36)
0.737
1,031
6,695
Template Control †
0.083 (3/36)
0.712
144
8,527
TaskVM advantage over baselines
vs Direct-CUA
+0.083 (+16.6%)
+0.109 (+14.3%)
⋅
⋅
Table 1. Stage 1 automated bench under skills injection OFF, 36 task instances per condition. TaskVM achieves the highest Contract-Outcome Verification (COV) strict pass rate and the highest atomic predicate mean; the lower block reports its advantage over each baseline.
Table 2. COV results split by task zone (skills injection ON). Zone-G families exercise intent landing, supersession, pause boundaries, and reversible restore. Subsets and denominators differ across conditions; rates describe the tested populations and are not matched contrasts.
Paired family flip
TaskVM
Direct-CUA
Planner-CUA
fail → pass (skills helped)
4
0
0
pass → fail (skills hurt)
3
0
0
unchanged
14
18
18
missing (no OFF data)
0
3
3
Net gain
+1
0
0
CFE non-inferiority, TaskVM vs Direct-CUA (skills-ON wave)
Table 3. Effects of procedural skills injection. Upper block: paired family-level outcome flips between the skills-OFF and skills-ON waves. Lower block: contract-faithful execution (CFE) non-inferiority of TaskVM against its strongest baseline within the skills-ON wave, cluster-bootstrapped (10,000 resamples, 35 paired clusters).
Measure
TaskVM
Conv CUA
Δ
Paired counts
p
dz
Subjective measures (1–7 unless noted; higher is better except NASA-TLX)
Trust (2-item TIA composite ↑ )
5.54
4.36
+1.18
21 / 7
.0030
+0.55
Recovery control (4 study-specific items ↑ )
5.56
4.63
+0.93
22 / 13
.0050
+0.51
Stop and redirect ( rc_stop_redirect )
5.68
4.43
+1.24
23 / 7
.0035
+0.58
Correct errors ( rc_correct )
5.49
4.59
+0.89
20 / 10
.0160
+0.42
Overall control ( rc_overall )
5.35
4.35
+1.00
19 / 9
.0099
+0.44
Table 4. Stage 2 within-subject user study evaluation ( N=37 ). We report means, paired differences ( Δ ), directional participant counts (in favor of TaskVM / in favor of Conversational CUA), Wilcoxon signed-rank p -values, and standardized effect sizes ( dz ). Bold entries are statistically significant at p<.05 . Write-probe margins are a manipulation check of the replay protocol: the environment and the replayed trajectory were identical across arms, so identical margins indicate that neither arm offered systematically easier intervention opportunities, not that the interfaces would execute with equivalent competence outside the rig.
Figure 5. Expanded governance interactions in TaskVM, shown as explanatory English redraws. (E1) Planned checkpoint modes. (E2) A rollback preview distinguishing restorable effects from non-compensatable residuals. (E3) An evidence drawer presenting execution and verification records. (E4) A command palette for governance actions. These panels illustrate interface functions; they are not a quantitative summary of Stage 3 use.
Paired contrast
mean Δ
dz
lower / higher
Wilcoxon p
Stage 3 vs Stage 2 TaskVM arm
−9.70
−0.83
12 / 2
.011
Stage 3 vs Stage 2 Conversational arm
−34.40
−1.43
14 / 0
< .001
Stage 3 vs Stage 2 two-arm mean
−22.05
−1.80
14 / 0
< .001
Table 5. Stage 3 exploratory follow-up with a live model on participant-authored tasks ( N=14 returning participants). NASA-TLX total workload is paired against each participant’s own Stage 2 scores. All 14 participants scored lower in Stage 3 than in their earlier conversational-arm sessions. Comparisons are reported descriptively, since task selection, practice, study procedures, and execution mode changed together across stages.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Predicate
Meaning (what it certifies)
required_final_state
The target fields specified by the task end in the desired state.
protected_non_interference
No observable field in the protected set was modified.
required_witness_trajectory
The execution visited the witness checkpoints that prove the right path was taken.
latest_intent_landed
The most recent user intent is the one reflected in the final world.
superseded_intent_suppressed
A superseded intent did not leave residue effects behind.
pause_stop_boundary
A pause or stop was honored at its transactional boundary.
Appendix
Table 6. COV predicates and what each certifies.
System
Governing object (what a task is)
Obligation created by an edit
Evidence of completion
Recovery model
Scope
Plan- and execution-centric oversight
Plover ( Venkatesan et al., 2026 )
A structured plan over executable steps, locally revisable
Replanning of upcoming steps; executed history stays immutable
Executor step outcomes plus terminal screenshot check
Retry of pending steps within plan scope
Desktop GUI environments
Sidekick ( Chang et al., 2026 )
Concurrent agent tracks and their status
Steering of which track continues and how
Agent-reported status indicators
Resumption of suspended tracks
Multi-task workflow context
AgentLens ( Kim et al., 2026 )
An adaptive visual stream of the mobile UI
Which view renders and which touches are forwarded
Native UI layout inspection
Not part of the reported model
Mobile OS application layer
ALLOY ( Li et al., 2025 )
A reusable workflow graph
Which workflow is replayed and with what bindings
Per-step execution assertions
Workflow modification; state rollback not in the reported model
Web browser DOM tasks
Malleable and generative views
Appendix
Table 7. TaskVM in the design space of interactive agent and malleable interface systems, organized by the primary object the user governs. The remaining columns trace what follows from that object: the obligation an edit creates, the evidence that counts as completion, and the meaning of recovery. The final row states the state-contract stance examined in this paper.
Figure 6. Stage 2 baseline Conversational CUA interface: an application screencast overlaid with a natural language chat drawer. Due to language localization requirements, the interface has been semantically redrawn in the target language.
Figure 8. Per-family strict COV pass rates across conditions on the frozen suite (36 instances in 12 structural families), grouped into Zone-S (structural and execution families) and Zone-G (governance-dense families).
Social feed: like the post mentioning the “overtime mobilization meeting”.
Experimenter announces the target changed to the “core CPI surprise drop” post; a similar decoy post exists (bound, pre-first-effect).
4 min
r8-t1b
pause–retarget
Task board: mark “organize the smoke-test checklist” as done.
Target changes to “update dependency list” (same trigger).
4 min
r8-t2a
mixed rollback
Social feed: like and bookmark the overtime-meeting post.
After the first effect lands, the experimenter requests an undo; the interface must display the reversibility boundary.
5 min
r8-t2b
mixed rollback
Chat: send a contact the message “weekly meeting moved to Thursday 2 pm”.
After send confirmation, undo is requested; a sent message is irreversible and must be honestly reported as such.
5 min
r8-t3a
external drift
Social feed: like the “team-building photo collection” post.
World-side drift is injected after target binding, before the effect lands; the participant must reconcile the conflict.
4 min
r8-t3b
external drift
Calendar: change the “daily standup” RSVP from accept to tentative .
Same drift injection (bound, pre-effect).
4 min
Appendix
Table 9. Stage 2 task battery (eight tasks, four pairs). Trigger points were determined by normalized execution events, not fixed wall-clock delays. Goal texts translated from the Chinese study script.
PID
Episodes
Model calls
#Apps
#Form.
P01
4
74
2
4
P02
3
52
2
2
P05
3
9
2
2
P06
4
42
1
3
P10
3
93
3
3
P13
3
149
3
2
Appendix
Table 10. Stage 3 usage per returning participant, as retained in the anonymized archive. #Form. = distinct goal formulations issued.
Theme
Content
Participants
Representative quote [translated]
F1
Faithful execution and precise intent understanding
P21, P37, P27, P25 (4)
“The system recognized my intent and made the keyword changes, and it did not over-guess what I wanted to do.” (P27)
F2
Governance features used and valued mid-flight
P35, P30 (2)
“Mid-execution I added an instruction to ‘ask Situ Nian whether she can work overtime’; the agent finished the original task, then correctly found the contact and surfaced the prompt. I found this delightful.” (P30)
“Automatically set a checkpoint at each node, so I can roll back at any time.” (P30); “I hope the system can autonomously identify the critical points where problems may occur.” (P27)
L1
Offline simulator has finite data coverage
P35, P06, P13 (3)
“I wanted to search for Valorant, cats, and the ‘Blue-and-White Porcelain’ MV, but the world model had none of them, which was quite frustrating.” (P13)
L2
Executor (model) reliability, orthogonal to the interface
P10, P20, P13, P22 (4)
A complex task “fell into a loop of repeated comments” (P20); repeated retries of the same send action (P13).
L3
UI flow and tutorial coverage
P06, P01, P05, P20, P13, P25, P02 (7)
Progress not visible during retries (P06); features discovered too late, i.e., tutorial coverage rather than missing capability (P01, P05).
Appendix
Table 11. Stage 3 open-ended response evidence map. F = favourable theme, L = limitation theme. [translated] from Chinese originals.
Figure 9. Stage 3 cockpit screenshot voluntarily submitted by participant P30 immediately after a self-directed session. The task was an ordered four-message chain in the simulated messaging application: rescheduling a quarterly budget review to two designated contacts, followed by a mid-chain correction. After the chain, the participant appended a self-authored final instruction asking one contact whether she could work overtime that week. The left cockpit shows all four plan nodes passed and the goal status done (73 model calls, matching the session ledger); the right panel shows the delivered messages inside the simulated app, including the appended overtime question, illustrating that a mid-flight user instruction was honored only after the original ordered chain had completed. Every personal name visible in this screenshot (e.g., the contact “Situ Nian”) is a programmatically injected fictional identity of the offline simulator’s seeded corpus; it does not denote any real person, any resemblance to a real individual is purely coincidental, and its appearance does not constitute a failure of anonymization.
Figure 10. Stage 3 screenshot voluntarily submitted by participant P06 at the moment the system reported task completion. The participant authored the task “search Bilibili for an Arena of Valor video and like it”; the image captures the verification-pass state (banner “Verification passed, task completed”; all six plan nodes verified, 19 model calls, zero governance interventions used), while the right panel shows the liked video inside the simulated video application. Any personal name that may appear in this screenshot is a programmatically injected fictional identity of the offline simulator’s seeded corpus; it does not denote any real person, any resemblance to a real individual is purely coincidental, and its appearance does not constitute a failure of anonymization.
Module
Item
Text
Scale
NASA-TLX
tlx_mental
How mentally demanding was the task?
0–100
tlx_physical
How physically demanding was the task?
0–100
tlx_temporal
How hurried or rushed was the pace of the task?
0–100
tlx_performance
How successful were you in accomplishing what you were asked to do?
0–100 (Perfect → Failure)
tlx_effort
How hard did you have to work to accomplish your level of performance?
0–100
tlx_frustration
How insecure, discouraged, irritated, stressed, and annoyed were you?
0–100
Appendix
Table 12. Condition-end questionnaire: all administered items (English strings verbatim).
Condition
Model roles
Notes
Stage 1 (automated benchmark, frozen waves)
TaskVM
GPT-5.6-sol × 4 roles: state compiler, task architect, CUA executor, independent model verifier
Verifier reads fresh screenshots only; per-trial role call ledgers retained
Direct-CUA
GPT-5.6-sol, single agentic loop
Same seeds, same substrate
Planner-CUA
GPT-5.6-sol, planner then executor
Same seeds, same substrate
Template Control
none (scripted gesture driver)
Assembly sanity check; zero LLM calls
Stage 2 (user study)
Appendix
Table 13. Model and runtime configuration by stage and condition. All LLM roles used the same production model; differences are architectural, not model-level.
Dimension
Category
n
Gender (self-reported)
Female / Male / Undisclosed
15 / 21 / 1
Age band
18–24 / 25–34 / 35–44
19 / 15 / 2
Affiliation sector
Academic / Industry
32 / 4
Field of study ( n=35 )
CS/AI disciplines
20
EE / basic sciences / other STEM
9
Business / humanities / other
6
Appendix
Table 14. Aggregate demographics and background of the Stage 2 sample ( N=37 ). Gender, age, and affiliation from the compensation record ( n=36 ); field and agent-related items from the background questionnaire ( n=35 ).
Item / code
Original option wording (abridged)
Q1 ai_ml
Artificial Intelligence / Machine Learning / Deep Learning
Q1 cv
Computer Vision / Image Processing
Q1 nlp
Natural Language Processing / Large Language Models
Q1 robotics
Robotics / Embodied Intelligence / Automation / Control
Computer Systems / Operating Systems / Distributed Systems
Appendix
Table 15. Mapping from dataset option codes to the original questionnaire wording (abridged English transcription; the full five-item instrument, including its warning boxes distinguishing GUI agents from other AI tools, is in the supplementary material).
Table 16. Per-participant demographics and background roster ( N=37 ). Age = participant’s band from the exact self-reported age; Sector: Acad. = university or public research institute, Ind. = industry employer (institution names withheld against re-identification); Field = questionnaire category with the participant’s free-text specialization in parentheses [translated]; AgentBG = agent professional background level (Table 15 ); GUI-use = self-reported computer-use/GUI-agent usage frequency over the past 6 months; S3 = returned for Stage 3.
GUI memory should preserve the results of past actions and learn reusable operations. This is difficult in long-horizon workflows, where agents must use information across applications and useful operations are interleaved with failed attempts. We introduce Task Advancement Memory (TAM), which defines task advancement as verified task progress, and supports ongoing task execution and learns reusable operations. During execution, Task Advancement State represents how verified task progress changes the remaining goals, allowing the agent to build on earlier achievements across applications. After each task, advancement supervision identifies and abstracts the interactions that produced a verified local result. The learned operations preserve the relation between required inputs, expected outcomes, and verification conditions. This retains locally successful operations from both successful and failed tasks. On MobileWorld GUI-only tasks, Advancement State alone raises success from 12.0% to 28.2%. Full TAM achieves a 17.6% relative gain in task success over the strongest evaluated baseline and reduces lifecycle tokens per solved task by 38.7%. On SPA-Bench, TAM achieves state-of-the-art task success in the single-app English Level 3 setting.
Ziyang Chen, Qian Li, Chengwei Qin
The Hong Kong University of Science and Technology (Guangzhou) · Joint SDU-NTU Centre for Artificial Intelligence Research (C-FAIR), Shandong University
Computer-use agents are usually improved by strengthening perception: better models for reading a screenshot and choosing where to click. Yet a screenshot is only a lossy rendering of the underlying program state, e.g., the files, application backends, and DOM that hold the task data. Different states can produce the same pixels, while code can inspect and modify that state directly. StateAct is a code-first, multi-agent harness built around this distinction. Its main agent works directly with program state by using code, while a dedicated GUI subagent handles screenshot-and-click interaction on the few subgoals that need it, just 28 of 108 tasks and 1.1% of main-agent steps. The same direct access to program state also supports verification: an independent finish gate double-checks the saved result for structural failures, e.g., output that is missing, unsaved, or written to the wrong path. To stay on track over hundreds of steps, the main agent hands subgoals to fresh subagents, keeping its own context focused. On OSWorld 2.0, StateAct lifts Claude Opus 4.8 from 20.6% to 26.9% on binary success, and from 54.8% to 61.6% on partial success, at ~ 9x lower cost per task than the same model driven by screenshots alone; a code-only variant with no GUI subagent reaches only 45.9% partial, below that screenshot-based baseline's 54.8%. In general, grounding action, verification, and memory in state, what we call state-grounding, shifts the main bottleneck from perception toward reasoning: failures depend more on what the agent thinks than on what it sees.
AI agents are moving from advisory interfaces into systems that execute consequential operations: issuing refunds, reserving scarce inventory, provisioning cloud resources, and initiating financial transfers. These workflows require governance over effects, not only over model outputs. Existing safeguards often decide whether an action is allowed from the information available when the action is requested. For stateful policies, that request-time view may be incomplete: budgets, inventory, approval status, and risk signals can change before the effect occurs, making an earlier authorization or approval stale. This paper studies stateful governance for concurrent agentic systems. We identify stale authorization as the core failure mode and define policy-state serializability, a correctness condition requiring committed effects to be explainable as authorized against the policy state immediately before they occur. We present MasuGate, a runtime architecture that keeps policies as reviewable programs while coordinating the state and effects needed to preserve their decisions. In experiments with a PostgreSQL-backed prototype of MasuGate, the system prevents stale authorizations missed by baselines that pass policy state as ordinary request context, preserves delayed approvals while unrelated work proceeds, keeps policy evolution mostly in policy text rather than trusted provider code, and avoids policy violations in a scripted, LLM-free procurement workflow where agent-governance baselines produce stale authorizations over shared budgets and inventory. More broadly, MasuGate suggests a path for integrating stateful governance boundaries into agent frameworks and provider-backed domains where agents act on shared resources.
Yuxiang Peng, Xiaodi Wu
Purdue University · West Lafayette, Indiana, USA · University of Maryland, College Park +1