Scaling GUI reinforcement learning requires environments that preserve real-app behavior. Yet coupling behavior inference with coding can turn partial traces into runnable mocks with incorrect state transitions. We introduce Trace2World, which separates global behavior reconstruction from implementation. It reconciles traces into a shared, source-linked app model, then compiles an executable environment. Independent replay of source workflows localizes mismatches for bounded implementation repair. Validated worlds then support learner-adaptive task and environment augmentation within source-supported behavior. From 1,570 traces, Trace2World constructs 50 environments. On 200 workflows across ten apps, 91.1% are reproduced end to end, versus 59.8% for the strongest matched task-local baseline. Source-grounded validation retains 94.0% of simulated training gains on real devices, versus 54.7% under self-validation. Learner guidance raises benchmark success from 56.9% to 61.1%. Training in Trace2World further improves the model by 20.2 points on our benchmark, 5.4--16.2 points across four external GUI benchmarks, and 16.3 points on 100 real-device tasks. Reliable reconstruction separates what behavior source evidence supports from whether the implementation preserves it.
Figures & tables
Figure 2: Same goal, different first transition. The source’s first Disable quick pay tap opens a confirmation; our matched PhoneWorld-style reconstruction saves the disabled value on that tap.
Figure 3: Semantic commitment before implementation, source acceptance afterward. Partial traces are reconciled into one source-linked model of shared app behavior; unresolved behavior stays non-executable. Bounded API calls compile the model into a package; code checks and replay of the original workflows only report defects, and repair edits the package while the model stays frozen. The validated world is indexed read-only and expanded with learner-guided tasks.
Figure 4: Two traces, one Home page. Xi’an and Beijing are supported initial values of the same city state field. The task selects a reset value; subsequent behavior uses the shared page and state-dependent action rules.
Figure 5: Learner-guided tasks from a validated app index. Task generation combines supported surfaces, operations, and dependencies from the validated package’s read-only index into a food-delivery task adapted to the current learner. Learner gaps guide sampling; source, GUI, solver, and learner-signal checks screen candidates. Missing entity, state, or UI slices are instantiated in the package; the app model is unchanged. Other icons by Freepik from Flaticon.
Method
Overall
Structure
State
Data
All six
Reach
Surface
State
Effect
Terminal
Direct w/o traces
5.2
28.4
14.8
22.7
15.0
15.9
12.8
Direct + traces
16.7
53.6
35.3
41.2
34.6
33.7
31.7
CUA-Gym w/o traces
8.2
34.4
20.0
29.0
22.6
21.5
19.6
CUA-Gym + traces
19.9
50.4
37.5
38.0
35.0
35.3
35.0
PhoneWorld-style
22.4
57.0
39.3
46.8
41.5
39.8
38.5
Table 1: End-to-end reconstruction fidelity (%). Ten-app macro means over the same 200 source workflows. All six (workflow fidelity) requires every check; Reach checks only the endpoint.
Figure 6: Learner-guided task supply. (a) Learning-signal share in selected tasks (four-attempt reward SD >0.1 ); gray labels above bars show the corresponding first-valid share before bounded refinement. Only learner-guided generation refines candidates; B1/B2 use the current branch learner. (b,c) Trace2World-Bench SR/PR for iteration/rollout-matched continuations; only the two augmentation branches also share H=45 . Error bars show three-seed SD.
Model
Trace2World-Bench (SR / PR)
MG SR / PR
AW SR
MW SR@50
AD SR
Entity repl.
Held-out workflow
Cross-app
Unseen apps
Overall
Proprietary models
Gemini 3.1 Pro
75.1 / 82.2
69.0 / 76.2
58.0 / 67.0
65.7 / 71.9
68.9 / 76.1
31.3 / 44.6
–
–
–
GPT-5.5
74.5 / 82.5
68.3 / 76.8
57.3 / 66.3
66.1 / 73.8
68.4 / 76.7
–
–
62.4
–
Qwen3.7-Plus
76.3 / 83.4
70.6 / 78.4
59.0 / 67.8
68.8 / 76.4
70.6 / 78.3
32.8 / 46.0
81.0
62.3
79.8
Open-source GUI agents
Table 2: Downstream policy performance (%). Trace2World-Bench and MobileGym (MG) report SR/PR; AndroidWorld (AW), MobileWorld (MW), and AndroidDaily (AD) report SR (MW: GUI-only SR@50). Reference agents give capability context; the fixed Qwen3.5-9B block isolates training gain. Bold marks the best in each block (Appendices E.3.1 , E.3.4 ).
Matched stage-1 source
Stage 3
Metric
Self- validation
Source-trace validation
Ours
Real gain
9.7 ± 2.1
15.7 ± 1.5
16.3 ± 2.1
Sim. gain
17.7 ± 1.5
16.7 ± 2.1
18.7 ± 1.5
Real/sim gain
54.7%
94.0%
87.5%
Sim-only pass
18.3%
10.0%
10.7%
Table 3: Real-device transfer on 100 paired tasks. Gains over base (pp; three-seed mean ± SD); real/sim is gain retention, and Sim-only pass is simulation-only success.
Appendix figures & tables70 assets
Supplementary material from the paper’s appendix.
Appendix
Real / designed
Topic / task-local
Trace-grounded
Axis
AndroidWorld
MobileGym
CUA-Gym
GUI-GENESIS
PhoneWorld
Trace2World
Construction input
real Android apps and task definitions
authored simulator and tasks
topic request
one traced task
real trajectories and screenshots
all app traces
Behavior representation
real app itself
authored state and actions
topic-based mock
task-local site
screen graph, page PRDs, mutable state
shared IDs, state/data, guards and effects
Cross-trace state/effect rule
native app
authored
topic specification
task-local
no explicit source-linked rule
guarded or unresolved
Implementation path
native apps
hand-authored simulator
Plan–Dev–Web agents
code agent
PRDs → coding agent
model → API compile → agentic repair
Acceptance evidence
real-app task oracle
authored task checks
generated verifier and filter
code-native reward and golden path
smoke tests and human comparison
replay of every source workflow
Appendix
Table A1: Semantics and acceptance across GUI environments. Columns group systems by the source of app behavior; rows distinguish input, representation, cross-trace semantic rules, implementation, and native acceptance evidence. Task checks in authored or synthesized worlds do not certify fidelity to an external app. PhoneWorld uses agentic construction with representative human QA; Trace2World fixes source-linked semantics before coding and replays each recorded workflow. CUA-Gym targets desktop and web software; the other systems target mobile applications. Counts use native units; paired denotes same-task simulation/device evaluation and unpaired a separate real benchmark (Appendix E.4 ). AndroidWorld and MobileGym memory figures are reported by MobileGym ( Wu et al., 2026 ) ; Trace2World’s is marginal PSS at 16 concurrent episodes across the 50-app portfolio (Appendix E.2.5 ). These are contextual cross-host figures; dashes mean no comparable figure here.
Figure A1: Selection halves volume while keeping workflow coverage (Meituan). Light bars count admitted candidate recordings and blue overlays count the traces selected for construction; right-side counts read candidate → selected. All 16 occupied buckets retain at least one selected trace. Buckets are a post-hoc keyword grouping of the recorded instruction text, not a registered product taxonomy.
Figure A2: Disposition of 16,472 recorded frames across the 1,570-trace corpus. Executable steps (77.0%) and terminal observations (9.5%) are retained as induction evidence; launcher/app-entry, transient-gate, no-effect, and human-handover frames are removed before semantic calls (13.5% combined).
Trace selection
Visual evidence
Stratum
Apps
Candidate recordings
Selected traces
Retention
Selected screenshots
Screens / trace
Multi-service
19
2,005
950
47.4%
5,206
5.48
Single-service
31
4,536
620
13.7%
4,712
7.60
Full corpus
50
6,541
1,570
24.0%
9,918
6.32
Appendix
Table A2: Source-trace and visual-evidence selection across the full construction corpus. Candidate recordings form the admissible pool, selected traces constitute the fixed 1,570-trace construction set, and selected screenshots form the page-level visual basis. Retention is selected traces divided by candidate recordings within each collection-budget stratum.
Layer
Evidence rule
App-specification commitment
Structure
Interaction topology that remains stable across tasks and runtime variation.
Canonical pages/surfaces, routes, containment, visible hosts, overlays, and bindings.
State
Launch-, guard-, or action-conditioned mutable variation owned by a canonical surface or entity.
Identity and attributes reused across pages or tasks, including stable resource bindings.
Canonical entities, attributes, relations, and resources rather than task-local aliases.
Appendix
Table A3: Structure–State–Data factorization. Rows distinguish stable interaction topology, mutable task/action-conditioned fields, and reusable domain entities after owner reconciliation. The final column states the supported app-specification relation, preventing task-local variation from becoming duplicated pages or data.
Figure A3: Dynamic interaction authority on the generated Meituan Home. Green marks every admitted host in the visible crop; blue singles out Group Buy as the example task target, and red marks unbound regions for illustration rather than a runtime overlay. The current DOM stack admits only visible, unobscured hosts on the active layer; other coordinates return NOT_INTERACTIVE before event dispatch.
Layer
Packaged form
Owns
Boundary
Presentation
Static HTML/CSS/JavaScript; one page per canonical route
Renders Viewp(x) and maps pointer or keyboard input to registered reads and actions
Disposable UI state only; no business state or task-specific product branches
State interface
One shared JavaScript client and typed same-origin API
Stable entity and affordance identity; registered reads, queries, and guarded transactions
No direct database or browser-storage writes; a failed guard cannot mutate state
State kernel
Packaged state module or same-origin service; immutable world plus one writable overlay per episode
Canonical entities, relations, fields, atomic transitions, revisions, diffs, and receipts
The world is immutable; overlays are session-isolated
Episode control
Private lifecycle and initial-state template
Creates, initializes, seals, resets, finalizes, and deletes an episode
Unreachable from pages; reset rebuilds the overlay and clears its ledger
Evaluation
Task/evaluator manifests and a read-only state view
Computes checkpoints and reward from finalized snapshots and diffs; receipts supplement order or count
Cannot mutate state or branch product behavior on task identity
Appendix
Table B1: Runtime architecture of an exported application package. All layers are served under one origin. “Owns” denotes authoritative responsibility; “Boundary” states what that layer cannot decide.
Probe
Required verdict
False pass it exposes
Zero-action initial state
no progress, no reward
reward granted at launch; leaked prior state
Each legal partial stage
progress only
terminal credit for route-only arrival
Wrong entity or value
fail
terminal check that ignores the bound object
Premature terminal
fail
success declared before the required mutation
Repeated action
judged on final state
double credit from a historical maximum
Correct terminal
pass
missing verifier-visible consequence
Appendix
Table B2: Mandatory probe matrix. Every probe runs on the same environment digest; the required verdict is judged on canonical business state, never on a screenshot.
Component
Scale
Role
Released artifact
Source traces
1,570 / 50 apps
construction evidence; external fidelity authority
preprocessing, reconciliation, compilation, validation, and repair code
Environment portfolio
50 apps / 10 domains
resettable GUI RL substrate
standalone packages with state and evaluator manifests
Training-task supply
1,280 seed + 768 learner-guided
policy optimization
task, launch, support, and evaluator manifests
Trace2World-Bench
1,044 tasks
frozen policy evaluation
task definitions, split identities, evaluators
Fidelity study
200 workflows / 10 apps
reconstruction correctness
source/execution packets, reviewer labels, human adjudication
Appendix
Table D1: Trace2World resource contract. Scale, role, and released artifact of each component. Source traces are construction evidence and the external fidelity authority; the environment portfolio is the reusable resource; Trace2World-Bench is one evaluation view over it.
Action
Arguments
Definition
click / double_click / long_press
coordinate
tap, double-tap, or 500 ms press at the point
type
text
enter text into the focused editable control
swipe / scroll
direction; optional coordinate
finger-motion direction / content-motion direction
drag
start, end coordinates
drag from start to end
open_app
app name
open the named application (episode manifest only)
system_button
{home, back, enter}
press the system button
Appendix
Table D2: Action space. The same public interface serves training and evaluation rollouts.
Field
Episode contract
Policy observation
Instruction, action history, three most recent screenshots, and feedback for the preceding rejected action; no trace, hidden state, or evaluator access.
Action unit
One call from Table D2 per policy decision. Invalid actions, wait , open_app , answer , and terminate each use one decision; a nonterminal call is followed by a new observation.
Geometry
The model uses [0,999]2 coordinates, mapped through the current screenshot to the browser CSS viewport; each package declares its viewport and device scale.
Settling
Declared navigation targets wait up to 5 s; pointer and gesture actions add 50 ms. Before a screenshot, the host waits for fonts and an animation frame. wait is a separate, budgeted action.
Stop and score
A terminal call or the class cap ends the attempt. In both cases the task evaluator scores the final business state and required receipts; a declared success does not override its verdict.
Horizon
H=30 source-derived single-app; 45 augmented single-app; 60 two-app return composition. Single-app caps match across screening, RL, and evaluation; two-app tasks enter only the benchmark.
Appendix
Table D3: Policy-facing rollout protocol. A step is one public action call, including rejected calls and terminal submissions; it cannot batch several GUI actions. The task construction class fixes the cap before policy outcomes.
Table D4: Application roster by domain and policy split. All 50 applications provide construction traces. The 40 policy-training applications supply optimization tasks; the ten held-out evaluation applications, one per domain, supply no seed, candidate, screening, or RL-update task.
Evaluation set
Application coverage
Size / partition
Fidelity (construction-exposed)
10-app reconstruction set
200 workflows (16–26/app)
Trace2World-Bench
all 50 apps
1,044 tasks
Policy-training-app subtotal
40 policy-training apps / 10 frozen pairs
868 tasks
Entity replacement
training-side workflow skeletons
256 tasks
Held-out workflow
20% signature split (320 seed + 192 aug.)
512 tasks
Cross-app
10 policy-training-app pairs
100 tasks (10/pair)
Appendix
Table D5: Evaluation sets, partitions, and denominators. This table is the sole size owner for the Trace2World-Bench result columns in Table E10 ; SR there averages three attempts per task. Entity replacements sample training-side workflow skeletons and do not reduce the complete 20% held-out-workflow split. Fidelity and device transfer retain separate populations.
Field
Value
Note
Initial policy
Qwen3.5-9B
no SFT warm-up
Objective
DAPO
clip-higher, dynamic sampling, token-level loss
Reward
checkpoint credit q
Eq. 6 ; no format or length terms
ϵlow / ϵhigh
0.2 / 0.28
DAPO defaults
KL / entropy coefficient
0 / 0
no reference model
Tasks / RL iteration
at most 40
shuffled once per stage; no replacement
Appendix
Table D6: Policy-optimization and rollout configuration. All matched training conditions use these settings; changing any field defines a new condition. The public-action horizon follows the frozen construction class in training, screening, Trace2World-Bench, and real-device episodes.
Task class
Tasks
Certified
Median
P90
Max
H
Source-derived single-app
656
656
9
17
24
30
Augmented single-app
288
288
18
29
36
45
Two-app return composition
100
100
27
39
44
60
Appendix
Table D7: Legal-path calibration of the frozen Trace2World-Bench horizons. Path lengths count public GUI actions from reset to terminal state. Every cross-app task is completed by a human through the agent’s action interface before admission, with a 45-action human cap; the agent receives 60. The two single-app classes use frozen legal paths and GUI checks.
Condition
Input
Behavior source
Validation
Direct Agent ± traces
app request; recordings in +traces
implicit in generated code
self-execution and debugging
CUA-Gym ± traces
topic/research; recordings in +traces
Plan-agent specification
Dev–Web loop against the plan
PhoneWorld-style (reimpl.)
recorded screens and transitions
screen types and page specifications
two independent screen-QA rounds with fixes
GUI-GENESIS-style (reimpl.)
one traced task per site
task-local site and co-written reward
static checks and own golden path
Trace2World, self-validation
same traces and construction stages
shared app specification
tests generated from the same interpretation decide validation
Trace2World
same real-app traces
reconciled Structure, State, and Data
independent source-trace review and state probes
Appendix
Table E1: Environment builders and controls under the shared experimental contract. Input, behavior source, and validation are distinct. Trace-access variants and the validation-evidence control each change one factor.
Method / audit statistic
Overall
Structure
State
Data
All six
Reach
Surface
State
Effect
Terminal
Direct Agent w/o traces
5.2
28.4
14.8
22.7
15.0
15.9
12.8
Direct Agent + traces
16.7
53.6
35.3
41.2
34.6
33.7
31.7
CUA-Gym w/o traces
8.2
34.4
20.0
29.0
22.6
21.5
19.6
CUA-Gym + traces
19.9
50.4
37.5
38.0
35.0
35.3
35.0
PhoneWorld-style (reimpl.)
22.4
57.0
39.3
46.8
41.5
39.8
38.5
Appendix
Table E2: Diagnostic fidelity and reviewer agreement (%). Method rows use the primary 200-workflow, ten-app evaluation set and report workflow fidelity plus the six P/F diagnostics assigned by the frozen GPT-5.6 Sol reviewer; missing or contradicted evidence is F, and the All six column requires every diagnostic. The final row reports exact automated–adjudicated-human label agreement on the same 1,200 method–workflow packets. It validates the labels and is not a second performance condition.
Figure E1: Validation-evidence control and per-application robustness. Both panels use the primary 200-task, ten-app evaluation set. (a) Grouped bars compare recheck pass and false-pass rates for source-trace validation and the matched self-validation control; higher and lower are better, respectively. (b) Each circle is one app’s paired workflow-fidelity difference between Trace2World and the comparator; diamonds are macro-app means, positive values favor our method, and the shaded region marks reversals. No resampled interval is estimated.
Method
Automated
Human
Δ
P/P
P/F
F/P
F/F
Direct Agent + traces
16.7
19.4
2.7
26
12
7
155
CUA-Gym + traces
19.9
22.1
2.2
31
12
7
150
PhoneWorld-style (reimpl.)
22.4
23.8
1.4
37
10
7
146
GUI-GENESIS-style (reimpl.)
59.8
63.6
3.8
108
18
10
64
Trace2World
91.1
93.5
2.4
178
9
4
9
Control (not our method)
Appendix
Table E3: Adjudicated human workflow-fidelity labels on the same 1,200 packets (%). Automated and human-adjudicated workflow fidelity are ten-app macro means over the 200 workflows of each condition; Δ is human minus automated in pp. Joint counts are human/automated labels pooled over the 200 workflows. The human column qualifies the automated reviewer’s labels; it is not a second performance condition.
Figure E2: Context-bounded induction across ten applications and nested 10/30/50-trace prefixes. The horizontal axis is the trace prefix; the value in parentheses is the mean number of in-app screenshots included at that prefix, which is the unit the context and call estimates are computed from. Lines are ten-application means and shaded bands span those applications. (a) Flat single-context input (gray) exceeds the 40K budget at every 30- and 50-trace point, whereas the largest hierarchical call (blue) remains within 28.7–33.2K tokens. (b) Bounded Reconcile / Factorize producer-plus-reducer calls; in-panel values are application means. Median audited semantic coverage at 10/30/50 traces is shown in the panel.
Figure E3: Shared-semantics reconstruction under expanding trace support. Low/Mid/Full use 10/25/50 traces per multi-service app and 10/15/20 per single-service app; panels report route recovery, evidence selection, and workflow fidelity against each budget’s own induced target.
Figure E4: Top-20 pagewise screenshot compression in Meituan. Pages from the 50-trace source-trace set are ranked by candidate observations; ties use the first source-trace occurrence and then filename. Light bars count candidate observations and blue overlays count the selected visual evidence set; right-side counts use candidate → selected order. These pages retain 105 of 241 observations and account for 136 of the 166 reductions across all 159 pages (81.9%).
Population
Evidence support
Fresh-GUI validation and repair
Training use
Seed workflows 1,570 across 50 apps
1,570 pass (100% of seeds)
1,171 at wave 0 (74.6%); +261/+103/+35 at waves 1/2/3; 1,570 validated, 0 unresolved
1,280 seeds from the 40 training apps
Generated candidates 1,152 proposals across 40 apps
985 pass (85.5% of proposals)
673 at wave 0 (68.3% of supported); +113/+46/+21 at waves 1/2/3; 853 validated (86.6%), 132 not validated by K=3
768 admitted (90.0% of validated)
Appendix
Table E4: Validation and admission funnels. Read each population from left to right. Percentages name their denominator; + counts are cases newly closed at that repair wave rather than cumulative totals. Fresh-GUI validation is capped at K=3 .
Builder
Hours
Calls/app
Direct w/o traces
0.92
248
Direct + traces
1.27
319
CUA-Gym w/o traces
1.82
344
CUA-Gym + traces
2.14
392
PhoneWorld-style
5.02
1,118
GUI-GENESIS-style
5.90
2,415
Appendix
Table E5: Mean construction budget and Trace2World time/call decomposition. (a) Per-application totals on the matched ten-app roster; Trace2World reports each total on the first line and (API pipeline + agentic workflow) below. (b) Share of the 602 mean API-pipeline calls by phase; wedge area and legend percentages use the pipeline-only denominator. Generated-task construction is accounted separately.
Metric
n
Median
P95
Package on disk, raw / app
50
18.9 MiB
46.3 MiB
Package on disk, tar.gz / app
50
8.2 MiB
20.8 MiB
Writable state / episode
50
0 KiB
412 KiB
Host process to healthy API
10
3.06 s
3.42 s
First task initialization after display acquisition
50
258 ms
555 ms
Warm task reinitialization, init only
150
229 ms
512 ms
Appendix
Table E6: Runtime cost of the full 50-app portfolio. Each app contributes to package and task-timing summaries; host starts use ten trials. Units and sample counts are explicit for each row.
Review condition
Defects found (%) ↑
Reported defects correct (%) ↑
Defective execs. missed / 200 ↓
Clean execs. flagged / 200 ↓
Execution only
43.2%
73.6%
34
42
+ source trace
68.4%
88.2%
8
28
+ contrastive rubric
86.0%
93.0%
1
20
Appendix
Table E7: Construction-time review ablation. All rows use the same model, output schema, and token cap; each lower row adds the named evidence or instruction. The 200 defective executions contain 599 adjudicated defects. Defects found is defect-level coverage; reported defects correct is precision. The last two columns use separate sets of 200 defective and 200 clean executions.
(a) Candidate funnel and fixed yield
Round
Proposed
Source- supported
Validated
Selected
R1
576
499
426
384
R2
576
486
427
384
Appendix
Table E8: Feedback rounds preserve training yield while shifting learner-relative difficulty and task composition. (a) Candidate funnel for R1/R2, which follow stages 1/2. (b) Mean pass over four fresh attempts per validated task and dependency-heavy share among selected; lower learner pass means harder tasks and fixed-solver pass measures solvability. Dependency-heavy comprises cross-surface, lifecycle-revisit, and recovery tasks.
(a) Refinement accounting
Outcome
R1
R2
Refinable / validated
60 / 426 (14.1%)
62 / 427 (14.5%)
Recovered after first
29
30
Recovered after second
7
7
Recovered total
36 (60.0%)
37 (59.7%)
Held: no supported hardening
9
9
Appendix
Table E9: Bounded hardening acts on about 14% of validated lineages, recovers about 60% within two rounds, and raises validated-lineage signal by 5.4–5.6 pp. (a) Outcomes compare R1/R2; recovered-total percentages use the refinable pool, and the recovery and held rows partition it. (b) Signal is the validated-lineage share with four-attempt reward SD >0.1 ; saturation uses the 384 selected tasks. Refinable means fixed-solver success and learner mean reward above 0.8. Each round separately excludes 12 solver-unsolved and 6 deferred lineages outside this pool. Hard and solver-unsolved lineages are never simplified, and the K=3 implementation repair is separate.
Figure E5: Alternative paths and launch states remain source-supported. (a) Of 611 terminal-reaching attempt cases on paths absent from the recorded traces, 486 are semantically equivalent; within those 486 cases, 402 already execute and 84 are materialized from supported edges. A/I/U denotes ambiguous, invalid, or unresolved. Counts are attempt-associated cases, not deduplicated paths. (b) Of 768 selected generated tasks, 61% declare a non-empty launch delta; among declared launch fields, 78% use directly observed contexts and 22% are reachable by supported operations. The task and field rows use different denominators; the field count is unavailable.
Figure E6: Task augmentation raises the prevalence of capability probes. Each row compares, within 1,280 seed tasks and 768 learner-guided augmented tasks, the share carrying the named probe; endpoint labels give count (share). Rows are overlapping task-level marginals, so one task may contribute to multiple rows and the shares do not form a partition.
Model
Training apps
Unseen apps
All apps
Entity replacement
Held-out workflow
Cross-app
Overall
Proprietary Models
Qwen3.7-Plus
76.3 / 83.4
70.6 / 78.4
59.0 / 67.8
71.0 / 78.7
68.8 / 76.4
70.6 / 78.3
GPT-5.5
74.5 / 82.5
68.3 / 76.8
57.3 / 66.3
68.9 / 77.3
66.1 / 73.8
68.4 / 76.7
Gemini 3.1 Pro
75.1 / 82.2
69.0 / 76.2
58.0 / 67.0
69.5 / 76.9
65.7 / 71.9
68.9 / 76.1
Gemini 3.5 Flash
72.4 / 81.1
66.2 / 74.6
55.7 / 64.9
66.8 / 75.4
62.5 / 70.7
66.1 / 74.6
Appendix
Table E10: Complete Trace2World-Bench model results. Each model occupies one row and each result cell reports terminal SR / mean partial reward PR, both in percent. Columns cover the three task strata, their overall result under Training apps, unseen applications, and the complete benchmark across all apps. Table D5 is the sole owner of partition sizes and the three-attempt protocol; Appendix E.3.1 defines the evaluation contract.
Model
SR
PR
L1
L2
L3
L4
Atomic
Seq.
Transfer
Deep
Proprietary Models
Gemini 3.1 Pro
31.3
44.6
100.0
56.2
20.5
2.5
77.3
41.8
7.1
19.1
Doubao-Seed-2.0-Pro
34.0
47.5
100.0
60.3
24.1
3.8
81.8
45.5
8.9
20.6
Qwen3.7-Plus
32.8
46.0
100.0
57.5
22.9
3.8
77.3
43.6
8.9
20.6
Open-source Models
UI-Venus-2-27B
60.5
–
–
–
–
–
–
–
–
–
Appendix
Table E12: MobileGym-Bench model and task-slice results (%). The 256 tasks comprise L1/L2/L3/L4 counts 20/73/83/80 and atomic/sequential/transfer/deep-dive counts 22/110/56/68. SR/PR and trial aggregation follow Appendix E.3.1 .
Model
Filter
Query
Analyze
Atomic
Comp.
Cond.
Low
Mid
High
Total
Proprietary Models
Qwen3.7-Plus †
–
–
–
–
–
–
–
–
–
79.8
Open-source Models
Qwen-UI-Agent-27B †
–
–
–
–
–
–
–
–
–
97.5
UI-TARS-1.5 †
57.6
66.0
36.7
61.4
13.6
60.4
57.1
54.9
57.9
56.6
Step-GUI-8B †
52.5
63.8
33.0
59.1
14.0
42.9
54.1
44.6
61.5
52.5
Appendix
Table E13: AndroidDaily-235 end-to-end results (%). Columns follow the benchmark task-type (106/93/36 tasks), complexity (174/30/31), and ambiguity (164/47/24) axes of the released task table. † : Step-GUI report; total-only references also use the Qwen-UI-Agent report ( Zhou et al., 2026 ) . Base and final policies form the matched pair.
Model
Transp. (78)
Shop. (61)
Social (43)
Content (37)
Local (16)
Total
Base policy
48.7
46.4
46.5
47.7
43.8
47.2
Ours
58.1
57.9
58.9
59.5
56.3
58.3
Gain (pp)
+9.4
+11.5
+12.4
+11.8
+12.5
+11.1
Appendix
Table E14: AndroidDaily-235 results by scenario (%). The matched base and final policies use the same 78/61/43/37/16 transportation/shopping/social/content/local-service tasks; gains are percentage points.
Figure E7: Failure-mode composition of our final policy by horizon class. Each pie normalizes the seven mutually exclusive terminal modes over failed attempts in that class; labeled shares are rounded to whole percentages, and slices below 7.5% remain in the legend without an in-slice label. Cross-app handoff errors are orthogonal tags, not an eighth slice.
Figure E8: Paired transfer on 100 tasks across 20 applications. The panel balances 50 entity rebindings and 50 Trace2World-Bench workflow holdouts; complementary 3:2 and 2:3 app-level mixes keep five tasks per app. (a) Base and stage-3 task success in both substrates; error bars are seed SD. (b) Mean paired outcome composition. Agreement includes both-pass and both-fail.
Slice
N
Base env.
Base real
Trained env.
Trained real
Agreement
Entity rebinding
50
58.7
54.0
79.3
73.3
78.0 / 84.7
Held-out workflow
50
34.0
30.7
50.7
44.0
84.7 / 85.3
All tasks
100
46.3
42.3
65.0
58.7
81.3 / 85.0
Appendix
Table E15: Balanced task-family transfer slices. The fixed panel contains 50 unseen entity/state rebindings and 50 workflows sampled directly from the 512-task Trace2World-Bench held-out-workflow slice. Percentages; agreement is base / trained over three seeds.
Figure F1: Complete same-task trajectories from the Meituan home (eight pages). (a–b) The frozen marked Reference establishes entity resolution, coupon arithmetic, and the pre-commit terminal; top-left numbers are local step indices, red dots mark the next recorded action, and the terminal panel identifies the stop. (c–d) GUI-GENESIS completes the workflow in its task-local site. (e–f) Trace2World completes the same workflow in one shared 100-task world. (g) One rubric below the aligned terminal frames specifies the deal, coupon-adjusted amount, and pre-commit payment boundary for all three traces; the Trace2World terminal payment indicators use an isolated display-only view correction, with the raw capture retained in the artifact. The original Trace2World package evaluator scores the frozen run 1.0. (h) Direct+Reference reaches a generic Group-buy page but exposes no Beauty/Hair control, CUA-Gym+Reference returns to the unchanged home shell, and PhoneWorld opens a sauna list. Every sequence starts at home; failed traces end at their first missing required action without an inferred suffix. This single case diagnoses mechanisms rather than failure frequencies.
Figure 54Figure 55Figure 56
Figure F2: Trace-wise real-app/generated-app execution for a seed 58Tongcheng task (four pages). The upper sequence is the human physical-device execution and terminal observation; the lower sequence is a successful execution in the Trace2World environment. The numbered cards follow each trace’s own action order; the unnumbered terminal observation is separate. The final panel lists the outcome-evaluator checks and the alignment boundary.
Figure 58Figure 59Figure 60
Figure F3: Complete augmented-task generated-app execution in 58Tongcheng (four pages). Each card shows a full marked pre-action viewport and a public-schema action. The 20-action trace retains three no-new-observation attempts as context-reuse cards and includes return, recomputation, branch expansion, one persistent like, and recovery from an unintended merchant page. The terminal evaluator requires the listed checks and vetoes calling, consultation, or coupon collection; the case remains qualitative.
Figure 62
Figure F4: 58Tongcheng reconstruction gallery (five matched page states across two figure pages). The paired pages cover service entry, the rental channel, VR filtering, a property detail, and the interactive panorama.
Figure 64Figure 65Figure 66Figure 67
Figure F5: Railway 12306 → Didi → Railway 12306 return-dependency case (20 actions and a terminal observation over five pages). Numeric card labels are local step indices. The final panel lists eight ordered evaluator checks for the Railway eligibility, Didi fare gate, persistent return, order binding, and unpaid terminal, together with the cross-application bindings and terminal vetoes. The panels assemble separately recorded app-local executions; no single frozen execution record spans the complete cross-application episode.
Figure 69
Figure F6: Xiaohongshu reconstruction gallery (five matched page states across two figure pages). The paired pages cover the discover shell, verified search result, verified profile, store-product picker, and service-rating sheet. Each pair shares one task, launch, and task-conditioned visible state; differences outside that state remain visible.
Figure 71
Figure F7: Bilibili reconstruction gallery (five matched page states across two figure pages). The paired pages cover video home, food-query results, the filter sheet, three selected long-duration buckets, and the resulting tutorial list.
Figure 73
Figure F8: Railway 12306 reconstruction gallery (five matched page states across two figure pages). The paired pages cover a date-and-route query, the D2643 result, waitlist confirmation, passenger binding, and the unpaid terminal order.
Figure 75
Figure F9: Dingdong Maicai reconstruction gallery (five matched page states across two figure pages). The paired pages cover grocery home, category entry, combined promotion filters, a product-variant sheet, and the resulting cart state.
Figure F10: Generated-app home screens (1/2). Default states of formal augmented browser environments; fresh browsers, common viewport, no task-specific initialization. Device shells are hidden; app content and controls remain. The twenty apps match the human-audit roster (Appendix D.3 ); these images illustrate entry diversity.
Figure F11: Generated-app home screens (2/2). Local-service, travel, grocery, food, shopping, and automotive entry surfaces captured from formal augmented browser environments at a common viewport. Decorative phone shells are hidden and app controls retained.
GUI agents require high-quality interaction trajectories to learn how software environments respond to actions, maintain state, and support multi-step workflows. However, the diversity of available trajectories is constrained by the applications, interface states, and workflows accessible in the underlying environments. Expanding this coverage requires deploying increasingly diverse and complex software, with specialized applications imposing additional installation, configuration, and runtime costs. We introduce AutoGUIWorld, a data generation framework that combines the visual priors of image generators with the task knowledge of a planner to synthesize GUI interaction trajectories without deploying or running the corresponding software environments. AutoGUIWorld samples initial GUI scenes from structured specifications of operating-system context, visual appearance, and interface state, and generates tasks conditioned on those scenes. A planner then specifies atomic actions and their intended visual consequences, while an image generator iteratively edits the current screenshot to produce subsequent observations. Action grounding and transition-level quality filtering yield 79,266 spatially annotated step-level training samples across Ubuntu, Windows, macOS, and Chrome. Fine-tuning Qwen3.5-35B-A3B on AutoGUIWorld trajectories improves the mean task score on OSWorld from 33.0% to 40.8% and the task success rate on ScienceBoard from 14.0% to 32.2%. These results show that generated trajectories improve GUI-agent performance on real desktop and scientific tasks.
Mobile GUI agents can operate apps through pixel perception and touch actions, making them a promising interface for collecting and improving long-horizon mobile interaction policies. However, real trajectories are difficult to obtain for sensitive apps and privacy-critical operations. At the same time, existing simulated environments are costly to scale up, and GUI world models still suffer from unstable generation, limited modality coverage, and inconsistent action-transition logic. To address these limitations, we propose AppDeltaWorld, a transition-grounded delta code world model that predicts the next GUI as a reachable code update rather than as an unconstrained image or text description. AppDeltaWorld retrieves app-specific Level-1 HTML references under an action-transition constraint, generates Level-2 executable HTML conditioned on the current screen, action, predicted next-screen text, and retrieved structure, and inserts generated visual assets into image slots before browser rendering. As a world model, AppDeltaWorld achieves the highest fidelity on CMGUIBench-500 under Code2World evaluation, with clear gains in structural layout and UI element reconstruction over image-only and code-only baselines. As a training environment, AppDeltaWorld supports filtered closed-loop SFT data construction that, when combined with public supervision, enables AppDeltaAgent to achieve state-of-the-art performance on AndroidLens and consistent gains on MobileGym and MobileWorld. Moreover, world-model-based test-time reinforcement learning enables policy adaptation and shows further improvements without additional interaction with real apps.
Weikai Xu, Yunren Feng, Haoxiang Lei +6
1Nanyang Technological University, Singapore · University of Electronic Science and Technology of China · 3Independent Researchers +2
GUI agents trained with reinforcement learning (RL) have showcased strong environment learning capabilities on mobile platforms. However, RL typically demands extensive real-environment interactions, leading to high resource costs and instability, especially in GUI scenarios. To address these, we propose WM-R1, the first reinforcement learning framework that trains mobile GUI agents with world models instead of real environments. Specifically, world models serve as the source of state transitions during all rollouts, replacing the real Android environment within the training loop. WM-R1 also embeds world models directly into the thinking process, enabling agents to reason about the consequences of candidate actions before committing to the final action. Crucially, WM-R1 eliminates the need for real-environment interaction, supports massively parallelized and step-level granularized trajectory generation grounded in world models, and introduces a multi-dimensional rule-based reward that jointly optimizes task success, trajectory efficiency, and world model utilization. For efficient training, we curate a high-quality dataset of 2000 challenging tasks. Experiments on Android mobile benchmarks demonstrate that WM-R1-trained agents significantly outperform GRPO-only baselines and inference-time simulation methods. Code is available at https://github.com/genalyu/WM-R1 .
Yu Han, Tianwen Qian
School of Computer Science and Technology, East China Normal University