cs.LGOct 22, 2026
SaveTrace2World: Reconstructing App Worlds from Real Interaction Traces for Scalable GUI Agent Training
Organizations: Nanjing University · Alibaba Token Hub, Alibaba Group · Beijing Institute of Technology
Abstract
Scaling GUI reinforcement learning requires environments that preserve real-app behavior. Yet coupling behavior inference with coding can turn partial traces into runnable mocks with incorrect state transitions. We introduce Trace2World, which separates global behavior reconstruction from implementation. It reconciles traces into a shared, source-linked app model, then compiles an executable environment. Independent replay of source workflows localizes mismatches for bounded implementation repair. Validated worlds then support learner-adaptive task and environment augmentation within source-supported behavior. From 1,570 traces, Trace2World constructs 50 environments. On 200 workflows across ten apps, 91.1% are reproduced end to end, versus 59.8% for the strongest matched task-local baseline. Source-grounded validation retains 94.0% of simulated training gains on real devices, versus 54.7% under self-validation. Learner guidance raises benchmark success from 56.9% to 61.1%. Training in Trace2World further improves the model by 20.2 points on our benchmark, 5.4--16.2 points across four external GUI benchmarks, and 16.3 points on 100 real-device tasks. Reliable reconstruction separates what behavior source evidence supports from whether the implementation preserves it.
Figures & tables
Figure 2: Same goal, different first transition. The source’s first Disable quick pay tap opens a confirmation; our matched PhoneWorld-style reconstruction saves the disabled value on that tap.
Figure 3: Semantic commitment before implementation, source acceptance afterward. Partial traces are reconciled into one source-linked model of shared app behavior; unresolved behavior stays non-executable. Bounded API calls compile the model into a package; code checks and replay of the original workflows only report defects, and repair edits the package while the model stays frozen. The validated world is indexed read-only and expanded with learner-guided tasks.
Figure 4: Two traces, one Home page. Xi’an and Beijing are supported initial values of the same city state field. The task selects a reset value; subsequent behavior uses the shared page and state-dependent action rules.
Figure 5: Learner-guided tasks from a validated app index. Task generation combines supported surfaces, operations, and dependencies from the validated package’s read-only index into a food-delivery task adapted to the current learner. Learner gaps guide sampling; source, GUI, solver, and learner-signal checks screen candidates. Missing entity, state, or UI slices are instantiated in the package; the app model is unchanged. Other icons by Freepik from Flaticon.
| Method | Overall | Structure | State | Data | |||
| All six | Reach | Surface | State | Effect | Terminal | ||
| Direct w/o traces | 5.2 | 28.4 | 14.8 | 22.7 | 15.0 | 15.9 | 12.8 |
| Direct + traces | 16.7 | 53.6 | 35.3 | 41.2 | 34.6 | 33.7 | 31.7 |
| CUA-Gym w/o traces | 8.2 | 34.4 | 20.0 | 29.0 | 22.6 | 21.5 | 19.6 |
| CUA-Gym + traces | 19.9 | 50.4 | 37.5 | 38.0 | 35.0 | 35.3 | 35.0 |
| PhoneWorld-style | 22.4 | 57.0 | 39.3 | 46.8 | 41.5 | 39.8 | 38.5 |
Table 1: End-to-end reconstruction fidelity (%). Ten-app macro means over the same 200 source workflows. All six (workflow fidelity) requires every check; Reach checks only the endpoint.
Figure 6: Learner-guided task supply. (a) Learning-signal share in selected tasks (four-attempt reward SD ); gray labels above bars show the corresponding first-valid share before bounded refinement. Only learner-guided generation refines candidates; B1/B2 use the current branch learner. (b,c) Trace2World-Bench SR/PR for iteration/rollout-matched continuations; only the two augmentation branches also share . Error bars show three-seed SD.
| Model | Trace2World-Bench (SR / PR) | MG SR / PR | AW SR | MW SR@50 | AD SR | ||||
| Entity repl. | Held-out workflow | Cross-app | Unseen apps | Overall | |||||
| Proprietary models | |||||||||
| Gemini 3.1 Pro | 75.1 / 82.2 | 69.0 / 76.2 | 58.0 / 67.0 | 65.7 / 71.9 | 68.9 / 76.1 | 31.3 / 44.6 | – | – | – |
| GPT-5.5 | 74.5 / 82.5 | 68.3 / 76.8 | 57.3 / 66.3 | 66.1 / 73.8 | 68.4 / 76.7 | – | – | 62.4 | – |
| Qwen3.7-Plus | 76.3 / 83.4 | 70.6 / 78.4 | 59.0 / 67.8 | 68.8 / 76.4 | 70.6 / 78.3 | 32.8 / 46.0 | 81.0 | 62.3 | 79.8 |
| Open-source GUI agents | |||||||||
Table 2: Downstream policy performance (%). Trace2World-Bench and MobileGym (MG) report SR/PR; AndroidWorld (AW), MobileWorld (MW), and AndroidDaily (AD) report SR (MW: GUI-only SR@50). Reference agents give capability context; the fixed Qwen3.5-9B block isolates training gain. Bold marks the best in each block (Appendices E.3.1 , E.3.4 ).
| Matched stage-1 source | Stage 3 | ||
| Metric | Self- validation | Source-trace validation | Ours |
| Real gain | 9.7 2.1 | 15.7 1.5 | 16.3 2.1 |
| Sim. gain | 17.7 1.5 | 16.7 2.1 | 18.7 1.5 |
| Real/sim gain | 54.7% | 94.0% | 87.5% |
| Sim-only pass | 18.3% | 10.0% | 10.7% |
Table 3: Real-device transfer on 100 paired tasks. Gains over base (pp; three-seed mean SD); real/sim is gain retention, and Sim-only pass is simulation-only success.
Appendix figures & tables70 assets
Supplementary material from the paper’s appendix.
Appendix
| Real / designed | Topic / task-local | Trace-grounded | ||||
| Axis | AndroidWorld | MobileGym | CUA-Gym | GUI-GENESIS | PhoneWorld | Trace2World |
| Construction input | real Android apps and task definitions | authored simulator and tasks | topic request | one traced task | real trajectories and screenshots | all app traces |
| Behavior representation | real app itself | authored state and actions | topic-based mock | task-local site | screen graph, page PRDs, mutable state | shared IDs, state/data, guards and effects |
| Cross-trace state/effect rule | native app | authored | topic specification | task-local | no explicit source-linked rule | guarded or unresolved |
| Implementation path | native apps | hand-authored simulator | Plan–Dev–Web agents | code agent | PRDs coding agent | model API compile agentic repair |
| Acceptance evidence | real-app task oracle | authored task checks | generated verifier and filter | code-native reward and golden path | smoke tests and human comparison | replay of every source workflow |
Table A1: Semantics and acceptance across GUI environments. Columns group systems by the source of app behavior; rows distinguish input, representation, cross-trace semantic rules, implementation, and native acceptance evidence. Task checks in authored or synthesized worlds do not certify fidelity to an external app. PhoneWorld uses agentic construction with representative human QA; Trace2World fixes source-linked semantics before coding and replays each recorded workflow. CUA-Gym targets desktop and web software; the other systems target mobile applications. Counts use native units; paired denotes same-task simulation/device evaluation and unpaired a separate real benchmark (Appendix E.4 ). AndroidWorld and MobileGym memory figures are reported by MobileGym ( Wu et al., 2026 ) ; Trace2World’s is marginal PSS at 16 concurrent episodes across the 50-app portfolio (Appendix E.2.5 ). These are contextual cross-host figures; dashes mean no comparable figure here.
Figure A1: Selection halves volume while keeping workflow coverage (Meituan). Light bars count admitted candidate recordings and blue overlays count the traces selected for construction; right-side counts read candidate selected. All 16 occupied buckets retain at least one selected trace. Buckets are a post-hoc keyword grouping of the recorded instruction text, not a registered product taxonomy.
Figure A2: Disposition of 16,472 recorded frames across the 1,570-trace corpus. Executable steps (77.0%) and terminal observations (9.5%) are retained as induction evidence; launcher/app-entry, transient-gate, no-effect, and human-handover frames are removed before semantic calls (13.5% combined).
| Trace selection | Visual evidence | |||||
| Stratum | Apps | Candidate recordings | Selected traces | Retention | Selected screenshots | Screens / trace |
| Multi-service | 19 | 2,005 | 950 | 47.4% | 5,206 | 5.48 |
| Single-service | 31 | 4,536 | 620 | 13.7% | 4,712 | 7.60 |
| Full corpus | 50 | 6,541 | 1,570 | 24.0% | 9,918 | 6.32 |
Table A2: Source-trace and visual-evidence selection across the full construction corpus. Candidate recordings form the admissible pool, selected traces constitute the fixed 1,570-trace construction set, and selected screenshots form the page-level visual basis. Retention is selected traces divided by candidate recordings within each collection-budget stratum.
| Layer | Evidence rule | App-specification commitment |
| Structure | Interaction topology that remains stable across tasks and runtime variation. | Canonical pages/surfaces, routes, containment, visible hosts, overlays, and bindings. |
| State | Launch-, guard-, or action-conditioned mutable variation owned by a canonical surface or entity. | Initialization deltas, fields, guards, reads/writes, reducers, projections, terminal conditions, and reset. |
| Data | Identity and attributes reused across pages or tasks, including stable resource bindings. | Canonical entities, attributes, relations, and resources rather than task-local aliases. |
Table A3: Structure–State–Data factorization. Rows distinguish stable interaction topology, mutable task/action-conditioned fields, and reusable domain entities after owner reconciliation. The final column states the supported app-specification relation, preventing task-local variation from becoming duplicated pages or data.
Figure A3: Dynamic interaction authority on the generated Meituan Home. Green marks every admitted host in the visible crop; blue singles out Group Buy as the example task target, and red marks unbound regions for illustration rather than a runtime overlay. The current DOM stack admits only visible, unobscured hosts on the active layer; other coordinates return NOT_INTERACTIVE before event dispatch.
| Layer | Packaged form | Owns | Boundary |
| Presentation | Static HTML/CSS/JavaScript; one page per canonical route | Renders and maps pointer or keyboard input to registered reads and actions | Disposable UI state only; no business state or task-specific product branches |
| State interface | One shared JavaScript client and typed same-origin API | Stable entity and affordance identity; registered reads, queries, and guarded transactions | No direct database or browser-storage writes; a failed guard cannot mutate state |
| State kernel | Packaged state module or same-origin service; immutable world plus one writable overlay per episode | Canonical entities, relations, fields, atomic transitions, revisions, diffs, and receipts | The world is immutable; overlays are session-isolated |
| Episode control | Private lifecycle and initial-state template | Creates, initializes, seals, resets, finalizes, and deletes an episode | Unreachable from pages; reset rebuilds the overlay and clears its ledger |
| Evaluation | Task/evaluator manifests and a read-only state view | Computes checkpoints and reward from finalized snapshots and diffs; receipts supplement order or count | Cannot mutate state or branch product behavior on task identity |
Table B1: Runtime architecture of an exported application package. All layers are served under one origin. “Owns” denotes authoritative responsibility; “Boundary” states what that layer cannot decide.
| Probe | Required verdict | False pass it exposes |
| Zero-action initial state | no progress, no reward | reward granted at launch; leaked prior state |
| Each legal partial stage | progress only | terminal credit for route-only arrival |
| Wrong entity or value | fail | terminal check that ignores the bound object |
| Premature terminal | fail | success declared before the required mutation |
| Repeated action | judged on final state | double credit from a historical maximum |
| Correct terminal | pass | missing verifier-visible consequence |
Table B2: Mandatory probe matrix. Every probe runs on the same environment digest; the required verdict is judged on canonical business state, never on a screenshot.
| Component | Scale | Role | Released artifact |
| Source traces | 1,570 / 50 apps | construction evidence; external fidelity authority | source recordings, action records, instructions, terminal contracts, collection metadata |
| Construction pipeline | one shared pipeline | trace reconciled specification environment | preprocessing, reconciliation, compilation, validation, and repair code |
| Environment portfolio | 50 apps / 10 domains | resettable GUI RL substrate | standalone packages with state and evaluator manifests |
| Training-task supply | 1,280 seed + 768 learner-guided | policy optimization | task, launch, support, and evaluator manifests |
| Trace2World-Bench | 1,044 tasks | frozen policy evaluation | task definitions, split identities, evaluators |
| Fidelity study | 200 workflows / 10 apps | reconstruction correctness | source/execution packets, reviewer labels, human adjudication |
Table D1: Trace2World resource contract. Scale, role, and released artifact of each component. Source traces are construction evidence and the external fidelity authority; the environment portfolio is the reusable resource; Trace2World-Bench is one evaluation view over it.
| Action | Arguments | Definition |
| click / double_click / long_press | coordinate | tap, double-tap, or 500 ms press at the point |
| type | text | enter text into the focused editable control |
| swipe / scroll | direction; optional coordinate | finger-motion direction / content-motion direction |
| drag | start, end coordinates | drag from start to end |
| open_app | app name | open the named application (episode manifest only) |
| system_button | {home, back, enter} | press the system button |
Table D2: Action space. The same public interface serves training and evaluation rollouts.
| Field | Episode contract |
| Policy observation | Instruction, action history, three most recent screenshots, and feedback for the preceding rejected action; no trace, hidden state, or evaluator access. |
| Action unit | One call from Table D2 per policy decision. Invalid actions, wait , open_app , answer , and terminate each use one decision; a nonterminal call is followed by a new observation. |
| Geometry | The model uses coordinates, mapped through the current screenshot to the browser CSS viewport; each package declares its viewport and device scale. |
| Settling | Declared navigation targets wait up to 5 s; pointer and gesture actions add 50 ms. Before a screenshot, the host waits for fonts and an animation frame. wait is a separate, budgeted action. |
| Stop and score | A terminal call or the class cap ends the attempt. In both cases the task evaluator scores the final business state and required receipts; a declared success does not override its verdict. |
| Horizon | source-derived single-app; augmented single-app; two-app return composition. Single-app caps match across screening, RL, and evaluation; two-app tasks enter only the benchmark. |
Table D3: Policy-facing rollout protocol. A step is one public action call, including rejected calls and terminal submissions; it cannot batch several GUI actions. The task construction class fixes the cap before policy outcomes.
| Domain | Policy-training apps (40) | Held-out eval app (10) |
| Social & community | WeChat; QQ; Weibo; Xiaohongshu | Zhihu |
| News & short video | Kuaishou; Bilibili; Toutiao; Tencent News | Douyin |
| Long video & audio | iQIYI; Youku; NetEase Cloud Music; Kugou Music; Ximalaya | QQ Music |
| Commerce & resale | Taobao; JD; Xianyu; Dewu | Pinduoduo |
| Local services & food | Meituan; Dianping; Dingdong Maicai; KFC; Hema | Ele.me |
| Mobility, travel & tickets | Amap; Ctrip; Didi; Railway 12306; Damai | Taopiaopiao |
Table D4: Application roster by domain and policy split. All 50 applications provide construction traces. The 40 policy-training applications supply optimization tasks; the ten held-out evaluation applications, one per domain, supply no seed, candidate, screening, or RL-update task.
| Evaluation set | Application coverage | Size / partition |
| Fidelity (construction-exposed) | 10-app reconstruction set | 200 workflows (16–26/app) |
| Trace2World-Bench | all 50 apps | 1,044 tasks |
| Policy-training-app subtotal | 40 policy-training apps / 10 frozen pairs | 868 tasks |
| Entity replacement | training-side workflow skeletons | 256 tasks |
| Held-out workflow | 20% signature split (320 seed + 192 aug.) | 512 tasks |
| Cross-app | 10 policy-training-app pairs | 100 tasks (10/pair) |
Table D5: Evaluation sets, partitions, and denominators. This table is the sole size owner for the Trace2World-Bench result columns in Table E10 ; SR there averages three attempts per task. Entity replacements sample training-side workflow skeletons and do not reduce the complete 20% held-out-workflow split. Fidelity and device transfer retain separate populations.
| Field | Value | Note |
| Initial policy | Qwen3.5-9B | no SFT warm-up |
| Objective | DAPO | clip-higher, dynamic sampling, token-level loss |
| Reward | checkpoint credit | Eq. 6 ; no format or length terms |
| / | 0.2 / 0.28 | DAPO defaults |
| KL / entropy coefficient | 0 / 0 | no reference model |
| Tasks / RL iteration | at most 40 | shuffled once per stage; no replacement |
Table D6: Policy-optimization and rollout configuration. All matched training conditions use these settings; changing any field defines a new condition. The public-action horizon follows the frozen construction class in training, screening, Trace2World-Bench, and real-device episodes.
| Task class | Tasks | Certified | Median | P90 | Max | |
| Source-derived single-app | 656 | 656 | 9 | 17 | 24 | 30 |
| Augmented single-app | 288 | 288 | 18 | 29 | 36 | 45 |
| Two-app return composition | 100 | 100 | 27 | 39 | 44 | 60 |
Table D7: Legal-path calibration of the frozen Trace2World-Bench horizons. Path lengths count public GUI actions from reset to terminal state. Every cross-app task is completed by a human through the agent’s action interface before admission, with a 45-action human cap; the agent receives 60. The two single-app classes use frozen legal paths and GUI checks.
| Condition | Input | Behavior source | Validation |
| Direct Agent traces | app request; recordings in +traces | implicit in generated code | self-execution and debugging |
| CUA-Gym traces | topic/research; recordings in +traces | Plan-agent specification | Dev–Web loop against the plan |
| PhoneWorld-style (reimpl.) | recorded screens and transitions | screen types and page specifications | two independent screen-QA rounds with fixes |
| GUI-GENESIS-style (reimpl.) | one traced task per site | task-local site and co-written reward | static checks and own golden path |
| Trace2World, self-validation | same traces and construction stages | shared app specification | tests generated from the same interpretation decide validation |
| Trace2World | same real-app traces | reconciled Structure, State, and Data | independent source-trace review and state probes |
Table E1: Environment builders and controls under the shared experimental contract. Input, behavior source, and validation are distinct. Trace-access variants and the validation-evidence control each change one factor.
| Method / audit statistic | Overall | Structure | State | Data | |||
| All six | Reach | Surface | State | Effect | Terminal | ||
| Direct Agent w/o traces | 5.2 | 28.4 | 14.8 | 22.7 | 15.0 | 15.9 | 12.8 |
| Direct Agent + traces | 16.7 | 53.6 | 35.3 | 41.2 | 34.6 | 33.7 | 31.7 |
| CUA-Gym w/o traces | 8.2 | 34.4 | 20.0 | 29.0 | 22.6 | 21.5 | 19.6 |
| CUA-Gym + traces | 19.9 | 50.4 | 37.5 | 38.0 | 35.0 | 35.3 | 35.0 |
| PhoneWorld-style (reimpl.) | 22.4 | 57.0 | 39.3 | 46.8 | 41.5 | 39.8 | 38.5 |
Table E2: Diagnostic fidelity and reviewer agreement (%). Method rows use the primary 200-workflow, ten-app evaluation set and report workflow fidelity plus the six P/F diagnostics assigned by the frozen GPT-5.6 Sol reviewer; missing or contradicted evidence is F, and the All six column requires every diagnostic. The final row reports exact automated–adjudicated-human label agreement on the same 1,200 method–workflow packets. It validates the labels and is not a second performance condition.
Figure E1: Validation-evidence control and per-application robustness. Both panels use the primary 200-task, ten-app evaluation set. (a) Grouped bars compare recheck pass and false-pass rates for source-trace validation and the matched self-validation control; higher and lower are better, respectively. (b) Each circle is one app’s paired workflow-fidelity difference between Trace2World and the comparator; diamonds are macro-app means, positive values favor our method, and the shaded region marks reversals. No resampled interval is estimated.
| Method | Automated | Human | P/P | P/F | F/P | F/F | |
| Direct Agent + traces | 16.7 | 19.4 | 2.7 | 26 | 12 | 7 | 155 |
| CUA-Gym + traces | 19.9 | 22.1 | 2.2 | 31 | 12 | 7 | 150 |
| PhoneWorld-style (reimpl.) | 22.4 | 23.8 | 1.4 | 37 | 10 | 7 | 146 |
| GUI-GENESIS-style (reimpl.) | 59.8 | 63.6 | 3.8 | 108 | 18 | 10 | 64 |
| Trace2World | 91.1 | 93.5 | 2.4 | 178 | 9 | 4 | 9 |
| Control (not our method) | |||||||
Table E3: Adjudicated human workflow-fidelity labels on the same 1,200 packets (%). Automated and human-adjudicated workflow fidelity are ten-app macro means over the 200 workflows of each condition; is human minus automated in pp. Joint counts are human/automated labels pooled over the 200 workflows. The human column qualifies the automated reviewer’s labels; it is not a second performance condition.
Figure E2: Context-bounded induction across ten applications and nested 10/30/50-trace prefixes. The horizontal axis is the trace prefix; the value in parentheses is the mean number of in-app screenshots included at that prefix, which is the unit the context and call estimates are computed from. Lines are ten-application means and shaded bands span those applications. (a) Flat single-context input (gray) exceeds the 40K budget at every 30- and 50-trace point, whereas the largest hierarchical call (blue) remains within 28.7–33.2K tokens. (b) Bounded Reconcile / Factorize producer-plus-reducer calls; in-panel values are application means. Median audited semantic coverage at 10/30/50 traces is shown in the panel.
Figure E3: Shared-semantics reconstruction under expanding trace support. Low/Mid/Full use 10/25/50 traces per multi-service app and 10/15/20 per single-service app; panels report route recovery, evidence selection, and workflow fidelity against each budget’s own induced target.
Figure E4: Top-20 pagewise screenshot compression in Meituan. Pages from the 50-trace source-trace set are ranked by candidate observations; ties use the first source-trace occurrence and then filename. Light bars count candidate observations and blue overlays count the selected visual evidence set; right-side counts use candidate selected order. These pages retain 105 of 241 observations and account for 136 of the 166 reductions across all 159 pages (81.9%).
| Population | Evidence support | Fresh-GUI validation and repair | Training use |
| Seed workflows 1,570 across 50 apps | 1,570 pass (100% of seeds) | 1,171 at wave 0 (74.6%); at waves 1/2/3; 1,570 validated, 0 unresolved | 1,280 seeds from the 40 training apps |
| Generated candidates 1,152 proposals across 40 apps | 985 pass (85.5% of proposals) | 673 at wave 0 (68.3% of supported); at waves 1/2/3; 853 validated (86.6%), 132 not validated by | 768 admitted (90.0% of validated) |
Table E4: Validation and admission funnels. Read each population from left to right. Percentages name their denominator; counts are cases newly closed at that repair wave rather than cumulative totals. Fresh-GUI validation is capped at .
| Builder | Hours | Calls/app |
| Direct w/o traces | 0.92 | 248 |
| Direct + traces | 1.27 | 319 |
| CUA-Gym w/o traces | 1.82 | 344 |
| CUA-Gym + traces | 2.14 | 392 |
| PhoneWorld-style | 5.02 | 1,118 |
| GUI-GENESIS-style | 5.90 | 2,415 |
Table E5: Mean construction budget and Trace2World time/call decomposition. (a) Per-application totals on the matched ten-app roster; Trace2World reports each total on the first line and (API pipeline + agentic workflow) below. (b) Share of the 602 mean API-pipeline calls by phase; wedge area and legend percentages use the pipeline-only denominator. Generated-task construction is accounted separately.
| Metric | Median | P95 | |
| Package on disk, raw / app | 50 | 18.9 MiB | 46.3 MiB |
| Package on disk, tar.gz / app | 50 | 8.2 MiB | 20.8 MiB |
| Writable state / episode | 50 | 0 KiB | 412 KiB |
| Host process to healthy API | 10 | 3.06 s | 3.42 s |
| First task initialization after display acquisition | 50 | 258 ms | 555 ms |
| Warm task reinitialization, init only | 150 | 229 ms | 512 ms |
Table E6: Runtime cost of the full 50-app portfolio. Each app contributes to package and task-timing summaries; host starts use ten trials. Units and sample counts are explicit for each row.
| Review condition | Defects found (%) | Reported defects correct (%) | Defective execs. missed / 200 | Clean execs. flagged / 200 |
| Execution only | 43.2% | 73.6% | 34 | 42 |
| source trace | 68.4% | 88.2% | 8 | 28 |
| contrastive rubric | 86.0% | 93.0% | 1 | 20 |
Table E7: Construction-time review ablation. All rows use the same model, output schema, and token cap; each lower row adds the named evidence or instruction. The 200 defective executions contain 599 adjudicated defects. Defects found is defect-level coverage; reported defects correct is precision. The last two columns use separate sets of 200 defective and 200 clean executions.
| (a) Candidate funnel and fixed yield | ||||
| Round | Proposed | Source- supported | Validated | Selected |
| R1 | 576 | 499 | 426 | 384 |
| R2 | 576 | 486 | 427 | 384 |
Table E8: Feedback rounds preserve training yield while shifting learner-relative difficulty and task composition. (a) Candidate funnel for R1/R2, which follow stages 1/2. (b) Mean pass over four fresh attempts per validated task and dependency-heavy share among selected; lower learner pass means harder tasks and fixed-solver pass measures solvability. Dependency-heavy comprises cross-surface, lifecycle-revisit, and recovery tasks.
| (a) Refinement accounting | ||
| Outcome | R1 | R2 |
| Refinable / validated | 60 / 426 (14.1%) | 62 / 427 (14.5%) |
| Recovered after first | 29 | 30 |
| Recovered after second | 7 | 7 |
| Recovered total | 36 (60.0%) | 37 (59.7%) |
| Held: no supported hardening | 9 | 9 |
Table E9: Bounded hardening acts on about 14% of validated lineages, recovers about 60% within two rounds, and raises validated-lineage signal by 5.4–5.6 pp. (a) Outcomes compare R1/R2; recovered-total percentages use the refinable pool, and the recovery and held rows partition it. (b) Signal is the validated-lineage share with four-attempt reward SD ; saturation uses the 384 selected tasks. Refinable means fixed-solver success and learner mean reward above 0.8. Each round separately excludes 12 solver-unsolved and 6 deferred lineages outside this pool. Hard and solver-unsolved lineages are never simplified, and the implementation repair is separate.
Figure E5: Alternative paths and launch states remain source-supported. (a) Of 611 terminal-reaching attempt cases on paths absent from the recorded traces, 486 are semantically equivalent; within those 486 cases, 402 already execute and 84 are materialized from supported edges. A/I/U denotes ambiguous, invalid, or unresolved. Counts are attempt-associated cases, not deduplicated paths. (b) Of 768 selected generated tasks, 61% declare a non-empty launch delta; among declared launch fields, 78% use directly observed contexts and 22% are reachable by supported operations. The task and field rows use different denominators; the field count is unavailable.
Figure E6: Task augmentation raises the prevalence of capability probes. Each row compares, within 1,280 seed tasks and 768 learner-guided augmented tasks, the share carrying the named probe; endpoint labels give count (share). Rows are overlapping task-level marginals, so one task may contribute to multiple rows and the shares do not form a partition.
| Model | Training apps | Unseen apps | All apps | |||
| Entity replacement | Held-out workflow | Cross-app | Overall | |||
| Proprietary Models | ||||||
| Qwen3.7-Plus | 76.3 / 83.4 | 70.6 / 78.4 | 59.0 / 67.8 | 71.0 / 78.7 | 68.8 / 76.4 | 70.6 / 78.3 |
| GPT-5.5 | 74.5 / 82.5 | 68.3 / 76.8 | 57.3 / 66.3 | 68.9 / 77.3 | 66.1 / 73.8 | 68.4 / 76.7 |
| Gemini 3.1 Pro | 75.1 / 82.2 | 69.0 / 76.2 | 58.0 / 67.0 | 69.5 / 76.9 | 65.7 / 71.9 | 68.9 / 76.1 |
| Gemini 3.5 Flash | 72.4 / 81.1 | 66.2 / 74.6 | 55.7 / 64.9 | 66.8 / 75.4 | 62.5 / 70.7 | 66.1 / 74.6 |
Table E10: Complete Trace2World-Bench model results. Each model occupies one row and each result cell reports terminal SR / mean partial reward PR, both in percent. Columns cover the three task strata, their overall result under Training apps, unseen applications, and the complete benchmark across all apps. Table D5 is the sole owner of partition sizes and the three-attempt protocol; Appendix E.3.1 defines the evaluation contract.
| Model | SR | PR | L1 | L2 | L3 | L4 | Atomic | Seq. | Transfer | Deep |
| Proprietary Models | ||||||||||
| Gemini 3.1 Pro | 31.3 | 44.6 | 100.0 | 56.2 | 20.5 | 2.5 | 77.3 | 41.8 | 7.1 | 19.1 |
| Doubao-Seed-2.0-Pro | 34.0 | 47.5 | 100.0 | 60.3 | 24.1 | 3.8 | 81.8 | 45.5 | 8.9 | 20.6 |
| Qwen3.7-Plus | 32.8 | 46.0 | 100.0 | 57.5 | 22.9 | 3.8 | 77.3 | 43.6 | 8.9 | 20.6 |
| Open-source Models | ||||||||||
| UI-Venus-2-27B | 60.5 | – | – | – | – | – | – | – | – | – |
Table E12: MobileGym-Bench model and task-slice results (%). The 256 tasks comprise L1/L2/L3/L4 counts 20/73/83/80 and atomic/sequential/transfer/deep-dive counts 22/110/56/68. SR/PR and trial aggregation follow Appendix E.3.1 .
| Model | Filter | Query | Analyze | Atomic | Comp. | Cond. | Low | Mid | High | Total |
| Proprietary Models | ||||||||||
| Qwen3.7-Plus † | – | – | – | – | – | – | – | – | – | 79.8 |
| Open-source Models | ||||||||||
| Qwen-UI-Agent-27B † | – | – | – | – | – | – | – | – | – | 97.5 |
| UI-TARS-1.5 † | 57.6 | 66.0 | 36.7 | 61.4 | 13.6 | 60.4 | 57.1 | 54.9 | 57.9 | 56.6 |
| Step-GUI-8B † | 52.5 | 63.8 | 33.0 | 59.1 | 14.0 | 42.9 | 54.1 | 44.6 | 61.5 | 52.5 |
Table E13: AndroidDaily-235 end-to-end results (%). Columns follow the benchmark task-type (106/93/36 tasks), complexity (174/30/31), and ambiguity (164/47/24) axes of the released task table. : Step-GUI report; total-only references also use the Qwen-UI-Agent report ( Zhou et al., 2026 ) . Base and final policies form the matched pair.
| Model | Transp. (78) | Shop. (61) | Social (43) | Content (37) | Local (16) | Total |
| Base policy | 48.7 | 46.4 | 46.5 | 47.7 | 43.8 | 47.2 |
| Ours | 58.1 | 57.9 | 58.9 | 59.5 | 56.3 | 58.3 |
| Gain (pp) | +9.4 | +11.5 | +12.4 | +11.8 | +12.5 | +11.1 |
Table E14: AndroidDaily-235 results by scenario (%). The matched base and final policies use the same 78/61/43/37/16 transportation/shopping/social/content/local-service tasks; gains are percentage points.
Figure E7: Failure-mode composition of our final policy by horizon class. Each pie normalizes the seven mutually exclusive terminal modes over failed attempts in that class; labeled shares are rounded to whole percentages, and slices below 7.5% remain in the legend without an in-slice label. Cross-app handoff errors are orthogonal tags, not an eighth slice.
Figure E8: Paired transfer on 100 tasks across 20 applications. The panel balances 50 entity rebindings and 50 Trace2World-Bench workflow holdouts; complementary 3:2 and 2:3 app-level mixes keep five tasks per app. (a) Base and stage-3 task success in both substrates; error bars are seed SD. (b) Mean paired outcome composition. Agreement includes both-pass and both-fail.
| Slice | Base env. | Base real | Trained env. | Trained real | Agreement | |
| Entity rebinding | 50 | 58.7 | 54.0 | 79.3 | 73.3 | 78.0 / 84.7 |
| Held-out workflow | 50 | 34.0 | 30.7 | 50.7 | 44.0 | 84.7 / 85.3 |
| All tasks | 100 | 46.3 | 42.3 | 65.0 | 58.7 | 81.3 / 85.0 |
Table E15: Balanced task-family transfer slices. The fixed panel contains 50 unseen entity/state rebindings and 50 workflows sampled directly from the 512-task Trace2World-Bench held-out-workflow slice. Percentages; agreement is base / trained over three seeds.
Figure F1: Complete same-task trajectories from the Meituan home (eight pages). (a–b) The frozen marked Reference establishes entity resolution, coupon arithmetic, and the pre-commit terminal; top-left numbers are local step indices, red dots mark the next recorded action, and the terminal panel identifies the stop. (c–d) GUI-GENESIS completes the workflow in its task-local site. (e–f) Trace2World completes the same workflow in one shared 100-task world. (g) One rubric below the aligned terminal frames specifies the deal, coupon-adjusted amount, and pre-commit payment boundary for all three traces; the Trace2World terminal payment indicators use an isolated display-only view correction, with the raw capture retained in the artifact. The original Trace2World package evaluator scores the frozen run 1.0. (h) Direct+Reference reaches a generic Group-buy page but exposes no Beauty/Hair control, CUA-Gym+Reference returns to the unchanged home shell, and PhoneWorld opens a sauna list. Every sequence starts at home; failed traces end at their first missing required action without an inferred suffix. This single case diagnoses mechanisms rather than failure frequencies.
Figure F2: Trace-wise real-app/generated-app execution for a seed 58Tongcheng task (four pages). The upper sequence is the human physical-device execution and terminal observation; the lower sequence is a successful execution in the Trace2World environment. The numbered cards follow each trace’s own action order; the unnumbered terminal observation is separate. The final panel lists the outcome-evaluator checks and the alignment boundary.
Figure F3: Complete augmented-task generated-app execution in 58Tongcheng (four pages). Each card shows a full marked pre-action viewport and a public-schema action. The 20-action trace retains three no-new-observation attempts as context-reuse cards and includes return, recomputation, branch expansion, one persistent like, and recovery from an unintended merchant page. The terminal evaluator requires the listed checks and vetoes calling, consultation, or coupon collection; the case remains qualitative.
Figure F4: 58Tongcheng reconstruction gallery (five matched page states across two figure pages). The paired pages cover service entry, the rental channel, VR filtering, a property detail, and the interactive panorama.
Figure F5: Railway 12306 Didi Railway 12306 return-dependency case (20 actions and a terminal observation over five pages). Numeric card labels are local step indices. The final panel lists eight ordered evaluator checks for the Railway eligibility, Didi fare gate, persistent return, order binding, and unpaid terminal, together with the cross-application bindings and terminal vetoes. The panels assemble separately recorded app-local executions; no single frozen execution record spans the complete cross-application episode.
Figure F6: Xiaohongshu reconstruction gallery (five matched page states across two figure pages). The paired pages cover the discover shell, verified search result, verified profile, store-product picker, and service-rating sheet. Each pair shares one task, launch, and task-conditioned visible state; differences outside that state remain visible.
Figure F7: Bilibili reconstruction gallery (five matched page states across two figure pages). The paired pages cover video home, food-query results, the filter sheet, three selected long-duration buckets, and the resulting tutorial list.
Figure F8: Railway 12306 reconstruction gallery (five matched page states across two figure pages). The paired pages cover a date-and-route query, the D2643 result, waitlist confirmation, passenger binding, and the unpaid terminal order.
Figure F9: Dingdong Maicai reconstruction gallery (five matched page states across two figure pages). The paired pages cover grocery home, category entry, combined promotion filters, a product-variant sheet, and the resulting cart state.
Figure F10: Generated-app home screens (1/2). Default states of formal augmented browser environments; fresh browsers, common viewport, no task-specific initialization. Device shells are hidden; app content and controls remain. The twenty apps match the human-audit roster (Appendix D.3 ); these images illustrate entry diversity.
Figure F11: Generated-app home screens (2/2). Local-service, travel, grocery, food, shopping, and automotive entry surfaces captured from formal augmented browser environments at a common viewport. Decorative phone shells are hidden and app controls retained.
More in this topic
RL for GUI Agents
Reinforcement learning for GUI agents trains screen-based AI to complete multi-step tasks across mobile, web, and desktop interfaces.
See all 50 papers