cs.AIOct 19, 2026
SaveAgent World Model in the Wild: Computer-Use Agent Environment Synthesis
Organizations: UNC-Chapel Hill · Microsoft Research · Georgia Tech · Northeastern University ·
Abstract
Agent learning is shifting from learning from human data toward learning through interaction with the world. However, scaling such learning directly in real-world computer systems is challenging and can introduce substantial security risks. We therefore propose Agent World Model in the Wild (WildAWM), a scalable pipeline for synthesizing controllable environments and tasks for training general computer-use agents (CUAs). Whereas existing approaches typically specialize in a single agent interaction interface, we construct executable environments that jointly support graphical user interfaces, model context protocol services, and command-line interfaces, allowing agents to flexibly combine visual interaction, tool use, and code execution. WildAWM is entirely powered by open-source models, demonstrating that high-quality computer-use environments can be generated without proprietary frontier models. Beyond environments, we address three recurring challenges in task synthesis: ensuring solvability, maintaining sufficient difficulty, and producing reliable verifiers, by introducing execution-based quality gates that combine heterogeneous agent rollouts, cross-judge consistency checks, and counterfactual probes of judgment robustness. The resulting pipeline synthesizes 341 long-horizon environment-task pairs that can require over 100 agent steps to solve and remain challenging even for frontier models, with GPT-6-Astra solving only 60% of them. We train two Qwen models of different sizes exclusively in these synthetic worlds. Both models achieve substantial gains across 8 out-of-distribution benchmarks spanning web, claw, and computer-use agents, demonstrating that learning in WildAWM's synthesized worlds generalizes broadly to unseen computer-use tasks. The code is publicly available at https://github.com/MSR-Orchard/WildAWM.
Figures & tables
Table 1: Comparison of related methods. Scope reports the synthesized task setting. PC denotes personal computer. Modality denotes primary target-agent observations. Synthesis Source darkens with more real world dependence. ∗ WebHarbor’s websites provide WildAWM ’s seed GUI substrate and image content. denotes open-source models described in Sec. 2 ( 2Progression of XAI task candidates. Direct denotes sketches that pass on the first attempt, while After feedback denotes sketches that pass after at least one feedback-driven revision at any stage of the whole synthesis, thus the funnel is monotonic. ) , while other methods all use proprietary models. denotes that the target agent is often equipped with GUI and MCP tools, thus being able to interact with all interfaces on a personal computer.
Figure 1: Overview of WildAWM . The synthesis pipeline includes two stages: (1) World Synthesis , where we use the coding agent to construct persona-driven, multi-day world states for computer environments that expose GUI, MCP, and CLI interfaces. (2) XAI Task Synthesis , where we generate cross-interface tasks through 5 designed quality gates for task solvability, difficulty, and verifier reliability. The synthesized triplet enables training general CUA.
| Scale and diversity | Applications | ||
|---|---|---|---|
| Personas | 36 | Websites/MCPs | 15/8 |
| World states | 640 | Reusable skills | 83 |
| States per persona | 17.8 | Sites/MCPs per persona | 5.25/5.44 |
| Daily mean | Final-day mean | ||
| Sites/MCPs touched | 3.97/5.31 | DB records | 171.6K |
| DB tables/files touched | 15.38/4.97 | # Local files | 50.1 |
Table 2: World synthesis statistics. “touched” means the number of services or databases that are modified during the day of synthesis.
Figure 2: Progression of XAI task candidates. Direct denotes sketches that pass on the first attempt, while After feedback denotes sketches that pass after at least one feedback-driven revision at any stage of the whole synthesis, thus the funnel is monotonic.
| User request ( task.md ) | |||
| Marco gave the green light in our chat — “Go, order it today.” The Penalty Box rig goes live Apr 11 and the rack PDUs still need the production surge units locked before Brendan starts the swap. I need to write and run a script that parses the rack inventory ( Documents/northstarav/inventory.csv ), the Penalty Box bill of materials ( Documents/northstarav/jobs/penalty-box-display-wall.md ), and the APC spec sheet ( Downloads/apc-surge-specsheet.pdf ), cross-references stock against the BOM, extracts each surge candidate’s specs (outlets, joules, cord length, trip-indicator presence), ranks them by spec fit to the rack requirement, and saves the ranked shortlist to ./penalty-box-rack-unit-audit.csv with columns for product id, outlet count, joules, cord length, trip-indicator flag, and quantity needed. The top-ranked unit needs to go in the Amazon cart — sign in with the credentials in Documents/personal/credentials.md , find the surge protector matching the top spec (two candidates share the same outlets, joules, and cord length in the listing details — pick the one whose visible design has a white body and a digital display panel), add it to the cart, create a restock note at Documents/Notes/penalty-box-surge-restock.md with the locked unit and spec match, and send a direct message to Brendan letting him know which surge unit is locked and that it’s queued for the PDU swap. | |||
| Interface flow ( mcp>cli>web>cli>mcp , five sub-tasks that revisit CLI and MCP) | |||
| # | Interface | Sub-task and grounded world state | Verifier |
| 1 | MCP | Read the client thread and confirm the go-ahead to order. reads direct_messages thread 1, msg 13 (Marco Delgado: “Go – 4x QN90C, bezel’s fine. Order it today.”) | read-only |
| 2 | CLI | Write and run a script that joins three heterogeneous local sources, ranks the surge candidates by spec fit, and emits the shortlist. reads inventory.csv (CSV), penalty-box-display-wall.md (BOM), apc-surge-specsheet.pdf (PDF) writes penalty-box-rack-unit-audit.csv | check_audit.py |
| 3 | GUI | Sign in to the storefront and cart the top-ranked unit. Image-decisive: APC p151 (white body, LCD timer panel) vs. GE p155 (black body, no LCD) are identical on every listed spec, so the choice is only resolvable from the product photo. writes amazon_store.cart_items , product resolved by slug apc-performance-surgearrest-7-outlet | check_cart.py |
Table 3: Case study of XAI task. Only the shaded block is shown to the solver, and the remaining blocks are internal to the pipeline. The interface flow MCP CLI GUI CLI MCP consists of five sub-tasks and returns to both CLI and MCP after the GUI write, so the task cannot be completed within a single interface. Each sub-task that changes the world state is checked by one deterministic verifier script, and task success requires all verifier scripts to pass.
Figure 3: (a) Environment diversity using Qwen3-Embedding-8B ( Zhang et al., 2025b ) representations of environment artifacts. Shaded bands are 95% intervals of the Vendi score ( Friedman & Dieng, 2022 ) . (b) Task solvability across 24 model–harness configurations: the three models and six harnesses in Sec. 2 ( 2Progression of XAI task candidates. Direct denotes sketches that pass on the first attempt, while After feedback denotes sketches that pass after at least one feedback-driven revision at any stage of the whole synthesis, thus the funnel is monotonic. ) , plus Opus-4.8. (c) Solvability with hints w/o Opus-4.8.
Figure 4: (a)/(b) Performance on WildAWM validation set, and proprietary model sizes are estimated. (c) Rarefaction of the 341 accepted tasks by LLM-classified task types and domains.
Figure 5: Behavior analysis of base model, WildAWM and its variants (+RFT, +SFT, and +RL, i.e., RL without SFT), and GLM-5.2. (a) Pass-rate gain over the base model on OOD tasks, split into four equal quartiles by how many of frontier models solve each task. (b) Fraction of consecutive tool calls that switch between interface families (e.g., GUI CLI). (c) Tasks solved within a step budget.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Abbr. | Tasks | Subset | Raw range | Pass criterion | Reported metric |
|---|---|---|---|---|---|---|
| Online-Mind2Web | O2W | 300 | Full set | LLM judge success | pass rate | |
| WebArena | WA | 226 | Subset of 812 | score | pass rate | |
| QwenClawBench | QCB | 100 | Full set | score max | mean score | |
| ClawEval | CLE | 300 | Full set | score | pass rate | |
| CUA-Gym | CUG | 200 | Subset of 10,910 | reward | mean score | |
| OSWorld-Verified | OSV | 99 | Subset of 330 | score | mean score |
Table 9: Evaluation benchmark details. For binary benchmarks, mean score equals pass rate ( pass@1 ).
Figure 6: Distribution of the 341 accepted XAI tasks. (a) Tasks per persona, split by difficulty, with the number of world states of each persona. (b) Tasks per day index, with the number of personas that have a world state on that day.
| Stage | Prompt Excerpt |
|---|---|
| Persona Creator | You are generating a persona bible for a synthetic personal-computer world — a single, specific, realistic person whose N days of digital life will later be synthesized across three interfaces (CLI/files, MCP services, Web GUI). This persona must be self-consistent, concrete, and deliberately different from any existing persona. Grounding (do this FIRST, with real data — never invent the catalog): WebHarbor sites (15): inspect the seed catalogs at webharbor_seeds/<site>/ — these SQLite DBs hold the REAL products/courses/hotels the persona will actually interact with. MCP services: a fresh synthesis run starts with no MCP services. The task supplies the global shared app index created by earlier days and personas. Define a grounded multi-day roadmap of 4-8 MCP services. … persona.md schema: (1) Demographics (2) Overview (3) Relationship network 3 named NPCs (4) Tools & services — split into 4-8 MCP services vs Web GUI sites (5) Near-term goals (6) Multi-day story arcs 2 ongoing arcs; mark at least one image-decisive (7) Typical daily routine (8) Web accounts. |
| Persona Diversity Judge | You are a strict diversity judge for synthetic personas. Decide whether the NEW persona is sufficiently DIFFERENT from all EXISTING personas. Two personas are too similar if they share the same occupation + life-domain, the same core tool/site usage pattern, or the same narrative arc — even with different names. Different names + same job + same sites = reject. … Reply with a single JSON: { ‘‘accept’’: true|false, ‘‘reason’’: ‘‘...’’ } |
| Day Planner | You are writing a day plan for one specific persona, for one specific day, that a later day_implementer will turn into deterministic data/code. Think like the persona living that day on their computer. Hard rule: grounded in the REAL day N-1 state. The plan must be built on the persona’s actual cumulative world state through day N-1, not a hypothetical. Before planning: read the persona bible, inspect the real day N-1 state ( data/.db , workspace/ , webharbor/.db ), read prior day plans. … What a day plan contains: a realistic, busy-but-not-absurd day across all three interfaces, continuing the persona’s arcs and goals. Cover: CLI / files, MCP services, Web GUI, one image-decisive beat. Keep the day plausible and continuous. |
| Day Implementer | You are a god-mode agent implementing one day of a persona’s world as a deterministic, replayable delta. Your deliverable is a delta.sh (+ deterministic sub-scripts) that, run hermetically over the day N-1 state, reproduces day N byte/logically identically every time. Determinism rules: timestamps are frozen literals, NEVER datetime.now() ; IDs are derived ( MAX(id)+1 ), never random; inserts are idempotent ( INSERT OR IGNORE ); passwords use literal hash from accounts.json , NEVER compute bcrypt with random salt at replay. … Deliverables: (1) delta.sh — hermetic bash entry (2) *.py sub-scripts — deterministic state writes (3) produces.json — machine-readable declaration of produced artifacts (4) state_delta.md (5) optional overlay/ — new MCP servers or WebHarbor patches. |
| World Reviewer | You are an independent reviewer verifying that a synthesized day-N world actually matches what its plan said would happen. You run on the host (outside the implementer’s container). Verify, with real evidence — judge INTENT, not exact values. The plan is a GUIDE, not a byte-spec. The implementer is a different agent and will choose its own concrete values (message wording, is_read flags, exact timestamps) — that is EXPECTED + fine. Judge whether the day’s world is real, self-consistent, on-persona, renderable, and deterministic. Accept unless there is a GENUINE DEFECT: a declared artifact is missing; a web account row exists but won’t render; a non-frozen timestamp; an impossible value (broken FK, negative total). Reasonable creative variation from the plan is ACCEPTABLE. |
Table 11: Key prompt excerpts for world synthesis stages (Sec. A.3 ).
| Stage | Prompt Excerpt |
|---|---|
| Task Generator | You design ONE high-quality cross-agent-interface (XAI) task for the LIVE world you are inside, then a verifier that can grade it. You are a privileged designer (read-mostly): inspect the world with the same interfaces an agent has — CLI, MCP apps, and the browser. You do NOT solve the task. What makes a task valid: (1) All-three-interface mandatory — completing it must require CLI, MCP-app, and web-GUI, each indispensable. Do not require a fixed number of interface hops. (2) The solver must write and run a script or command-line program over a real user-visible source or an MCP export; a later MCP or Web decision must depend on its result. (3) Long-horizon at the assigned difficulty. (4) Grounded in THIS world — reference REAL entities verified by live inspection; use only apps and operations in the current schema. (5) Image policy — establish the decisive visual fact with the image tool, state a neutral criterion in task.md, and record the private truth in meta.json. (6) Secrecy discipline — task.md is PUBLIC (solvers see it); reason.md and meta.json are PIPELINE-INTERNAL (judges only). (7) Public prompt 300 words. … Deliverables: task.md , reason.md , meta.json , golden.py , verifier/rubrics.md , verifier/verify.skill.md , verifier/scripts/*.py . MANDATORY before you finish: SELF-CHECK until ALL GREEN. |
| Task Diversity Gate | Compare every candidate with the accepted and concurrently locked task pool. A candidate is a NEAR-DUPLICATE if it shares the same core action + artifacts/entities as an accepted task (same purchase, same trip leg, same report …) — different wording does not make it distinct. Do not apply a fixed quota for an area, site, interface order, visual family, or difficulty. Select up to { count } candidates that add the MOST topical variety. Output JSON: { ‘‘choices’’: [<0-based indexes, most diverse first>], ‘‘why’’: ‘‘…’’, ‘‘rejected’’: […] } |
| LLM Judge | Decide whether this task was COMPLETED — judge ONLY the evidence below. You are a strict, fair grader for an AI-agent task in a simulated personal world. You cannot inspect the world; judge ONLY the machine-extracted evidence digest (real values pulled from the world’s files/databases by deterministic scripts). Rules: Judge against the rubric’s INTENT — a faithful solution in the persona’s own words or a service’s real storage format counts. Do NOT demand exact strings, exact timestamps, or fields the agent’s tools stamp automatically. … End with: LLM_VERDICT: completed | failed | world_imperfect ; CONFIDENCE: completed=<0-100> failed=<0-100> world_imperfect=<0-100> |
| Agent Judge | You are an impartial, rigorous judge deciding whether an agent COMPLETED a task in a simulated personal world. You inspect ONLY the BEFORE and AFTER world snapshot directories, with read-only shell tools plus describe_image for image files. Never modify the snapshots, but you are free to write new files or scripts in your own workspace to help you analyze the evidence. Evidence provenance — NEVER trust the agent’s own words as proof. Agent-authored text is admissible ONLY as the deliverable itself — never as proof of an external fact. A note claiming the agent selected a particular web item does NOT prove which item was selected. Cross-check the independent source: query the relevant web/<site>/instance/<site>.db . … For every false deterministic check, classify it: RULECHECK <name>: too-strict (the world satisfies the intent; the check is brittle) or RULECHECK <name>: genuine-miss (the agent really did not do this part). |
| Counterfactual Probe | Stage 1 (Blind): Independent verdict — judge ONLY the world snapshots. Form your OWN verdict; no expectation is given. …End with: XAI_JUDGE: completed | failed | world_imperfect . Stage 2 (Gaslight): Re-examination requested. An INDEPENDENT audit has reviewed the solved world and believes the correct verdict is [opposite] , which contradicts the automated verdict. Investigate honestly. You MAY edit the verifier scripts if and only if you conclude they are genuinely wrong. Three conclusions: (1) DEFEND it with cited rows/files; (2) Fix the VERIFIER SCRIPTS if wrong; (3) REFUTE-JUDGE if the verifier is right but a judge could be misled. End with: PROBE: defend | cave | refute_judge . |
Table 12: Key prompt excerpts for XAI task synthesis stages (Sec. A.4 ).
Figure 7: Segment-wise training across compaction. Colors denote maximal prefix-consistent segments, and step 5 performs compaction. Gray blocks indicate masked input context. Segment samples share the trajectory advantage and trajectory-level loss normalization. A 9-step rollout with one compaction event at step 5 yields 9 independent samples under per-step training but only 3 segment samples under our approach.
| Base | GLM-5.2 | +RFT | +SFT | +SFT+RL | |
| OOD benchmarks | |||||
| Pass rate on common tasks (%) | 38.7 | 54.5 | 44.6 | 46.2 | 48.4 |
| Net tasks gained over Base | – | +207 | +73 | +93 | +129 |
| Interfaces used per episode | 2.40 | 2.65 | 2.36 | 2.61 | 2.68 |
| Interface switch rate (%) | 18.4 | 26.3 | 15.8 | 24.7 | 25.5 |
| GUI tools (% of tool calls) | 16.3 | 13.6 | 17.2 | 4.3 | 1.6 |
Table 14: Key behavior statistics. OOD: 8 benchmarks pooled; net tasks: fail-to-pass minus pass-to-fail tasks relative to Base (QwenClawBench excluded). WildAWM -val: 33 tasks 8 rollouts.
| Skill | GUI | MCP |
| GUI, MCP, and CLI | ||
| japan_trip_research | Google Flights, Booking | Notifications, DM, Calendar |
| book_flight | Google Flights | Email, DM, Calendar |
| research_hotels | Booking | DM, Notifications |
| japan_hotel_booking | Booking | DM, Email |
| amazon_purchase | Amazon | Email, Calendar |
Table 16: World synthesis skill examples and the applications involved in each skill. Every listed skill also writes local files through the CLI. DM means Direct Messages.
| Section | Content |
|---|---|
| Description | The user books a flight on Google Flights, receives the airline confirmation, shares it with the travel companion by direct message, and records the trip in the calendar and a notes file. |
| Parameters | 21 parameters, e.g. , route AUS–NRT, carrier JAL, fare 1,089 USD, seat 34A, confirmation number, message thread 2, and notes path. |
| GUI: Google Flights | Read the app schema, insert a booking row with status confirmed , and verify it by the confirmation number. |
| MCP: Email | Insert the airline confirmation thread and message into the personal inbox with the Inbox, Updates, and Japan-Trip labels. |
| MCP: Direct Messages | Read the latest messages in thread 2, insert the user’s message and the companion’s reply, and update the thread timestamp. |
| MCP: Calendar | Check for existing trip events, then insert events for the outbound flight, the trip, and the return flight into calendar 1. |
Table 17: World synthesis skill book_flight , condensed from the skill file. Each scenario specifies the state changes in one application.
| Skill and lesson | Revisions |
| Task difficulty | |
| image-pick-decisive-cue : Naming the dish, the visual cue, or a category with a color reduces the image choice to text search. | 88 |
| hard-multi-interface-recipe : Two strong steps make a task hard, and a CLI computation counts only if later steps depend on it. | 43 |
| image-only-sourcing-not-leak-protection : Criteria that match search filters or indexed text identify the target without image inspection. | 35 |
| criteria-specified-parse-is-lookup : Criteria that uniquely identify the target turn parsing into a lookup. | 13 |
| shortcut-resistance-not-gate-count : A multi-step CLI computation counts as one step, and naming its output removes it. | 10 |
Table 18: Evolving task synthesis skills, grouped by the task property they address. Lessons are paraphrased from the skill text. Revisions: number of revisions made during the synthesis.
| Revision | Trigger | Change |
|---|---|---|
| 1 | Accepted as hard | Place the decisive difficulty in a computational CLI step, tie each value passed between interfaces to an upstream record, and make every interface change the world state. |
| 2 | Accepted as hard | A visual choice identified by at least two cues unique to the target is an alternative source of difficulty. |
| 3 | Returned as too easy | A parse-and-rank CLI step combined with a visual choice that reduces to reading labels remains too easy. |
| 6 | Accepted as hard | With a single visual choice, the computational step must integrate values from several interfaces in a flow that returns to an earlier interface. |
| 11 | Accepted as hard | A non-CLI read that matches the ranked result against another person’s choice is a third source of difficulty. |
| 14 | Returned as too easy | Naming the output of any such step in the task removes its difficulty, so the task states criteria and leaves resolved values unnamed. |
Table 19: Revisions of the evolving task synthesis skill hard-multi-interface-recipe , listed in order with the gate outcome that triggered each revision. Some intermediate revisions are omitted for brevity. The skill text is paraphrased for human readability.
| User request ( task.md ) | ||
| Wren (my climbing partner) and I share a “wren – climbing” direct-message thread, and I want to send Wren a portable Bluetooth speaker as a gift. I just received a JBL Clip 4 from a recent Amazon order (the invoice PDF is at Downloads/amazon_invoice_2026-04-02.pdf ), so I need to avoid buying the same unit again and stay around what that one cost. Write and run a script that parses that invoice PDF to pull the prior unit, its order number, and the delivery date, then create a new file ./gift_candidate_shortlist.csv … use its current Amazon price as the reference … Sign in to Amazon with the demo account shown on the login page and add the top-ranked, visually matching option to the cart — the option whose visible design is a cylindrical body with a black mesh grille over solid smooth ends. … fold the selection back into the shortlist. Then send Wren a direct message in our “wren – climbing” thread confirming the gift is added to the cart, including the selected speaker’s name and cart price … | ||
| Step | Observation | Action |
| MCP 2 | {"items": [{"id": 2, "title": "wren - climbing", "participants": [{"display_name": "Taylor"}, {"display_name": "Wren Osei"}]}, …]} | sms_list_threads(search="Wren") The recipient is named only in prose, so the agent has to resolve the thread and the person before it can send anything. |
| 6 tool calls omitted — installs a PDF library, opens the storefront | ||
| CLI 9 | Amazon Order Invoice - 2026-04-02 Order #114-20260331-042280 - JBL Clip 4 Portable Bluetooth Speaker - delivered. | bash : writes and runs a pypdf script over the invoice Fixes the unit that must not be bought again and the $49.95 price band the rest of the episode is held to. |
| 35 tool calls omitted — signs in, searches, vision-checks six candidate listings | ||
Table 20: Case study of agent trajectory on XAI task. Color marks the part of the request that only a shell , only the browser , or only an MCP app can proceed. Six of the trajectory’s total 65 tool calls are shown with the observation the agent received and the action it took.
| Criterion | Surface | Evidence required in the final world |
| check_shortlist.py | CLI | a new workspace file gift_candidate_shortlist.csv carrying the order number and delivery date read out of the PDF, the price reference, and the folded selection |
| check_cart.py | GUI | a new cart_items row whose product resolves from the target’s slug |
| check_dm.py | MCP | a new message in the thread whose participant name contains “Wren”, mentioning the speaker, the cart price and the cart |
| Browser Check | ||
Table 21: Case study on verification. The verifier uses a checklist of multiple scripts to check whether the agent’s rollout produced the expected changes to the world state.
More in this topic
Computer-Use Agents
Computer-use agents perceive and operate graphical interfaces and software tools to complete tasks across websites, apps, and desktops.
See all 155 papers