Terrarium: Growing Personal Assistants via Scalable Synthesis of App Worlds
Authors: Mengzhou Wu, Yuzhe Guo, Jiawen Tao, Xiaokun Yuan, Yuan Cao, Gang Cao, Yuetang Deng, Zhi Gong, Dezhi Ran, Linyi Li, Wei Yang, Tao Xie
Organizations: School of Computer Science, Peking University, Beijing, China · Beijing Tongming Lake Information Technology Application Innovation Center, Beijing, China · WeChat, Tencent Inc., China · Simon Fraser University, Canada · Institute of Systems for Advanced Computing, Fudan University, China
Advances in large language models (LLMs) have brought the vision of a general-purpose personal assistant closer to reality. Such assistants could provide a unified natural-language interface to a broad ecosystem of applications. However, real-world applications do not readily support training at scale because access is often restricted and their state is difficult to reset. To address this challenge, we introduce Terrarium, a framework that synthesizes executable App Worlds from real-world application information, generates training task packages with evaluator-side rubrics from these worlds, and iteratively validates and refines both through agent rollouts. For App World synthesis, Terrarium organizes public Web information and in-application observations into structured app specifications that guide the construction of self-contained runtimes with callable operations and mutable, resettable state. For task synthesis, Terrarium constructs tasks grounded in App World state together with evaluator-side outcome rubrics that assess success without prescribing reference trajectories. We construct a training suite of 275 App Worlds and 13,344 tasks for outcome-based reinforcement learning across three Qwen3 model scales (4B, 8B, and 14B parameters). Across these scales, models trained with Terrarium outperform the corresponding Qwen3 models by an average of 18.34 percentage points on the τ2-bench and 10.94 percentage points on the AppWorld bench. These results suggest that app-grounded world and task synthesis offers a scalable path toward general-purpose assistants capable of operating across broad application ecosystems.
Figures & tables
Figure 1: From 275 applications, Terrarium constructs executable App Worlds and 13,344 tasks for assistant training.
Figure 2: Overview of the Terrarium construction pipeline.
Model
τ2 -bench
AppWorld
Airline
Retail
Telecom
Overall
Normal
Challenge
Overall
Qwen3-4B (Thinking)
36.00
39.47
26.32
33.45
16.67
9.59
11.62
AWM-4B
22.00
41.23
28.07
32.37
15.48
9.35
11.11
EnvScaler-4B
44.00
58.77
28.07
43.53
15.48
12.71
13.50
Terrarium -4B (Ours)
50.00
62.28
54.39
56.83 (+23.38)
30.36
21.10
23.76 (+12.14)
Qwen3-8B (Thinking)
34.00
46.49
37.72
40.65
23.21
16.31
18.29
Table 1: Results on τ2 -bench and AppWorld. Parentheses show Terrarium gains over the corresponding Qwen3 (Thinking) model, and bold marks the best result at each scale.
Figure 3: Category distribution of 275 App Worlds.
Figure 4: Complexity of 275 App Worlds. Bars show the interquartile range, with markers for the median and P90.
Configuration
τ2 -bench
AppWorld
Airline
Retail
Telecom
Overall
Normal
Challenge
Overall
A. Training strategy
Qwen3-4B (Thinking)
36.00
39.47
26.32
33.45
16.67
9.59
11.62
SFT on accepted rollouts
28.00
40.35
5.26
23.74
26.79
13.67
17.44
SFT on all rollouts
36.00
49.12
8.77
30.22
31.55
15.83
20.34
GRPO
50.00
62.28
54.39
56.83
30.36
21.10
23.76
Table 2: Ablations of training strategy and supervision design on Qwen3-4B. Panels A and B compare training strategies and supervision variants, respectively.
Figure 5: Performance versus the number of training App Worlds on τ2 -bench and AppWorld.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Failure class
Classification rule
Required action
Policy failure
The task, tool schema, seed, runtime, rubric, evidence, and judge wiring are coherent, but the rollout does not satisfy the outcome
Retain as a policy-failure example
Task/rubric defect
The query is ambiguous or infeasible, the rubric is too strict/loose, the evidence targets the wrong object or state, or process requirements replace outcome requirements
Repair the query, rubric, expected evidence, or judge-input construction; rerun every affected task
Seed/runtime defect
Required entities or relations are absent, enrichment is inconsistent, a valid transition is impossible, the tool contract disagrees with implementation, or reset/diff is incorrect
Repair the seed, enrichment, operation schema, transition, or state-diff provider; reset and rerun affected tasks
Infrastructure failure
API, runner, adapter, serialization, timeout, path, or judge-schema failure occurs independently of task semantics
Repair or retry infrastructure; exclude the episode from task difficulty and outcome statistics
Appendix
Table 3: Calibration failure classification and routing.
Quantity
Mean
P25
Median
P75
P90
Maximum
Capabilities / world
7.90
4
7
11
14
26
Steps / capability
4.60
4
4
5
7
13
Entity types / world
10.21
6
9
13
19
40
State tables / world
11.11
6
10
15
19
40
Operations / world
15.81
10
14
22
28
46
Observation
9.62
5
8
13
17
29
Appendix
Table 4: Complete structural profile of the final accepted corpus.
Category
Count (%)
Category description
Travel
45 (16.4)
Mobility, trip planning, transportation, and logistics.
Entertainment
44 (16.0)
Media, sports, events, and other leisure activities.
Commerce
37 (13.5)
Shopping, retail, and related commercial transactions.
Services
39 (14.2)
Government and public services, utilities, and information access.
Lifestyle
41 (14.9)
Food, hospitality, local services, and social or community activities.
Business
27 (9.8)
Workplace productivity, business operations, finance, and payments.
Appendix
Table 5: The eight App World categories used in the main paper and their descriptions. Counts and percentages are over 275 App Worlds.
Model
Base
Miss. Func.
Miss. Param.
Long Context
Overall
Unmodified Qwen3
Qwen3-4B
39.00
28.00
32.00
21.00
30.00
Qwen3-8B
45.00
43.00
36.00
23.50
36.88
Qwen3-14B
50.50
37.00
36.50
29.50
38.38
Data scaling
Terrarium -4B (35 App Worlds)
47.00
27.50
36.50
29.00
35.00
Appendix
Table 6: BFCL Multi-Turn accuracy (%). Each split contains 200 examples, and Overall is the macro-average across Base, Missing Function, Missing Parameter, and Long Context. All models are evaluated with thinking enabled and thinking history preserved.
Terminal agents extend Large Language Models with the ability to execute tasks directly in command-line environments, but their progress is bottlenecked by the scarcity of high-quality training data. Existing approaches bootstrap from partial sources such as human-defined seeds or GitHub repositories to instantiate one component and then complete the rest, producing tasks confined to narrow seed distributions, environments misaligned with task semantics, and inefficient trajectories from unguided exploration. To address these limitations, we introduce Terminal-World, a fully automated pipeline that uses agent skills as the central synthesis primitive, which jointly encode what to accomplish, when to apply (preconditions and environment state), and how to execute, enabling task instructions, environments, and teacher trajectories to be co-derived. To further broaden the synthesis space, Terminal-World composes skills into skill teams and skill graphs for multi-role and cross-domain task synthesis. Using this pipeline, we construct 5,723 training environments and train Terminal-World-8B/14B/32B, evaluated across 6 benchmarks where the Terminal-World series consistently outperforms terminal-agent baselines. Notably, using the same teacher model and only 1.2% of the training data, Terminal-World-32B surpasses Nemotron-Terminal-32B on Terminal-Bench 2.0 by +4.5 Pass@1 (31.5) and achieves 43.8 Pass@3.
Zihao Cheng, Hongru Wang, Zeming Liu +6
School of Computer Science and Engineering, Beihang University, Beijing, China · Independent Researcher · 3Beijing Institute of Technology +2
Web agents require massive trajectories to generalize, yet real-world training is constrained by network latency, rate limits, and safety risks. We introduce WebWorld series, the first open-web simulator trained at scale. While existing simulators are restricted to closed environments with thousands of trajectories, WebWorld leverages a scalable data pipeline to train on 1M+ open-web interactions, supporting reasoning, multi-format data, and long-horizon simulations of 30+ steps. For intrinsic evaluation, we introduce WebWorld-Bench with dual metrics spanning nine dimensions, where WebWorld achieves simulation performance comparable to Gemini-3-Pro. For extrinsic evaluation, Qwen3-14B trained on WebWorld-synthesized trajectories improves by +9.2% on WebArena, reaching performance comparable to GPT-4o. WebWorld enables effective inference-time search, outperforming GPT-5 as a world model. Beyond web simulation, WebWorld exhibits cross-domain generalization to code, GUI, and game environments, providing a replicable recipe for world model construction.
Training terminal agents at scale requires diverse, verifiable terminal tasks and high-quality interaction trajectories, yet acquiring such data remains a significant challenge. Existing synthesis methods face two key limitations: (1) weak reliability caused by the disconnect between task generation and real execution, and (2) limited diversity and scalability due to dependence on existing repositories. We propose Meta-Task, a framework that redefines terminal task synthesis as a Terminal-Bench-format task itself: an agent operates within a real container environment to iteratively generate, execute, and verify tasks, so that synthesized components are checked for internal consistency and executability within the generation loop itself. Building upon this, we decouple the target task requirements along multiple dimensions, introduce a multi-phase mechanism that dynamically designs novel task specifications before producing the actual tasks, and incorporate optional external material support to enhance diversity and realism. We additionally apply LLM-as-Judge filtering to ensure the quality of the final training data. Experiments on Terminal-Bench 2.0 show that fine-tuning on only 3,221 Meta-Task synthesized trajectories achieves 22.5% and 31.8% Avg Pass@1 for Qwen3-14B and Qwen3-32B respectively, outperforming concurrent approaches with significantly less training data.
Zhihong Pan, Jiyuan He, Kai Zhang +5
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China · Meituan