Terrarium: Growing Personal Assistants via Scalable Synthesis of App Worlds
Authors: Mengzhou Wu, Yuzhe Guo, Jiawen Tao, Xiaokun Yuan, Yuan Cao, Gang Cao, Yuetang Deng, Zhi Gong, Dezhi Ran, Linyi Li, Wei Yang, Tao Xie
Organizations: School of Computer Science, Peking University, Beijing, China · Beijing Tongming Lake Information Technology Application Innovation Center, Beijing, China · WeChat, Tencent Inc., China · Simon Fraser University, Canada · Institute of Systems for Advanced Computing, Fudan University, China
Advances in large language models (LLMs) have brought the vision of a general-purpose personal assistant closer to reality. Such assistants could provide a unified natural-language interface to a broad ecosystem of applications. However, real-world applications do not readily support training at scale because access is often restricted and their state is difficult to reset. To address this challenge, we introduce Terrarium, a framework that synthesizes executable App Worlds from real-world application information, generates training task packages with evaluator-side rubrics from these worlds, and iteratively validates and refines both through agent rollouts. For App World synthesis, Terrarium organizes public Web information and in-application observations into structured app specifications that guide the construction of self-contained runtimes with callable operations and mutable, resettable state. For task synthesis, Terrarium constructs tasks grounded in App World state together with evaluator-side outcome rubrics that assess success without prescribing reference trajectories. We construct a training suite of 275 App Worlds and 13,344 tasks for outcome-based reinforcement learning across three Qwen3 model scales (4B, 8B, and 14B parameters). Across these scales, models trained with Terrarium outperform the corresponding Qwen3 models by an average of 18.34 percentage points on the τ2-bench and 10.94 percentage points on the AppWorld bench. These results suggest that app-grounded world and task synthesis offers a scalable path toward general-purpose assistants capable of operating across broad application ecosystems.
Figures & tables
Figure 1: From 275 applications, Terrarium constructs executable App Worlds and 13,344 tasks for assistant training.
Figure 2: Overview of the Terrarium construction pipeline.
Model
τ2 -bench
AppWorld
Airline
Retail
Telecom
Overall
Normal
Challenge
Overall
Qwen3-4B (Thinking)
36.00
39.47
26.32
33.45
16.67
9.59
11.62
AWM-4B
22.00
41.23
28.07
32.37
15.48
9.35
11.11
EnvScaler-4B
44.00
58.77
28.07
43.53
15.48
12.71
13.50
Terrarium -4B (Ours)
50.00
62.28
54.39
56.83 (+23.38)
30.36
21.10
23.76 (+12.14)
Qwen3-8B (Thinking)
34.00
46.49
37.72
40.65
23.21
16.31
18.29
Table 1: Results on τ2 -bench and AppWorld. Parentheses show Terrarium gains over the corresponding Qwen3 (Thinking) model, and bold marks the best result at each scale.
Figure 3: Category distribution of 275 App Worlds.
Figure 4: Complexity of 275 App Worlds. Bars show the interquartile range, with markers for the median and P90.
Configuration
τ2 -bench
AppWorld
Airline
Retail
Telecom
Overall
Normal
Challenge
Overall
A. Training strategy
Qwen3-4B (Thinking)
36.00
39.47
26.32
33.45
16.67
9.59
11.62
SFT on accepted rollouts
28.00
40.35
5.26
23.74
26.79
13.67
17.44
SFT on all rollouts
36.00
49.12
8.77
30.22
31.55
15.83
20.34
GRPO
50.00
62.28
54.39
56.83
30.36
21.10
23.76
Table 2: Ablations of training strategy and supervision design on Qwen3-4B. Panels A and B compare training strategies and supervision variants, respectively.
Figure 5: Performance versus the number of training App Worlds on τ2 -bench and AppWorld.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Failure class
Classification rule
Required action
Policy failure
The task, tool schema, seed, runtime, rubric, evidence, and judge wiring are coherent, but the rollout does not satisfy the outcome
Retain as a policy-failure example
Task/rubric defect
The query is ambiguous or infeasible, the rubric is too strict/loose, the evidence targets the wrong object or state, or process requirements replace outcome requirements
Repair the query, rubric, expected evidence, or judge-input construction; rerun every affected task
Seed/runtime defect
Required entities or relations are absent, enrichment is inconsistent, a valid transition is impossible, the tool contract disagrees with implementation, or reset/diff is incorrect
Repair the seed, enrichment, operation schema, transition, or state-diff provider; reset and rerun affected tasks
Infrastructure failure
API, runner, adapter, serialization, timeout, path, or judge-schema failure occurs independently of task semantics
Repair or retry infrastructure; exclude the episode from task difficulty and outcome statistics
Appendix
Table 3: Calibration failure classification and routing.
Quantity
Mean
P25
Median
P75
P90
Maximum
Capabilities / world
7.90
4
7
11
14
26
Steps / capability
4.60
4
4
5
7
13
Entity types / world
10.21
6
9
13
19
40
State tables / world
11.11
6
10
15
19
40
Operations / world
15.81
10
14
22
28
46
Observation
9.62
5
8
13
17
29
Appendix
Table 4: Complete structural profile of the final accepted corpus.
Category
Count (%)
Category description
Travel
45 (16.4)
Mobility, trip planning, transportation, and logistics.
Entertainment
44 (16.0)
Media, sports, events, and other leisure activities.
Commerce
37 (13.5)
Shopping, retail, and related commercial transactions.
Services
39 (14.2)
Government and public services, utilities, and information access.
Lifestyle
41 (14.9)
Food, hospitality, local services, and social or community activities.
Business
27 (9.8)
Workplace productivity, business operations, finance, and payments.
Appendix
Table 5: The eight App World categories used in the main paper and their descriptions. Counts and percentages are over 275 App Worlds.
Model
Base
Miss. Func.
Miss. Param.
Long Context
Overall
Unmodified Qwen3
Qwen3-4B
39.00
28.00
32.00
21.00
30.00
Qwen3-8B
45.00
43.00
36.00
23.50
36.88
Qwen3-14B
50.50
37.00
36.50
29.50
38.38
Data scaling
Terrarium -4B (35 App Worlds)
47.00
27.50
36.50
29.00
35.00
Appendix
Table 6: BFCL Multi-Turn accuracy (%). Each split contains 200 examples, and Overall is the macro-average across Base, Missing Function, Missing Parameter, and Long Context. All models are evaluated with thinking enabled and thinking history preserved.