Organizations: National University of Singapore · Peking University · University of California, Santa Cruz · Shanghai Jiao Tong University · Tencent · Emory University
LLM agents learn by interacting with environments, yet existing environments are designed for specific tasks and fail to generalize to new ones. Recent methods automate environment construction with LLMs. However, they generate each environment from scratch, neither reusing parts shared across environments nor carrying a fix from one environment to the rest. To address this, we propose a new paradigm of evolving environment atoms, which composes environments from reusable atoms (intent, rule, data and server) and evolves the atoms when an environment fails to give reliable feedback (e.g., an application lacks a tool the task requires). To instantiate this paradigm, we introduce LegoEnv, which builds 5,465 verified environments over 223 applications, and one round of evolution raises the share of solvable environments from 71.6% to 88.3%. On AutomationBench, SFT on these environments lifts Qwen3.6-27B from 16.0% to 32.0%, and RL with their verifiers lifts Qwen3.8-27B from 37.6% to 40.3%. Further analyses show steady gains with corpus size, transfer to unseen applications, and failures concentrated in a few shared root causes.
Figures & tables
Figure 1: Overview of LegoEnv. Four pools of atoms are collected from real sources and composed into skeletons under structural constraints. A builder agent turns each skeleton into an environment, which is admitted only after its verifier passes a correctness check. Environments that yield no successful trajectory are diagnosed, and the result files an issue against an atom and waits for its fix, supplements a pool, or revises the builder; tasks that are merely hard are kept.
Figure 2: The four atom pools on one worked example (pay invoices over 10,000 only if approved). The brown links, numbered 1–4, are the four structural constraints of § 3.3 .
Figure 3: Structural recombination. A seed skeleton σ mined from a procedure is edited by replacing an intent ( i8→i12 ), replacing the rule or adding an atom; the candidate is kept only if constraints 1–4 of Figure 2 all hold, and duplicates are removed.
AutomationBench
MCP-Mark
BFCL v4
Model
Fin.
HR
Mkt.
Ops.
Sal.
Sup.
Avg
File
GitH.
Not.
Play.
Post.
Avg
Srch.
Mem.
MultT.
NonL.
Live
Rel.
Irr.
Avg
Open-weight
DeepSeek-V4.1-Flash
54.0
58.0
57.0
60.0
46.0
32.0
51.2
61.7
48.9
79.5
57.0
84.5
66.1
54.0
79.8
51.5
86.4
78.4
93.8
64.7
65.2
GLM-5.3-Flash
49.0
48.0
54.0
55.0
45.0
34.0
47.5
82.5
83.7
88.4
74.0
89.3
83.5
79.5
78.9
55.9
85.8
79.9
79.7
71.4
72.2
DeepSeek-V4-Pro ⋄
16.0
22.0
36.0
30.0
25.0
29.0
26.3
76.7
45.7
72.3
71.0
92.9
71.7
72.0
80.4
53.4
85.0
77.2
85.9
70.1
69.7
DeepSeek-V4-Flash ‡
30.0
32.0
35.0
47.0
23.0
27.0
32.3
75.8
68.5
56.3
42.0
86.9
65.4
76.0
72.5
52.6
87.0
78.1
89.1
43.9
66.4
Table 1: Main results. All numbers are measured by us: AutomationBench harness 1.0.6, api toolset, 50 steps, one rollout per task, 600 tasks (6 domains × 100); MCP-Mark, 127 tasks over five MCP servers; BFCL v4, all seven official categories plus the official weighted average. Our rows’ AutomationBench cells are means of one to four evaluations (Table 4 ), the baselines’ single evaluations. ⋄ Served through a commercial gateway that drops the model’s reasoning between tool calls, which lowers some models by several points (Appendix E ); not comparable with the other rows. MCP-Mark and BFCL v4 cells are means of four evaluations per model, whose standard deviations are in Table 5 . Rel. is computed over only 16 tasks and should be read as noise. Open-weight baselines are run at their highest available reasoning setting. † teacher that collected the SFT trajectories and the base for RL; \lx@sectionsign students; ‡ the model that ran every stage of the pipeline, including building the environments. Each LegoEnv row is one checkpoint evaluated on all three suites: Qwen3.5-27B trained on the round-1 + 3k corpus (5,229 trajectories), Qwen3.6-27B on the round-2 corpus (6,110), Qwen3.8-27B at GRPO step 200 (§ 4.1 , Appendix E ). The round-3 corpus (34.8%) scores higher on AutomationBench but lower on MCP-Mark and BFCL v4; GRPO step 200 (40.3%) scores 1.9 above step 240 on AutomationBench and 1.9 and 1.6 below it on MCP-Mark and BFCL v4 (Appendix E ).
Figure 4: (a) AutomationBench pass rate as the corpus grows. Qwen3.6-27B rises to 34.8% and Qwen3.5-27B to 26.3%. Bands are evolution rounds. (b) What each round adds; brown marks a value that grew, grey repeats the previous value. Operation kinds are counted under the final taxonomy and rule archetypes are the closed vocabulary introduced in round 2 (Appendix L ).
Environments
AB
vs. ours
MCP-Mark
BFCL v4
LegoEnv (round 2)
29.5
—
72.4 ± 1.9
68.2 ± 0.6
LegoEnv (round 1)
25.5
47:69
—
—
AWM ( Wang et al., 2026b )
21.5
30:76
63.8 ± 2.3
67.6 ± 0.6
EnvFactory ( Xu et al., 2026a )
20.4
33:81
63.8 ± 2.8
66.3 ± 0.8
EnvScaler ( Song et al., 2026 )
16.8
24:97
61.4 ± 1.3
63.8 ± 0.4
EnvCraft ( Zeng et al., 2026 )
15.8
27:100
59.1 ± 1.7
67.8 ± 0.7
Table 2: One teacher, student, recipe and budget of 2,229 trajectories; only the environments differ. AB : AutomationBench pass rate, mean of one to four evaluations (Table 4 ). vs. ours : per-task wins:losses against the round-2 row, first evaluation of each. MCP-Mark , BFCL v4 : mean ± s.d. of four evaluations (round-1 arm not evaluated). Environment counts, teacher pass rates, turns and token counts per arm are in Table 6 .
Figure 5: GRPO on Qwen3.8-27B with the task verifiers as a binary reward. Left: training reward, the fraction of 96 sampled episodes the verifier accepts. Right: AutomationBench pass rate at seven checkpoints against the 37.6% base; from step 40 on it stays in a 38.3–41.3% band while reward keeps climbing.
Atoms
AutomationBench
MCP-Mark
BFCL v4
env.
tokens
real
30.3 ± 0.7
54.9 ± 2.7
66.3 ± 0.4
859
7.6M
generated data
28.0 ± 0.9
51.2 ± 2.8
65.2 ± 0.4
759
7.3M
generated intents
27.3 ± 0.5
52.4 ± 2.5
65.9 ± 0.9
748
7.4M
Table 3: One atom family replaced by model-generated content, the rest fixed; Qwen3.6-27B trained on 885 trajectories per arm. Means ± standard deviation over four evaluations; one training run per arm. env. : distinct environments among the 885 trajectories; tokens : supervised tokens.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: The corpus. (a) Servers per environment; almost every environment spans two or more. (b) Use of each server, ranked; a dozen servers for spreadsheets, mail, chat and boards carry most of the load, as they do in real automation templates. (c) Rule archetypes and carriers over the 4,794 skeletons. (d) The twelve most frequent of the 110 operation kinds.
Base
Checkpoint
Dropped
Replayed
n
vs. base
Qwen3.5-27B
base
2.7
3.2
1
round 0 (1,261)
5.7
10.2
1
round 1 (2,229)
9.0
16.2
1
3,229
16.0
21.0
1
4,228
16.8
23.7
1
round 1 + 3k (5,229) ⋆
21.0
23.2
1
129:9
Appendix
Table 4: AutomationBench pass rate with the model’s reasoning dropped between tool calls (an earlier configuration of our gateway, one evaluation each) and with it replayed (the protocol of Tables 1 and 2 ); n is the number of replayed evaluations, with mean and standard deviation where n>1 . vs. base : tasks won and lost against the base under the replayed protocol, first evaluation of each; in the Table 2 block, tasks won and lost by our round-2 arm against that row. Checkpoints are named by their corpus (§ 4.1 ), with trajectory counts in parentheses; ⋆ marks the rows of Table 1 .
Model
MCP-Mark
BFCL v4
DeepSeek-V4.1-Flash
0.6
0.6
GLM-5.3-Flash
3.4
0.4
DeepSeek-V4-Pro
1.9
0.7
DeepSeek-V4-Flash
2.3
0.4
GLM-5.2
2.6
0.5
Kimi-K2.6
2.0
0.8
Appendix
Table 5: Standard deviation, in points, of the MCP-Mark pass rate and the BFCL v4 weighted average over the four evaluations behind each row of Table 1 .
Figure 7: Training reward by prompt group during GRPO, in bins of 20 steps: the 980 native environments, the 316 repaired environments in the same teacher band, and the 220 of the exploration set.
Arm
pass@8
env.
turns
tokens
loss
MCP
BFCL
LegoEnv (round 2)
—
1,173
13
43.3M
14.3M
72.4
68.2
AWM
0.59
556
6
40.4M
7.6M
63.8
67.6
EnvFactory
0.72
142
4
21.4M
5.5M
63.8
66.3
EnvScaler
0.36
47
8
41.1M
13.9M
61.4
63.8
EnvCraft
0.87
61
18
22.5M
9.1M
59.1
67.8
Qwen3.6-27B (no training)
—
—
—
—
—
42.5
63.0
Appendix
Table 6: Five of the arms of Table 2 , 2,229 trajectories each, over 2,138, 767, 818, 763 and 768 distinct tasks respectively. pass@8 : share of tasks the teacher solved at least once in eight attempts. env. : distinct worlds ( env_id ) behind the trajectories; several of our tasks share a world. turns : median assistant turns that carry loss (EnvFactory’s prefix turns, median 9 including them, carry none). tokens , loss : total tokens and tokens that carry loss. MCP , BFCL : MCP-Mark pass rate and BFCL v4 weighted average of the trained student, means of four evaluations as in Table 2 .
Measure
ours vs. AB
AB vs. itself
sharing a 13-gram
0 / 5,465 (0.0%)
351 / 600 (58.5%)
sharing an 8-gram
81 / 5,465 (1.5%) ∗
401 / 600 (66.8%)
TF-IDF max cosine, median
0.198
0.344
TF-IDF max cosine, max
0.442
0.680
embedding max cosine, median
0.719
0.779
embedding max cosine, max
0.930
0.965
Appendix
Table 7: Overlap between our environments and AutomationBench (AB). Rows 1–2: environments (left) or tasks (right) that share at least one word n -gram with some task; ∗ all six shared 8-grams are timestamps or “today is ⟨ date ⟩ ”, and the count is 0 when digits are not normalised. Rows 3–6: for each benchmark task, the maximum cosine to our requests (left) and to the other 599 tasks (right), under TF-IDF and under a dense embedding (Qwen3-Embedding-4B).
Stage (agent on the builder model)
tried
admitted
≥ 1/8
build, with rule atom
100
92
69/92
build, without rule atom
100
98
78/98
dispatch → build
30
30
14/20 ∗
onboard → build
10
10
9/9 ∗
instance repair
100
68
44/51 ∗
server repair (15 servers)
188 of 203 recovered, 0 of 90 broken
Appendix
Table 8: Each stage of LegoEnv run by an agent on DeepSeek-V4-Flash. Admitted : passes the verifier correctness check. ≥ 1/8 : environments with at least one verifier-accepted trajectory in eight teacher attempts. ∗ Teacher attempts were run only on the environments that also passed a review of the request (20 of 30, 9 of 10, 51 of 68).
source of each part
Work
schema
tool
data
intent
cross-app
verifier
RL
EnvScaler
✗
✗
✗
✗
✗
prog.
✓
AgentScaler
∼
∼
✗
✗
✗
prog.
✗
TOUCAN ‡
—
✓
✓
✗
✓
hybrid
✗
Agent-World
∼
✗
∼\lx@sectionsign
✗
✗
hybrid
✓
AWM
✗
✗
✗
✗
✗
hybrid
✓
Appendix
Table 9: Where the content of a synthesised agentic environment comes from. ✓ = derived from a real external artefact (source code, a dataset, a published template or procedure); ∼ = only names or interface descriptions are real, with behaviour or values written by an LLM; ✗ = written by an LLM. verifier : what decides whether a trajectory solved its task; prog. a program alone, hybrid an LLM judge takes part in the decision (filters that discard tasks or trajectories are not counted). ‡ runs against live services, so schema, tools and returned data are real and no initial state is fixed; C-World’s simulated mode is ∼ /✗. \lx@sectionsign rows mined from the web, complexified with generated rows. † additionally tested: every verifier must reject the initial world and accept the reference solution before its environment is admitted (§ 3.4 ).
Figure 10: Repair of the raneto server atom: every field that differs between the original and the repaired specification, except the free-text description of page_render ( - before, + after).
Layer
Root cause
Repairs
check-kind semantics (582)
verifier_wrong_selection_or_arithmetic
110
verifier_conflicts_visible_instruction
86
verifier_unstated_exact_value
81
verifier_format_normalization
63
verifier_equivalent_expression
42
verifier_overchecks_linked_text
37
Appendix
Table 10: The 33 root causes of the 997 repairs, by layer. 89 repairs also carry a secondary cause, which is not counted.
Step
Kind added
Implemented by
31 → 37
add_label
planka: add_card_label, create_label
create_metadata
paperless_ngx: tag_create
post_batch
ledgersmb: batch_post
record_status_change
akaunting: document_sent, …
37 → 57
reservation_approve
librebooking: reservationseries_approve
check_pause
healthchecks: check_pause
Appendix
Table 11: Growth of the operation-kind vocabulary, with four of the kinds added at each step. Steps 1 and 2 give the server and tools that implement the kind; step 3 gives the number of historical write steps classified into it. The steps are vocabulary snapshots at onboarding, not evolution rounds.
Table 12: Ten sources. Template, documentation and procedure rows give the rule atom (archetype, carrier) and the intents (kind on server) of the skeleton mined from the page; dataset rows give the dataset card’s domain and columns; repository rows give the server atom compiled from the code.