Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis

Andrew Szot; Samira Abnar; Omar Attia; Abbas Kazerouni; Michael Kirchhof; Cheng-Yu Hsieh; Raviteja Vemulapalli; Alexander Toshev

cs.LGOct 27, 2026

Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis

Authors: Andrew Szot, Samira Abnar, Omar Attia, Abbas Kazerouni, Michael Kirchhof, Cheng-Yu Hsieh, Raviteja Vemulapalli, Alexander Toshev

Organizations: Apple

Abstract

We explore how to build performant large language model (LLM) tool-use agents that complete complex, multi-step tasks. Specifically, we study how data composition in synthetically generated tool-use environments affects the training of such agents. To do so, we propose the SynthAPI pipeline, which explicitly controls the diversity of generated environments along the axes of tools, tasks, and the databases on which the tools operate. By systematically varying SynthAPI's generation controls in 458,752 tasks across 849 trained models of four different sizes, we derive a scaling law that predicts downstream tool-use agent generalization based on the scale and composition of the synthesized environments. These predictable relationships reveal several lessons for scaling tool-use agent data. As data budgets grow, scaling tool and database diversity becomes more effective than increasing the number of tasks, and increasing data complexity yields larger gains. Scaling according to these findings yields agents that outperform models trained on data from prior work in large-scale tool-use agent and synthetic environment generation across four evaluation benchmarks.

Figures & tables

Figure 1: Overview of SynthAPI. On the left, we show the tool-use agent tasks we focus on, where given an instruction, an agent must reason and execute tool calls to complete the task. In the middle, the SynthAPI pipeline generates synthetic tool-use agent environments at scale. SynthAPI generates this data while controlling the diversity of the tools, tasks, and starting database states. These controls allow deriving the scaling laws on the right (presented in detail in Figure 3 ).

Panel of Figure 2 — Figure 2: Effect of app-to-task allocation, database breadth, and database depth on the 9B student. The x-axis shows the number of tasks $N$ , and the y-axis shows aggregate downstream performance. (a) Curves correspond to different allocations of $A$ and $T$ at a fixed total task count $N=AT$ ; at a fixed value of $N$ , the highest curve identifies the best allocation. (b) Curves correspond to different numbers of database instantiations $S$ per task. (c) Curves compare minimal task state ( $D=1$ ), moderate distractor data ( $D=2$ ), and a larger distractor set ( $D=4$ ).

Figure 3: Visualization of the proposed data composition dependent scaling law and its extrapolation to higher compute budgets. The x-axis reports the training FLOPs of the SFT training and the y-axis is the normalized score indicating the combined generalization performance across various benchmarks. The points are normalized scores for models trained on an optimal setting, meaning the data setting that performs the best at that FLOP budget, representing the FLOP/performance allocation frontier. The lines are a visualization of the per parameter count scaling model, fit without the held-out runs in the top 25% of training FLOPs.

	AppWorld			ASTRA-Bench			BFCL			SynthAPI-Holdout
Approach	4B	9B	27B	4B	9B	27B	4B	9B	27B	4B	9B	27B
Zero-shot	14.7	26.4	61.5	57.4	71.8	80.2	27.3	39.3	54.4	0.3	1.5	20.0
Nemotron	17.5	20.8	-	54.4	67.7	-	15.9	39.7	-	1.0	0.5	-
SynthAPI-SubOpt	38.4	53.0	73.8	71.1	74.2	77.3	44.9	47.5	59.6	51.3	56.0	64.0
SynthAPI-Opt	45.0	56.2	74.1	72.3	75.5	79.3	48.8	49.9	61.5	56.0	63.0	68.0
SynthAPI-Big	66.4	77.0	81.5	74.2	76.8	82.3	51.1	53.6	65.0	53.0	64.0	69.8

Table 1: Unnormalized success rates on downstream tool-use agent benchmarks. AppWorld reports the Test-C split (see Table 4 for both splits). SynthAPI rows are trained with SynthAPI data; “-Opt” uses the optimal data composition from our scaling analysis, and “-Big” is a scaled-up version.

BFCL	4B	8B
Zero-shot	33.5	41.3
Agent World Model	40.8	42.3
EnvScaler	45.0	51.9
EnvFactory	48.5	49.0
SynthAPI-Big	50.5	55.6

Table 2: Comparison to per-benchmark baselines from prior work on BFCL (left) and AppWorld (right) showing success rates. BFCL results are on Qwen3 and AppWorld results are on Qwen3.5.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: Statistics of the generated apps and their APIs. The top row shows per-app statistics and the bottom row shows per-API and per-parameter statistics. The sample used here is the complete set of 64 apps used in all the experiments.

Panel of Figure 5 — Figure 5: Statistics of the GLM-5.2 expert trajectories in the SFT dataset, both computed on a subsample of the overall set of training trajectories. (a) Distribution of the number of tokens per expert action. (b) Distribution of the number of unique function calls per trajectory.

Model size	$\mathbf{c_{0}}$	$\mathbf{c_{A}}$	$\mathbf{b_{0}}$	$\mathbf{b_{T}}$	$\bm{\nu}$
2B	$-2.46$	0.181	0.54	0.595	20.00
4B	$-0.57$	0.247	$-0.66$	0.401	1.67
9B	$-0.25$	0.236	$-0.41$	0.495	1.46
27B	0.20	0.392	$-9.96$	1.475	0.05

Appendix

Table 3: Fitted scaling-law coefficients of Equation 4.2 by model size.

Panel of Figure 6 — Figure 6: Allocation experiment for the 2B model.

Panel of Figure 7 — Figure 7: Allocation experiment for the 4B model.

Panel of Figure 8 — Figure 8: Allocation experiment for the 9B model.

Panel of Figure 9 — Figure 9: Allocation experiment for the 27B model.

	Qwen3.5-4B		Qwen3.5-9B		Qwen3.5-27B
Approach	Test-N	Test-C	Test-N	Test-C	Test-N	Test-C
Zero-shot	47.5	14.7	45.4	26.4	81.8	61.5
SimSAT	66.8	58.8	74.5	65.4	-	-
Nemotron	26.0	17.5	31.9	20.8	-	-
ToolACE	20.2	10.9	18.6	12.4	-	-
ToolAlpaca	25.7	17.3	24.4	15.7	-	-

Appendix

Table 4: Full AppWorld results on the Test-Normal (Test-N) and Test-Challenge (Test-C) splits. The SimSAT, ToolACE, and ToolAlpaca results are from Lee et al. (2026) .

Model	Approach	Ref.	Func.	Info.	Overall
Qwen3.5-4B	Zero-shot	55.5	33.5	60.3	57.4
	Nemotron	50.1	27.5	58.6	54.4
	SynthAPI-SubOpt	69.7	48.1	74.4	71.1
	SynthAPI-Opt	71.4	51.0	74.8	72.3
	SynthAPI-Big	73.7	53.2	76.3	74.2
Qwen3.5-9B	Zero-shot	69.5	50.3	74.2	71.8

Appendix

Table 5: Full ASTRA-Bench results broken down by task category. Note that in ASTRA-Bench, a single task can count for multiple categories, thus the overall score is not the average of the individual columns.

Model	Approach	Shop.	Msg.	Fin.	Music	Avg.
Qwen3.5-4B	Zero-shot	0.0	0.0	1.0	0.0	0.3
	Nemotron	0.0	0.0	1.0	3.0	1.0
	SynthAPI-SubOpt	50.0	51.0	60.0	44.0	51.3
	SynthAPI-Opt	64.0	74.0	36.0	50.0	56.0
	SynthAPI-Big	60.0	72.0	32.0	48.0	53.0
Qwen3.5-9B	Zero-shot	1.0	1.0	3.0	1.0	1.5

Appendix

Table 6: Full SynthAPI-Holdout results broken down by app category.

Figure 10: Comparing the allocation between apps and tasks per-app on another data synthesis pipeline SimSAT ( Lee et al., 2026 ) .

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 289 papers

DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use
Mar 10, 2026Aili Chen, Chi Zhang, Junteng Liu +11Tool-Augmented Language Model Agents LLM Tool Use
Fudan University · MiniMax · Independent
EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL
May 18, 2026Minrui Xu, Zilin Wang, Mengyi DENG +12Tool-Augmented Language Model Agents Agentic RL
LARK, HKUST (GZ) · University of Cambridge · UCL +2
How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies
Oct 8, 2026Bowei He, Yankai Chen, Xiaokun Zhang +4Agentic RL Synthetic Data Generation for LLM Agents
MBZUAI · McGill University · CityUHK +1

Mar 10, 2026J/K move · Enter open · S save

DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use

Aili Chen, Chi Zhang, Junteng Liu, Jiangjie Chen+10

Fudan University · MiniMax · Independent

Tool-Augmented Language Model Agents LLM Tool Use LLM Agent Training Synthetic Data Generation for Language Models Cross-Domain Generalization

Open paper AskPDF arXiv

Recent work synthesizes agentic tasks for post-training tool-using LLMs, yet robust generalization under shifts in tasks and toolsets remains an open challenge. We trace this brittleness to insufficient diversity in synthesized tasks. Scaling diversity is difficult because training requires tasks to remain executable and verifiable, while generalization demands coverage of diverse tool types, toolset combinations, and heterogeneous tool-use patterns. We propose DIVE, an evidence-driven recipe that inverts synthesis order, executing diverse, real-world tools first and reverse-deriving tasks strictly entailed by the resulting traces, thereby providing grounding by construction. DIVE scales structural diversity along two controllable axes, tool-pool coverage and per-task toolset variety, and an Evidence Collection--Task Derivation loop further induces rich multi-step tool-use patterns across 373 tools in five domains. Training Qwen3-8B on DIVE data (48k SFT + 3.2k RL) improves by +22 average points across 9 OOD benchmarks and outperforms the strongest 8B baseline by +68. Remarkably, controlled scaling analysis reveals that diversity scaling consistently outperforms quantity scaling for OOD generalization, even with 4x less data.

Abstract

Tool-Using Agents

Explore similar work

DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use

EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL

How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies