Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning

Siyuan Xu; Shiyang Li; Xin Liu; Tianyi Liu; Yixiao Li; Zhan Shi; Zixuan Zhang; Zilong Wang

cs.AIApr 10, 2026

Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning

Authors: Siyuan Xu, Shiyang Li, Xin Liu, Tianyi Liu, Yixiao Li, Zhan Shi, Zixuan Zhang, Zilong Wang,

Organizations: The Pennsylvania State University · Amazon · Georgia Institute of Technology

Abstract

Existing synthetic tool-use corpora are primarily designed for offline supervised fine-tuning, yet reinforcement learning (RL) requires executable environments that support reward-checkable online rollouts. We propose COVERT, a two-stage pipeline that first generates reliable base tool-use trajectories through self-evolving synthesis with multi-level validation, and then applies oracle-preserving augmentations that systematically increase environmental complexity. These augmentations introduce distractor tools, indirect or ambiguous user queries, and noisy, multi-format, or erroneous tool outputs, while strictly preserving oracle tool calls and final answers as ground truth. This design enables automatic reward computation via reference matching for standard cases and lightweight judge-assisted verification for special behaviors such as error detection, supporting RL optimization of tool-calling policies. On Qwen2.5-Instruct-14B, COVERT-RL improves overall accuracy on BFCL v3 from 56.5 to 59.9 and on ACEBench from 53.0 to 59.3, with minimal regressions on general-ability benchmarks; when stacked on SFT, it further reaches 62.1 and 61.8, confirming additive gains. These results suggest that oracle-preserving synthetic environments offer a practical RL refinement stage, complementary to SFT, for improving tool-use robustness under ambiguity and unreliable tool feedback.

Figures & tables

Figure 1: Overview of the proposed COVERT pipeline. Stage 1 generates reliable base tool-use trajectories with simple queries and single tools via self-evolution and validation. Stage 2 applies oracle-preserving augmentations to produce complex multi-tool scenarios with noisy and multi-format outputs, while preserving oracle tool behaviors for reward computation during RL training.

Figure 2: Reliable tool-use trajectory generation pipeline (Stage I) with diverse prompt construction, LLM-based generation, multi-level trajectory checking, and self-evolving seed pool expansion.

Figure 3: Illustration of oracle-preserving augmentation. Starting from a simple and reliable base tool-calling trajectory (left), we apply controlled perturbations to the user query and tool environment (right), including indirect entity references, distractor tools, and injected tool-call failures.

Capability	Modified	Oracle preserved	Knobs	Verifier
Distractor tools	$\mathcal{T}$	$c^{},a^{}$	#distractors, overlap	Exact
Query rewriting	$u$	$c^{},a^{}$	indirection level	Exact
Multi-fmt parsing	$o^{*}$ format	$c^{},a^{}$	format family	Exact
Noisy outputs	$o^{*}$ content	$c^{},a^{}$	noise length	Exact
Erroneous outputs	$o^{*}$	$c^{*}$ , label $k$	error mode	Judge
Problematic queries	$u$	$c^{*}$ , label $k$	failure type	Judge

Table 1: Augmentation families in Stage II. “Modified”: perturbed component; “Oracle preserved”: fields for reward verification; “Knobs”: controllable parameters; “Verifier”: exact or judge-assisted.

	Overall	Non-Live AST	Live	Multi Turn	Relevance Detection	Irrelevance Detection
Qwen2.5-Instruct-7B	54.1	84.2	72.2	10.0	72.2	67.9
with TOUCAN-SFT	57.0	78.5	75.0	15.6	66.7	75.1
with COVERT-RL (ours)	57.2	85.8	76.4	8.8	72.2	80.1
Qwen2.5-Instruct-14B	56.5	83.5	73.6	15.6	83.3	69.3
with TOUCAN-SFT	59.5	85.3	76.4	18.3	66.7	76.3
with COVERT-RL (ours)	59.9	86.8	78.1	14.9	72.2	81.2

Table 2: BFCL v3 accuracy based on Qwen2.5-Instruct-7B/14B, with TOUCAN-SFT or COVERT-RL. All numbers are accuracies (%). Bold for the best performance.

	Overall Acc	Atom	Normal	Special	Agent
Qwen2.5-Instruct-7B	42.6	60.0	52.6	36.5	10.2
with TOUCAN-SFT	41.8	61.7	53.2	26.7	17.5
with COVERT-RL (ours)	51.2	73.3	59.8	50.7	9.2
Qwen2.5-Instruct-14B	53.0	61.3	55.0	68.6	14.0
with TOUCAN-SFT	48.8	76.1	61.9	27.3	24.0
with COVERT-RL (ours)	59.3	76.3	66.5	70.4	13.0

Table 3: ACEBench accuracy on Qwen2.5-Instruct-7B/14B. All numbers are accuracies (%).

	LiveCodeBench -v5	AIME 2024	AIME 2025	Math 500	IFEval	GPQA- Diamond	MMLU -Pro	GSM8K
Qwen2.5-Instruct-7B	26.1	13.3	6.7	76.6	57.7	33.8	43.1	80.7
with TOUCAN-SFT	27.3	10.0	6.7	72.6	53.1	25.8	38.3	76.0
with COVERT-RL (ours)	25.8	13.3	6.7	76.4	58.4	35.4	43.1	80.8
Qwen2.5-Instruct-14B	36.8	20.0	13.3	80.6	62.3	44.4	48.7	82.3
with TOUCAN-SFT	35.8	10.0	6.7	79.4	55.1	38.4	42.0	80.2
with COVERT-RL (ours)	35.9	20.0	13.3	80.4	63.0	43.4	48.2	82.4

Table 4: Performance on general-ability benchmarks: LiveCodeBench-v5, AIME 2024, AIME 2025, Math500, IFEval (Prompt-Strict), GPQA-Diamond, MMLU-Pro, and GSM8K.

	BFCL v3	ACEBench
Qwen2.5-Instruct-7B	54.1	42.6
with COVERT-RL (ours)	57.2	51.2
with TOUCAN-SFT	57.0	41.8
with TOUCAN-SFT + COVERT-RL (ours)	59.1	53.9
Qwen2.5-Instruct-14B	56.5	53.0
with COVERT-RL (ours)	59.9	59.3

Table 5: Summary of overall accuracy on BFCL v3 and ACEBench for all Qwen2.5-Instruct-7B/14B variants.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: System prompt used for evaluation.

	Rollout-time reward	Perturbation control	Oracle- preserving
APIGen	×	×	×
ToolACE	×	×	×
TOUCAN	×	×	×
ToolRL	✓	×	×
ToolZero	✓	×	×
COVERT	✓	✓	✓

Appendix

Table 6: Positioning of COVERT relative to representative prior work along three design axes.

	Overall	Non-Live AST	Live	Multi Turn	Relevance Detection	Irrelevance Detection
Qwen2.5-Instruct-7B	54.1	84.2	72.2	10.0	72.2	67.9
with COVERT-RL (ours)	57.2	85.8	76.4	8.8	72.2	80.1
with TOUCAN-SFT	57.0	78.5	75.0	15.6	66.7	75.1
with TOUCAN-SFT + COVERT-RL (ours)	59.1	84.2	76.5	15.0	72.2	81.6
Qwen2.5-Instruct-14B	56.5	83.5	73.6	15.6	83.3	69.3
with COVERT-RL (ours)	59.9	86.8	78.1	14.9	72.2	81.2

Appendix

Table 7: Full BFCL v3 accuracy for all Qwen2.5-Instruct-7B/14B variants (extends Table 5 ). All numbers are accuracies (%).

	Overall Acc	Atom	Normal	Special	Agent
Qwen2.5-Instruct-7B	42.6	60.0	52.6	36.5	10.2
with COVERT-RL (ours)	51.2	73.3	59.8	50.7	9.2
with TOUCAN-SFT	41.8	61.7	53.2	26.7	17.5
with TOUCAN-SFT + COVERT-RL (ours)	53.9	78.7	62.6	49.3	17.0
Qwen2.5-Instruct-14B	53.0	61.3	55.0	68.6	14.0
with COVERT-RL (ours)	59.3	76.3	66.5	70.4	13.0

Appendix

Table 8: Full ACEBench accuracy for all Qwen2.5-Instruct-7B/14B variants (extends Table 5 ). All numbers are accuracies (%).

Figure 5: Training reward curves for COVERT-RL initialized from Qwen2.5-Instruct (blue) and from TOUCAN-SFT (orange), at 7B (left) and 14B (right) scales.

Panel of Figure 6 — Figure 6: Examples 1: Raw tool-calling data examples of layered symbolic reasoning.

Panel of Figure 7 — Figure 7: Examples 2: Raw tool-calling data example of parallel tool calling.

Panel of Figure 8 — Figure 8: Examples 3: Raw tool-calling data example of multi-turn tool calling.

Panel of Figure 9 — Figure 9: Raw case-study conversations (Case 1) comparing base vs. COVERT-RL models.

Panel of Figure 10 — Figure 10: Raw case-study conversations (Case 2) comparing base vs. COVERT-RL models.

Tool Use

Tool use studies how AI agents select, invoke, and combine external tools to solve tasks beyond their internal knowledge.

See all 91 papers

EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL
May 18, 2026Minrui Xu, Zilin Wang, Mengyi DENG +12Tool-Augmented Language Model Agents Agentic RL
LARK, HKUST (GZ) · University of Cambridge · UCL +2
ToolLoop: Closed-Loop Tool-Use Data Synthesis via Decomposed Generation and Dynamic Self-Feedback
Sep 8, 2026Min Zeng, Yuzhou Liu, Zhenyu Cao +5Tool-Augmented Language Model Agents Synthetic Data Generation
vivo AI Lab
CoVe: Training Interactive Tool-Use Agents via Constraint-Guided Verification
Mar 2, 2026Jinpeng Chen, Cheng Gong, Hanbo Li +9Tool-Using Agents LLM Agent Training
Huawei Research · Independent Researcher

May 18, 2026J/K move · Enter open · S save

EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL

Minrui Xu, Zilin Wang, Mengyi DENG, Zhiwei Li+11

LARK, HKUST (GZ) · University of Cambridge · UCL+2

Tool-Augmented Language Model Agents Agentic RL LLM Agent Trajectory Synthesis LLM Agents LLM Agent Training RL for Tool Use

Open paper AskPDF arXiv

Equipping LLMs with tool-use capabilities via Agentic Reinforcement Learning (Agentic RL) is bottlenecked by two challenges: the lack of scalable, robust execution environments and the scarcity of realistic training data that captures implicit human reasoning. Existing approaches depend on costly real-world APIs, hallucination-prone LLM simulators, or synthetic environments that are often single-turn or depend on pre-collected documents. Moreover, synthetic trajectories are frequently over-specified, resembling instruction sequences rather than natural human intents, reducing their effectiveness for RL training. We introduce EnvFactory, a fully automated framework that addresses both challenges. EnvFactory autonomously explores and verifies stateful, executable tool environments from authentic resources, and synthesizes natural multi-turn trajectories through topology-aware sampling and calibrated refinement, producing grounded queries with implicit intents. Using only 85 verified environments across 7 domains, EnvFactory generates 2,575 SFT and RL trajectories. Despite using significantly fewer environments than prior work, which are often 5 times more, EnvFactory achieves superior training efficiency and downstream performance, improving Qwen3-series models by up to +15% on BFCLv3, +8.6% on MCP-Atlas, and +6% on conversational benchmarks including

τ^2

-Bench and VitaBench. By fully automating both environment construction and trajectory synthesis, EnvFactory provides a scalable, extensible, and robust foundation for Agentic RL.

Abstract

Tool Use

Explore similar work

EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL

ToolLoop: Closed-Loop Tool-Use Data Synthesis via Decomposed Generation and Dynamic Self-Feedback

CoVe: Training Interactive Tool-Use Agents via Constraint-Guided Verification