Joint Optimization of Tool Creation and Use for Large Language Model Agents

Zhi Rui Tam; Chieh-Yen Lin; Yun-Nung Chen; Shao-Hua Sun; Hung-yi Lee

cs.AIAug 25, 2026

Joint Optimization of Tool Creation and Use for Large Language Model Agents

Authors: Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, Shao-Hua Sun, Hung-yi Lee

Organizations: Appier AI Research · National Taiwan University

Abstract

Tool-augmented language models are bounded by the APIs humans bothered to write; existing tool-creation systems patch this by prompting a frozen LLM at inference time, leaving the model that writes a tool decoupled from the one that uses it, with no signal that the schemas it produces are schemas it can invoke. We propose SMITH (Schema-grounded Multi-task Iterative Tool Honing), a reinforcement learning framework that jointly trains tool creation and tool use inside a single policy. Each rollout is either a build task (write a tool from a few examples) or a use task (invoke a pooled tool on a held-out question). Three separate reward axes catch schema, code, and outcome failures independently, so each failure mode contributes its own gradient. A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.8 macro-average accuracy on held-out tasks, the best across all evaluated methods and ahead of an untrained 30B-A3B tool-writer. It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline), without any visual or tabular training data. Tools written by our 4B models also lifted the performance of LFM-2.5-350M and Qwen3-30B-A3B under same reasoning tasks.

Figures & tables

Figure 1: SMITH jointly trains tool creation and tool use in a single shared policy. The build task rewards schema correctness and execution accuracy; the use task rewards answer correctness using only the schema. Because the same model that writes a tool must also invoke it, ambiguous or broken schemas are penalized directly—a feedback loop that prompting-only approaches cannot provide.

	Seen	Unseen RG
Method	Avg	Logic	Game	Algebra	Arith	Algo	Avg	I/O
Standard CoT	58.0	49.9	60.3	56.8	62.7	48.6	55.7	173 / 3,206
LATM* [ 2 ]	77.6	53.9	55.5	38.6	53.0	90.2	58.3	607 / 174
LATM* - Qwen3-30B-A3B	74.0	68.7	64.2	97.3	56.5	84.0	74.1	659 / 405
CRAFT [ 35 ]	74.1 $\pm$ 0.7	27.4 $\pm$ 1.3	89.5 $\pm$ 0.0	94.2 $\pm$ 1.2	76.6 $\pm$ 1.1	95.0 $\pm$ 0.0	76.5 $\pm$ 0.4	1,226 / 418
Trove [ 31 ]	52.6 $\pm$ 0.4	60.7 $\pm$ 2.4	10.6 $\pm$ 1.3	51.2 $\pm$ 3.5	59.8 $\pm$ 0.8	97.0 $\pm$ 0.0	55.9 $\pm$ 0.6	347 / 575

Table 1: Reasoning-Gym results (Qwen3-4B-Instruct); CRAFT, TroVE, KTCE, ReTool, LATM-distill, and SMITH report mean

\pm

std over seeded re-evaluations. SMITH leads held-out generalization (RG Unseen 79.9) over distilled models (ReTool 63.2, LATM-distill 65.8) and every scaffolding baseline, using 32

\times

fewer output tokens than standard CoT. Fixing baseline grading bugs raises TroVE’s RG (Unseen) from 40.6 to 55.9 and KTCE’s from 48.7 to 65.1. RG (Seen/Unseen) : macro-average over training/held-out tasks. ∗ LATM prompt rewritten for schema clarity. I/O: input / output tokens.

Method	RG (Seen)	RG (Unseen)	TabMWP	GQA
LFM2.5-350M (no tool)	14.2	11.6	0.0	0.1
+ Qwen3-4B-Instruct (tool writer)	36.8	23.4	4.2	0.1
+ Qwen3-30B-A3B (tool writer)	10.4	41.5	4.1	0.1
+ RL on LFM2.5-350M	39.1	30.2	4.2	0.1
+ RL 4B (Ours)	38.9	42.9	4.1	0.1

Table 2: Tools from SMITH’s RL-trained 4B writer enable a 350M model (LFM2.5) to match a 30B tool writer on held-out tasks (42.9 vs. 41.5 RG Unseen), showing that SMITH’s tools encode genuinely generalizable solutions that transfer across model families. First row is the no-tool baseline. RG (Seen)/(Unseen): macro-average accuracy (%). Best per column in bold .

Method	Held-out RG (10 tasks)	Seen RG (13 tasks)	TabMWP-Hard	GQA	Overall
LATM ∗ (30B writes its own tool)	71.9	78.1	0.7	20.2	70.2
SMITH-4B to 30B consumer	74.5	84.6	38.8	30.2	76.6

Table 3: Tools written by SMITH’s RL-trained 4B writer lift a Qwen3-30B-A3B-Instruct consumer above the same 30B model writing tools for itself (LATM ∗ ) on every group, with the largest gain on TabMWP-Hard (0.7

\to

38.8), showing the 4B writer is a viable drop-in tool provider even when a much larger model is available to consume its tools. Overall : task-count-weighted average over the 25 shared tasks (10 held-out RG + 13 seen RG + TabMWP-Hard + GQA). Best per column in bold .

	Baselines						Distilled
Task	CoT	LATM	LATM* (30B)	CRAFT	TroVE	KTCE	ReTool	LATM (distill)	SMITH
TabMWP-Hard	7.2	19.7	0.7	30.0	36.4	27.2	3.0	7.1	40.4
GQA	11.5	35.0	29.8	21.9	21.4	0.0	26.1	56.0	42.6

Table 4: OOD generalization (Qwen3-4B-Instruct). SMITH leads on TabMWP-Hard (40.4 vs. next best TroVE 36.4) and ranks second on GQA (42.6), making it the only 4B method without oracle distillation to top either column. Bold = best, underline = second-best.

Method	RG (Seen)	RG (Unseen)	TabMWP	GQA
Qwen3-8B (baseline)	72.6	72.2	42.4	17.3
SMITH: Qwen3-8B	79.4	81.7	56.7	28.7
SMITH: Self-Judge	74.7	85.9	54.5	16.3
Granite-3.3-8B (baseline)	31.2	22.0	3.9	7.8
SMITH: Granite-3.3-8B	39.1	28.5	4.5	11.7

Table 5: SMITH improves over the base model on every backbone tested. On Qwen3-8B it raises held-out RG from 72.2 to 81.7 and TabMWP-Hard from 42.4 to 56.7; Granite-3.3-8B follows the same trend despite a weaker starting point, confirming the training signal is not model-family specific. RG (Seen/Unseen) : macro-average over training/held-out tasks. Best per column in bold .

	Qwen3-4B-Instruct		Qwen3-8B		Granite-3.3-8B
Metric	Base	SMITH	Base	SMITH	Base	SMITH
BFCLv4	45.1	48.6	43.3	55.8	36.3	38.7

Table 6: SMITH improves external tool-calling (BFCL v4, no-web) across all backbones, with the largest gain on Qwen3-8B (43.3

\to

55.8), despite never seeing BFCL schemas, multi-turn traces, or judges during RL training. Bold = best, underline = second-best.

Method	$\boldsymbol{\pi^{\mathrm{eval}}}$	Tool Use	RG (S)	RG (U)	TabMWP	GQA
Qwen3-4B-Instruct	–	–	61.85	47.01	19.70	20.86
Tool Create	30B-A3B	No	77.43	59.39	15.60	37.01
Tool Create	4B	No	73.88	68.80	17.70	35.82
Decoupled Create/Use	30B-A3B	Yes	76.44	58.93	13.90	35.04
SMITH : No Sync	4B	Yes	80.29	66.89	25.81	24.58
SMITH : No LLM Judge	self	Yes	82.57	67.81	18.30	42.63

Table 7: Reward-structure ablation (Qwen3-4B-Instruct).

\pi^{\mathrm{eval}}

: model that invokes the generated tool at eval ( 30B-A3B / 4B : separate model; self : same RL’d policy). Tool Use : whether

\pi^{\mathrm{eval}}

calls the tool rather than only writing it. Decoupled build/use does not beat single-model Tool Create on RG (Unseen) (

58.9

vs.

68.8

), so joint training, not specialization, drives the gain; only the full SMITH objective wins on aggregate , with individual benchmarks split between K=1 (TabMWP) and No LLM Judge (GQA). Best per column bold , second-best underlined . RG(S)/(U): Seen/Unseen.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Step	env_reward	eval_reward	step1_fail	fn_mismatch	param_mismatch	judge_reward
1	1.363	0.386	34.9%	23.4%	11.5%	0.460
2	1.809	0.466	10.4%	8.3%	2.1%	0.614
3	2.026	0.557	2.1%	1.0%	1.0%	0.670
4	2.043	0.684	9.4%	5.2%	3.6%	0.710
5	2.123	0.662	2.6%	0.0%	2.6%	0.722
LoRA sync applied after step 5

Appendix

Table 8: Key build-task metrics per optimizer step. The LoRA sync that loads eval_lora_step5 into the evaluator server fires after step 5’s metrics are logged. Steps 6–7 are the first rollouts with the synced evaluator.

Component	Condition	Penalty
Structural verifier	fn name absent	$r=0$ , trajectory terminates
LLM judge	fn name absent	$r\times 0.5$ (partial credit)

Appendix

Table 9: Penalty for a schema that names a function absent from the code.

#	Principle	Implication
1	Hard failures must be hard everywhere.	If the verifier zeros a trajectory, the judge must also return zero for that condition. Never apply partial credit to a fatal structural error.
2	Align the judge prompt to verifier constraints.	Every binary constraint enforced by the verifier should appear verbatim in the judge prompt, with a scored negative example.
3	Log structural sub-metrics as first-class signals.	Track each constraint-violation rate independently. Set alerts on violation rates, not only on aggregate reward.
4	Gate the judge signal on verifier pass.	Apply $r_{j}=0$ whenever the verifier assigns $r=0$ to prevent gradient leaks from judge partial credit.
5	Structural constraints belong in the verifier, not the judge.	RL policies transfer styles associated with high reward. If a structural property can be enforced exactly by a rule, enforce it as a hard verifier check rather than as a soft judge preference.

Appendix

Table 10: Design principles for combining a structural verifier with an LLM-as-judge reward signal in RL training for code generation.

Task	Category	Induction band (easy)	Evaluation band (hard)	Key difficulty axis
Bitwise arithmetic	Arithmetic	Expression depth 2	Expression depth 4–5	Nesting depth of the bit-operation expression tree
Cryptarithmetic	Arithmetic	$\leq\!8$ unique letters (easy/medium puzzles)	$\geq\!9$ unique letters, including 10-letter puzzles such as FORTY+TEN+TEN=SIXTY	Number of unique letters (exponential search space growth)
Bit counting	Algorithms	Integer values $1$ – $10^{6}$	Integer values $10^{7}$ – $10^{8}$	Magnitude of the input integer
LCM	Algorithms	2 numbers, values 1–50	3–4 numbers, values 100–500	Count and magnitude of operands
GCD	Algorithms	2 numbers, values 1–500	3–4 numbers, values 1,000–5,000	Count and magnitude of operands
Base conversion	Algorithms	Bases 2–10, values 1–500	Bases 2–16, values 2,000–5,000	Target base range and magnitude of the number

Appendix

Table 11: Induction context and evaluation difficulty bands for all 13 SMITH training tasks. Induction band defines the difficulty of examples shown in the

N{=}4

build-task context. Evaluation band defines the difficulty of instances used to compute

r^{\mathrm{eval}}

. Key difficulty axis is the parameter that grows from easy to hard.

Hyperparameter	Value
Base model	Qwen3-4B-Instruct
Training dataset	JoeYing/ReTool-SFT
Adapter	QLoRA (4-bit NF4, double quantisation)
LoRA rank $r$	64
LoRA $\alpha$	128
LoRA dropout	0.05

Appendix

Table 12: ReTool fine-tuning hyperparameters.

Hyperparameter	Value
Base model	Qwen3-4B-Instruct
Adapter	None (full fine-tuning)
Epochs	4
Sequence length	16,000 tokens
Gradient accumulation	18
Micro-batch size	2

Appendix

Table 13: LATM (distill GPT-4.1) fine-tuning hyperparameters.

System	Tool creation timing	Weights changed?	Tools survive across problems?
KTCE	Fully offline (before any test problem)	No	Yes (fixed toolset)
CRAFT	Fully offline (before any test problem)	No	Yes (fixed library)
TroVE	Online / streaming (during test inference)	No	Yes (library grows sequentially)
SMITH	Trained offline via RL	Yes	Yes (quality-filtered pool)

Appendix

Table 14: Tool-creation timing, model modification, and cross-problem reuse for each system.

Dimension	KTCE	CRAFT	TroVE	SMITH
Tool creation trigger	Per knowledge-subfield cluster of training data	Per training problem (diversity-sampled across epochs)	Per test problem that yields a novel, executable function	RL training rollout on build tasks
Tool structure	Python function + name / docstring / experience_pool	Python function + docstring + 4 SimCSE embedding vectors	Python function + docstring + frequency counter	Python function + OpenAI JSON schema
Tool verification	Execution + majority-vote during creation; loss-tracked during optimization	Execution + answer correctness at both specific and abstract stages	Implicit: tool enters library only if full solution executes correctly and wins 3-way selection	Execution accuracy as RL reward; format consistency as separate reward axis
Optimization / refinement	Explicit 5-iteration evolutionary loop: evaluate $\to$ compute loss $\to$ LLM delete / modify / add $\to$ rollback if worse	None (one-shot: create, validate, deduplicate)	Implicit frequency-based forgetting: low-reuse tools pruned every 500 examples	RL training is the optimization loop; gradient updates improve the tool-writing policy itself
Retrieval mechanism	Two-stage: (1) subfield lookup from metadata; (2) LLM reads numbered list and selects tools	Multi-view SimCSE similarity across 4 views (name, explanation, docstring, original question)	None; entire current library injected into the import-mode prompt, library size bounded by trimming	None at test time; model generates the required tool directly (trained to do so)
Tool reuse across problems	Yes (fixed toolset shared across all test problems)	Yes (fixed library shared across all test problems)	Yes (tools created for problem $i$ available for problem $i{+}1$ onward)	Yes (shared execution-verified pool)

Appendix

Table 15: Implementation-level comparison of agentic tool-creation designs.

Task	Acc. (%)	N	Real LLM calls	Stub tool	Failure mode
ab	0.0	210	64 / 210	Yes	FM2 + FM3
base_conversion	0.0	210	0 / 210	Yes	FM2 + FM3
bitwise_arithmetic	100.0	280	0 / 280	N/A	FM1 (solver)
caesar_cipher	82.4	210	0 / 210	No	FM1 (solver)
calendar_arithmetic	66.7	198	198 / 198	Yes	FM2
chinese_theorem	100.0	100	0 / 100	No	FM1 (solver)

Appendix

Table 16: Per-task KTCE diagnostic breakdown (Qwen3-4B-Instruct-2507). Real LLM calls counts entries with completion_tokens

>0

in the per-question inference output files. Stub tool indicates the generated tool body unconditionally returns an empty string.

Task	N	v1	v2 (bundled)	v3 (punct.)	v4 (example)
bitwise_arithmetic a	280	55.0	27.5 ∗	40.4 ∗	45.4 ∗
caesar_cipher	196	59.2	excl.	49.5 ∗	51.5 ∗
chinese_theorem	100	96.0	100.0	96.0	96.0
count_bits	210	100.0	100.0	100.0	100.0
isomorphic_string	350	100.0	100.0	100.0	100.0
knights_knaves	210	51.0	50.5	51.0	52.9

Appendix

Table 17: TroVE prompt-tuning ablation (Qwen3-4B-Instruct-2507, frozen-toolbox import/skip evaluation, strict exact-match grading). ∗ : significantly different from v1 (

p<0.05

, two-sided exact McNemar test, paired by question). caesar_cipher ’s on-disk v2 run is an 8-example smoke test that predates the full sweep and is excluded as unreliable (“excl.”); its v1/v3/v4 columns all use the same full 196-question matched set. No configuration ever significantly outperforms v1.

Tool-Augmented Language Model Agents

Tool-augmented language model agents use external tools, APIs, and environments to solve tasks beyond text generation alone.

See all 166 papers

Jan 7, 2026

SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models

Training reliable tool-augmented agents remains a significant challenge, largely due to the difficulty of credit assignment in multi-step reasoning. While process-level reward models offer a promising direction, existing LLM-based judges often produce noisy and inconsistent signals because they lack fine-grained, task-specific rubrics to distinguish high-level planning from low-level execution. In this work, we introduce SCRIBE (Skill-Conditioned Reward with Intermediate Behavioral Evaluation), a reinforcement learning framework that intervenes at a novel mid-level abstraction. SCRIBE grounds reward modeling in a curated library of skill prototypes, transforming open-ended LLM evaluation into a constrained verification problem. By routing each subgoal to a corresponding prototype, the reward model is equipped with precise, structured rubrics that substantially reduce reward variance. Experimental results show that SCRIBE achieves state-of-the-art performance across a range of reasoning and tool-use benchmarks. In particular, it improves the AIME25 accuracy of a Qwen3-4B model from 43.3% to 63.3%, and significantly increases success rates in complex multi-turn tool interactions. Further analysis of training dynamics reveals a co-evolution across abstraction levels, where mastery of mid-level skills consistently precedes the emergence of effective high-level planning behaviors. Finally, we demonstrate that SCRIBE is additive to low-level tool optimizations, providing a scalable and complementary pathway toward more autonomous and reliable tool-using agents.

Yuxuan Jiang, Francis Ferraro

University of Maryland, Baltimore County

Process Reward Models RL for Language Model Reasoning LLM Agent Skill Learning LLM Agent Training RL for Tool Use

Ask PDF

Jun 11, 2026

HyperTool: Beyond Step-Wise Tool Calls for Tool-Augmented Agents

Tool-augmented LLM agents commonly rely on step-wise atomic tool calls, where each invocation, observation, and value transfer is exposed in the main reasoning trace. This creates an execution-granularity mismatch: locally deterministic tool workflows are unfolded into repeated model-visible decisions, consuming context and forcing the model to manage low-level dataflow in the trace. We introduce HyperTool, a unified executable MCP-style tool interface that changes the model-visible unit of tool execution. A model invokes HyperTool with a code block that can call existing tools through their original schemas, manipulate returned values, and pass intermediate results locally, folding deterministic tool subroutines into a single outer call. To train models to use this interface, we synthesize HyperTool-format trajectories from cross-tool compositional tasks and verify them in real MCP environments. On MCP-Universe, HyperTool improves average accuracy from 15.69% to 35.29% on Qwen3-32B and from 9.93% to 33.33% on Qwen3-8B, and surpass GPT-OSS and Kimi-k2.5 on average accuracy, showing that our HyperTool can substantially improve multi-step tool use.

Yaxin Du, Yifan Zhou, Yujie Ge +7

Shanghai Jiao Tong University · IQuest Research · Beijing University of Aeronautics and Astronautics

Tool-Augmented Language Model Agents LLM Agent Workflow Optimization LLM Tool Use LLM Agent Training Tool Use

Ask PDF

Oct 22, 2026

Training Small Language Models for Transferable Tool Selection

As LLM agents operate over corpora of more than ten thousand tools, selecting the right tool for a complex multi-step user instruction becomes a central bottleneck. Tool descriptions cannot all fit in context, and relevance-based retrievers only rank tools without knowing which are actually needed. Recent work instead leverages proprietary commercial LLMs like Claude or ChatGPT to drive a retriever, decomposing the instruction and selecting over retrieved candidates. We show that a small, open model can be trained via reinforcement learning to perform the same functionality. On ToolBench, a Qwen3-8B model post-trained with our recipe outperforms Llama-4-Maverick (402B). In addition, the trained model transfers directly to different retrievers and generalizes to new tool corpora in different domains.

Zhaoyi Zhou, Akshayaa Magesh, Jeongyeol Kwon +4

Meta · Carnegie Mellon University

Tool Selection for Language Model Agents Small Language Models RL for Tool Use Language Model Post-Training

Ask PDF

Abstract

Tool-Augmented Language Model Agents

Explore similar work

SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models

HyperTool: Beyond Step-Wise Tool Calls for Tool-Augmented Agents

Training Small Language Models for Transferable Tool Selection