AgentTune: A Unified Training Lifecycle for Tool-Using LLM Agents

R E Zera Marveen Lyngkhoi; Abhivansh Gupta; Vidushee Vats; Ram Mohan Rao Kadiyala; Vinay Kumar Sankarapu; Pratinav Seth

cs.LGOct 9, 2026

AgentTune: A Unified Training Lifecycle for Tool-Using LLM Agents

Authors: R E Zera Marveen Lyngkhoi, Abhivansh Gupta, Vidushee Vats, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth

Organizations: Lexsi Labs

Abstract

Tool-using agents increasingly require a common substrate for building, training, evaluating, distilling, and maintaining agentic systems, yet existing tooling typically addresses these stages in isolation. We present AgentTune, a source-available Python library that provides a unified framework for developing and optimizing tool-using agents. At its core, AgentTune introduces a two-tier EventLog representation that preserves agent trajectories while supporting consistent projections for training, evaluation, and auditing. Building on this substrate, the library provides a shared agent runtime, tool-calling reinforcement learning, agentic distillation, retrieval-augmented generation, decision workflows, evaluation utilities, and a designed path toward self-healing agents. These components are exposed through composable interfaces so that agent strategies, training methods, evaluation procedures, and recovery mechanisms can be combined without requiring separate representations or pipelines. We demonstrate the framework across agentic tasks including legal contract search and domain-specific applications, illustrating its use for trajectory-based optimization, tool-use evaluation, and cross-task agent development. AgentTune is released as a source-available library at https://github.com/Lexsi-Labs/AgentTune

Figures & tables

Layer	Package	Contents	Described in
1 Interfaces	cli/	the agenttune console commands and the decide sub-app; the Python SDK and the [service] FastAPI app are the other three entry points	§ 3.12
2 Agent runtime	agentic/	five strategies, three memory backends and TrajectoryStore , two harnesses, and the Project lifecycle	§ 3.4
3 Unified trajectory	agentic/ events.py	the EventLog schema, its two tiers, and the six projections in and out	§ 3.3
4 Learning and evaluation	core/ , backends/ , rag/ , langgraph/ , eval/	create_agentic_trainer and the five TrlAgentic* wrappers; three SearchBackend implementations and the six-stage synthesis pipeline; AgentTuneGraph ; BaseEvaluator / RLEvaluator , TrajectoryEvaluator , the lm-eval runner and run_eval ; rollout engines and the 26-entry reward registry	§ 3.5 –§ 3.7 , § 3.10 , § 3.11
5 Production and improvement	decide/	GraphRunner , the eight stage types, 24 templates, the append-only audit log, and closed_loop/	§ 3.8 , § 3.9
—	data/ , utils/ , scenarios/	DataManager , device and environment diagnostics, checkpoint helpers, scripted domain scenarios	out of scope (§ 3.1 )

Table 1: The five layers of Figure 1 , the packages of src/agenttune/ that implement each, and the subsection describing them.

Figure 1: AgentTune in five layers. Interfaces (1) drive the agent runtime (2), whose episodes normalize into the EventLog (3) in a light or full tier; the schema projects out to SFT rows and to the evaluator and audit views, which learning and evaluation (4) consume to produce adapters, metrics, and traces. Production (5) runs DECIDE workflows and the detect–classify–generate–retrain–gate cycle, whose audited failures and corrected trajectories return to Layer 3 as new training data. Agentic distillation is drawn beside the shared RL core rather than inside it because it reaches TRLSFTTrainer through a caller-supplied trainer_factory instead of create_agentic_trainer .

Projection	Direction	Tier produced / required
from_trajectory	RL rollout $\rightarrow$ EventLog	full
from_pipeline_state	DECIDE state $\rightarrow$ EventLog	light
from_eval_dict	eval dict $\rightarrow$ EventLog	light
as_dataset_rows("sft")	EventLog $\rightarrow$ SFT rows	requires full
to_eval_dict	EventLog $\rightarrow$ eval dict	either
to_audit_records	EventLog $\rightarrow$ audit log	either

Table 2: The six EventLog projections. The three from_* classmethods only read their input; none of the native formats is modified.

Event kind	Chat role	Segment weight
REASONING / TEXT	assistant	$1.0$
TOOL_CALL	assistant	$2.0$
TOOL_RESULT	tool	$0.0$
OBSERVATION	user	$0.0$

Table 3: Segment weights emitted by as_dataset_rows("sft") ; loss_mask is [w == 0.0 for w] . Note the polarity: unlike attention_mask or TRL’s completion_mask , True here means excluded from the loss.

Algorithm	Needs	Rollout modes
GRPO [ 15 ]	reward function(s)	agentic tool loop; combine_rewards applied
DPO [ 16 ]	chosen / rejected , or a reward fn	offline; reward-ranked; agentic
PPO [ 17 ]	reward fn/model and value fn/model	agentic; patches batch_generation
RLOO [ 18 ]	reward function	agentic; reward fn forwarded as-is
BCO [ 19 ]	completion / label , or a reward fn	offline; online (regenerates per epoch)

Table 4: The five dispatchable algorithms behind create_agentic_trainer . GRPO is the default for agentic training. Only GRPO routes reward_funcs through the weighted combine_rewards combinator.

Stage type	What it does
llm_call	one model call, reply validated against an output_schema
router	LLM-based (has a model: ) or rules-based (no model: ) dispatch to one of several on_result targets, through one stage type
llm_judge	scores an upstream output on a rubric
rules	deterministic gate: a list of condition strings over stage_outputs , evaluated by a hand-written AST walker (only == != < <= > >= and or not , dotted lookups, JSON literals; no eval() , no function calls); first failing rule short-circuits with goto + inject
parallel	fan-out to independent branches, run concurrently via asyncio.gather , merged back into stage_outputs by branch id
tool_call	invoke a tool from the same registry the spine uses, with interpolated args and a per-stage timeout

Table 5: The eight DECIDE stage types, all registered and executable. A straight llm_call

\to

output chain is the easy case; real pipelines branch with rules and router .

#	Name	Fires when
T1	total_failures_exceeded	buffer size $\geq$ threshold
T2	dominance:<type>	one root_cause $\geq$ the dominance ratio of the buffer
T3	enough_examples	accepted-example count $\geq$ minimum
T4	reward_drift	rolling mean episode reward drifts below baseline
T5	novel_failure:<type>	a never-seen root_cause recurs $\geq$ a minimum count
T6	staleness	time since last retrain $\geq$ a window, with a minimum buffer

Table 6: The six RetrainingTrigger conditions; any one fires a retrain. Two safety gates run first.

Metric	Measures
TAC	tool-argument correctness, jsonschema-validated against registered schemas
ARR	API-redundancy ratio: duplicate calls / total
SCSR	self-correction success rate: recovered from a trailing error
RAD	reasoning-to-action density (character-count proxy)
LCF	loop-collapse frequency: 3+ identical consecutive calls
PAS	plan adherence: planned steps that appear in the trajectory

Table 7: Deterministic trajectory metrics computed by TrajectoryEvaluator with no LLM; Project.evaluate_agentic() surfaces these.

Path	Setup	Result
Distillation (SFT)	SmolLM2-360M [ 41 ] , 16 teacher trajectories, LoRA, $\sim$ 7 s	train loss $3.60{\to}0.25$ ; token acc. $0.47{\to}0.94$ ; contract learned
GRPO (on-policy)	SmolLM2-360M, 90 steps, contract-only reward	mean reward $0.03{\to}0.66$ ; contract rate $25\%{\to}100\%$ ; correct-label $0\%{\to}75\%$
Self-heal $\rightarrow$ DPO	12 corrective preference pairs from the heal loop	reward acc. $0.50{\to}1.00$ ; final reward margin $1.97$

Table 8: Representative training runs shipped with AgentTune , captured on a single NVIDIA RTX PRO 6000 Blackwell Max-Q workstation GPU. These are smoke tests and the denominators are small: the contract and correct-label rates are four sampled completions on one probe input (

1/4\rightarrow 4/4

and

0/4\rightarrow 3/4

), and the distillation row reports in-sample training loss at epochs

1.25

and

25.0

over the same 16 trajectories, not a held-out result. They demonstrate that each trainer path executes end to end and moves the policy, not that AgentTune sets a benchmark.

Figure 2: Answer reachability under one-shot top-8 hybrid retrieval and perfect retrieval on the multi-hop CUAD questions.

Figure 3: Dataset curation from 5,361 candidates to 3,992 verified multi-hop questions.

Model	Exact match (%)	Searches / question
Base model	60.0	3.00
+ GRPO (LoRA)	69.0	2.90

Table 9: Held-out multi-hop CUAD questions under greedy decoding. GRPO raises exact match while reducing search use.

	Doc@8		Searches / question
Corpus	Base	+ GRPO	Base	+ GRPO
ContractNLI [ 48 ]	0.58	0.69	4.2	3.8
MAUD [ 46 ]	0.98	1.00	5.0	4.5
PrivacyQA [ 47 ]	0.83	0.98	4.0	3.5

Table 10: Transfer of the frozen CUAD-trained policy to three unseen LegalBench-RAG corpora [ 45 ] . Doc@8 improves and searches fall on every corpus; the MAUD gain is a ceiling effect.

Figure 4: Greedy-decoding performance on held-out multi-hop CUAD questions. GRPO improves exact match while slightly reducing search use.

Figure 5: Transfer to three unseen LegalBench-RAG corpora. The frozen GRPO policy improves Doc@8 and reduces searches on each dataset.

CUAD
Model	Training variant	Precision	Recall	Pass@1
0.5B	Base	36.8	34.1	29.4
0.5B	Answer SFT	38.2	24.1	28.5
0.5B	QA + reasoning	45.1	42.0	36.5
0.5B	Trajectory, no mask	39.4	58.2	41.0
0.5B	Masked trajectory	48.5	52.6	45.8

Table 11: Agentic-distillation ablation on CUAD and LexGLUE ECTHR. The 0.5B student is evaluated on CUAD and the 1.5B student on ECTHR. Masked trajectory training excludes retrieved tool outputs from the loss while retaining them as conditioning context. Bold values indicate the best result within each benchmark.

Capability	AgentTune	TRL	verl	SkyRL / OpenRLHF	LangChain / CrewAI
Trains a model at all	✓	✓	✓	✓	–
GRPO/PPO/DPO/RLOO/BCO via one entry point	✓	–	partial	partial	–
Agentic tool-calling rollouts native to every algorithm	✓	–	✓	✓	executes tools, no training
One trajectory schema across eval / train / distill / heal	✓	–	–	–	–
Agentic distillation (behavior, not weights alone)	✓	–	–	–	–
Synthetic multi-hop data with verification	✓	–	–	–	–

Table 12: AgentTune against the tools it composes with, assessed in September 2026 against each project’s then-current documented feature set. TRL supplies the optimizers; verl, SkyRL [ 67 ] , and OpenRLHF scale RL to clusters; LangChain [ 68 ] and CrewAI [ 69 ] orchestrate agents without a training loop. AgentTune reuses TRL and LangGraph rather than reimplementing them, and adds the schema and the stages built on it. Entries record whether a capability is exposed as a first-class, documented entry point, not whether it can be assembled with user code; “partial” means some but not all of the five algorithms. The last row is the one quantitative cell: connecting six lifecycle stages pairwise would need up to fifteen format converters, where one schema needs six projections. AgentTune is source-available under LSAL 1.1 and is therefore not open source in the OSI sense, unlike every other column.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

EventKind	Meaning
TEXT	model-emitted final text
REASONING	model-emitted reasoning / thought
TOOL_CALL	a tool / action invocation
TOOL_RESULT	the tool’s returned output
OBSERVATION	environment / state fed to the agent
TURN_COMPLETE	end of one agent turn

Appendix

Table 13: The eight EventKind values. REWARD is the only kind with a meaningful scope .

Registry key	Behavior
correctness_reward	parses yes / no ; $+0.6$ matches answer , $-1.0$ wrong, $-0.5$ no match
structure_reward	call $\to$ response $\to$ content shape: $+0.1$ / $+0.05$ well-formed, $-0.15$ call with no response
query_reward	SQL heuristic: penalizes $>3$ queries, LIMIT 1 , missing WHERE
reward_correct_answer	numeric match: $1.5$ exact ( $\pm 10^{-3}$ ), $0.5$ within $1\%$
reward_tool_used	$1.0$ if any message has role: tool
reward_concise_answer	$0.3$ if the final answer is $\leq 80$ words

Appendix

Table 14: All 26 registered reward functions. The finqa_* entries are the numerically careful versions of the plainly-named ones and are reachable under prefixed keys.

Key	Contents
trajectories	list[Trajectory] , full step-by-step record incl. tool calls
responses	list[str] , final assistant text per rollout
rewards	list[float] , all zero unless reward_fn= was passed
prompt_ids / completion_ids	token ids (the GRPO contract)
logprobs	per-token logprobs
tool_call_counts	list[int] , tool calls per rollout

Appendix

Table 15: Keys in a rollout output dict. A bare rollout_func must return at least prompt_ids , completion_ids , and logprobs ; the rest are forwarded to reward functions as kwargs.

Data
Parameter	Value
Source corpus	90 CUAD training contracts [ 31 ]
Generated candidates	5,361
After verification	4,364
After answerability gate	3,992 (2–5 hops)
Train/validation split	gold-chunk disjoint

Appendix

Table 16: Configuration for the legal contract-search use case of § 5.1 .

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 289 papers

AT-AT: Automatic Tool Optimization from Agent Traces
Oct 26, 2026Maya Ravichandran, Pragya Tripathi, Dharamendra Kumar +2Tool-Using Agents LLM Agent Efficiency
Netflix
UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents
Apr 13, 2026Yijuan Liang, Xinghao Chen, Yifan Ge +5LLM Agent Evaluation AI Agent Benchmarks
University of Science and Technology of China · Zhejiang Key Laboratory of Industrial Intelligence and Digital Twin, Eastern Institute of Technology, Ningbo · Department of Computing, The Hong Kong Polytechnic University
Learning to Rewrite Tool Descriptions for Reliable LLM-Agent Tool Use
Feb 23, 2026Ruocheng Guo, Kaiwen Dong, Xiang Gao +1LLM Tool Use LLM Agent Training
Intuit AI Research, Mountain View, CA, USA

Oct 26, 2026J/K move · Enter open · S save

AT-AT: Automatic Tool Optimization from Agent Traces

Maya Ravichandran, Pragya Tripathi, Dharamendra Kumar, Adith Swaminathan+1

Netflix

Tool-Using Agents LLM Agent Efficiency

Open paper AskPDF arXiv

Efforts to improve LLM-based agents typically focus on modifying the agent itself, yet success also depends on the environment in which an agent acts. Tools mediate this interaction: a tool's description shapes how an agent calls it, and its implementation determines the results and errors the agent receives afterwards. Many tools, such as APIs and MCP servers, are shared by many agents and maintained by providers who cannot change those agents. We introduce AT-AT (Automatic Tool Optimization from Agent Traces), an agentic tool optimizer that improves a shared toolset offline from historical agent executions while holding the agents fixed. A lightweight coding agent reads agent trajectories and their evaluation results, and selectively edits tool descriptions and, when permitted, implementations. On

τ^2

-bench-verified, AT-AT achieves the highest task completion rates in the Airline and Telecom domains at lower cost than strong prompt- and tool-optimizer baselines, and its Telecom gain is statistically significant under paired permutation tests. Optimizing the tools within a coding agent harness also improves held-out performance on TerminalBench-2 and TBLite. When applied to three production MCP servers at a large enterprise, AT-AT lowered cost or tool error rates on every server and increased held-out task passes on one. By turning accumulated execution traces into updates to shared tool environments, AT-AT provides a path toward continual improvement of agent systems through environment-side learning.

Abstract

Tool-Using Agents

Explore similar work

AT-AT: Automatic Tool Optimization from Agent Traces

UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents

Learning to Rewrite Tool Descriptions for Reliable LLM-Agent Tool Use