AT-AT: Automatic Tool Optimization from Agent Traces

Maya Ravichandran; Pragya Tripathi; Dharamendra Kumar; Adith Swaminathan; Nathan Kallus

cs.LGOct 26, 2026

AT-AT: Automatic Tool Optimization from Agent Traces

Authors: Maya Ravichandran, Pragya Tripathi, Dharamendra Kumar, Adith Swaminathan, Nathan Kallus

Organizations: Netflix

Abstract

Efforts to improve LLM-based agents typically focus on modifying the agent itself, yet success also depends on the environment in which an agent acts. Tools mediate this interaction: a tool's description shapes how an agent calls it, and its implementation determines the results and errors the agent receives afterwards. Many tools, such as APIs and MCP servers, are shared by many agents and maintained by providers who cannot change those agents. We introduce AT-AT (Automatic Tool Optimization from Agent Traces), an agentic tool optimizer that improves a shared toolset offline from historical agent executions while holding the agents fixed. A lightweight coding agent reads agent trajectories and their evaluation results, and selectively edits tool descriptions and, when permitted, implementations. On $τ^2$ -bench-verified, AT-AT achieves the highest task completion rates in the Airline and Telecom domains at lower cost than strong prompt- and tool-optimizer baselines, and its Telecom gain is statistically significant under paired permutation tests. Optimizing the tools within a coding agent harness also improves held-out performance on TerminalBench-2 and TBLite. When applied to three production MCP servers at a large enterprise, AT-AT lowered cost or tool error rates on every server and increased held-out task passes on one. By turning accumulated execution traces into updates to shared tool environments, AT-AT provides a path toward continual improvement of agent systems through environment-side learning.

Figures & tables

Figure 1: AT-AT optimizes tools from historical agent traces. (A) An agent completes a task by calling tools, implemented in its harness or the external environment, producing a trajectory. (B) In a sandbox holding evaluated traces and the current tool source, a Pi coding agent driven by an optimizer LLM selectively edits tool descriptions and implementations.

	Optimizer	Test reward (%)	Reward $p$	Test cost (USD/sim.)	Cost $p$	Optimizer cost (USD)
Airline	Baseline	52.17%	—	0.0618	—	—
	AT-AT	59.17%	0.2305	0.0675 *	0.002155	2.7800
	Without past traces	50.33%	0.3575	0.1205 *	0.000002	0.2330
	Edit descriptions only	51.33%	0.7609	0.1176 *	0.000002	0.3839
	Without coding agent harness	52.50%	0.9446	0.0825 *	0.000002	2.2481
	DRAFT	53.50%	0.5988	0.0822 *	0.000002	2.4367

Table 1: AT-AT achieves the highest test reward in Airline and Telecom. Held-out results over

n=30

trials per task. Test cost is agent plus user-simulator cost per simulation; optimizer cost is the one-time cost of producing the toolset.

p

: two-sided task-paired permutation test against the domain baseline;

*

marks

p<0.05

. Bold: best value per domain (optimizer cost excludes baseline).

Domain	Optimizer	Test reward (%)	Reward $p$	Test cost (USD/sim.)	Cost $p$	Optimizer cost (USD)
Airline	Baseline	58.00%	—	0.1135	—	—
Airline	AT-AT	59.67%	0.3584	0.1239 *	0.000557	2.4029
Retail	Baseline	73.83%	—	0.1221	—	—
Retail	AT-AT	76.42%	0.1662	0.1315 *	0.000010	2.3330
Telecom	Baseline	41.42%	—	0.2457	—	—
Telecom	AT-AT	40.67%	0.7320	0.2602 *	0.000100	1.8308

Table 2: AT-AT’s tool edits improve held-out reward in 2 of 3

\tau^{2}

-bench-verified domains when averaged across five customer service agent LLMs. Results are averaged across models within each task, then across tasks, with 30 trials per task. Columns and

*

are as in Tab. 1 .

	Train				Test
	TerminalBench-2		TBLite		TerminalBench-2		TBLite
$k$	Baseline	Optimized	Baseline	Optimized	Baseline	Optimized	Baseline	Optimized
1	45.78%	48.44%	72.80%	72.80%	49.32%	51.36%	75.60%	77.20%
3	58.43%	58.69%	81.38%	81.95%	59.43%	61.34%	88.65%	89.63%
10	68.89%	68.89%	84.00%	86.00%	68.18%	70.45%	94.00%	94.00%

Table 3: AT-AT improves held-out pass@

k

on TerminalBench-2 and TBLite. Pass@

k

from each benchmark’s native evaluator before (Baseline) and after (Optimized) optimizing OpenCode’s tools, on the train split used for optimization and the held-out test split.

	Dev Tools ( $n{=}10$ )		Collab Toolbox ( $n{=}5$ )		Eng. Context ( $n{=}11$ )
Metric	Orig.	AT-AT	Orig.	AT-AT	Orig.	AT-AT
Task pass rate (%)	80.0	80.0	0.0	0.0	54.5	72.7
Mean reward (%)	90.0	90.0	22.3	35.1	77.3	86.4
Content score (%)	88.3	80.0	37.7	36.7	64.7	69.0
Tool calls	176	133	121	86	114	76
Tool errors	15	10	73	37	0	0

Table 4: AT-AT improves task pass rate on one out of three production MCP servers, while reducing tool call error rates and/or cost on all three MCP servers. Held-out results with original (Orig.) and AT-AT-optimized descriptions;

n

is the number of scored test tasks. Mean reward and content score are shown as percentages. Session cost covers the agent and user simulator; optimizer cost is one-time.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Domain	Optimizer	Optimizer LLM	Test reward (%)	Reward $p$	Test cost (USD/sim.)	Cost $p$	Optimizer cost (USD)
Airline	Baseline	—	52.17%	—	0.0618	—	—
	AT-AT	Opus 4.7	59.17%	0.2305	0.0675 *	0.002155	2.7800
		GPT-5.6 Sol	47.33%	0.3743	0.1194 *	0.000002	1.2866
		Kimi K3	54.33%	0.3135	0.1246 *	0.000002	4.7419
	ToolObserver	Opus 4.7	55.83%	0.1650	0.0870 *	0.000002	16.2171
		GPT-5.6 Sol	52.83%	0.7461	0.1498 *	0.000002	10.1014

Appendix

Table 5: AT-AT’s improvements are robust to the choice of optimizer LLM in

\tau^{2}

-bench-verified’s Telecom, where both GPT-5.6 Sol and Kimi K3 produce statistically significant gains. The customer service agent and user simulator are GPT-4.1 and GPT-5.1, respectively; only the optimizer LLM varies. Results are computed over

n=30

trials per held-out test task. Test cost is mean agent plus user-simulator inference cost in USD per simulation, and optimizer cost is the one-time cost of producing the candidate toolset.

*

marks a statistically significant change from the domain baseline at

p<0.05

under a two-sided task-paired permutation test. Best reward (highest), test cost (lowest), and optimizer cost (lowest non-baseline value) in each domain are bold.

Domain	Optimizer	Test reward (%)	Reward $p$	Test cost (USD/sim.)	Cost $p$	Optimizer cost (USD)
Airline	Baseline	52.17%	—	0.0618	—	—
	AT-AT	59.17%	0.2305	0.0675 *	0.002155	2.7800
	Without generalization prompt or full traces	51.33%	0.8099	0.0605	0.363811	2.7800
Retail	Baseline	77.00%	—	0.0707	—	—
	AT-AT	74.67%	0.2313	0.0724	0.070629	1.9300
	Without generalization prompt or full traces	75.75%	0.3186	0.0710	0.683283	1.9300

Appendix

Table 6: Pi optimizer without the generalization prompt or full file system traces underperforms AT-AT’s reward gains in Airline and Telecom. Held-out results over

n=30

trials per task; baseline and AT-AT rows are repeated from Tab. 1 for comparison. Test cost is agent plus user-simulator cost per simulation; optimizer cost is the one-time cost of producing the toolset.

p

: two-sided task-paired permutation test against the domain baseline;

*

marks

p<0.05

. Bold: best test reward and test cost per domain.

Domain		Full AT-AT	Descriptions only
Airline	Tools edited	book_reservation D cancel_reservation D get_flight_status D get_reservation_details C+D get_user_details D search_direct_flight C+D search_onestop_flight C send_certificate D update_reservation_baggages D update_reservation_flights D update_reservation_passengers D	update_reservation_baggages D update_reservation_flights D
Airline	Qualitative changes	• Added timestamp parsing and reservation-time helpers. • Reservation lookup now returns current time, booking age, 24-hour eligibility, flight-date bounds, and past/upcoming status. • Direct and one-stop search results now carry their flight dates. • Descriptions add or clarify policy and usage guidance for booking, cancellation, lookup, search, certificate, baggage, flight-change, and passenger-change operations.	• Clarified that baggage totals are absolute. • Documented paid-bag charges and certificate restrictions. • Added flight-change guidance covering basic-economy restrictions, downgrade refunds, baggage recomputation, and passenger changes.
Retail	Tools edited	exchange_delivered_order_items D get_item_details C modify_pending_order_items C+D	cancel_pending_order D exchange_delivered_order_items D find_user_id_by_name_zip D modify_pending_order_items D return_delivered_order_items D
Retail	Qualitative changes	• Corrected item lookup for object-backed products ( get_item_details ). • Corrected pending-order modification to apply the requested replacement variant’s price and options. • Clarified that exchanges and pending-order modifications only swap variants of the same product and cannot add or remove items.	• Added explicit confirmation and irreversibility guidance. • Distinguished whole-order cancellation from item modification. • Required batching one-time return or exchange actions. • Clarified same-product variant constraints and preferred customer lookup.
Telecom	Tools edited	get_line_by_phone new get_customer_by_phone C+D get_data_usage C+D enable_roaming C+D disable_roaming D refuel_data C+D	get_details_by_id D send_payment_request D enable_roaming D disable_roaming D
Telecom	Qualitative changes	• Added formatting-insensitive phone normalization and a direct phone-to-line/customer lookup. • Customer lookup now tolerates formatted numbers. • Roaming responses now expose current and prior state. • Usage responses add remaining data and an exceeded flag. • Refueling responses add the resulting allowance and exceeded state.	• Expanded object-status guidance in generic lookup. • Advised re-fetching bills after payment requests. • Distinguished carrier-side roaming from the device’s Data Roaming setting.

Appendix

Table 8: Tools edited by full-scope AT-AT and by the descriptions-only ablation. Tags mark which part of each tool changed: D = description, C = code, C+D = both, new = new tool.

Trajectory step	Full AT-AT	Descriptions only
Tool feedback	created_at=2024-05-02 , current_datetime=2024-05-15 , hours_since_booking=320.95 , within_24h_of_booking=False	created_at=2024-05-02 ; no elapsed time or eligibility flag
Agent response	“Booked more than 24 hours ago (about 13 days ago, not 10 hours ago)”; refuses refund	“Booking made about 10 hours ago (within 24 hours)”; offers and executes cancellation
Outcome	Correctly preserves reservation; reward 1	Incorrectly cancels and refunds; reward 0

Appendix

Table 9: Airline task: computed state returned by an existing tool prevents an unsupported refund.

Trajectory step	Full AT-AT	Descriptions only
Line lookup	New tool get_line_by_phone(555-123-2002) returns customer_id=C1001 , line L1002	get_customer_by_phone(555-123-2002) returns line_ids=[L1001,L1002,L1003] ; agent queries the first, L1001 , which returns phone_number=555-123-2001
Agent action	Enables roaming on L1002	Overlooks the mismatched number; enables roaming on L1001
Diagnosis	Usage on L1002 : 15.1/15.0 GB ; attributes outage to exhausted data and refuels 2 GB	Usage on L1001 : 3.2/5.0 GB ; concludes the cap was not reached and transfers to a human agent
Outcome	Reward 1	Reward 0

Appendix

Table 10: Telecom task: adding a new tool enables a direct line lookup that avoids misdiagnosing the problem.

Trajectory step	Full AT-AT	Descriptions only
Tool feedback	get_line_by_phone directly returns customer_id=C1001 , line_id=L1002 , roaming_enabled=False	Customer lookup returns line_ids=[L1001,L1002,L1003] without identifying the relevant line
Agent action	Immediately calls enable_roaming(C1001, L1002)	Asks the user to identify the line, then queries L1001 and L1002 separately before enabling roaming
Outcome	Reward 1	Reward 1

Appendix

Table 11: Telecom task: the new lookup tool removes exploratory line searches.

MCP server	Representative tasks
Core Developer Tools	• Interpret an application performance metric • Investigate VPN-related host reachability • Diagnose repeated software installation failures
Observability Collaboration Toolbox	• Classify date-scoped support threads • Evaluate automated support answer quality and recommend fixes • Compare activity across support channels
Enterprise Engineering Context and Best Practices	• Assess whether dependency changes caused canary latency • Diagnose a linked CI/CD failure • Reconcile data compaction settings with an underlying messaging topic

Appendix

Table 12: Representative tasks for each enterprise MCP server.

Tool category	Paraphrased guidance added by AT-AT
Dev Tools	Use namespace-specific retrieval only when the exact registered namespace is known. Avoid guessing namespaces from product or team names; otherwise, prefer cross-source or dedicated search tools.
Collab Toolbox	Normalize channel references to the identifier format expected by each tool and resolve display names before calling tools that require channel IDs. Distinguish access-denied or unindexed channels from genuine zero-result searches, and treat an unresolved workspace or channel identifier as an identity or access problem rather than an empty result.
Eng. Context	Clarify that the tool searches indexed documentation, code, and discussion content rather than live executions, incidents, dashboards, metrics, or logs. Avoid near-duplicate retries after no relevant context is found, and use focused calls for separate parts of a multi-part request.

Appendix

Table 13: Paraphrased examples of AT-AT description updates for enterprise MCP tools.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 289 papers

AgentTune: A Unified Training Lifecycle for Tool-Using LLM Agents
Oct 9, 2026R E Zera Marveen Lyngkhoi, Abhivansh Gupta, Vidushee Vats +3Tool-Using Agents LLM Agent Training
Lexsi Labs
Learning to Rewrite Tool Descriptions for Reliable LLM-Agent Tool Use
Feb 23, 2026Ruocheng Guo, Kaiwen Dong, Xiang Gao +1LLM Tool Use LLM Agent Training
Intuit AI Research, Mountain View, CA, USA
ToolAtlas: Learning Once, Reusing Everywhere with Tool-Side Memory
Jul 13, 2026Yue Fang, Zhibang Yang, Fangkai Yang +5LLM Tool Use
School of Computer Science, Peking University, Beijing, China · Microsoft

Oct 9, 2026J/K move · Enter open · S save

AgentTune: A Unified Training Lifecycle for Tool-Using LLM Agents

R E Zera Marveen Lyngkhoi, Abhivansh Gupta, Vidushee Vats, Ram Mohan Rao Kadiyala+2

Lexsi Labs

Tool-Using Agents LLM Agent Training LLM Agent Harnesses LLM Agent Evaluation

Open paper AskPDF arXiv

Tool-using agents increasingly require a common substrate for building, training, evaluating, distilling, and maintaining agentic systems, yet existing tooling typically addresses these stages in isolation. We present AgentTune, a source-available Python library that provides a unified framework for developing and optimizing tool-using agents. At its core, AgentTune introduces a two-tier EventLog representation that preserves agent trajectories while supporting consistent projections for training, evaluation, and auditing. Building on this substrate, the library provides a shared agent runtime, tool-calling reinforcement learning, agentic distillation, retrieval-augmented generation, decision workflows, evaluation utilities, and a designed path toward self-healing agents. These components are exposed through composable interfaces so that agent strategies, training methods, evaluation procedures, and recovery mechanisms can be combined without requiring separate representations or pipelines. We demonstrate the framework across agentic tasks including legal contract search and domain-specific applications, illustrating its use for trajectory-based optimization, tool-use evaluation, and cross-task agent development. AgentTune is released as a source-available library at https://github.com/Lexsi-Labs/AgentTune

Abstract

Tool-Using Agents

Explore similar work

AgentTune: A Unified Training Lifecycle for Tool-Using LLM Agents

Learning to Rewrite Tool Descriptions for Reliable LLM-Agent Tool Use

ToolAtlas: Learning Once, Reusing Everywhere with Tool-Side Memory