AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agent

Wenyue Hua; Sripad Karne; Qian Xie; Armaan Agrawal; Nikos Pagonas; Kostis Kaffes; Tianyi Peng

cs.LGApr 7, 2026

AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agent

Authors: Wenyue Hua, Sripad Karne, Qian Xie, Armaan Agrawal, Nikos Pagonas, Kostis Kaffes, Tianyi Peng

Organizations: Microsoft Research, AI Frontiers · Columbia University · Cornell University

Abstract

AI agents are increasingly deployed in real-world applications, including systems such as Manus, OpenClaw, and coding agents. Existing research has primarily focused on server-side efficiency, proposing methods such as caching, speculative execution, traffic scheduling, and load balancing to reduce the cost of serving agentic workloads. However, as users increasingly construct agents by composing local tools, remote APIs, and diverse models, an equally important optimization problem arises on the client side. Client-side optimization asks how developers should allocate the resources available to them, including model choice, local tools, and API budget across pipeline stages, subject to application-specific quality, cost, and latency constraints. Because these objectives depend on the task and deployment setting, they cannot be determined by server-side systems alone. We introduce AgentOpt, the first framework-agnostic Python package for client-side agent optimization. We first study model selection, a high-impact optimization lever in multi-step agent pipelines. Given a pipeline and a small evaluation set, the goal is to find the most cost-effective assignment of models to pipeline roles. This problem is consequential in practice: at matched accuracy, the cost gap between the best and worst model combinations can reach 13-32x in our experiments. To efficiently explore the exponentially growing combination space, AgentOpt implements ten search algorithms, including UCB-E, UCB-E with Low-Rank Factorization, Arm Elimination, Epsilon-LUCB, Threshold Successive Elimination, and Bayesian Optimization. Across four benchmarks, UCB-E recovers near-optimal accuracy while reducing evaluation budget by 62-76% relative to brute-force search. Code and benchmark results available at https://agentoptimizer.github.io/agentopt/.

Figures & tables

Figure 1: Overview of AgentOpt . (a) A model combination

c=(m_{1},\dots,m_{N})

assigns one model per pipeline role. The full trajectory

\tau(c)

is evaluated end-to-end. (b) The optimization loop iteratively selects, executes, and measures combinations, returning the Pareto frontier over performance, cost, and latency.

\mathbf{c}_{t}=\arg\max_{\mathbf{c}\in\mathcal{C}}\alpha(\mathbf{c};g,\mathcal{H})

Algorithm 6 Bayesian Optimization for model-combination search

Benchmark	Tuple	Combos	Best Combination	Acc.	BF Cost
GPQA Diamond	1	9	Claude Opus 4.6	74.75%	$4.71
BFCL	1	9	Claude Opus 4.6 (tied; Qwen3 Next: $32\times$ cheaper)	70.00%	$84.80
HotpotQA	2	81	Ministral 3 8B + Claude Opus 4.6	74.27%	$51.90
MathQA	2	81	Claude Opus 4.6 + Claude Haiku 4.5	98.84%	$123.87

Table 1: Cross-benchmark summary of exhaustive search. “Tuple” denotes the number of pipeline roles jointly optimized, “Combos” is the total number of model combinations, “Acc.” is the best observed task accuracy, and “BF Cost” is the total API cost of brute-force evaluation over the full combination space.

Benchmark	Best Combo	Interpretation
HotpotQA	Ministral 3 8B (planner) + Opus 4.6 (solver)	The weakest and cheapest planner performs best. When Opus is used as planner, it often bypasses the downstream solver and its search tools, leading to consistently poor end-to-end accuracy.
MathQA	Opus 4.6 (answer) + Haiku 4.5 (critic)	Critic identity has limited effect once the answer model is sufficiently strong. With Opus as answerer, all nine critics lie within a narrow 2.9-point range.
BFCL	Opus 4.6 / Kimi / Qwen3 Next (tied)	Three models achieve identical 70% accuracy. Qwen3 Next costs 32 $\times$ less than Opus, making model selection purely a cost decision at this accuracy level.
GPQA	Opus 4.6 (single)	This is the one setting where raw standalone capability remains predictive, and the strongest model is also the best end-to-end choice.

Table 2: Representative cases in which the optimal model combination differs from naive capability-based expectations. The table illustrates that model quality is role-dependent and must be evaluated at the level of the full workflow rather than inferred from standalone model rankings.

Rank	Planner	Solver	Acc.	Avg. Lat. (s)	Cost
71	Claude Opus 4.6	Kimi K2.5	31.96%	4.72	$2.02
72	Claude Opus 4.6	Ministral 3 8B	31.96%	4.72	$2.02
73	Claude Opus 4.6	Qwen3 32B	31.96%	4.72	$2.02
74	Claude Opus 4.6	Qwen3 Next 80B A3B	31.96%	4.72	$2.02
75	Claude Opus 4.6	gpt-oss-120b	31.95%	4.60	$2.02
76	Claude Opus 4.6	gpt-oss-20b	31.88%	4.57	$2.03

Table 3: Bottom 11 combinations on HotpotQA. Most of these configurations use Claude Opus 4.6 as planner and cluster near 32% accuracy regardless of the solver, illustrating that a strong model can be systematically mismatched to a particular pipeline role.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Selector	Budget	Mean Acc	Mean Evals	Mean Cost	Cost Savings
Brute Force (ref)	–	74.75%	1,782	$4.71	0.0%
Matrix UCB-E	0.5	74.75%	891	$4.18	11.4%
Matrix UCB-E-LRF	0.5	74.75%	891	$4.17	11.6%
LM Proposal	–	74.75%	198	$2.47	47.6%
Hill Climbing	–	74.55%	1,501	$4.03	14.4%
Matrix UCB-E	0.3	74.34%	535	$3.11	33.9%

Appendix

Table 4: GPQA Diamond selector comparison (198 samples, 9 models).

Selector	Budget	Mean Acc	Mean Evals	Mean Cost	Cost Savings
Brute Force (ref)	–	70.00%	1,800	$84.80	0.0%
Hill Climbing	–	70.00%	1,664	$72.12	15.0%
Matrix UCB-E	0.5	70.00%	900	$73.10	13.8%
Matrix UCB-E	0.2	69.90%	360	$26.14	69.2%
Matrix UCB-E	0.3	69.90%	540	$43.43	48.8%
Epsilon-LUCB	–	69.90%	399	$40.03	52.8%

Appendix

Table 5: BFCL v3 selector comparison (200 samples, 9 models).

Selector	Budget	Mean Acc	Mean Evals	Mean Cost	Cost Savings
Brute Force (ref)	–	74.27%	16,168	$51.90	0.0%
Matrix UCB-E	0.5	74.27%	8,084	$23.18	55.3%
Matrix UCB-E	0.3	74.25%	4,851	$18.41	64.5%
Matrix UCB-E	0.2	73.54%	3,234	$12.49	75.9%
Bayesian Opt	–	73.33%	3,996	$12.29	76.3%
Arm Elimination	–	73.19%	4,283	$16.92	67.4%

Appendix

Table 6: HotpotQA selector comparison (199 samples, 81 combinations).

Selector	Budget	Mean Acc	Mean Evals	Mean Cost	Cost Savings
Brute Force (ref)	–	98.84%	14,961	$123.87	0.0%
Matrix UCB-E	0.5	98.84%	7,481	$89.10	28.1%
Matrix UCB-E-LRF	0.5	98.84%	7,481	$89.48	27.8%
Matrix UCB-E	0.3	98.84%	4,489	$62.12	49.9%
Arm Elimination	–	98.83%	3,356	$51.86	58.1%
Hill Climbing	–	98.76%	3,926	$54.22	56.2%

Appendix

Table 7: MathQA selector comparison (200 samples, 81 combinations).

Rank	Model	Accuracy	Avg. Lat. (s)	Cost
1	Claude Opus 4.6	74.75%	9.16	$2.47
2	Kimi K2.5	72.73%	16.41	$1.13
3	gpt-oss-120b	68.18%	6.46	$0.19
4	Claude Haiku 4.5	59.60%	3.70	$0.52
5	Qwen3 Next 80B A3B	51.01%	10.33	$0.13
6	gpt-oss-20b	50.00%	6.21	$0.13

Appendix

Table 8: GPQA Diamond brute force results (198 samples, 9 models).

Rank	Planner	Solver	Acc	Avg. Lat. (s)	Cost
1	Ministral 3 8B	Claude Opus 4.6	74.27%	4.97	$2.64
2	Claude 3 Haiku	Claude Opus 4.6	73.25%	4.52	$2.79
3	Qwen3 32B	Claude Opus 4.6	73.02%	4.26	$2.65
4	Qwen3 Next 80B A3B	Claude Opus 4.6	72.10%	4.67	$2.67
5	Qwen3 Next 80B A3B	gpt-oss-120b	71.83%	3.07	$0.13
6	Qwen3 32B	gpt-oss-120b	70.04%	2.66	$0.13

Appendix

Table 10: HotpotQA brute force results (199 samples, 81 planner–solver combinations).

Rank	Answer Model	Critic Model	Acc	Avg. Lat. (s)	Cost
1	Claude Opus 4.6	Claude Haiku 4.5	98.84%	16.15	$6.19
2	Claude Opus 4.6	Qwen3 Next 80B A3B	98.82%	14.30	$5.77
3	Claude Opus 4.6	Ministral 3 8B	98.72%	14.03	$5.26
4	Claude Opus 4.6	gpt-oss-20b	98.28%	16.50	$5.93
5	Claude Opus 4.6	gpt-oss-120b	97.77%	15.40	$6.30
6	Claude Opus 4.6	Qwen3 32B	97.28%	15.05	$6.68

Appendix

Table 11: MathQA brute force results (200 samples, 81 answer–critic combinations).

LLM Agent Workflow Optimization

LLM agent workflow optimization studies how to design and run multi-step agents for better task success, reliability, latency, and cost.

See all 243 papers

Aug 7, 2026

The Optimizer Is the Agent: Reasoning-Driven Search across Prompts, Programs, and ML Workflows

Recent systems for optimizing prompts, programs, and ML workflows typically rely on explicit outer-loop controllers such as evolutionary search, bandits, or textual-gradient methods. We ask a fundamentally different question: how much of this search policy can be internalized by a single tool-using agent? We present ReASearch, a unified framework for reasoning-driven optimization in which the agent autonomously decides what to evaluate, how to diagnose failures, which edits to make, and when to verify or restart. Rather than serving only as a proposal generator guided by hand-designed heuristics, the agent actively analyzes outcomes, allocates budget, and refines its strategy over long horizons through persistent memory. With a shared agent loop and domain-specific tools, ReASearch instantiates the exact same scaffold to optimize prompts, programs, and ML workflows. Across 14 diverse tasks, it is competitive with and mostly better than specialized optimization systems, achieving gains of 2% to 40% over strong domain-specific baselines, and in some cases discovering solutions that improve on prior human best-known results. Crucially, we observe that complex search behaviors, which are typically implemented by explicit controllers, emerge naturally from the agent's reasoning process.

Junbo Li, Boyi Liu, Canwen Xu +5

The University of Texas at Austin · Snowflake

Large Language Model-Guided Optimization LLM Agent Workflow Optimization Automatic Prompt Optimization Automated Machine Learning

Ask PDF

Dec 9, 2025

Evolving Excellence: Automated Optimization of LLM-based Agents

Agentic AI systems built on large language models (LLMs) offer significant potential for automating complex workflows, from software development to customer support. However, LLM agents often underperform due to suboptimal configurations; poorly tuned prompts, tool descriptions, and parameters that typically require weeks of manual refinement. Existing optimization methods either are too complex for general use or treat components in isolation, missing critical interdependencies. We present ARTEMIS, a no-code evolutionary optimization platform that jointly optimizes agent configurations through semantically-aware genetic operators. Given only a benchmark script and natural language goals, ARTEMIS automatically discovers configurable components, extracts performance signals from execution logs, and evolves configurations without requiring architectural modifications. We evaluate ARTEMIS on four representative agent systems: the ALE Agent for competitive programming on AtCoder Heuristic Contest, achieving a \textbf{

13.6\%

improvement} in acceptance rate; the Mini-SWE Agent for code optimization on SWE-Perf, with a statistically significant 10.1% performance gain; and the CrewAI Agent for cost and mathematical reasoning on Math Odyssey, achieving a statistically significant \textbf{

36.9\%

reduction} in the number of tokens required for evaluation. We also evaluate the MathTales-Teacher Agent powered by a smaller open-source model (Qwen2.5-7B) on GSM8K primary-level mathematics problems, achieving a 22% accuracy improvement and demonstrating that ARTEMIS can optimize agents based on both commercial and local models.

Paul Brookes, Vardan Voskanyan, Rafail Giavrimis +18

TurinTech AI, London, UK · University of Surrey, Guildford, United Kingdom · University of Leeds, Leeds, UK +4

Large Language Model-Guided Optimization LLM Agent Workflow Optimization LLM Agents Genetic Algorithms

Ask PDF

Apr 3, 2026

Let's Have a Conversation: Designing and Evaluating LLM Agents for Interactive Optimization

Optimization is as much about modeling the right problem as solving it. Identifying the right objectives, constraints, and trade-offs demands extensive interaction between researchers and stakeholders. Large language models can empower decision-makers with optimization capabilities through interactive optimization agents that can propose, interpret and refine solutions. However, it is fundamentally harder to evaluate a conversation-based interaction than traditional one-shot approaches. This paper proposes a scalable and replicable methodology for evaluating optimization agents through conversations. We build LLM-powered decision agents that role-play diverse stakeholders, each governed by an internal utility function but communicating like a real decision-maker. We generate thousands of conversations in a school scheduling case study. Results show that one-shot evaluation is severely limiting: the same optimization agent converges to much higher-quality solutions through conversations. Then, this paper uses this methodology to demonstrate that tailored optimization agents, endowed with domain-specific prompts and structured tools, can lead to significant improvements in solution quality in fewer interactions, as compared to general-purpose chatbots. These findings provide evidence of the benefits of emerging solutions at the AI-optimization interface to expand the reach of optimization technologies in practice. They also uncover the impact of operations research expertise to facilitate interactive deployments through the design of effective and reliable optimization agents.

Joshua Drossman, Alexandre Jacquillat, Sébastien Martin

Operations Research Center and Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA. · Kellogg School of Management, Northwestern University, Evanston, IL

LLM Agent Evaluation Human-in-the-Loop Evaluation

Ask PDF

Abstract

LLM Agent Workflow Optimization

Explore similar work

The Optimizer Is the Agent: Reasoning-Driven Search across Prompts, Programs, and ML Workflows

Evolving Excellence: Automated Optimization of LLM-based Agents

Let's Have a Conversation: Designing and Evaluating LLM Agents for Interactive Optimization