Let's Have a Conversation: Designing and Evaluating LLM Agents for Interactive Optimization

Joshua Drossman; Alexandre Jacquillat; Sébastien Martin

cs.AIApr 3, 2026

Let's Have a Conversation: Designing and Evaluating LLM Agents for Interactive Optimization

Authors: Joshua Drossman, Alexandre Jacquillat, Sébastien Martin

Organizations: Operations Research Center and Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA. · Kellogg School of Management, Northwestern University, Evanston, IL

Abstract

Optimization is as much about modeling the right problem as solving it. Identifying the right objectives, constraints, and trade-offs demands extensive interaction between researchers and stakeholders. Large language models can empower decision-makers with optimization capabilities through interactive optimization agents that can propose, interpret and refine solutions. However, it is fundamentally harder to evaluate a conversation-based interaction than traditional one-shot approaches. This paper proposes a scalable and replicable methodology for evaluating optimization agents through conversations. We build LLM-powered decision agents that role-play diverse stakeholders, each governed by an internal utility function but communicating like a real decision-maker. We generate thousands of conversations in a school scheduling case study. Results show that one-shot evaluation is severely limiting: the same optimization agent converges to much higher-quality solutions through conversations. Then, this paper uses this methodology to demonstrate that tailored optimization agents, endowed with domain-specific prompts and structured tools, can lead to significant improvements in solution quality in fewer interactions, as compared to general-purpose chatbots. These findings provide evidence of the benefits of emerging solutions at the AI-optimization interface to expand the reach of optimization technologies in practice. They also uncover the impact of operations research expertise to facilitate interactive deployments through the design of effective and reliable optimization agents.

Figures & tables

Panel of Figure 1 — Figure 1: Visualization of approaches to optimization in practice.

Panel of Figure 2 — Figure 2: Example of LLM-based autoformulation: a school principal describes an optimization problem in precise technical language and the tool returns a mathematical formulation of the problem.

Panel of Figure 3 — Figure 3: Example of a conversational interaction: the principal is far less precise in their goals and requirements and the tool navigates the problem’s landscape to guide them to a good solution.

Panel of Figure 4 — Figure 4: LLM-optimization evaluation frameworks. One-shot evaluation: an optimization agent creates a formulation based on one question, and aims to match a ground truth. Empirical evaluation: an optimization agent is scored based on interactions with human decision-makers. Our approach: an optimization agent is scored based on interactions with decision agents based on verifiable and rigorous criteria.

Panel of Figure 5 — Figure 5: The optimization agent’s architecture includes an LLM, prompt, and toolkit, allowing the agent to read/write (R/W) from/to the model, pass the model to the solver, derive its output, and summarize the solution for the decision-maker. This illustration corresponds to the T-P-P design.

Name	Model Interaction	Prompt	Solution Processing
C-N-R	Code (Python)	No	No
C-P-R	Code (Julia)	Yes	No
C-P-P	Code (Julia)	Yes	Yes
T-P-P	Tools	Yes	Yes

Table 1: Controlled experimental setup: four optimization agent designs with increasing levels of structure.

	Default Schedule		Baseline One-Shot		Optimization-Aware One-Shot		Conversation
Comm. style	Avg. score	Success rate	Avg. score	Success rate	Avg. score	Success rate	Avg. score	Success rate
Vague	-	-	0.67	15.6%	0.71	19.3%	0.91	56.9%
Precise	-	-	0.73	17.9%	0.79	33.5%	0.95	73.4%
All	0.56	8.3%	0.70	16.7%	0.75	26.3%	0.93	65.1%

Table 2: Performance of the T-P-P optimization agent under three evaluation paradigms.

Panel of Figure 6 — Figure 6: Distribution of scores when evaluating the T-P-P agent under two evaluation paradigms.

		T-P-P		C-P-P		C-P-R		C-N-R
Comm. Style	check_utility Implementation	Avg. Score	Success rate	Avg. Score	Success rate	Avg. Score	Success rate	Avg. Score	Success rate
Vague	Binary (Y/N)	0.87	44.0%	0.86	42.2%	0.82	35.8%	0.70	24.8%
	Rich	0.94	69.7%	0.88	56.0%	0.85	52.3%	0.83	37.6%
Precise	Binary (Y/N)	0.95	70.6%	0.89	53.2%	0.85	49.5%	0.82	32.1%
	Rich	0.95	76.1%	0.90	67.9%	0.85	63.3%	0.82	31.1%
All		0.93	65.1%	0.88	54.8%	0.84	50.2%	0.81	31.4%

Table 3: Solution performance of four designs of optimization agents.

Panel of Figure 7 — Figure 7: Distribution of scores for each optimization agent by communication style.

Panel of Figure 8 — Figure 8: Example of interaction between a decision agent and the C-P-P optimization agent.

Panel of Figure 10 — Figure 10: Example of interaction between a decision agent and the T-P-P optimization agent.

Comm. Style	check_utility Implementation	T-P-P	C-P-P	C-P-R	C-N-R
Vague	Binary (Y/N)	8.52	7.36	8.20	10.74
Vague	Rich	6.17	5.96	6.31	0 8.83
Precise	Binary (Y/N)	6.11	6.06	6.05	0 8.74
Precise	Rich	5.29	5.00	4.67	0 7.04
All		6.52	6.10	6.31	0 8.84

Table 4: Average conversation lengths.

Figure 11: Distribution of conversation length for each optimization agent across decision agents’ personalities.

	Avg. score			Success rate			T–P–P behavior
Model	T-P-P	C-N-R	% $\Delta$	T-P-P	C-N-R	$\Delta$ (pp)	Number of Turns	Calls to Solver
GPT-3.5-turbo	0.83	0.61	36.1%	34.4%	11.0%	23.4pp	12.61	4.60
GPT-4.1	0.93	0.81	14.8%	65.1%	31.4%	33.7pp	6.52	6.23
GPT-5	0.93	0.88	0 5.7%	63.3%	44.7%	18.6pp	6.43	13.35

Table 5: Performance of T–P–P vs. C–N–R and T–P–P interaction behavior across GPT model generations.

ID	School Name	Grade Level	Total Enrollment	Current Start
1	Muir (John) PK	PK	242	9:30 AM
2	Ortega (Jose) PK	PK	399	9:20 AM
3	McCoppin (Frank) PK	PK	225	9:20 AM
4	Transition Training Center (Access)	HS	5	8:00 AM
5	Balboa HS	HS	1226	8:15 AM
6	Galileo HS	HS	1851	8:00 AM

Table 6: School enrollment and current start times.

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems
Jul 12, 2026Yongchang Fu, Xinjie Huang, Chengjun Dai +3Large Language Model-Guided Optimization LLM Agent Evaluation
Ding Talk, Alibaba Group, Hangzhou, China · Zhejiang University, Hangzhou, China
ParetoChat: An LLM-Based Interface for Interactive Multi-Objective Optimization
Oct 16, 2026Gokhan CeyhanMulti-Objective Optimization Multi-Criteria Decision Analysis
Just Eat Takeaway.com
From Soliloquy to Agora: Memory-Enhanced LLM Agents with Decentralized Debate for Optimization Modeling
Apr 28, 2026Jianghao Lin, Zi Ling, Chenyu Zhou +4Multi-Agent Debate Multi-Agent LLMs
Antai College of Economics and Management, Shanghai Jiao Tong University · University of Chicago Booth School of Business · School of Economics and Management, Tsinghua University +1

Jul 12, 2026J/K move · Enter open · S save

Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems

Yongchang Fu, Xinjie Huang, Chengjun Dai, Chengzhe Feng+2

Ding Talk, Alibaba Group, Hangzhou, China · Zhejiang University, Hangzhou, China

Large Language Model-Guided Optimization LLM Agent Evaluation AI Agent Benchmarks Natural Language to Optimization Modeling

Open paper AskPDF arXiv

LLM-based agents are increasingly deployed to solve optimization problems, yet existing benchmarks evaluate them on pre-structured mathematical formulations that bypass the most critical challenge: translating complex business requirements into correct models and solve efficiently. We introduce Opti-Agent-Bench, an end-to-end benchmark that evaluates Large Language Models (LLMs) across the complete optimization R&D pipeline, from understanding business-language descriptions through mathematical modeling, algorithm selection, and code implementation, to solution report generation. Our design rests on three pillars: (1) businesssemantic authenticity with anti-template traps that defeat pattern matching; (2) modular evaluation with cross-module consistency checking across Problem Understanding, Formal Modeling, Implementation, and Reporting; and (3) the ORAC bi-level validity framework that simultaneously ensures task quality and scoring integrity. Across several industrialscale tasks spanning integer programming, robust optimization, stochastic programming, and non-convex optimization, we expose critical failure modes of current models, including constraint omission, model-code inconsistency, and report-implementation divergence, that remain invisible under conventional single-metric evaluation.

Abstract

LLM Agent Evaluation

Explore similar work

Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems

ParetoChat: An LLM-Based Interface for Interactive Multi-Objective Optimization

From Soliloquy to Agora: Memory-Enhanced LLM Agents with Decentralized Debate for Optimization Modeling