Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling

Zhong Li; Xin Huang; Jinhui Wan; Xiangyi Wang; Shenkai Zhang; Ruiqi Chen; Wenyu Liu; Zaiwen Wen

cs.LGSep 30, 2026

Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling

Authors: Zhong Li, Xin Huang, Jinhui Wan, Xiangyi Wang, Shenkai Zhang, Ruiqi Chen, Wenyu Liu, Zaiwen Wen, +1 more

Organizations: Great Bay University · Beijing Jiaotong University · Beihang University · Peking University

Abstract

Optimization modeling formulates real-world decision problems as mathematical programs that solvers can use to find optimal decisions. Large language models (LLMs) can automate this process, but the resulting correct formulations can require substantial time and memory to construct and solve, limiting practical scalability. Therefore, we systematically investigate whether LLMs can identify problem structure from natural-language descriptions and apply suitable optimization modeling techniques to generate mathematical models and solver code that solve the problems correctly and efficiently. To this end, we first curate OptTips, a knowledge base of 50 expert modeling techniques in eight families. Using this knowledge, we develop OptDachshund, a multi-agent framework that transforms problems from existing optimization benchmarks into new tasks for evaluating LLMs' use of modeling techniques. It constructs conventional and expert mathematical models with solver code for the same task and data, providing baselines for correctness and computational cost. The resulting EfficientOpt benchmark contains 561 expert-reviewed tasks with paired reference implementations. Evaluation of 11 representative LLMs reveals an efficiency gap on correctly solved tasks with comparable measurements: for every LLM, most generated programs take longer to solve than their expert counterparts. Within the comparable reference-size subset, 57% of programs with correct objective values and fewer variables and linear constraints have longer recorded solver times. Case studies show that different modeling techniques can achieve the same optimal value at similar recorded cost. Faster solving may not reduce execution time if the code takes longer to prepare data and build the model. LLM optimization modeling should therefore be evaluated for both correctness and computational efficiency.

Figures & tables

Panel of Figure 3 — Figure 3: OptTips: 50 cards in eight families, with two abbreviated examples showing the seven-field structure. Full card examples appear in Appendix J (Figures 12 and 13 ).

Panel of Figure 5 — Figure 5: Task composition and reference model sizes for the 561 tasks in EfficientOpt. (a–c) Problem classes, formulation types of ordinary references, and application fields. (d,e) Variable and constraint counts for ordinary (blue, above) and expert (orange, below) references, shown on logarithmic horizontal axes. Appendix B gives the classifications and counting rules. (The visual presentation is inspired by the elegant design of Fig. 1 in Kong et al. (2026a) .)

Model	Count $\uparrow$	Rate (%) $\uparrow$	Ordinary $\downarrow$	Expert $\downarrow$	Expert $\downarrow$	Expert $\downarrow$
	Correct answers		Cost ratio (LLM/reference)
			Runtime		Work	Build+opt
Gemini 3.1 Pro	512	91.3	0.52	1.68	1.98	1.43
GPT-5.5	447	79.7	0.58	1.79	2.08	1.64
Claude Opus 4.6	442	78.8	0.46	1.49	1.77	1.42
DeepSeek-V4 Flash	441	78.6	0.49	1.62	1.78	1.47

Table 1: Numerical accuracy and computational cost. Accuracy uses all 561 tasks. Cost ratios use complete measurements on correctly solved tasks within the 543-task reference cost subset (LLM/reference; below 1 means lower cost). See Appendix D for metric definitions and cost-ratio calculations.

Panel of Figure 6 — Figure 6: Runtime distributions and cumulative time. Bars show the percentage of runs in each time interval (left axis). Blue curves show the share of total Runtime used by runs at or below each runtime (right axis). Orange highlights the slowest $\sim$ 10%; arrows show their contribution to total time. Only correctly solved tasks with valid timings are included. See Appendix C.3 for per-model runtime statistics.

Panel of Figure 7 — Figure 7: Size and technique use. (a) Among correct programs with smaller formulations, how many are slower than the expert? Labels give slower/total; the striped bar combines all 11 LLMs. (b) Technique-use labels for 6,171 programs. Appendix C.4 analyzes size and runtime; Appendix F.1 details technique use.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Primary problem category	$n$	Application field	$n$
Resource allocation & production	177	Mobility & logistics	114
Routing & network flow	121	Manufacturing & industry	106
Packing & selection	88	Unspecified	76
Covering & facility planning	68	Public services & research	61
Assignment & matching	45	Business & resource planning	47
Scheduling & control	37	Health & life sciences	46

Appendix

Table 2: Task counts by problem category (left) and application field (right), each totaling 561. Figure 5 (a) combines fitting and estimation with layout planning as Other (25 tasks).

	Mutually exclusive outcomes			Solver status
Model	Correct	Objective mismatch	Other outcome	OPTIMAL
Gemini 3.1 Pro	512	10	39	522
GPT-5.5	447	23	91	470
Claude Opus 4.6	442	36	83	478
DeepSeek-V4 Flash	441	25	95	466
Kimi K2.6	433	23	105	456

Appendix

Table 3: Numerical outcomes on all 561 tasks per model. Correct, Objective mismatch, and Other outcome sum to 561. The OPTIMAL column is the sum of Correct and Objective mismatch.

Panel of Table 4 — Table 4: Task groups and numerical accuracy. $n$ is the number of tasks. Correct gives the number of correctly solved cases out of all $11n$ model–task pairs; the last column shows the corresponding percentage. Descriptions list the modeling variants represented in each group.

	Runtime (s)			Work	Build	Build+opt	Shared Runtime
Model	Median	Mean	P90	Median	Median	Median	Mean ( $n=47$ )
Gemini 3.1 Pro	32.51	197.66	414.53	22.66	0.63	37.80	77.04
GPT-5.5	28.84	208.50	671.57	21.57	0.97	37.77	197.44
Claude Opus 4.6	23.52	177.59	369.99	18.84	0.87	37.38	130.45
DeepSeek-V4 Flash	23.11	190.92	430.92	18.19	0.82	32.71	109.45
Kimi K2.6	29.70	210.28	599.96	20.36	0.76	36.10	166.83

Appendix

Table 5: Recorded costs on correctly solved tasks within the reference cost subset. Each metric uses that model’s available measurements; medians are calculated separately. The final column uses the same 47 tasks for all 11 models and the expert reference. Times are in seconds; Work is in solver work units.

Panel of Figure 9 — Figure 9: How long runs contribute to total solver time. The three models have the lowest, middle, and highest shares of time spent in their slowest tenth. Top: runtime histograms on a logarithmic time axis, with the slowest $\lceil n/10\rceil$ runs highlighted. Bottom: cumulative share of solver time as runs are ordered from fastest to slowest; shading marks the same slowest runs. Each model uses its correctly solved tasks with valid timings.

Models with correct answers	Tasks	Runtime pairs	/Expert $\downarrow$	Slower (%)
1–3	26	67	1.69	58.2
4–7	192	1,169	1.58	57.5
8–10	276	2,449	1.67	66.8
11 (all models)	47	517	2.82	82.2

Appendix

Table 6: Solver costs by the number of models answering each task correctly. The four groups contain 541 tasks in the reference cost subset, each solved correctly by at least one model. /Expert is the shifted geometric Runtime ratio (Appendix D.2 ). Slower (%) is the percentage of valid LLM–expert pairs with longer LLM runtime.

Panel of Figure 10 — Figure 10: Model comparisons on shared tasks and runtime distributions. (a) Percentage of tasks solved correctly by both models with valid timings on which the row model is faster than the column model ( $10^{-6}$ s tie tolerance; 126–420 tasks per pair). Opposite cells need not sum to 100% because ties remain in the denominator. (b) Median, mean, and 90th-percentile Runtime on tasks solved correctly by each model, and the slowest tenth’s share of measured Runtime . Whiskers summarize the distribution; they are not confidence intervals. Model order follows Table 1 .

		No larger on three counts			Fewer variables and rows
Model	Eligible	$n$	Slower	%	$n$	Slower	%
Gemini 3.1 Pro	465	100	55	55.0	45	24	53.3
GPT-5.5	401	45	26	57.8	19	11	57.9
Claude Opus 4.6	395	82	44	53.7	41	19	46.3
DeepSeek-V4 Flash	395	59	31	52.5	29	10	34.5
Kimi K2.6	390	70	40	57.1	28	13	46.4

Appendix

Table 7: Model size and solver time relative to the expert reference. Eligible counts pairs before applying the size criteria;

n

counts pairs meeting each criterion. Slower counts pairs where LLM Runtime exceeds the expert’s runtime by more than

10^{-6}

seconds; percentages use

n

as the denominator. Rows are linear constraints; nonzeros are nonzero coefficients in their constraint matrix.

Model	LLM runs	Build longer $n$ (%)	LLM–expert pairs	Faster to slower	Slower to faster
Gemini 3.1 Pro	504	58 (11.5)	499	0	49
GPT-5.5	441	73 (16.6)	435	8	33
Claude Opus 4.6	435	76 (17.5)	430	21	29
DeepSeek-V4 Flash	433	88 (20.3)	428	14	18
Kimi K2.6	427	71 (16.6)	423	14	22
Qwen 3.6 27B	391	88 (22.5)	386	24	15

Appendix

Table 8: Preparation time and comparisons with the expert. LLM runs have recorded preparation and solver-call times; LLM–expert pairs also have the corresponding expert measurements. Build longer counts runs where preparation exceeds solver-call time, with percentages over LLM runs. The last two columns count changes in the LLM’s faster/slower relation to the expert when preparation is included.

Model	$n_{A}/n_{T}$	API latency (s)	Prompt	Completion	Reasoning	$n_{R}$
Gemini 3.1 Pro	559/558	55.4	743.5	6,714.5	5,827	558
GPT-5.5	535/534	61.2	715.5	1,431	516	522
Claude Opus 4.6	561/561	17.2	937	748	0	561
DeepSeek-V4 Flash	561/561	81.9	714	11,327	10,562	561
Kimi K2.6	561/561	82.2	714	11,680	10,714	561
Qwen 3.6 27B	558/558	194.9	724	8,319	8,193	190

Appendix

Table 9: Generation latency and token counts for all available responses, including incorrect programs. Values are medians.

n_{A}

n_{T}

, and

n_{R}

count records with latency, input/output-token, and reasoning-token measurements, respectively. Prompt and Completion denote input and output tokens. Token definitions vary by provider; reasoning tokens may already be included in Completion, and a reported zero does not establish that no internal reasoning occurred.

	All correct cost-analysis runs			Matched subset with process RSS
		Gurobi peak			RSS snapshot max		Gurobi peak
Model	$n$	Median	P90	$n$	Median	P90	Median	P90
Gemini 3.1 Pro	504	124.7	2,397.9	210	79.2	2,011.6	72.0	2,403.3
GPT-5.5	441	136.3	2,198.6	186	70.8	2,069.4	58.7	2,440.1
Claude Opus 4.6	435	118.4	2,510.0	176	78.8	2,082.5	57.8	2,507.5
DeepSeek-V4 Flash	433	130.4	1,938.5	184	75.9	1,234.1	69.6	1,660.2

Appendix

Table 10: Solver and process memory in MiB. The full set contains all 4,202 correct runs used for cost analysis. The matched subset contains 1,769 runs with both solver memory and process RSS sampled before preparation, before solving, and after solving; RSS is the largest of these three snapshots. Gurobi peak measures peak allocation within the solver environment. P90 is the 90th percentile. Overall pools runs across models; each model uses its own correctly solved tasks.

Model	Correct $n$ (%)	OPTIMAL	Static failure	Code error	Time limit	Memory limit	Other
OptMATH (32B)	137 (24.42)	189	11	212	51	57	41
SIRL (32B)	118 (21.03)	188	34	164	77	72	26
LLMOPT (14B)	36 (6.42)	77	20	377	28	35	24
OptMATH (7B)	1 (0.18)	4	401	146	3	6	1
SIRL (7B)	0 (0.00)	0	504	57	0	0	0

Appendix

Table 11: Accuracy and execution outcomes for five fine-tuned models, with 561 tasks per model. Correct counts programs that finish with OPTIMAL status and the verified objective value. It is a subset of OPTIMAL , not an additional outcome category. The six columns from OPTIMAL through Other sum to 561 in each row.

Model	$n$	Median (s)	Total (h)	$k$	Time share (%)	Slower	LLM/expert
	All eligible runs			Slowest $\sim$ 10%
Gemini 3.1 Pro	504	32.5	27.67	51	72.4	48/51	5.43
GPT-5.5	441	28.8	25.54	45	70.2	43/45	7.71
Claude Opus 4.6	435	23.5	21.46	44	72.7	42/44	5.31
DeepSeek-V4 Flash	433	23.1	22.96	44	72.9	43/44	5.80
Kimi K2.6	427	29.7	24.94	43	68.4	42/43	4.97

Appendix

Table 12: Solver-time concentration for each LLM. Time share is the selected

k

runs’ percentage of total Runtime . Slower gives the number exceeding expert time by more than

10^{-6}

seconds, out of

k

. LLM/expert divides summed LLM time by summed expert time on these same

k

tasks; it is not the shifted geometric ratio in Table 1 .

Model	Task	LLM (s)	Expert (s)	Share (%)
Gemini 3.1 Pro	T24_002	4,677.80	303.66	4.7
GPT-5.5	T25_019	4,071.28	16.38	4.4
Claude Opus 4.6	T48_009	4,718.84	22.63	6.1
DeepSeek-V4 Flash	T15_012	5,431.07	18.95	6.6
Kimi K2.6	T06_001	4,324.49	2,905.46	4.8
Qwen 3.6 27B	T07_003	7,065.35	15.21	8.4

Appendix

Table 13: The slowest eligible run for each LLM and the expert’s runtime on the same task. Times are Gurobi Runtime . Share is the selected run’s percentage of that LLM’s total eligible solver time. Runs are selected by LLM time, irrespective of the expert time.

Quantity	Recorded measurement	What it captures
Numerical check	OPTIMAL status and a finite objective matching the verified reference value	Numerical agreement on the tested instance; full task semantics and required decisions need separate verification.
Technique use	Automated labels from code and task context	Identifies target, alternative, none, or N/A; does not establish correctness or efficiency (Appendix F.1 ).
Solver cost	Gurobi Runtime (s), Work (units)	Time and solver effort for the recorded optimization calls.
Build+opt	Preparation time + separately timed solver calls (s)	An execution subtotal; excludes work outside the recorded intervals.
Generation	API latency (s); provider-reported Tokens	Waiting time for the final recorded call; token usage reported for its response. Earlier calls are excluded.
Model size	Variables, constraints, linear-matrix nonzeros	Dimensions before presolve. Figure 5 and Appendix C.4 use different counting scopes, explained below.

Appendix

Table 14: Evaluation metrics and what they measure.

Panel of Figure 11 — Figure 11: Dataset quality control and expert verification. Checks cover required decision recovery and valid termination, not objective agreement alone. Repairs trigger all affected checks. Appendix B.2 summarizes reference-answer verification and cost coverage.

	LLM technique label				Combined diagnostic category
Model	Target	Alt.	None	N/A	C1	C2	C3	C4	C5
Claude Opus 4.6	311	131	98	21	253	52	66	68	122
Gemini 3.1 Pro	297	173	79	12	272	67	99	73	50
DeepSeek-V4 Flash	296	139	100	26	247	62	62	71	119
Qwen 3.6 27B	286	132	128	15	196	44	75	79	167
GPT-5.5	284	147	106	24	223	68	71	82	117

Appendix

Table 15: Technique-use assessments by LLM. Each model row classifies the same 561 records in two ways: technique labels (left) and combined diagnostic categories (right; Appendix F.2 ). The two column blocks each sum to 561 and must not be added together.

ID	Target OptTips group	$n$	Target (%)	Alt.	None	N/A
T01	Equality/inequality transformations	352	283 (80.4)	10	39	20
T02	Auxiliary variables and epigraphs	165	156 (94.5)	0	1	8
T03	Bound tightening and scaling	209	12 (5.7)	115	79	3
T04	LP relaxation	275	23 (8.4)	94	147	11
T05	Big- $M$ modeling	209	171 (81.8)	7	10	21
T06	Indicators, semicontinuous variables, SOS	231	58 (25.1)	149	15	9

Appendix

Table 16: Automated technique labels by OptTips group.

n

includes all model–task records from 11 LLMs, including failed executions. Target gives count (% of

n

); other columns give counts.

Task and output	Implemented choice	What the evidence supports
T10_011, GPT-5.5	Binary pairwise products with separate product inequalities; the expert uses continuous interactions with row–column marginals.	Both formulations enforce the same products at integer assignments, but their LP relaxations differ. The GPT model does not use the expert’s marginal formulation.
T13_006, Gemini	Primal resource-allocation LP with explicit implied upper bounds; the expert uses a dual LP.	The approaches attain the same objective with similar recorded Build+opt times.
T20_011, Kimi / Qwen 3.5 122B	Kimi uses cumulative auxiliary variables; Qwen uses direct coverage constraints.	Eliminating Kimi’s auxiliaries preserves feasible integer shift-start decisions and the objective. Qwen’s longer construction time largely offsets its solver-time advantage.

Appendix

Table 17: Selected modeling choices (RQ3). The cases are illustrative, not a random sample.

Code	Methodological family	Cards
A	Formulation standardization and dimension reduction	6
B	Relaxation strengthening and convexification	5
C	Duality, optimality, and equilibrium	4
D	Specialized combinatorial and structured models	8
E	Discrete and logical modeling	4
F	Convex, nonlinear, and structure-preserving modeling	9

Appendix

Table 18: Primary reporting families of OptTips. Codes identify the assignments in Table 19 ; techniques can apply across families.

Panel of Figure 12 — Figure 12: OptTips cards for families A–D. Each expanded card includes all seven components of the schema. Paired models highlight the transformation, with shared terms described in the card. Panel (b) retains the GPT-5.5–expert comparison from Figure 1 ; the other panels use ordinary and expert references. Recorded solver Runtime values have matching reported objective values.

Panel of Figure 13 — Figure 13: OptTips cards for families E–H. The same seven-component schema continues with four further examples. Panel (g) retains the GPT-5.5–expert comparison from Figure 1 ; other panels use ordinary and expert references. Times report solver Runtime under the execution protocol in Appendix D ; panel (h) lacks historical input-hash verification.

ID	Technique	Family	ID	Technique	Family
T01	Equality/inequality equivalent transformation	A	T26	Complementarity and KKT/MPEC modeling	E
T02	Auxiliary-variable epigraph/hypograph modeling	A	T27	Multi-objective and lexicographic optimization	H
T03	Bound tightening and scaling	A	T28	Penalty, slack, and feasibility-relaxation modeling	H
T04	LP relaxation of IP/MIP	E	T29	Solver-native modeling constructs	H
T05	Big- $M$ modeling	E	T30	Warm starts and incumbent injection	H
T06	Indicator, semicontinuous, and SOS constraints	E	T31	Algorithm–structure matching	H

Appendix

Table 19: All 50 OptTips techniques. Family codes refer to Table 18 . The catalog includes formulation, algorithmic, and solver-aware choices; full cards are available in the public repository.

Explore similar work

OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale
Jul 29, 2024Ali AhmadiTeshnizi, Wenzhi Gao, Herman Brunborg +3Optimization Modeling Solvers
a School of Management Science and Engineering, Stanford University, Stanford, California 94305 · b Institute for Computational and Mathematical Engineering, Stanford University, Stanford, California 94305
Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems
Jul 12, 2026Yongchang Fu, Xinjie Huang, Chengjun Dai +3Optimization Modeling Agentic Benchmarks
Ding Talk, Alibaba Group, Hangzhou, China · Zhejiang University, Hangzhou, China
OR for AI That Does OR: Routing LLMs up the Escalator inside the OSCAR Framework
Oct 1, 2026Jinzhi Bu, Haixin Tang, Huanan ZhangOptimization Modeling
Department of Logistics and Maritime Studies, The Hong Kong Polytechnic University, Hong Kong · Leeds School of Business, University of Colorado Boulder, Boulder, CO, 80309, USA

Jul 29, 2024, cs.AIJ/K move · Enter open · S save

OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale

Ali AhmadiTeshnizi, Wenzhi Gao, Herman Brunborg, Shayan Talaei+2

a School of Management Science and Engineering, Stanford University, Stanford, California 94305 · b Institute for Computational and Mathematical Engineering, Stanford University, Stanford, California 94305

Optimization Modeling Solvers

Open paper AskPDF arXiv

Optimization problems are pervasive in sectors from manufacturing and distribution to healthcare. However, most such problems are still solved heuristically by hand rather than optimally by state-of-the-art solvers because the expertise required to formulate and solve these problems limits the widespread adoption of optimization tools and techniques. We introduce a Large Language Model (LLM)-based system designed to formulate and solve (mixed integer) linear programming problems from their natural language descriptions. Our system can develop mathematical models, write and debug solver code, evaluate the generated solutions, and improve efficiency and correctness of its model and code based on these evaluations. OptiMUS is designed as a productivity tool for optimization practitioners who understand the problem domain and can describe it precisely, but seek to accelerate the modeling and implementation workflow. OptiMUS-0.3 utilizes a modular structure to process problems, allowing it to handle problems with long descriptions and complex data without long prompts. Experiments demonstrate that OptiMUS-0.3 outperforms direct-prompting baselines by over 43% on easy and 18% on hard instances. It remains competitive with fine-tuned specialist models on benchmark problems, and outperforms them on real-world case studies (28.6% vs. 0%) where fine-tuned models fail to generalize. Ablation studies show that modular architecture with error correction is central to these gains. A key finding is that system architecture is a stronger driver of performance than model capability. Structured decomposition with targeted error correction enables weaker models to match stronger models under naive prompting, and remains competitive with fine-tuned specialist models without retraining costs.

Abstract

Explore similar work

OptiMUS-0.3: Using Large Language Models to Model and Solve Optimization Problems at Scale

Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems

OR for AI That Does OR: Routing LLMs up the Escalator inside the OSCAR Framework