Harnessing Language Models as Optimization Modeling Agents

Yunxiang Luo; Xin Xu; Tianhao Chen; Zhixi Xiong; Peiqi Jiang; Fan Zhang; Wei Shao; Xuejian Gao

cs.AIOct 22, 2026

Harnessing Language Models as Optimization Modeling Agents

Authors: Yunxiang Luo, Xin Xu, Tianhao Chen, Zhixi Xiong, Peiqi Jiang, Fan Zhang, Wei Shao, Xuejian Gao,

Organizations: The Hong Kong University of Science and Technology · Huawei

Abstract

Large language models (LLMs) can translate natural-language optimization problems into solver programs, but reliably producing correct formulations and implementations remains challenging. Existing approaches improve optimization modeling through agentic workflows or task-specific post-training. This raises a practical question: can a lightweight harness using a single model role achieve competitive performance without additional training? We introduce OPTICYCLE and OPTISTATE, two lightweight inference harnesses for optimization modeling, and evaluate them on nine benchmarks comprising 2,634 problems using 11 Qwen variants spanning 0.6B-27B parameters. Our results show that execution-guided revision consistently improves macro accuracy over direct generation for all evaluated models, with gains of up to 49.46 percentage points. Despite their simplicity, both harnesses outperform six adapted agentic workflows in same-model comparisons and four evaluated post-trained optimization models at comparable model sizes, without additional training. Explicit formulation state provides further but model-dependent improvements, suggesting that additional structure is not universally beneficial.

Figures & tables

Figure 1: Three inference interfaces for the same frozen model. OptiCycle adds execution feedback and persistent Python state; OptiState additionally maintains a revisable five-element formulation. The hidden grader evaluates submitted answers and provides no feedback during inference.

Method	Complex OR	Industry OR	Opti Bench	MAMO easy	MAMO complex	NL4Opt	NLP4LP	OptMath	ICML Comp.	Macro
	avg@64	avg@16	avg@4	avg@4	avg@8	avg@8	avg@8	avg@8	avg@8	Avg.
Qwen3-4B
Direct Generation	29.08	33.88	54.38	80.06	27.61	72.45	72.88	9.41	72.62	50.26
CAFA	46.09	35.06	60.79	79.49	30.15	79.24	69.16	11.14	76.77	54.21
Reflexion	43.84	25.50	50.99	82.52	13.27	77.17	70.71	7.15	70.52	49.07
Chain-of-Experts	28.30	23.31	48.84	71.09	6.93	71.25	72.68	4.14	65.12	43.52

Table 1: Benchmark accuracy (%) for Direct Generation, adapted agent baselines, and our inference harnesses with shared backbones. Values are avg@

k

; Macro is the unweighted mean over nine benchmarks. Bold and underline mark the highest and second-highest values within each backbone and column, respectively, including ties.

Figure 2: Accuracy gains and token consumption across nine benchmarks. Left, upper: OptiCycle gains over direct generation; left, lower: OptiState gains over OptiCycle, in percentage points. Background bands separate model series; the gain panels use different horizontal scales and differences between rounded scores. Right: macro accuracy versus mean recorded input plus output tokens per trajectory on Qwen3.5-9B, with both metrics averaged equally across benchmarks. Dashed lines mark OptiCycle’s accuracy and token consumption.

Figure 3: Model-size scaling within the Qwen3 (left) and Qwen3.5 (right) families. Model size is shown in billions of parameters on a logarithmic scale; accuracy is the macro average across nine benchmarks. Blue solid lines with circles denote OptiCycle; orange dashed lines with squares denote OptiState. Shaded bands mark three qualitative regimes: Capability Bottleneck, State Advantage, and Method Convergence.

Model	direct	OptiCycle	OptiState
Qwen3-0.6B	0.06%	0.01%	0.01%
Qwen3-1.7B	2.93%	13.28%	26.79%
Qwen3-4B	50.26%	62.74%	64.22%
Qwen3-8B	58.01%	65.49%	65.00%
Qwen3-14B	17.53%	66.99%	66.38%
Qwen3.5-0.8B	0.00%	2.48 %	0.71%

Table 2: Macro-average accuracy (%) across nine optimization benchmarks.

\dagger

marks evaluations with missing scores counted as failures; see Appendix A.1 .

Figure 4: Macro accuracy (%) across nine benchmarks, ordered by accuracy. Blue and orange highlight OptiCycle and OptiState; gray denotes released models. Bold and underline mark the highest and second-highest values, respectively.

Figure 5: Macro accuracy (%) across nine benchmarks on non-Qwen models, including GPT-OSS-20B. Values are unweighted means of benchmark avg@

k

scores.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Trajectories per problem ( $k$ )
ComplexOR	64
IndustryOR	16
OptiBench	4
MAMO-easy	4
MAMO-complex	8
NLP4LP	8

Appendix

Table 3: Per-problem sampling counts used to estimate average accuracy.

Method	Adapted procedure	Budget
Direct Generation	Generate a complete solver program and execute it once.	One model call
CAFA	Use compact OR-specific prompting and examples.	One model call
Reflexion	Generate tests and an initial program, execute it, reflect on the observations, and revise.	At most one revision
Chain-of-Experts	A conductor selects three of seven expert types; a reducer combines their outputs.	Three experts and a reducer
OptiMUS	A manager schedules Formulator, Programmer, and Evaluator roles over structured state.	At most ten scheduling steps
AutoFormulation	Use UCT search over parameters, variables, the objective, equalities, and inequalities.	Eight search steps; branching factor three

Appendix

Table 4: Implemented workflows and method-specific inference budgets.

Qwen3-0.6B
Method	Complex OR	Industry OR	OptiBench	MAMO easy	MAMO complex	NLP4LP	NL4Opt	ICML Comp.	OptMath	Macro
	avg@64	avg@16	avg@4	avg@4	avg@8	avg@8	avg@8	avg@8	avg@8	Avg.
Direct Generation	0.00	0.12	0.04	0.00	0.18	0.05	0.00	0.00	0.15	0.06
OptiCycle	0.00	0.06	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.01
OptiState	0.00	0.00	0.04	0.04	0.00	0.00	0.05	0.00	0.00	0.01
Qwen3-1.7B

Appendix

Table 5: Qwen3 benchmark accuracy (%). Model blocks compare the three inference interfaces.

Qwen3.5-0.8B
Method	Complex OR	Industry OR	OptiBench	MAMO easy	MAMO complex	NLP4LP	NL4Opt	ICML Comp.	OptMath	Macro
	avg@64	avg@16	avg@4	avg@4	avg@8	avg@8	avg@8	avg@8	avg@8	Avg.
Direct Generation	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
OptiCycle	0.43	0.31	2.73	8.82	0.00	3.31	3.21	3.29	0.23	2.48
OptiState	0.17	0.00	0.74	2.80	0.06	0.72	0.82	1.07	0.00	0.71
Qwen3.5-2B

Appendix

Table 6: Qwen3.5 and Qwen3.6 benchmark accuracy (%), grouped by model.

\dagger

marks evaluations with missing scores counted as failures.

Large Language Model-Guided Optimization

Large Language Model-Guided Optimization uses language models to propose, steer, and refine solutions to hard search and design problems.

See all 605 papers

Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling
Sep 30, 2026Zhong Li, Xin Huang, Jinhui Wan +6LLM Evaluation Large Language Model-Guided Optimization
Great Bay University · Beijing Jiaotong University · Beihang University +1
Strategy-Aware Optimization Modeling with Reasoning LLMs
May 4, 2026Ruiqing Zhao, Fengzhi Li, Yuan Zuo +5RL for Code Generation LLM-Based Program Synthesis
School of Computer Science and Engineering, Beihang University, Beijing, China · JIUTIAN Research, Beijing, China · MIIT Key Laboratory of Data and Decision Intelligence, Beihang University, Beijing, China
Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems
Jul 12, 2026Yongchang Fu, Xinjie Huang, Chengjun Dai +3Large Language Model-Guided Optimization LLM Agent Evaluation
Ding Talk, Alibaba Group, Hangzhou, China · Zhejiang University, Hangzhou, China

Sep 30, 2026J/K move · Enter open · S save

Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling

Zhong Li, Xin Huang, Jinhui Wan, Xiangyi Wang+5

Great Bay University · Beijing Jiaotong University · Beihang University+1

LLM Evaluation Large Language Model-Guided Optimization Natural Language to Optimization Modeling

Open paper AskPDF arXiv

Optimization modeling formulates real-world decision problems as mathematical programs that solvers can use to find optimal decisions. Large language models (LLMs) can automate this process, but the resulting correct formulations can require substantial time and memory to construct and solve, limiting practical scalability. Therefore, we systematically investigate whether LLMs can identify problem structure from natural-language descriptions and apply suitable optimization modeling techniques to generate mathematical models and solver code that solve the problems correctly and efficiently. To this end, we first curate OptTips, a knowledge base of 50 expert modeling techniques in eight families. Using this knowledge, we develop OptDachshund, a multi-agent framework that transforms problems from existing optimization benchmarks into new tasks for evaluating LLMs' use of modeling techniques. It constructs conventional and expert mathematical models with solver code for the same task and data, providing baselines for correctness and computational cost. The resulting EfficientOpt benchmark contains 561 expert-reviewed tasks with paired reference implementations. Evaluation of 11 representative LLMs reveals an efficiency gap on correctly solved tasks with comparable measurements: for every LLM, most generated programs take longer to solve than their expert counterparts. Within the comparable reference-size subset, 57% of programs with correct objective values and fewer variables and linear constraints have longer recorded solver times. Case studies show that different modeling techniques can achieve the same optimal value at similar recorded cost. Faster solving may not reduce execution time if the code takes longer to prepare data and build the model. LLM optimization modeling should therefore be evaluated for both correctness and computational efficiency.

Abstract

Large Language Model-Guided Optimization

Explore similar work

Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling

Strategy-Aware Optimization Modeling with Reasoning LLMs

Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems