NLData2Opt: Benchmarking Parameter Derivation and Prediction for Modeling Optimization Problems from Natural Language and Data

Lijia Liu; Kyohei Atarashi; Jiyi Li; Koh Takeuchi; Shunji Umetani; Hisashi Kashima

cs.CLOct 16, 2026

NLData2Opt: Benchmarking Parameter Derivation and Prediction for Modeling Optimization Problems from Natural Language and Data

Authors: Lijia Liu, Kyohei Atarashi, Jiyi Li, Koh Takeuchi, Shunji Umetani, Hisashi Kashima

Organizations: Kyoto University · Hokkaido University · Recruit Co., Ltd.

Abstract

Formulating optimization problems from raw inputs of problem descriptions and data is time-consuming and costly. While Large Language Models (LLMs) offer a promising way to automate this process, existing benchmarks overestimate their capabilities by relying on idealized inputs where all data is relevant and complete. This diverges from reality, where data-centric reasoning is required to extract, compute, and predict parameters from noisy sources. To address this, we introduce NLData2Opt, a benchmark to evaluate LLMs on the multi-stage reasoning inherent in practical optimization workflows. NLData2Opt decomposes problems into problem descriptions and data, and requires models to (1) extract relevant parameters from data containing misleading attributes, (2) compute missing parameters derived by transforming existing data attributes, and (3) predict unavailable future values from historical observations. Evaluation results show substantial performance decline relative to existing benchmarks, indicating that benchmark progress does not directly translate to formulation accuracy in practical optimization workflows. NLData2Opt is available at https://github.com/sumarivyn/NLData2Opt.

Figures & tables

Figure 1: An optimization task in NLData2Opt with its components.

Figure 2: Transformation pipeline for prediction and derivation cases.

Figure 3: AMPL formulation of an example optimization task with model–data separation.

Figure 4: Ground-truth AMPL formulation construction framework.

Figure 5: Average EBC scores across task cases. The heatmap illustrates functional correctness achieved on the nine task cases of NLData2Opt (rows, cases with irrelevant attributes are marked with "+ Irr"), when evaluated with an LLM model or baseline method (columns). The columns are ranked to reflect the worst to best performing models. Higher saturation represents a higher EBC score. While models with smaller capacity did not generate executable code, stronger models only reached scores as high as 0.77, highlighting the challenges of the tasks.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

	Simple Forecasting			Oracle Prediction			LLM
Case	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
One-Variable	0.8665	0.6144	0.5377	0.8654	0.7305	0.5858	0.8836	0.7357	0.5572
One-Variable + Irr	0.8551	0.4796	0.4655	0.8436	0.6538	0.5018	0.8441	0.5331	0.4602
Multi-Variable	0.8739	0.5810	0.5552	0.8643	0.6257	0.5886	0.8626	0.6225	0.5300
Multi-Variable + Irr	0.8424	0.4972	0.4482	0.8325	0.5898	0.4769	0.8683	0.6393	0.4671

Appendix

Table 1: Performance breakdown for Claude Opus 4.6 across prediction baselines (Simple Forecasting, Oracle Prediction, and LLM).

	Simple Forecasting			Oracle Prediction			LLM
Case	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
One-Variable	0.8968	0.2213	0.5390	0.8548	0.2224	0.6043	0.3788	0.2386	0.5368
One-Variable + Irr	0.9208	0.1671	0.4878	0.8765	0.1945	0.4716	0.4259	0.2555	0.4443
Multi-Variable	0.9535	0.2229	0.5239	0.8582	0.2798	0.5830	0.3837	0.2680	0.5522
Multi-Variable + Irr	0.8714	0.1512	0.4692	0.8500	0.1533	0.4833	0.3419	0.2071	0.4434

Appendix

Table 2: Performance breakdown for GPT-4o Mini across prediction baselines (Simple Forecasting, Oracle Prediction, and LLM).

	Simple Forecasting			Oracle Prediction			LLM
Case	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
One-Variable	0.8756	0.0962	0.4874	0.8007	0.1847	0.5626	0.3486	0.1824	0.4932
One-Variable + Irr	0.8004	0.0733	0.4308	0.8046	0.0999	0.4493	0.1557	0.1180	0.4176
Multi-Variable	0.8087	0.1387	0.5107	0.8681	0.2397	0.5739	0.4154	0.2825	0.5045
Multi-Variable + Irr	0.6693	0.1066	0.4305	0.7159	0.1425	0.4543	0.2925	0.2008	0.3960

Appendix

Table 3: Performance breakdown for Meta LLaMA 3.3 70B Instruct across prediction baselines (Simple Forecasting, Oracle Prediction, and LLM).

Model/Method	NLP4LP	Mamo E.	Mamo C.
GPT-4o-mini	0.000	0.000	0.000
GPT-4o	0.425	0.367	0.467
Gemini-2.5-Pro	0.000	0.067	0.000
Claude Haiku 4.5	0.600	0.587	0.593
Claude Sonnet 4.6	0.573	0.533	0.307
Claude Opus 4.6	0.567	0.547	0.440

Appendix

Table 4: Percentage of correctly identified optimization models on derivation case.

	One-variable				Multiple-variables
	Standard		Irr. attributes		Standard		Irr. attributes
Model/Method	EM	SC	EM	SC	EM	SC	EM	SC
GPT-4o Mini	0.433	0.779	0.322	0.711	0.371	0.526	0.287	0.506
GPT-4o	0.611	0.892	0.400	0.845	0.408	0.560	0.258	0.511
Gemini 2.5 Pro	0.078	0.219	0.022	0.125	0.102	0.146	0.069	0.107
Claude Haiku 4.5	0.556	0.972	0.478	0.961	0.603	0.744	0.521	0.718

Appendix

Table 5: Percentage of Exact Matches (EM) and continuous Smoothed Correctness (SC) across four prediction cases.

Figure 6: Distribution of AMPL execution errors across nine task cases. The breakdown highlights the relative frequency of specific compilation and logical errors, such as undefined entities, undeclared data assignments, and syntax errors.

Case	Compilation Success	Exact Match Rate	Syntax Error	Structural Error
NL	51%	23%	11%	43%
NL + Data	45%	22%	24%	46%
NL + Data + Irr	48%	25%	16%	46%
Derivation	42%	8%	20%	50%
Derivation + Irr	43%	10%	12%	49%
Prediction (1-Var)	37%	7%	17%	54%

Appendix

Table 6: Failure type breakdown across all task cases.

	NLP4LP			Mamo Easy			Mamo Complex
Model/Method	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
GPT-4o Mini	0.582	0.818	$0.568\pm 0.056$	0.729	0.867	$0.481\pm 0.055$	0.809	0.130	$0.548\pm 0.061$
GPT-4o	0.968	0.150	$0.667\pm 0.047$	0.687	0.767	$0.383\pm 0.050$	0.870	0.468	$0.670\pm 0.034$
Gemini 2.5 Pro	0.413	0.000	$0.282\pm 0.056$	0.370	0.000	$0.203\pm 0.059$	0.301	0.000	$0.390\pm 0.047$
Claude Haiku 4.5	0.588	0.702	$0.463\pm 0.051$	0.825	0.845	$\bm{0.638\pm 0.060}$	0.865	0.619	$0.625\pm 0.043$
Claude Sonnet 4.6	0.880	0.450	$0.403\pm 0.039$	0.850	0.600	$0.610\pm 0.046$	0.905	0.616	$0.596\pm 0.042$

Appendix

Table 7: Comprehensive metrics for performance on NL case.

	NLP4LP			Mamo Easy			Mamo Complex
Model/Method	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
GPT-4o Mini	0.931	0.400	$0.421\pm 0.049$	0.700	0.345	$0.419\pm 0.049$	0.876	0.393	$0.529\pm 0.049$
GPT-4o	0.940	0.150	$0.645\pm 0.049$	0.670	0.639	$0.361\pm 0.056$	0.887	0.612	$\bm{0.658\pm 0.049}$
Gemini 2.5 Pro	0.427	0.000	$0.317\pm 0.046$	0.239	0.000	$0.147\pm 0.044$	0.327	0.000	$0.396\pm 0.061$
Claude Haiku 4.5	0.896	0.733	$0.644\pm 0.045$	0.734	0.753	$\bm{0.543\pm 0.059}$	0.872	0.630	$0.625\pm 0.042$
Claude Sonnet 4.6	0.945	0.290	$0.515\pm 0.050$	0.670	0.553	$0.495\pm 0.062$	0.879	0.794	$0.596\pm 0.042$

Appendix

Table 8: Comprehensive metrics for performance on NL+Data case.

	NLP4LP			Mamo Easy			Mamo Complex
Model/Method	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
GPT-4o Mini	0.737	0.266	$0.421\pm 0.049$	0.600	0.490	$0.419\pm 0.049$	0.811	0.336	$0.529\pm 0.049$
GPT-4o	0.723	0.627	$0.451\pm 0.047$	0.627	0.513	$0.403\pm 0.053$	0.801	0.385	$0.558\pm 0.066$
Gemini 2.5 Pro	0.879	0.273	$0.347\pm 0.055$	0.635	0.219	$0.299\pm 0.059$	0.800	0.032	$0.411\pm 0.055$
Claude Haiku 4.5	0.824	0.613	$0.438\pm 0.047$	0.667	0.697	$0.461\pm 0.056$	0.875	0.551	$0.554\pm 0.043$
Claude Sonnet 4.6	0.861	0.223	$0.355\pm 0.074$	0.701	0.284	$0.503\pm 0.049$	0.876	0.678	$0.539\pm 0.055$

Appendix

Table 9: Comprehensive metrics for performance on NL+Data case with irrelevant attributes.

	NLP4LP			Mamo Easy			Mamo Complex
Model/Method	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
GPT-4o Mini	0.901	0.259	$0.476\pm 0.053$	0.637	0.151	$0.289\pm 0.035$	0.730	0.132	$0.433\pm 0.040$
GPT-4o	0.913	0.125	$0.553\pm 0.062$	0.609	0.328	$0.310\pm 0.050$	0.792	0.416	$0.502\pm 0.052$
Gemini 2.5 Pro	0.378	0.000	$0.318\pm 0.043$	0.222	0.000	$0.131\pm 0.045$	0.282	0.000	$0.372\pm 0.049$
Claude Haiku 4.5	0.782	0.542	$0.486\pm 0.041$	0.696	0.560	$0.459\pm 0.040$	0.818	0.438	$0.511\pm 0.037$
Claude Sonnet 4.6	0.864	0.289	$0.416\pm 0.034$	0.646	0.215	$0.408\pm 0.038$	0.845	0.542	$0.543\pm 0.042$

Appendix

Table 10: Comprehensive metrics for performance on derivation case.

	NLP4LP			Mamo Easy			Mamo Complex
Model/Method	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
GPT-4o Mini	0.672	0.194	$0.389\pm 0.041$	0.499	0.190	$0.353\pm 0.040$	0.654	0.048	$0.453\pm 0.037$
GPT-4o	0.685	0.444	$0.396\pm 0.052$	0.537	0.283	$0.388\pm 0.034$	0.695	0.275	$0.447\pm 0.042$
Gemini 2.5 Pro	0.914	0.128	$0.290\pm 0.065$	0.708	0.025	$0.209\pm 0.063$	0.800	0.033	$0.373\pm 0.052$
Claude Haiku 4.5	0.766	0.510	$0.398\pm 0.042$	0.653	0.478	$0.394\pm 0.051$	0.827	0.357	$0.491\pm 0.033$
Claude Sonnet 4.6	0.795	0.225	$0.341\pm 0.038$	0.561	0.276	$0.381\pm 0.047$	0.851	0.426	$\bm{0.493\pm 0.049}$

Appendix

Table 11: Comprehensive metrics for performance on derivation case with irrelevant attributes.

	NLP4LP			Mamo Easy			Mamo Complex
Model/Method	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
GPT-4o Mini	0.326	0.240	$0.531\pm 0.058$	0.399	0.385	$0.471\pm 0.044$	0.442	0.204	$0.559\pm 0.048$
GPT-4o	0.676	0.554	$\bm{0.575\pm 0.049}$	0.546	0.548	$0.505\pm 0.066$	0.710	0.404	$0.548\pm 0.041$
Gemini 2.5 Pro	0.405	0.322	$0.472\pm 0.043$	0.297	0.293	$0.385\pm 0.057$	0.113	0.038	$0.424\pm 0.046$
Claude Haiku 4.5	0.891	0.355	$0.528\pm 0.057$	0.709	0.473	$0.468\pm 0.044$	0.902	0.629	$0.574\pm 0.056$
Claude Sonnet 4.6	0.769	0.164	$0.439\pm 0.039$	0.712	0.557	$0.505\pm 0.054$	0.844	0.602	$0.531\pm 0.043$

Appendix

Table 12: Comprehensive metrics for performance on prediction case (one variable).

	NLP4LP			Mamo Easy			Mamo Complex
Model/Method	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
GPT-4o Mini	0.423	0.247	$0.418\pm 0.049$	0.330	0.298	$0.426\pm 0.043$	0.451	0.257	$0.482\pm 0.044$
GPT-4o	0.670	0.480	$0.435\pm 0.036$	0.527	0.585	$0.446\pm 0.057$	0.754	0.410	$0.513\pm 0.052$
Gemini 2.5 Pro	0.330	0.065	$0.312\pm 0.068$	0.242	0.268	$0.315\pm 0.068$	0.024	0.035	$0.411\pm 0.045$
Claude Haiku 4.5	0.800	0.401	$0.407\pm 0.047$	0.684	0.422	$0.416\pm 0.044$	0.851	0.504	$\bm{0.529\pm 0.054}$
Claude Sonnet 4.6	0.751	0.317	$0.364\pm 0.037$	0.716	0.493	$0.468\pm 0.056$	0.854	0.694	$0.498\pm 0.055$

Appendix

Table 13: Comprehensive metrics for performance on prediction case (one variable) with irrelevant attributes.

	NLP4LP			Mamo Easy			Mamo Complex
Model/Method	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
GPT-4o Mini	0.334	0.283	$0.568\pm 0.056$	0.323	0.298	$0.481\pm 0.055$	0.461	0.242	$0.548\pm 0.061$
GPT-4o	0.776	0.596	$0.600\pm 0.050$	0.531	0.538	$0.503\pm 0.062$	0.711	0.447	$0.574\pm 0.047$
Gemini 2.5 Pro	0.410	0.199	$0.480\pm 0.039$	0.245	0.184	$0.395\pm 0.063$	0.137	0.000	$0.426\pm 0.041$
Claude Haiku 4.5	0.879	0.630	$0.640\pm 0.039$	0.632	0.394	$0.454\pm 0.040$	0.907	0.467	$0.557\pm 0.056$
Claude Sonnet 4.6	0.842	0.328	$0.489\pm 0.051$	0.622	0.493	$0.465\pm 0.046$	0.853	0.598	$0.528\pm 0.046$

Appendix

Table 14: Comprehensive metrics for performance on prediction case (multiple variables).

	NLP4LP			Mamo Easy			Mamo Complex
Model/Method	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
GPT-4o Mini	0.397	0.264	$0.426\pm 0.035$	0.250	0.181	$0.415\pm 0.050$	0.292	0.140	$0.472\pm 0.050$
GPT-4o	0.743	0.569	$0.453\pm 0.037$	0.492	0.510	$0.419\pm 0.059$	0.711	0.471	$0.485\pm 0.052$
Gemini 2.5 Pro	0.300	0.206	$0.333\pm 0.055$	0.192	0.130	$0.298\pm 0.074$	0.090	0.034	$0.395\pm 0.058$
Claude Haiku 4.5	0.778	0.599	$0.428\pm 0.048$	0.610	0.345	$0.409\pm 0.045$	0.825	0.389	$0.503\pm 0.045$
Claude Sonnet 4.6	0.803	0.253	$0.331\pm 0.069$	0.683	0.429	$0.444\pm 0.052$	0.773	0.605	$0.473\pm 0.048$

Appendix

Table 15: Comprehensive metrics for performance on prediction case (multiple variables) with irrelevant attributes.

	NLP4LP			Mamo Easy			Mamo Complex
Case	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
NL	0.5488	0.7231	0.4190	0.7933	0.7517	0.6199	0.8062	0.7066	0.6224
NL + Data	0.8342	0.7080	0.6033	0.7268	0.6463	0.5453	0.8319	0.7241	0.6148
NL + Data + Irr	0.7640	0.5451	0.3848	0.7143	0.5813	0.4926	0.8550	0.7068	0.5743
Derivation	0.7589	0.5547	0.4595	0.6676	0.3369	0.3955	0.8043	0.5772	0.5443
Derivation + Irr	0.6921	0.3794	0.3519	0.6362	0.3118	0.3698	0.7848	0.5151	0.4881

Appendix

Table 16: Evaluation results for Claude Opus 4.6 on the full benchmark.

	NLP4LP			Mamo Easy			Mamo Complex
Case	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
NL	0.5172	0.0129	0.4554	0.7853	0.0484	0.6422	0.6145	0.1378	0.5933
NL + Data	0.7512	0.4018	0.6506	0.6280	0.4861	0.5301	0.6822	0.2486	0.6205
NL + Data + Irr	0.7097	0.3552	0.4253	0.5781	0.4292	0.4368	0.6216	0.2468	0.5376
Derivation	0.6503	0.2272	0.4502	0.5233	0.2200	0.4094	0.5393	0.1769	0.4886
Derivation + Irr	0.5797	0.1991	0.3636	0.4748	0.1738	0.3671	0.5504	0.1390	0.4419

Appendix

Table 17: Evaluation results for GPT-4o Mini on the full benchmark.

	NLP4LP			Mamo Easy			Mamo Complex
Case	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES	EF1	EBC	LLM-ES
NL	0.6900	0.2745	0.5193	0.6711	0.1124	0.6403	0.6371	0.2787	0.6095
NL + Data	0.4554	0.2995	0.6033	0.5043	0.3895	0.5202	0.6037	0.2272	0.5914
NL + Data + Irr	0.4710	0.2009	0.4004	0.4215	0.3046	0.4354	0.5017	0.1382	0.4881
Derivation	0.4192	0.1636	0.4498	0.3829	0.1930	0.4097	0.5778	0.1262	0.4990
Derivation + Irr	0.3707	0.0847	0.3699	0.3103	0.1161	0.3628	0.4487	0.0749	0.4305

Appendix

Table 18: Evaluation results for LLaMA 3.3 70B Instruct on the full benchmark.

Natural Language to Optimization Modeling

Natural language to optimization modeling translates human requirements into formal mathematical models and solver-ready code.

See all 76 papers

Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems
Jul 12, 2026Yongchang Fu, Xinjie Huang, Chengjun Dai +3Large Language Model-Guided Optimization LLM Agent Evaluation
Ding Talk, Alibaba Group, Hangzhou, China · Zhejiang University, Hangzhou, China
Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling
Sep 30, 2026Zhong Li, Xin Huang, Jinhui Wan +6LLM Evaluation Large Language Model-Guided Optimization
Great Bay University · Beijing Jiaotong University · Beihang University +1
Harnessing Language Models as Optimization Modeling Agents
Oct 22, 2026Yunxiang Luo, Xin Xu, Tianhao Chen +7Large Language Model-Guided Optimization Natural Language to Optimization Modeling
The Hong Kong University of Science and Technology · Huawei

Jul 12, 2026J/K move · Enter open · S save

Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems

Yongchang Fu, Xinjie Huang, Chengjun Dai, Chengzhe Feng+2

Ding Talk, Alibaba Group, Hangzhou, China · Zhejiang University, Hangzhou, China

Large Language Model-Guided Optimization LLM Agent Evaluation AI Agent Benchmarks Natural Language to Optimization Modeling Operations Research

Open paper AskPDF arXiv

LLM-based agents are increasingly deployed to solve optimization problems, yet existing benchmarks evaluate them on pre-structured mathematical formulations that bypass the most critical challenge: translating complex business requirements into correct models and solve efficiently. We introduce Opti-Agent-Bench, an end-to-end benchmark that evaluates Large Language Models (LLMs) across the complete optimization R&D pipeline, from understanding business-language descriptions through mathematical modeling, algorithm selection, and code implementation, to solution report generation. Our design rests on three pillars: (1) businesssemantic authenticity with anti-template traps that defeat pattern matching; (2) modular evaluation with cross-module consistency checking across Problem Understanding, Formal Modeling, Implementation, and Reporting; and (3) the ORAC bi-level validity framework that simultaneously ensures task quality and scoring integrity. Across several industrialscale tasks spanning integer programming, robust optimization, stochastic programming, and non-convex optimization, we expose critical failure modes of current models, including constraint omission, model-code inconsistency, and report-implementation divergence, that remain invisible under conventional single-metric evaluation.

Abstract

Natural Language to Optimization Modeling

Explore similar work

Opti-Agent-Bench: Benchmarking End-to-End Optimization R&D Agents on Real-World Business Problems

Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling

Harnessing Language Models as Optimization Modeling Agents