There is growing interest in leveraging large language models (LLMs) for text-to-model translation and optimization tasks. This paper aims to advance this line of research by introducing Text2Model and Text2Zinc. Text2Model is a suite of copilots based on several LLM strategies with varying complexity, along with an online leaderboard. Text2Zinc is a cross-domain dataset for capturing optimization and satisfaction problems specified in natural language, along with an interactive editor with built-in AI assistant. While there is an emerging literature on using LLMs for translating combinatorial problems into formal models, our work is the first attempt to integrate both satisfaction and optimization problems within a unified architecture and dataset. Moreover, our approach is solver-agnostic unlike existing work that focuses on translation to a solver-specific model. To achieve this, we leverage MiniZinc's solver-and-paradigm-agnostic modeling capabilities to formulate combinatorial problems. We conduct comprehensive experiments to compare execution and solution accuracy across several single- and multi-call strategies, including; zero-shot prompting, chain-of-thought reasoning, intermediate representations via knowledge-graphs, grammar-based syntax encoding, and agentic approaches that decompose the model into sequential sub-tasks. Our copilot strategies are competitive, and in parts improve, recent research in this domain. Our findings indicate that while LLMs are promising they are not yet a push-button technology for combinatorial modeling. We contribute Text2Model copilots and leaderboard, and Text2Zinc and interactive editor to open-source to support closing this performance gap.
Figures & tables
Figure 1: Text2Model copilots leveraging the Text2Zinc dataset.
Figure 2: Distribution of problems across application domains.
Figure 3: An example input with description, parameters, metadata, and output fields.
Figure 4: An example MiniZinc model.
Figure 5: A data instance (left) and the corresponding output produced by the MiniZinc model (right).
Figure 6: Text2Zinc Editor for dataset curation and validation with built-in AI Assistant.
IndustryOr
Mamo-Easy
Mamo-Complex
Nl4Opt
Orlm Pass@1
38.0%
82.3%
37.4%
85.7%
Our CoT Pass@1 (from Table 2 )
49.0%
83.90%
39.34%
78.37%
Orlm Pass@2
40.0%
83.7%
49.8%
88.6%
Orlm Pass@4
44.0%
85.9%
56.9%
91.4%
Orlm Pass@8
49.0%
88.4%
72.1%
93.0%
Our Best Pass@5 (from Table 2 )
57.0%
84.97%
55.92%
78.37%
Table 4: Comparison with Orlm results from the original paper ( Huang et al. (2024a) )
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
# LP
# MIP
# CP
NLP4LP
54
13
0
ComplexOR
25
12
0
LPWP
1101
0
0
Text2Zinc
64
31
15
Appendix
Table 5: Distribution of problem types across datasets (only displaying verified Text2Zinc instances).
copilot Strategy
LLM
Execution Accuracy (%)
Solution Accuracy (%)
#LLM Query
Single-Call Strategies
Baseline
GPT-4
32.73
17.27
1
Chain-of-Thought (CoT)
GPT-4o
60.91
34.55
1
GPT-4
57.27
28.18
1
Multi-Call Strategies
Knowledge Graph
GPT-4
48.18
25.45
2
Appendix
Table 9: Experimental results with GPT-4 and GPT-4o comparing execution and solution accuracy across different prompting strategies and models on the Text2Zinc dataset. Underlined indicates best performance within each strategy and Bold indicates the overall best performance. We also report the number of LLM queries required.
Error Type
Commentary
Syntax Errors
These errors indicate issues with the MiniZinc code syntax. They may include unexpected tokens, missing semicolons, or incorrect use of language constructs. For example: Error: syntax error, unexpected where, expecting ’)’ .
Undefined Identifiers
These errors are due to the use of variables or identifiers that have not been declared or are out of scope. Example: Error: type error: undefined identifier ‘i’, did you mean ‘X’? .
Array and Indexing Issues
Errors related to improper use of arrays or indexing problems. This can include out-of-bounds errors or incorrect array dimensions. Example: Error: type error: initialisation value for ‘Downtime’ has invalid type-inst: expected ‘array[int] of int’, actual ‘array[int,int] of int’ .
Function Not Found Errors
These errors occur when attempting to use functions or predicates that do not exist in the current MiniZinc environment or have incorrect signatures. This can happen with missing imports or incorrect function names. Example: Error: type error: no function or predicate with this signature found: ‘..o(float)’ .
Variable Redefinition Errors
These errors happen when the same variable is assigned or defined multiple times inappropriately, violating MiniZinc ’s variable declaration rules. Example: Error: type error: multiple assignment to the same variable .
Flattening Errors
These errors occur during the model flattening phase when MiniZinc attempts to convert the high-level model into a lower-level representation. Often related to unbounded variables or overly complex expressions that cannot be properly linearized. Example: Error: flattening error: unbounded coefficient in linear expression .
Large language models (LLMs) can translate natural-language optimization problems into solver programs, but reliably producing correct formulations and implementations remains challenging. Existing approaches improve optimization modeling through agentic workflows or task-specific post-training. This raises a practical question: can a lightweight harness using a single model role achieve competitive performance without additional training? We introduce OPTICYCLE and OPTISTATE, two lightweight inference harnesses for optimization modeling, and evaluate them on nine benchmarks comprising 2,634 problems using 11 Qwen variants spanning 0.6B-27B parameters. Our results show that execution-guided revision consistently improves macro accuracy over direct generation for all evaluated models, with gains of up to 49.46 percentage points. Despite their simplicity, both harnesses outperform six adapted agentic workflows in same-model comparisons and four evaluated post-trained optimization models at comparable model sizes, without additional training. Explicit formulation state provides further but model-dependent improvements, suggesting that additional structure is not universally beneficial.
Yunxiang Luo, Xin Xu, Tianhao Chen +7
The Hong Kong University of Science and Technology · Huawei
Large Language Models (LLMs) struggle to solve complex combinatorial problems through direct reasoning, so recent neuro-symbolic systems increasingly use them to synthesize executable solvers. A central design question is how the LLM should represent the solver, and whether it should also attempt to optimize search. We introduce CP-SynC-XL, a benchmark of 100 combinatorial problems (4,577 instances), and evaluate three solver-construction paradigms: native algorithmic search (Python), constraint modeling through a Python solver API (Python + OR-Tools), and declarative constraint modeling (MiniZinc + OR-Tools). We find a consistent representational divergence: Python + OR-Tools attains the highest correctness across LLMs, while MiniZinc + OR-Tools has lower absolute coverage despite using the same OR-Tools back-end. Native Python is the most likely to return a schema-valid solution that fails verification, whereas solver-backed paths preserve higher conditional fidelity. On the heuristic axis, prompting for search optimization yields only small median speed-ups (1.03-1.12x) and a strongly bimodal effect: many instances slow down, and correctness drops sharply on a long tail of problems. A paired code-level audit traces these regressions to a recurring heuristic trap. Under an efficiency-oriented prompt, the LLM may replace complete search with local approximations (Python), inject unverified bounds (Python + OR-Tools), or add redundant declarative machinery that overwhelms or over-constrains the model (MiniZinc + OR-Tools). These findings support a conservative design principle for LLM-generated combinatorial solvers: use the LLM primarily to formalize variables, constraints, and objectives for verified solvers, and separately check any LLM-authored search optimization before use.
Haoyu Wang, Yuliang Song, Tao Li +5
University of Pennsylvania · University of Toronto · Google DeepMind +1
Optimization problems are pervasive in sectors from manufacturing and distribution to healthcare. However, most such problems are still solved heuristically by hand rather than optimally by state-of-the-art solvers because the expertise required to formulate and solve these problems limits the widespread adoption of optimization tools and techniques. We introduce a Large Language Model (LLM)-based system designed to formulate and solve (mixed integer) linear programming problems from their natural language descriptions. Our system can develop mathematical models, write and debug solver code, evaluate the generated solutions, and improve efficiency and correctness of its model and code based on these evaluations. OptiMUS is designed as a productivity tool for optimization practitioners who understand the problem domain and can describe it precisely, but seek to accelerate the modeling and implementation workflow. OptiMUS-0.3 utilizes a modular structure to process problems, allowing it to handle problems with long descriptions and complex data without long prompts. Experiments demonstrate that OptiMUS-0.3 outperforms direct-prompting baselines by over 43% on easy and 18% on hard instances. It remains competitive with fine-tuned specialist models on benchmark problems, and outperforms them on real-world case studies (28.6% vs. 0%) where fine-tuned models fail to generalize. Ablation studies show that modular architecture with error correction is central to these gains. A key finding is that system architecture is a stronger driver of performance than model capability. Structured decomposition with targeted error correction enables weaker models to match stronger models under naive prompting, and remains competitive with fine-tuned specialist models without retraining costs.
Ali AhmadiTeshnizi, Wenzhi Gao, Herman Brunborg +3
a School of Management Science and Engineering, Stanford University, Stanford, California 94305 · b Institute for Computational and Mathematical Engineering, Stanford University, Stanford, California 94305