Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.
Figures & tables
Figure 1: Three gaps and corresponding improvements in OR-License . Each column links an evaluation gap (top) to its improvement (bottom). The client dialogue is illustrative.
Figure 2: The OR-License evaluation framework. Experts curate tasks and reference solutions from real-world OR projects. Agents may query ORSim throughout problem solving. Final submissions undergo hidden evaluation of feasibility and expert-relative solution quality.
Figure 3: Overview of the dataset. (a): The two-ring donut summarizes the covered industries (outer ring) and OR problem types (inner ring). Industry categories are based on the three-digit subsectors of the 2022 North American Industry Classification System (NAICS) ( Office of Management and Budget, 2022 ) , with selected labels shortened for readability. (b): Source-engagement statistics summarize the financial value and human effort associated with the original client projects. (c): The benchmark covers six core OR expert capabilities reflected in the curated tasks.
Table 1: Main results under Claude Code on 31 tasks with five runs per model–task pair (%). Means weight tasks equally. For feasibility and DAQ, ± reports the mean within-task sample standard deviation; quality and its spread are DAQ and its spread divided by mean feasibility. Right: task-equal reliability–quality profiles; the expert reference is 1.
Figure 4: Research-chain analysis for RQ1. (a) Task-equalized stopping locations along the HCAEU chain. (b) Mean stage quality among opportunities with a complete HCAEU chain versus model-average DAQ.
Figure 5: Left: question timing by workflow stage. Segment width shows the task-equalized question share; height shows the pooled proportion with confirmed issue resolution. Right: bar length shows the infeasible rate from Table 1 ; colored segments show the fixed annotated error-event distribution, scaled to that length. A trajectory may contain multiple events.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Annotation
Unit
Raw agreement
Krippendorff’s α
Evidence support
Claim
98.53%
0.920
Reference fulfillment
Reply
98.10%
0.831
Disclosure compliance
Reply
99.31%
0.866
Derived reply success
Reply
97.58%
0.873
Appendix
Table 2: Pre-adjudication inter-annotator agreement. Raw agreement is the percentage of units receiving identical labels from their two assigned reviewers, including both positive and negative agreement, rounded to two decimal places. Evidence support is evaluated over 5,293 factual claims; the other dimensions are each evaluated over 3,060 replies. These units cover only the three selector variants; the full-statement baseline is excluded. Krippendorff’s α is computed using Equation 17 and rounded to three decimal places.
Rule selection
Response quality
Reliability
Variant
P ↑
R ↑
Faith. ↑
Viol. ↓
pass1↑
pass5↑
Standard queries
Flat
95.64%
98.01%
98.71%
0.34%
95.46%
94.12%
Summary + Detail
93.43%
96.05%
87.50%
3.19%
89.24%
79.83%
Summary + KG
92.04%
94.45%
74.50%
2.86%
87.90%
81.51%
Full statement
—
—
94.47%
28.24%
64.54%
41.18%
Appendix
Table 3: Simulator reliability for the three selector variants and the full-statement direct-answer baseline. Each variant is evaluated on 119 standard and 85 adversarial queries, with five runs per query. Selector P/R are macro-averaged over queries with nonempty reference rule sets: 119 standard and 54 adversarial queries, corresponding to 595 and 270 runs per selector variant. The remaining 31 adversarial queries are assessed using reply-level criteria. “—” indicates that selector P/R are inapplicable to the full-statement baseline; its output-level rule coverage is reported in Table 4 . Faith. is the percentage of supported factual claims. Viol. and pass1 are the percentages of violating and successful replies, respectively; pass5 is the percentage of queries with five successful replies. Annotation agreement in Table 2 covers the three selector variants. Percentages and percentage-point differences are calculated before rounding and reported to two decimal places.
Query category
Pout↑
Rout↑
Faith. ↑
Viol. ↓
pass1↑
pass5↑
Standard
50.41%
95.94%
94.47%
28.24%
64.54%
41.18%
Adversarial
39.30%
94.31%
87.03%
47.29%
41.41%
18.82%
Appendix
Table 4: Full-statement direct-answer diagnostic. Pout and Rout are reply-level macro-averages over 119 standard and 54 adversarial queries with nonempty reference rule sets, with five runs per query. They must not be interpreted as the selector-level P/R in Table 3 . The remaining four metrics repeat the full-statement rows of that table, using all 119 standard and 85 adversarial queries.
Model
Temperature
Top- p
Top- k
Max output tokens
Max context
Thinking strength
Claude Opus 5 (xhigh)
–
–
–
128K
1M
xhigh
DeepSeek-V4.1-Flash (high)
–
1.0
–
384K
1M
high
Gemini 3.8 Flash (high)
–
–
–
64K
1M
high
GLM-5.3-Flash (max)
1.0
0.95
–
128K
1M
max
Grok 4.6 (high)
1.0
1.0
Off
–
0.5M
high
GPT-5.6 Sol (xhigh)
–
–
–
128K
1M
xhigh
Appendix
Table 5: Model configurations and specifications.
Table 6: Mean agent resource usage under Claude Code. API costs are calculated using official list prices.
Table 7: Codex-harness results on OR-License.
Figure 6: Solution-technique adoption across LLMs and the expert reference. Rows represent models and the expert; columns represent categories T1–T7. Each cell reports the percentage of the 31 tasks receiving that label, using one selected solution scheme per model–task pair. Labels may coexist, so row percentages need not sum to 100%. Color intensity indicates adoption rate on a 0–100% scale.
Figure 7: Objective handling (a) and problem-structure exploitation (b) across LLMs and the expert reference. Each cell reports adoption as a percentage of the 31 tasks, using one selected solution scheme per model–task pair. O1 and O2 are mutually exclusive and sum to 100% within each row; S1–S4 may coexist. Both panels use a 0–100% color scale.
Benchmark
Task / Data Source
Original Client Materials
Agent?
Client Interaction
Client-Accepted References
Operations research and combinatorial optimization benchmarks
NL4Opt ( Ramamonjison et al., 2023 )
Annotated LP word problems
✗
✗
None
✗
Mamo ( Huang et al., 2024 )
Academic materials
✗
✗
None
✗
IndustryOR ( Huang et al., 2025 )
Abstracted industrial optimization problems
✗
✗
None
✗
OptiBench ( Yang et al., 2025 )
Textbooks, assignments, and exams
✗
✗
None
✗
OptMATH-Bench ( Lu et al., 2025 )
Bidirectional data synthesis
✗
✗
None
✗
Appendix
Table 8: Comparison with related OR and professional-work benchmarks. Original Client Materials requires original project requirements and inputs preserved through de-identification only; a cross marks benchmarks that fall outside this definition. Agent? indicates whether the benchmark evaluates LLM agents that use tools or interact with an environment over multiple steps ( ✓ : yes; ✗ : no). Client Interaction distinguishes fact elicitation from event updates and rubric feedback, excluding ordinary file access and solver feedback. Client-Accepted References must have been accepted in original professional engagements before benchmark construction ( ✓ : yes; ✗ : no).
Task
Integer variables
Continuous variables
Constraints
1
1,546
90
2,168
2
1,600
5
3,561
3
360
60
125
4
6,820
1
4,633
5
6,160
5
12,115
6
1,049,600
0
1,658
Appendix
Table 9: Estimated scale of the optimization models.
Task
Scenario
Verifier summary
Evaluator summary
Tags
1
Multi-pot brine scheduling
Checks submission format, schedule structure, resource/time consistency, and demand feasibility.
Recomputes the overall score under the official scheduling objective.
K2,K4,K6
2
Short-horizon multi-batch food production
Checks batch-schedule structure, resource constraints, and demand/round limits.
Recomputes the score under the official production-efficiency and priority objective.
K1,K4
3
Single-product multi-batch production
Checks batches, process order, lifecycle consistency, and demand constraints.
Recomputes the score under the official completion-time objective.
K1,K4
4
Long-horizon multi-product food production
Checks multi-product scheduling, transitions, ordering, and temporal constraints.
Recomputes the score under the official completion, ordering, and soft-preference objective.
K2,K4,K6
5
Multi-batch production with a brine-flow graph
Checks production flow, precedence, continuity, and demand constraints.
Recomputes the score under the official production-timing objective.
K1,K4
6
40-day IC aging-board schedule
Checks submission structure, resource assignment, capacity, timing, and process constraints.
Recomputes the score under the official scheduling-efficiency/pressure objective.
K2,K4
Appendix
Table 10: Verifier, evaluator, and competency coverage for the 31 benchmark tasks.
Large language models are increasingly deployed as autonomous agents for multi-step tasks in executable environments, yet their ability to perform realistic operations research (OR) work remains unclear. Existing OR evaluations often decouple modeling from solving, rely on pre-formalized or text-only instances, and rarely test the full workflow from operational artifacts to validated decisions. In this work, we introduce ORAgentBench, an execution-grounded benchmark for evaluating autonomous agents on challenging end-to-end operations research tasks. It contains 107 human-reviewed tasks across diverse operational scenarios, each packaged in an isolated environment with a natural-language brief, multi-file data, configuration artifacts, and a required submission schema. Agents must write and run solution code, and their submissions are evaluated by hidden validators for schema validity, hard-constraint feasibility, and normalized objective quality. Experiments with fourteen frontier agent-model configurations show that current agents remain far from reliable OR practice. The best agent passes only 35.51% of all tasks and 20.59% of hard tasks, and many feasible submissions still fall below the required quality threshold. Failure analysis further shows that errors are dominated by strategic weaknesses, including missed operational rules, brittle formulations, weak feasible-solution construction, and insufficient solution improvement. OR-specific procedural skills increase hard-task feasibility, but do not reliably improve solution quality or pass rate. These results suggest that progress in OR agents requires moving beyond plausible optimization code toward dependable, high-quality operational decision-making.
Jiajun Li, Mingshu Cai, Yixuan Li +5
School of Computer Science and Engineering, Southeast University · Waseda University · Nanyang Technological University +1
Operations Research (OR) provides a rigorous framework for high-stakes decision-making, but effective OR modeling requires substantial domain knowledge, mathematical abstraction, and solver expertise. Recent LLM-based systems automate parts of this pipeline, yet remain limited by low accuracy on complex problems, opaque outputs, and narrow solver support. We propose COOPA (COoperative OPerations Agent), a modular LLM-agent architecture for interpretable and scalable OR decision support. It combines three components: iterative confidence-based modeling, which generates multiple candidate formulations, self-evaluates them across modeling dimensions, and selects one using a max-min confidence criterion; element-level provenance and confidence explanations, which link variables, parameters, constraints, and objectives to quoted source text and provide an audit trail for human verification; and multi-solver routing to specialized optimizer agents for different OR problem classes. Across three OR benchmarks, eight LLM backbones, and four baselines under identical conditions, COOPA achieves the best macro-average accuracy on six of eight backbones and improves over the strongest baseline by up to 6.7 percentage points. A within-system ablation isolates the contribution of iterative confidence-based modeling, while additional analyses and case studies illustrate the value of source traceability and multi-solver dispatch.
Chuanhao Li, Xiaoan Xu, Dirk Bergemann +3
1Tsinghua University · 2Duke University · 3Yale University
Large language model (LLM) agents are increasingly used to assist with operations research (OR) modeling, yet existing OR-oriented benchmarks often reduce evaluation to one-shot translation from a self-contained problem statement into a mathematical formulation or solver program. Such settings abstract away two characteristics of real industrial OR workflows: persistent multi-artifact workspaces and multi-stage task lifecycles. We introduce OR-Space, a full-lifecycle workspace benchmark for evaluating industrial optimization agents across model construction, model revision, and grounded explanation. Each instance is an executable workspace containing business documents, structured data, optional code artifacts, solver outputs, and task-specific evaluators distributed across interdependent files. OR-Space defines three task modes: Build, where agents construct solver-ready optimization models from heterogeneous artifacts; Revise, where agents modify existing models under changing requirements or solver feedback while preserving valid prior logic; and Explain, where agents answer grounded questions about solutions, constraints, and business implications using evidence spread across workspace artifacts. By combining persistent workspaces with lifecycle-oriented tasks, OR-Space evaluates whether agents can perform reliable optimization work beyond end-to-end text generation. We describe the benchmark design, evaluation protocol, and quality-control pipeline, and position OR-Space as a benchmark for studying the reliability, failure modes, and practical readiness of LLM agents in industrial OR workflows.
Chenyu Zhou, Xinyun Lu, Jiangyue Zhao +3
Shanghai Jiao Tong University · Shanghai University of Finance and Economics · Stanford University