OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

Peisong Wang; Feixue Liu; Yize Meng; Bowen Liu; Ruoyu Chen; Gaoyang Dai; Xing He; Aiqian Li

cs.CLOct 16, 2026

OR-License: Evaluating LLM Agents as Real-World Operations Research Practitioners

Authors: Peisong Wang, Feixue Liu, Yize Meng, Bowen Liu, Ruoyu Chen, Gaoyang Dai, Xing He, Aiqian Li,

Organizations: HKUST-GZ · HKU · Qingzhi Optimization (Jinan) Information Technology Co., Ltd. · Hunyuan, Tencent ·

Abstract

Can large language model (LLM) agents deliver expert-level operations research (OR) solutions directly from original client materials, without prior expert reformulation? Existing OR benchmarks typically begin with self-contained descriptions or predefined optimization tasks, underrepresenting the translation of original business materials into professionally validated solutions. We introduce OR-License, an end-to-end benchmark of 31 instances across 10 industrial scenarios from paid OR engagements totaling USD 3.68 million. It preserves original requirements, input data, deliverable formats, and information boundaries after de-identification. To study information acquisition, we introduce ORSim, a controlled client simulator that discloses business facts without supplying OR expertise. Hidden evaluation checks feasibility and compares objective quality against original expert solutions accepted by clients before benchmark construction. Deployment-Adjusted Quality (DAQ) averages expert-relative quality across instances, assigning zero to infeasible deliveries. Across 11 frontier LLMs using coding-agent harnesses, several exceed the expert reference in conditional objective quality, but none matches its DAQ; the strongest primary Claude Code configuration achieves 88.39% strict feasibility and a DAQ of 0.87 against the expert reference of 1. Client information acquisition extends beyond clarification into solution development and validation, with some later stages showing higher confirmed issue-resolution proportions. Trajectory analysis identifies formulation and implementation failures, while task-equalized research-chain completion rates range from 24% to 45% across models on evaluable tasks. These findings motivate sustained information-gap resolution, faithful problem formulation, and evidence-linked self-improvement for reliable long-horizon agents.

Figures & tables

Figure 1: Three gaps and corresponding improvements in OR-License . Each column links an evaluation gap (top) to its improvement (bottom). The client dialogue is illustrative.

Figure 2: The OR-License evaluation framework. Experts curate tasks and reference solutions from real-world OR projects. Agents may query ORSim throughout problem solving. Final submissions undergo hidden evaluation of feasibility and expert-relative solution quality.

Panel of Figure 3 — Figure 3: Overview of the dataset. (a): The two-ring donut summarizes the covered industries (outer ring) and OR problem types (inner ring). Industry categories are based on the three-digit subsectors of the 2022 North American Industry Classification System (NAICS) ( Office of Management and Budget, 2022 ) , with selected labels shortened for readability. (b): Source-engagement statistics summarize the financial value and human effort associated with the original client projects. (c): The benchmark covers six core OR expert capabilities reflected in the curated tasks.

Panel of Table 1 — Table 1: Main results under Claude Code on 31 tasks with five runs per model–task pair (%). Means weight tasks equally. For feasibility and DAQ, $\pm$ reports the mean within-task sample standard deviation; quality and its spread are DAQ and its spread divided by mean feasibility. Right: task-equal reliability–quality profiles; the expert reference is 1.

Figure 4: Research-chain analysis for RQ1. (a) Task-equalized stopping locations along the HCAEU chain. (b) Mean stage quality among opportunities with a complete HCAEU chain versus model-average DAQ.

Figure 5: Left: question timing by workflow stage. Segment width shows the task-equalized question share; height shows the pooled proportion with confirmed issue resolution. Right: bar length shows the infeasible rate from Table 1 ; colored segments show the fixed annotated error-event distribution, scaled to that length. A trajectory may contain multiple events.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Annotation	Unit	Raw agreement	Krippendorff’s $\alpha$
Evidence support	Claim	$98.53\%$	$0.920$
Reference fulfillment	Reply	$98.10\%$	$0.831$
Disclosure compliance	Reply	$99.31\%$	$0.866$
Derived reply success	Reply	$97.58\%$	$0.873$

Appendix

Table 2: Pre-adjudication inter-annotator agreement. Raw agreement is the percentage of units receiving identical labels from their two assigned reviewers, including both positive and negative agreement, rounded to two decimal places. Evidence support is evaluated over 5,293 factual claims; the other dimensions are each evaluated over 3,060 replies. These units cover only the three selector variants; the full-statement baseline is excluded. Krippendorff’s

\alpha

is computed using Equation 17 and rounded to three decimal places.

	Rule selection		Response quality		Reliability
Variant	P $\uparrow$	R $\uparrow$	Faith. $\uparrow$	Viol. $\downarrow$	$\mathrm{pass}^{1}$ $\uparrow$	$\mathrm{pass}^{5}$ $\uparrow$
Standard queries
Flat	$95.64\%$	$98.01\%$	$98.71\%$	$0.34\%$	$95.46\%$	$94.12\%$
Summary + Detail	$93.43\%$	$96.05\%$	$87.50\%$	$3.19\%$	$89.24\%$	$79.83\%$
Summary + KG	$92.04\%$	$94.45\%$	$74.50\%$	$2.86\%$	$87.90\%$	$81.51\%$
Full statement	—	—	$94.47\%$	$28.24\%$	$64.54\%$	$41.18\%$

Appendix

Table 3: Simulator reliability for the three selector variants and the full-statement direct-answer baseline. Each variant is evaluated on 119 standard and 85 adversarial queries, with five runs per query. Selector P/R are macro-averaged over queries with nonempty reference rule sets: 119 standard and 54 adversarial queries, corresponding to 595 and 270 runs per selector variant. The remaining 31 adversarial queries are assessed using reply-level criteria. “—” indicates that selector P/R are inapplicable to the full-statement baseline; its output-level rule coverage is reported in Table 4 . Faith. is the percentage of supported factual claims. Viol. and

\mathrm{pass}^{1}

are the percentages of violating and successful replies, respectively;

\mathrm{pass}^{5}

is the percentage of queries with five successful replies. Annotation agreement in Table 2 covers the three selector variants. Percentages and percentage-point differences are calculated before rounding and reported to two decimal places.

Query category	$P^{\mathrm{out}}\uparrow$	$R^{\mathrm{out}}\uparrow$	Faith. $\uparrow$	Viol. $\downarrow$	$\mathrm{pass}^{1}\uparrow$	$\mathrm{pass}^{5}\uparrow$
Standard	50.41%	95.94%	94.47%	28.24%	64.54%	41.18%
Adversarial	39.30%	94.31%	87.03%	47.29%	41.41%	18.82%

Appendix

Table 4: Full-statement direct-answer diagnostic.

P^{\mathrm{out}}

and

R^{\mathrm{out}}

are reply-level macro-averages over 119 standard and 54 adversarial queries with nonempty reference rule sets, with five runs per query. They must not be interpreted as the selector-level P/R in Table 3 . The remaining four metrics repeat the full-statement rows of that table, using all 119 standard and 85 adversarial queries.

Model	Temperature	Top- $p$	Top- $k$	Max output tokens	Max context	Thinking strength
Claude Opus 5 (xhigh)	–	–	–	128K	1M	xhigh
DeepSeek-V4.1-Flash (high)	–	1.0	–	384K	1M	high
Gemini 3.8 Flash (high)	–	–	–	64K	1M	high
GLM-5.3-Flash (max)	1.0	0.95	–	128K	1M	max
Grok 4.6 (high)	1.0	1.0	Off	–	0.5M	high
GPT-5.6 Sol (xhigh)	–	–	–	128K	1M	xhigh

Appendix

Table 5: Model configurations and specifications.

Panel of Table 6 — Table 6: Mean agent resource usage under Claude Code. API costs are calculated using official list prices.

Panel of Table 7 — Table 7: Codex-harness results on OR-License.

Figure 6: Solution-technique adoption across LLMs and the expert reference. Rows represent models and the expert; columns represent categories T1–T7. Each cell reports the percentage of the 31 tasks receiving that label, using one selected solution scheme per model–task pair. Labels may coexist, so row percentages need not sum to 100%. Color intensity indicates adoption rate on a 0–100% scale.

Figure 7: Objective handling (a) and problem-structure exploitation (b) across LLMs and the expert reference. Each cell reports adoption as a percentage of the 31 tasks, using one selected solution scheme per model–task pair. O1 and O2 are mutually exclusive and sum to 100% within each row; S1–S4 may coexist. Both panels use a 0–100% color scale.

Benchmark	Task / Data Source	Original Client Materials	Agent?	Client Interaction	Client-Accepted References
Operations research and combinatorial optimization benchmarks
NL4Opt ( Ramamonjison et al., 2023 )	Annotated LP word problems	✗	✗	None	✗
Mamo ( Huang et al., 2024 )	Academic materials	✗	✗	None	✗
IndustryOR ( Huang et al., 2025 )	Abstracted industrial optimization problems	✗	✗	None	✗
OptiBench ( Yang et al., 2025 )	Textbooks, assignments, and exams	✗	✗	None	✗
OptMATH-Bench ( Lu et al., 2025 )	Bidirectional data synthesis	✗	✗	None	✗

Appendix

Table 8: Comparison with related OR and professional-work benchmarks. Original Client Materials requires original project requirements and inputs preserved through de-identification only; a cross marks benchmarks that fall outside this definition. Agent? indicates whether the benchmark evaluates LLM agents that use tools or interact with an environment over multiple steps ( ✓ : yes; ✗ : no). Client Interaction distinguishes fact elicitation from event updates and rubric feedback, excluding ordinary file access and solver feedback. Client-Accepted References must have been accepted in original professional engagements before benchmark construction ( ✓ : yes; ✗ : no).

Task	Integer variables	Continuous variables	Constraints
1	1,546	90	2,168
2	1,600	5	3,561
3	360	60	125
4	6,820	1	4,633
5	6,160	5	12,115
6	1,049,600	0	1,658

Appendix

Table 9: Estimated scale of the optimization models.

Task	Scenario	Verifier summary	Evaluator summary	Tags
1	Multi-pot brine scheduling	Checks submission format, schedule structure, resource/time consistency, and demand feasibility.	Recomputes the overall score under the official scheduling objective.	K2,K4,K6
2	Short-horizon multi-batch food production	Checks batch-schedule structure, resource constraints, and demand/round limits.	Recomputes the score under the official production-efficiency and priority objective.	K1,K4
3	Single-product multi-batch production	Checks batches, process order, lifecycle consistency, and demand constraints.	Recomputes the score under the official completion-time objective.	K1,K4
4	Long-horizon multi-product food production	Checks multi-product scheduling, transitions, ordering, and temporal constraints.	Recomputes the score under the official completion, ordering, and soft-preference objective.	K2,K4,K6
5	Multi-batch production with a brine-flow graph	Checks production flow, precedence, continuity, and demand constraints.	Recomputes the score under the official production-timing objective.	K1,K4
6	40-day IC aging-board schedule	Checks submission structure, resource assignment, capacity, timing, and process constraints.	Recomputes the score under the official scheduling-efficiency/pressure objective.	K2,K4

Appendix

Table 10: Verifier, evaluator, and competency coverage for the 31 benchmark tasks.

Operations Research

See all 36 papers

ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?
Jun 18, 2026Jiajun Li, Mingshu Cai, Yixuan Li +5LLM Agent Evaluation AI Agent Benchmarks
School of Computer Science and Engineering, Southeast University · Waseda University · Nanyang Technological University +1
COOPA: A Modular LLM Agent Architecture for Operations Research Problems
Jun 25, 2026Chuanhao Li, Xiaoan Xu, Dirk Bergemann +3Multi-Agent LLM Systems Multi-Agent Orchestration
1Tsinghua University · 2Duke University · 3Yale University
OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents
May 27, 2026Chenyu Zhou, Xinyun Lu, Jiangyue Zhao +3LLM Agent Evaluation AI Agent Benchmarks
Shanghai Jiao Tong University · Shanghai University of Finance and Economics · Stanford University

Jun 18, 2026J/K move · Enter open · S save

ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?

Jiajun Li, Mingshu Cai, Yixuan Li, Yu Ding+4

School of Computer Science and Engineering, Southeast University · Waseda University · Nanyang Technological University+1

LLM Agent Evaluation AI Agent Benchmarks Operations Research Long-Horizon Agent Tasks

Open paper AskPDF arXiv

Large language models are increasingly deployed as autonomous agents for multi-step tasks in executable environments, yet their ability to perform realistic operations research (OR) work remains unclear. Existing OR evaluations often decouple modeling from solving, rely on pre-formalized or text-only instances, and rarely test the full workflow from operational artifacts to validated decisions. In this work, we introduce ORAgentBench, an execution-grounded benchmark for evaluating autonomous agents on challenging end-to-end operations research tasks. It contains 107 human-reviewed tasks across diverse operational scenarios, each packaged in an isolated environment with a natural-language brief, multi-file data, configuration artifacts, and a required submission schema. Agents must write and run solution code, and their submissions are evaluated by hidden validators for schema validity, hard-constraint feasibility, and normalized objective quality. Experiments with fourteen frontier agent-model configurations show that current agents remain far from reliable OR practice. The best agent passes only 35.51% of all tasks and 20.59% of hard tasks, and many feasible submissions still fall below the required quality threshold. Failure analysis further shows that errors are dominated by strategic weaknesses, including missed operational rules, brittle formulations, weak feasible-solution construction, and insufficient solution improvement. OR-specific procedural skills increase hard-task feasibility, but do not reliably improve solution quality or pass rate. These results suggest that progress in OR agents requires moving beyond plausible optimization code toward dependable, high-quality operational decision-making.

Abstract

Operations Research

Explore similar work

ORAgentBench: Can LLM Agents Solve Challenging Operations Research Tasks End to End?

COOPA: A Modular LLM Agent Architecture for Operations Research Problems

OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents