ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization

Junbo Jacob Lian; Yujun Sun; Huiling Chen; Chaoyu Zhang; Hanzhang Qin; Chung-Piaw Teo

cs.SEFeb 17, 2026

ReLoop: Structured Modeling and Behavioral Verification for Reliable LLM-Based Optimization

Authors: Junbo Jacob Lian, Yujun Sun, Huiling Chen, Chaoyu Zhang, Hanzhang Qin, Chung-Piaw Teo

Organizations: Institute of Operations Research and Analytics, National University of Singapore · McCormick School of Engineering, Northwestern University · Wenzhou Buyi Pharmacy Chain Co., Ltd. · College of Computer Science and Artificial Intelligence, Wenzhou University · Department of Decision Analytics and Operations, City University of Hong Kong

Abstract

Large language models (LLMs) can translate natural-language problem descriptions into optimization code, but the code is prone to silent failures: it executes and returns a solver-feasible solution while encoding a semantically incorrect formulation. On compositional problems, the resulting feasibility-correctness gap reaches 90 percentage points. We introduce ReLoop, which combines two mechanisms. Structured generation decomposes code production into a four-stage reasoning chain (understand, formalize, synthesize, verify) to reduce formulation errors during generation. Behavioral verification detects the errors that remain by testing whether the formulation responds correctly to solver-based parameter perturbation, a signal that comes from the solver rather than from LLM self-review and requires no ground truth. The two mechanisms address different error structures: structured generation gives the largest gain on compositional problems (+8.5pp accuracy on RetailOpt-190 with Claude Opus 4.6), and behavioral verification gives its largest gain on localized defects (+4.4pp on MAMO-ComplexLP). With diagnostic execution recovery, ReLoop reaches 100% executable code on Claude Opus 4.6, and relative to direct generation it raises or preserves every reported metric of the three chat-tuned foundation models on all three benchmarks. For the narrowly fine-tuned SFT model we test, the chain-of-thought prompt conflicts with its learned output format and lowers its accuracy on MAMO-ComplexLP; we document and analyze this interaction. We release RetailOpt-190, 190 compositional retail optimization scenarios in which several constraints interact.

Figures & tables

Method	Detects Silent	External	No Ground	Iterative
	Failures	Signal	Truth	Repair
Solver feedback [ 10 ]	✗	✓	✓	✗
OptiChat [ 12 ]	✗	✓	✓	✓
Self-Refine [ 1 ]	✗	✗	✓	✓
Reflexion [ 3 ]	✗	✓	✓	✓
SAC-Opt [ 14 ]	✓	✗	✓	✓

Table 1: Verification approaches for LLM-generated optimization. † Requires ground-truth labels.

Figure 1: ReLoop overview. Structured Generation follows expert modeling practice: understand the problem, formalize the mathematical model with explicit variable-type reasoning, synthesize Gurobi code with data extraction, and self-verify completeness. Behavioral Verification : L1 checks execution correctness ( Fatal blocks output); L2 tests constraint (CPT) and objective (OPT) presence via solver-based perturbation ( Warning / Pass ). Diagnosis-Guided Repair : Fatal triggers diagnosis-specific regeneration (syntax tracebacks, IIS, or unbounded rays); Warning triggers targeted repair with regression rollback. When the budget $N$ is exhausted, ReLoop returns the best verified code.

Layer	Check	Severity	Action
L1: Execution (blocking)	Syntax / runtime error	Fatal	Regenerate
	Infeasible	Fatal	Regen. (IIS)
	Unbounded	Fatal	Regen. (ray)
	Duality gap †	Info	None
L2: CPT (diagnostic)	Missing ( $r<5\%$ )	Warning	Repair
L2: CPT (diagnostic)	Uncertain ( $5\%{\leq}r{\leq}30\%$ ) †	Info	None

Table 2: Severity matrix across verification layers. ( † ) Reference only: marked “do not fix.”

Defect	L2 behavior
Missing parameter-governed constraint	Detected (CPT)
Missing objective term	Detected (OPT)
Present-but-wrong constraint reusing the same parameters	Partial / not detected
Coefficient-magnitude errors	Not detected
Equivalent-looking but structurally different decompositions	Not detected
Errors inducing infeasibility or unboundedness	Handled by L1 (IIS / rays)

Table 3: Detection scope of L2 behavioral testing.

Benchmark	Domain	Inst.	Multi-period	Compositional	Data-Code Sep.
NL4Opt [ 22 ]	Generic	289	Few	✗	✗
MAMO [ 23 ]	Generic	$\sim$ 800	Some	✗	✗
IndustryOR [ 7 ]	Mixed	100	Some	✗	✗
OptMATH [ 9 ]	Mixed	$\sim$ 360	Some	✗	✗
RetailOpt-190	Retail	190	All	✓	✓

Table 4: Comparison with existing benchmarks.

ID	Family	Arch.	Key Mechanisms
F1	Core Operations	4	Multi-period inventory, perishability, lost sales
F2	Assortment & Substitution	6	Product substitution, promotions, price bands
F3	Resource Constraints	4	Storage bottleneck, supply limits, volumetric
F4	Demand Dynamics	6	Demand surge, supply risk, quality holds
F5	Feasibility Stress	4	Impossible demand, storage overflow, service traps
F6	Discrete Logistics	4	Lead time, MOQ, pack size, fixed order cost

Table 5: RetailOpt-190 scenario families (

38\times 5=190

instances).

		Exec%			Acc%( $\epsilon\!=\!10^{-4}$ )			Acc%( $\epsilon\!=\!10^{-2}$ )
Type	Model	Base	CoT	ReLoop	Base	CoT	ReLoop	Base	CoT	ReLoop
Foundation	Claude Opus 4.6	72.1	93.7	100.0	22.6	31.1	31.1	26.8	34.7	35.3
	DeepSeek-V3.2	91.1	53.2	97.4	0.5	3.7	5.8	3.7	5.8	11.1
	Qwen3-32B	0.0	0.0	2.1	0.0	0.0	0.0	0.0	0.0	0.0
Offline SFT	OptMATH-32B	2.6	2.6	17.9	0.0	0.0	0.0	0.0	0.0	0.5
Online RL	SIRL-32B	0.0	0.0	1.6	0.0	0.0	0.0	0.0	0.0	0.0

Table 6: Main results on RetailOpt-190 (pass@1, greedy decoding,

N\!=\!3

		MAMO-ComplexLP			IndustryOR
Type	Model	Base	CoT	+ReLoop	Base	CoT	+ReLoop
Foundation	Claude Opus 4.6	70.4	73.9	79.8	66.0	66.0	68.0
	DeepSeek-V3.2	60.1	59.6	62.6	50.0	54.0	62.0
	Qwen3-32B	40.4	37.4	46.3	43.0	43.0	46.0
Offline SFT	OptMATH-32B	56.2	30.0	31.0	34.0	31.0	34.0
Online RL	SIRL-32B	53.2	46.8	54.2	40.0	40.0	43.0

Table 7: Cross-benchmark generalization (Acc%,

\epsilon\!=\!10^{-6}

, pass@1).

	RetailOpt-190						MAMO-ComplexLP ( $\epsilon\!=\!10^{-6}$ )
	Claude Opus 4.6			DeepSeek-V3.2			Claude Opus 4.6		DeepSeek-V3.2
Config	Exec	Acc ${}_{10^{-4}}$	Acc ${}_{10^{-2}}$	Exec	Acc ${}_{10^{-4}}$	Acc ${}_{10^{-2}}$	Exec	Acc	Exec	Acc
Direct	72.1	22.6	26.8	91.1	0.5	3.7	94.1	70.4	93.6	60.1
+CoT	93.7	31.1	34.7	53.2	3.7	5.8	95.6	73.9	87.7	59.6
+CoT+L1	99.5	31.1	35.3	97.4	5.8	10.5	98.0	75.4	88.7	60.6
+CoT+L1+L2	100.0	31.1	35.3	97.4	5.8	11.1	98.0	79.8	88.7	62.6

Table 8: Ablation across benchmarks (pass@1). Each row adds one component; L2 = CPT + OPT behavioral testing.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Variable	Domain	Interpretation
$I_{p,l,t,k}$	$\mathbb{R}_{\geq 0}$	Start-of-period inventory of product $p$ at location $l$
		in period $t$ with $k$ periods of remaining life
$y_{p,l,t,k}$	$\mathbb{R}_{\geq 0}$	Sales from remaining-life bucket $k$ in period $t$
$Q_{p,l,t}$	$\mathbb{R}_{\geq 0}$	Order quantity placed for product $p$ at location $l$ in period $t$
$W_{p,l,t}$	$\mathbb{R}_{\geq 0}$	Waste (expired units) at end of period $t$
$L_{p,l,t}$	$\mathbb{R}_{\geq 0}$	Lost sales (unmet demand) in period $t$

Appendix

Table 9: Decision variables. Variables marked with † are created only when the corresponding mechanism is active (i.e., the relevant JSON field is non-default).

Family	Constraint	Design Rationale
C1. Initialization	$I_{p,l,1,k}=0\hskip 9.24994pt\forall\,k<\text{SL}_{p}$ $I_{p,l,1,\text{SL}_{p}}=A_{p,l,1}$	Without zero initialization for non-fresh buckets, unbounded phantom inventory yields objective $\approx 0$ (D1).
C2. Fresh inflow	$I_{p,l,t,\text{SL}_{p}}=A_{p,l,t}+\Delta^{\text{tr}}_{p,l,t}+R_{p,l,t}$ where $A_{p,l,t}=Q_{p,l,t-\text{LT}_{p}}$ if $t{-}\text{LT}_{p}\geq 1$ , else $0$ ; $\Delta^{\text{tr}}_{p,l,t}=\sum_{l^{\prime}:(l^{\prime},l)\in\mathcal{E}^{\text{tr}}}X_{p,l^{\prime}\to l,t}-\sum_{l^{\prime}:(l,l^{\prime})\in\mathcal{E}^{\text{tr}}}X_{p,l\to l^{\prime},t}$ ; $R_{p,l,t}=\rho_{p}\sum_{k}y_{p,l,t-1,k}$ if $t>1$ , else $0$ .	Fresh inventory enters only via ordering, transshipment, and returns. LLMs that subtract sales in this equation effectively double-count demand fulfillment.
C3. Aging dynamics	$I_{p,l,t+1,k}=I_{p,l,t,k+1}-y_{p,l,t,k+1}$ $\hskip 9.24994pt\forall\,t\in\{1,\dots,T{-}1\},\;k\in\{1,\dots,\text{SL}_{p}{-}1\}$	Remaining life decrements each period: bucket $k{+}1$ today becomes bucket $k$ tomorrow, minus sales. Incrementing $k$ instead causes fresh stock to expire immediately (D2).
C4. Expiration	$W_{p,l,t}=I_{p,l,t,1}-y_{p,l,t,1}$	Only bucket $k{=}1$ can expire. Unsold units from the oldest bucket become waste; waste is not carried forward.
C5. Sales availability	$y_{p,l,t,k}\leq I_{p,l,t,k}\hskip 9.24994pt\forall\,k$	Cannot sell more than on-hand inventory in each remaining-life bucket. Without this, the solver can create phantom sales.
C6. Demand conservation	$\sum_{k=1}^{\text{SL}_{p}}y_{p,l,t,k}+L_{p,l,t}=d_{p,l,t}+S^{\text{in}}_{p,l,t}-S^{\text{out}}_{p,l,t}$ with $S^{\text{out}}_{p,l,t}=\sum_{p^{\prime}\in\mathcal{N}^{\text{out}}_{p}}S_{p\to p^{\prime},l,t}$ , $S^{\text{in}}_{p,l,t}=\sum_{p^{\prime}\in\mathcal{N}^{\text{in}}_{p}}S_{p^{\prime}\to p,l,t}$ , $S^{\text{out}}_{p,l,t}\leq d_{p,l,t}$ (demand-route bound), and $S^{\text{in}}_{p,l,t}\leq\sum_{k=1}^{\text{SL}_{p}}y_{p,l,t,k}$ (inventory-backed substitution).	Reversing edge direction is a common modeling mistake (D4): $S_{p\to p^{\prime}}$ means $p$ ’s demand exported to $p^{\prime}$ ’s inventory, not the reverse. The demand-route and inventory-backed bounds together prevent unbounded substitution flow.

Appendix

Table 10: Core constraint families with explicit equations and design rationale. Each rationale column explains the specific modeling choice; see § A.7 for detailed justification.

ID	Family	#Arc.	Perish.	Capacity	Subst.	Discrete	Network	Primary Test
F1	Core Operations	4	✓	$\circ$	$\circ$	–	–	Aging dynamics
F2	Assortment	6	✓	✓	✓	$\circ$	–	Substitution $\times$ capacity
F3	Resources	4	✓	✓	$\circ$	–	–	Multi-resource coupling
F4	Dynamics	6	✓	✓	$\circ$	–	–	Temporal propagation
F5	Feasibility Stress	4	✓	✓	–	–	–	Stress robustness
F6	Discrete Logistics	4	✓	$\circ$	$\circ$	✓	–	Integer constraints

Appendix

Table 11: Compositional design matrix. Each family tests specific mechanism interactions. ✓ = structurally active (affects optimal solution);

\circ

= present but not binding in default data.

Family	Archetype ID	Modification from Base Scenario
F1	f1_base	Baseline: 20 periods, 3 SKUs, 5 DCs, seasonal Gaussian demand
	f1_high_waste	Waste cost $\times 20$ for all products
	f1_jit_logic	Holding cost $\times 20$ for all products
	f1_52_weeks	Horizon extended to $T{=}52$ (demand/capacity arrays tiled from base)
F2	f2_no_substitution	Substitution edges removed ( $\mathcal{E}^{\text{sub}}=\emptyset$ )
	f2_circular_sub	Circular substitution ring: Basic $\to$ Premium $\to$ ShortLife $\to$ Basic

Appendix

Table 12: Complete archetype specifications. “Modification” describes the parameter change relative to the base scenario ( get_base_scenario() in the generator). All 38 archetypes share the same JSON schema and are solved by the same universal MILP.

Parameter	Value	Description
$T$	20	Planning periods
$\|\mathcal{P}\|$	3	SKU_Basic, SKU_Premium, SKU_ShortLife
$\|\mathcal{L}\|$	5	DC1 through DC5
Shelf life	$\{10,8,4\}$	Periods remaining at production
Lead time	$\{0,0,0\}$	Same-period arrival (default)
Demand curve	Gaussian: $\lfloor 1000\exp\!\big(\!-\frac{(t-10)^{2}}{18}\big)+300\rfloor$	Seasonal peak at mid-horizon ( $t$ is 0-indexed); SKU_Premium $\times 0.5$ , SKU_ShortLife $\times 0.4$ ; values cast to int in generator

Appendix

Table 13: Base scenario parameters shared across all archetypes (before archetype-specific modifications).

Format	Data Location	Content	Use Case	File Suffix
Schema-based	External (runtime)	Narrative + JSON schema (types only)	Scalability, production	.scenario.txt
Data-embedded	In prompt	Narrative + full JSON data	Benchmark comparison	.full.txt

Appendix

Table 14: Two prompt formats for different evaluation scenarios. The same semantic information is conveyed in both formats; only the data delivery method differs.

Provided in Prompt	Left for LLM to Derive
Business narrative with structure cues	Complete formulation (variables, objective, constraints)
Data schema (field names, types)	Capacity, demand, and budget constraints
Data access patterns (indexing, nesting)	Substitution and inventory dynamics constraints
Output format specification (GurobiPy)	Boundary conditions / edge cases
Inventory inflow accounting (URS-aligned)	Common error warnings

Appendix

Table 15: Information provided in baseline prompts vs. left for the LLM to derive.

Field	Type	Example	Semantics
periods	int	20	Number of planning periods $T$
products	[str]	["SKU_Basic",...]	Product identifiers
locations	[str]	["DC1",...]	Location identifiers
shelf_life	{str:int}	{"SKU_Basic":10}	Remaining-life capacity per SKU
lead_time	{str:int}	{"SKU_Basic":0}	Order-to-arrival delay (periods)
demand_curve	{str:[float]}	[303,311,...]	Aggregate demand per SKU per period (0-indexed)

Appendix

Table 16: Complete JSON schema field reference. Types: int = integer, float = real, str = string, [T] = array of type T, {K:V} = dict, null = absent/inactive.

Prompt Section	Schema-based	Data-embedded
[SCENARIO] header	✓	✓
[BUSINESS DESCRIPTION]	✓	✓
[DATA SCHEMA] (types)	✓	–
[DATA ACCESS] (runtime)	✓	–
[DATA] (full JSON inline)	–	✓
[OUTPUT FORMAT]	✓	✓

Appendix

Table 17: Section-level comparison of the two prompt formats for a single instance. ✓ = present, – = absent.

Parameter	Value	Purpose
TimeLimit	60 seconds	Prevent stalling on complex MIPs (F6/F7)
MIPGap	1%	Tolerance for near-optimal solutions
Threads	1	Single-threaded for reproducibility
Seed	0	Fixed random seed
OutputFlag	0	Suppress solver output for batch processing

Appendix

Table 18: Gurobi solver parameters for ground truth computation, as set in universal_retail_solver.py .

Families	Problem Type	Tolerance $\epsilon$	Rationale
F1–F5, F7–F8	LP / easy MIP	$10^{-4}$ (strict) / $10^{-2}$ (practical)	LP relaxation tight; optimal is exact
F6	Hard MIP (MOQ, pack size)	$10^{-2}$ (both tiers)	60s time limit may yield near-optimal solutions; pack-size rounding creates inherent gaps

Appendix

Table 19: Accuracy tolerances by problem structure.

Layer	Parameter	Value	Description
L1 Execution	timeout	60 s	Subprocess execution timeout
	max_regenerations	3	Regeneration attempts on Fatal
	duality_gap_threshold	0.01	Primal–dual gap threshold (1%, Info only)
L2 CPT (Constraint Presence)	missing_threshold	0.05	$<5\%$ change $\to$ Warning
	uncertain_threshold	0.30	5–30% change $\to$ Info
	max_candidates	10	Max constraints to test per problem

Appendix

Table 20: Complete parameter configuration for ReLoop verification layers.

Benchmark	Families	Problem Type	Tolerance $\epsilon$
RetailOpt-190	F1–F5, F7–F8	LP	$10^{-4}$ (strict) / $10^{-2}$ (practical)
RetailOpt-190	F6	MIP	$10^{-2}$ (both tiers)
MAMO-ComplexLP	–	LP	$10^{-6}$
IndustryOR	–	Mixed	$10^{-6}$

Appendix

Table 21: Tolerance thresholds by benchmark and family.

Type	Model	Provider	Max Tokens	Notes
Foundation	Claude Opus 4.6	Anthropic API	8192
Foundation	DeepSeek-V3.2	DeepSeek API	8192
Foundation	Qwen3-32B	Local (vLLM [ 27 ] )	8192	BF16
Offline SFT	OptMATH-Qwen2.5-32B	Local (vLLM)	8192	BF16
Online RL	SIRL-Qwen2.5-32B	Local (vLLM)	8192	BF16

Appendix

Table 22: Model configurations used in experiments. All models use greedy decoding for reproducibility.

Benchmark	Instances	Avg Tokens	Tolerance	Format
RetailOpt-190	190	$\sim$ 2,900	$10^{-4}$ / $10^{-2}$	Data-embedded
MAMO-ComplexLP	203	$\sim$ 459	$10^{-6}$	Data-embedded
IndustryOR	100	$\sim$ 267	$10^{-6}$	Data-embedded

Appendix

Table 23: Benchmark statistics.

		Claude Opus 4.6		DeepSeek V3.2		Qwen3-32B		OptMATH-32B		SIRL-32B
Family	#	Base	+ReLoop	Base	+ReLoop	Base	+ReLoop	Base	+ReLoop	Base	+ReLoop
F1 Core Ops	20	55.0	95.0	5.0	5.0	0.0	0.0	0.0	0.0	0.0	0.0
F2 Assort & Sub	30	50.0	53.3	0.0	3.3	0.0	0.0	0.0	0.0	0.0	0.0
F3 Resource	20	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
F4 Demand Dyn	30	3.3	20.0	0.0	10.0	0.0	0.0	0.0	0.0	0.0	0.0
F5 Feasibility	20	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0

Appendix

Table 24: Per-family results on RetailOpt-190 (Acc%,

\epsilon\!=\!10^{-4}

). Base = direct generation for foundation models, native format for SFT/RL models. +ReLoop = CoT + L1–L2 for all models.

	Typed routing			Taxonomy-free
	$\times 10^{-4}$	$\times 10^{-3}$	$\times 10^{-2}$	(bidirectional)
Recall	238/240	238/240	238/240	238/240
Intact-code FP	63/240	64/240	64/240	54/240

Appendix

Table 25: Injected-defect detection across shrink multipliers (240 knockouts; 38 intact baselines).

Explore similar work

Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
Jul 18, 2026Haifeng Li, Mo HaiOptimization Modeling Falsehood
School of Information, Central University of Finance and Economics
SemOPT: Fixing Semantic Errors in LLM-based Optimization Modeling via Reward-Guided Search
Sep 29, 2026Zetong Zhou, Wentao Zhang, Jingyuan Wang +3Optimization Modeling Functional Code Solvers
School of Computer Science and Engineering, Beihang University, Beijing, China · MIIT Key Laboratory of Data and Decision Intelligence, Beihang University, Beijing, China · School of Data Science, The Chinese University of Hong Kong, Shenzhen, China +1
Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling
Sep 30, 2026Zhong Li, Xin Huang, Jinhui Wan +6Optimization Modeling Large Language Model Benchmarks
Great Bay University · Beijing Jiaotong University · Beihang University +1

Jul 18, 2026, cs.SEJ/K move · Enter open · S save

Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits

Haifeng Li, Mo Hai

School of Information, Central University of Finance and Economics

Optimization Modeling Falsehood Lattice Mutation

Open paper AskPDF arXiv

Large language models now translate natural-language descriptions of decision problems into solver-ready optimization models, but they fail silently. A generated model often runs and still formulates the wrong problem. This paper develops a theory of falsification-based verification for this setting. Every numeric quantity in the description is a typed slot, and a candidate model is tested only through solver calls on slot-transformed instances; no reference model or label is consulted. From duality, comparative statics, and polyhedral limit arguments we derive a battery of test classes covering directions, curvature, crush probes, prohibitive limits, annihilation, and exchange. Every test is sound, so a violation certifies unfaithfulness and the false-positive rate is zero by design. We characterize what such verification can never see, give conditions under which the canonical error classes are detected with certainty, and prove that no fixed-threshold perturbation tester is simultaneously sound and nontrivial. Experiments on 326 ground-truth models from NL4OPT and four benchmark families confirm the theory. The battery attains a 0.0% false-positive rate against 54.9% for a threshold tester, detects 70.0% of certified conditional-class mutants, convicts 40.4% of the mutants invisible to execution-accuracy scoring, and reproduces the predicted detectability pattern including its zeros.

Abstract

Explore similar work

Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits

SemOPT: Fixing Semantic Errors in LLM-based Optimization Modeling via Reward-Guided Search

Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling