Optimization modeling is the process of translating real-world decision problems, often described in natural language, into formal mathematical formulations and executable solver code. While recent advances in large language models have shown promise in automating this process, many approaches focus on one-shot generation or rely on predefined execution-and-repair workflows, without explicitly learning a policy for tool use and revision. This stands in sharp contrast to real-world optimization modeling, which is inherently interactive and proceeds through repeated solve-debug-revise cycles. We introduce PEARL, a system for multi-turn solver-interactive optimization modeling that uses Python execution and mathematical programming solvers inside this loop. Rather than relying on a fixed repair workflow, PEARL learns when to test partial models, how to revise from solver diagnostics, and when to stop. It operates in a multi-turn tool-integrated setting where intermediate execution results, feasibility signals, and solution checks are used to improve both formulations and solver code before finalization. Across diverse optimization benchmarks, PEARL substantially improves executable objective agreement over the same-scale base model and is competitive with strong one-shot and tool-augmented baselines. Notably, PEARL-Qwen3-4B outperforms the much larger DeepSeek-V3.2-685B baseline under direct prompting on both macro- and micro-averaged Pass@1. Code and datasets are provided at https://github.com/AuroraLHL/PEARL.
Figures & tables
Figure 1: Macro-averaged Pass@1 versus reported model scale across representative optimization-modeling methods. PEARL (Ours) uses a 4B backbone yet matches or exceeds much larger general-purpose models such as DeepSeek-V3.2-685B, highlighting that solver-mediated interaction and optimization-specific training can deliver strong accuracy in a compact model regime that is more practical for repeated agentic invocation ( Kwon et al., 2023 ; Abdin et al., 2024 ; Belcak et al., 2025 ) .
Figure 2: Workflow of PEARL . Colors indicate elements of the POMDP ⟨S,O,A,P,R,T⟩ : observations O (blue), agent policy πθ (red), actions A (purple), environment transitions P (orange), reward R (green, dashed), and termination T (gray). The latent state S is not shown. Terminal scoring executes the final code and compares its objective with the reference; this comparison is not an in-loop tool observation.
Method
NL4Opt
MAMO-E
MAMO-C
NLP4LP
OptMATH
IndustryOR
ComplexOR
Resocratic
Macro Avg
Micro Avg
Direct Prompt
Qwen-3-4B-Instruct
48.13 ± 1.40
2.20 ± 0.45
13.51 ± 2.20
55.62 ± 1.25
2.41 ± 0.50
16.66 ± 3.10
5.56 ± 2.78
33.50 ± 1.80
22.20 ± 0.75
22.42 ± 0.55
Qwen-3-4B-Think
48.60 ± 2.80
27.34 ± 2.50
6.31 ± 1.90
59.55 ± 2.10
1.81 ± 0.45
30.95 ± 4.10
22.22 ± 5.56
50.12 ± 2.50
30.86 ± 0.95
35.06 ± 0.80
Qwen3-235B-A22B-Instruct-2507
74.92 ± 1.77
91.62 ± 0.69
38.74 ± 5.41
88.95 ± 0.33
20.88 ± 3.09
58.73 ± 5.50
38.89 ± 0.00
70.64 ± 1.41
60.42 ± 1.13
72.26 ± 0.69
Qwen3-235B-A22B-Think-2507
79.59 ± 5.42
89.79 ± 3.07
40.84 ± 6.64
91.20 ± 5.54
30.72 ± 1.59
71.43 ± 4.12
50.00 ± 5.56
74.36 ± 4.50
65.99 ± 1.68
74.95 ± 1.39
DeepSeek-V3.2-685B
83.18 ± 3.07
94.07 ± 2.29
50.75 ± 9.38
92.70 ± 2.81
25.10 ± 3.03
58.73 ± 11.98
51.85 ± 6.42
75.68 ± 2.52
66.51 ± 1.88
77.07 ± 1.22
DeepSeek-V3.2-Think-685B
85.20 ± 1.18
95.53 ± 0.59
53.15 ± 3.12
97.38 ± 0.86
30.32 ± 1.25
64.29 ± 2.39
50.00 ± 5.56
77.67 ± 0.43
69.19 ± 0.65
79.58 ± 0.41
Table 1: Pass@1 accuracy (%) across benchmarks (higher is better). Each entry reports the mean ± standard deviation over three runs.
Figure 3: Main ablations for PEARL training choices. Tool use converges early, response length continues to grow under GRPO, SFT mainly serves as a warm-start for less aligned models, and aggressive trajectory filtering is not beneficial in this solver-feedback setting.
Figure 4: Failure attribution before and after PEARL training under the main pass@1 evaluation setting. Bars are normalized within failed cases; attribution setup, category definitions, and detailed interpretation are provided in Appendix C .
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Method
Learning target
Role of solver feedback
SIRL
Outcome-based learning of complete generations
Terminal execution, feasibility, and objective-based scoring.
OR-R1
Output-level RL
Solver outcomes score generated solutions.
MURKA
GRPO-trained Extractor and distilled Solver
A separate Checker iteratively verifies solutions through solver feedback.
StepORLM
Generative process supervision and preference learning
Solver outcomes and process feedback guide reasoning-trajectory learning.
STORM
Curriculum-guided output learning
Outcome signals support curriculum-shaped alignment.
PEARL
Multi-turn policy for execution, revision, and termination
Solver output conditions later actions; trajectory rewards combine format and objective agreement.
Appendix
Table 2: Comparison of RL-based methods for optimization modeling. The table summarizes what each method learns and how it uses solver feedback during training or inference.
Figure 5: Overview of the PEARL training and inference loop for optimization modeling with tool use. The right-hand branch depicts external training-time scoring; its objective-value probe does not certify optimal solver termination.
Hyperparameter
Value
Base model
Qwen3-4B-Instruct-2507
Hardware
1 node, 4 high-performance GPUs, Ray-based training
Rollout engine
SGLang tool-call rollout (colocated), 4 GPUs per engine
Algorithm
GRPO with decoupled clipping ( ϵ=0.2 , ϵhigh=0.28 )
KL regularization
disabled in main runs, coefficient 0.0
Entropy regularization
coefficient 0.0
Appendix
Table 3: Main hyperparameters for Qwen3-4B-Instruct-2507 PEARL training.
Parameter
Value
Provider
vLLM (optrl_vllm)
Model
Qwen/Qwen3-4B-Instruct-2507
Tokenizer
Qwen/Qwen3-4B-Instruct-2507
Max model length
16384 tokens
Max output tokens
16384 tokens
LLM configuration timeout
300 seconds (see note below)
Appendix
Table 4: Inference parameters for Qwen3-4B-Instruct-2507 PEARL evaluation.
Method
NL4Opt
MAMO-E
MAMO-C
NLP4LP
OptMATH*
IndustryOR
ComplexOR
Resocratic
Macro Avg
Micro Avg
Qwen-3-8B
38.32 ± 1.60
38.53 ± 2.20
12.61 ± 2.30
46.07 ± 1.90
3.01 ± 0.55
2.38 ± 1.35
5.56 ± 2.78
37.47 ± 1.70
22.99 ± 0.70
32.56 ± 0.60
PEARL-Qwen-3-8B
86.45 ± 1.10
94.68 ± 0.75
57.66 ± 3.40
94.38 ± 1.20
24.10 ± 1.65
64.29 ± 6.20
38.89 ± 4.80
81.39 ± 0.90
67.73 ± 0.55
79.61 ± 0.30
PEARL-Qwen-3-4B-Instruct-2507
85.28 ± 0.99
91.65 ± 0.65
76.12 ± 3.19
93.25 ± 1.59
16.87 ± 1.70
67.86 ± 8.42
41.66 ± 3.92
84.99 ± 0.52
69.71 ± 0.13
79.84 ± 0.25
Appendix
Table 5: Pass@1 accuracy (%) for 8B-related comparisons. The 8B rows are reported in the appendix to keep the main table focused on the primary 4B PEARL model and representative baselines.
Figure 6: Training reward curve when directly applying PEARL to Qwen3-8B-Base without prior instruction tuning. The reward begins to decline after 60 steps, indicating training collapse due to the model’s inability to consistently follow the required output format.
Figure 7: Analyzing the role of SFT prior to agentic RL. Results are obtained using the same base model, Qwen3-4B-Instruct, under three training regimes: (i) SFT only, (ii) SFT followed by agentic RL (SFT+ARL), and (iii) direct agentic RL without SFT (ARL).
Method
NL4Opt
MAMO-E
MAMO-C
NLP4LP
OptMATH
IndustryOR
ComplexOR
Resocratic
Macro Avg
Micro Avg
SFT-Qwen-3-4B-Instruct
83.64
92.66
55.86
93.82
14.46
52.38
44.44
79.90
64.64
76.86
PEARL-SFT-Qwen-3-4B-Instruct
85.98
91.19
76.57
94.38
23.49
61.90
38.89
84.62
69.63
80.32
PEARL-Qwen-3-4B-Instruct
86.45
91.01
63.96
94.93
30.12
61.90
50
84.86
70.40
80.38
Appendix
Table 6: Ablation study on the role of supervised fine-tuning (SFT) before PEARL training. We compare three controlled ablation settings using Qwen3-4B-Instruct as the base: (1) SFT only, (2) SFT followed by PEARL training, and (3) direct PEARL training without SFT. This ablation is separate from the final main-result checkpoint in Table 1 . Results show that while SFT provides substantial gains, the ultimate performance ceiling is similar whether SFT is used as an intermediate step or not.
Figure 8: Impact of trajectory filtering strategies on PEARL training. We compare three settings—Normal, Filter Truncated, and Filter All—and report average Pass@1 accuracy on MAMO-C and IndustryOR throughout training.
Figure 9: Impact of trajectory filtering strategies on PEARL training. We compare three settings: Normal (no filtering), Filter Truncated (filtering only truncated negative samples), and Filter All (filtering both truncated and format-violating negative samples). (a) Average Pass@1 accuracy on MAMO-C and IndustryOR benchmarks. (b) Training set raw reward. (c) Rollout time per step. (d) Truncated ratio during training.
Figure 10: Training dynamics of PEARL using Qwen3-4B-Instruct-2507 as the base model. The model is trained on open-source and curated data. (a) Pass@1 accuracy across three evaluation benchmarks (IndustryOR, MAMO-C, and NL4LP), showing consistent improvement during training. (b) Raw reward from the training set, demonstrating increasing performance that saturates in later stages. (c) Response length increases throughout training under GRPO optimization. (d) Tool calls rapidly converge, indicating the model quickly learns to utilize tools effectively. Online accuracy uses absolute objective error ≤10−3 , without format bonuses.
Benchmark
Acc.
Tool Calls
Base
PEARL
Base
PEARL
NL4Opt
68.22
85.28
0.47
1.31
MAMO-Easy
41.28
91.65
0.85
1.42
MAMO-Complex
12.61
76.12
0.15
1.55
NLP4LP
81.29
93.25
0.19
1.29
OptMATH
5.42
16.87
0.23
4.43
Appendix
Table 7: Scaffold-control comparison under the same PEARL inference process. Accuracy is pass@1 (%); tool calls are averaged per instance.
Figure 11: Distribution of tool-call counts for PEARL across the Table 1 benchmarks. Counts are concentrated at one or two calls. The eight-turn budget includes final-answer generation, so the eight-call bin must not be interpreted as eight calls followed by an additional final-answer turn.
Problem pool
Approx. count
Sources and preparation
Cleaned public data
10k
OR-Instruct/ORLM, OptMATH, and OptiBench-style resources; cleaning and reference-objective verification.
Expert-curated data
3.8k
Public OR articles and websites; LLM-assisted description rewriting, expert formulation checks, filtering, and solver verification.
Appendix
Table 8: Training problem pools and documented preparation steps. Counts refer to underlying problems, not online RL trajectories.
Benchmark
Instances
Released PEARL file
NL4Opt
214
nl4opt.jsonl
MAMO EasyLP
545
mamo_easy_lp_clean.jsonl
MAMO ComplexLP
111
mamo_complex.jsonl
NLP4LP
178
nl4lp.jsonl
OptMATH-Bench
166
OptMATH_Bench.jsonl
IndustryOR
42
IndustryOR.jsonl
Appendix
Table 9: Benchmark instance counts used in the evaluation.
Benchmark
n
PEARL
Official SIRL
Retrained SIRL
Pyomo
Gurobi
Pyomo
NL4Opt
214
90.19
87.85
84.51
MAMO-E
545
94.13
96.15
94.50
MAMO-C
111
87.39
79.28
44.14
NLP4LP
178
91.01
94.38
89.33
OptMATH-Bench
166
3.61
27.71
1.20
Appendix
Table 10: Qwen2.5-7B comparisons (Pass@1, %). Official SIRL uses its native Gurobi API and original training data; retrained SIRL and PEARL use Pyomo and the same 13.8k problems. Evaluation shares benchmark instances, validator, tolerance, and a common prompt template with interface-specific instructions. Results are point estimates, reported separately from Table 1 .
Training problem pool
Macro
Micro
Cleaned public 10k
64.12
78.95
Curated 3.8k
67.61
79.23
Combined 13.8k
70.40
80.38
Appendix
Table 11: Effect of training data on PEARL-Qwen3-4B. Training and evaluation settings are unchanged across rows. Scores are macro- and micro-averaged Pass@1 (%). The combined-data row matches the direct-RL results in Table 6 .
Method
Mean tokens
Execution success (%)
Pass@1 (%)
PEARL-Qwen3-4B
9,868.75
90.88
79.43
DeepSeek-V3.2
1,465.51
91.29
80.68
DeepSeek-V3.2-Think
2,323.56
91.06
79.90
Appendix
Table 12: Mean generated tokens, execution success, and Pass@1 on the same evaluation instances. All methods use the same solver, code parser, and objective-value check, but retain their own inference procedures. Token counts cover all model responses for each instance, including intermediate turns, and are averaged across instances.
Benchmark
Pass@1 (%)
NL4Opt
4.21
MAMO-E
8.62
MAMO-C
7.21
NLP4LP
5.62
OptMATH-Bench
0.00
IndustryOR
4.76
Appendix
Table 13: Pass@1 (%) for Qwen3-4B using LLMOPT’s inference workflow without LLMOPT-specific fine-tuning. These scores do not represent the released LLMOPT-Qwen2.5-14B model.
a School of Management Science and Engineering, Stanford University, Stanford, California 94305 · b Institute for Computational and Mathematical Engineering, Stanford University, Stanford, California 94305