Learning to Optimize through Solver-Grounded Self-Play

Xia Jiang; Yaoxin Wu; Chenyu Zhou; Mengzhu Xu; Wim P. M. Nuijten; Yingqian Zhang

cs.LGSep 28, 2026

Learning to Optimize through Solver-Grounded Self-Play

Authors: Xia Jiang, Yaoxin Wu, Chenyu Zhou, Mengzhu Xu, Wim P. M. Nuijten, Yingqian Zhang

Organizations: Eindhoven University of Technology · Shanghai Jiaotong University

Abstract

Optimization modeling is central to many decision-making scenarios, but traditionally requires extensive domain expertise. While Large Language Models (LLMs) have shown promise in automating this process, current training paradigms mainly rely on human-annotated or teacher-generated datasets. This dependence introduces a Generalization Ceiling, where models overfit to narrow data distributions, and Capability Anchoring, where models' reasoning is bounded by annotator proficiency and teacher model capability. In response, we propose OPT-Zero, the first fully self-play training framework for optimization modeling that requires zero external training data. OPT-Zero employs a single LLM in a dual-role closed loop: a Proposer that synthesizes increasingly challenging optimization problems alongside their mathematical formulations and solving code, and a Solver that attempts to resolve the problems given only natural-language problem descriptions. Grounded in execution feedback from external optimization solvers, we alternately train both roles using reinforcement learning. This process fosters an auto-curriculum in which the Proposer and Solver co-evolve: generating harder valid problems by the Proposer seamlessly enhances the structural reasoning ability of the Solver. Extensive results indicate that with zero curated data, OPT-Zero matches state-of-the-art data-dependent methods while exhibiting substantially stronger generalizability, establishing self-play training as a highly scalable paradigm for advancing LLM reasoning in modeling and solving optimization problems.

Figures & tables

Figure 1 : Average performance across six benchmarks vs. annotated training data size.

Method	Data Size †	Data Source	Human/Teacher Cost	Paradigm
ORLM ( Huang et al., 2025 )	32k	Expert + GPT-4	8 Experts + Teacher LLM	SFT
LLMOPT ( Jiang et al., 2025a )	$\sim$ 30k	Expert + GPT-4	12 Experts + Teacher LLM	SFT + Alignment
OptMATH ( Lu et al., 2025 )	200k	Deepseek-V3 Synthesis	Experts ‡ + Teacher LLM	SFT
SIRL ( Chen et al., 2025a )	10k	Solver-Assisted Synthesis	Experts ‡ + Teacher LLM	Online RL
StepORLM ( Zhou et al., 2026 )	50k	Expert + GPT-4o	Experts + Teacher LLM	SFT + Alignment
OR-PRM ( Wang et al., 2026 )	20k	Expert + GPT-4o	Experts ‡ + Teacher LLM	SFT + Alignment

Table 1 : Comparison of Data Requirements and Training Paradigms. † Data size with annotated ground-truth code. ‡ Experts-curated seed data is used.

Figure 2 : The overall framework of OPT-Zero .

Model	NL4Opt	EasyLP	ComplexLP	NLP4LP	IndOR	ReSocratic	Macro Avg.	Micro Avg.
Zero-shot LLMs
o4-mini	78.9	90.5	56.8	87.6	69.0	74.9	76.3	81.2
DeepSeek-R1	78.9	89.5	59.5	87.6	59.5	84.6	76.6	83.4
GLM-5	85.0	91.4	70.3	89.9	69.1	83.1	81.5	85.9
Gemma4-31B	87.3	89.4	66.7	87.6	69.0	83.9	80.7	85.1
Qwen3-14B	70.4	87.5	45.0	79.8	54.8	73.0	68.4	76.1

Table 2 : The overall performance of OPT-Zero and baselines with Pass@1 accuracy (%). *: results of the model are drawn from the original literature.

Model	NL4Opt	EasyLP	ComplexLP	NLP4LP	IndOR	ReSocratic	Macro Avg.	Micro Avg.
OPT-Zero	93.0	96.7	58.6	96.6	54.8	89.8	81.6	90.3
w/o Solver training	77.9	93.8	45.0	75.8	45.2	76.7	69.1	79.8
w/o Proposer training	87.8	93.6	48.7	89.3	50.0	80.9	75.1	84.3
w/o Reference buffer	84.5	94.0	40.5	87.1	50.0	81.4	72.9	83.2
w. Binary Solver reward	83.1	94.9	24.3	83.1	47.6	75.2	68.0	79.9
w. Binary Proposer reward	91.5	95.8	50.5	93.8	45.2	83.6	76.7	86.9

Table 3 : Ablation study of OPT-Zero on Pass@1 accuracy (%).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Test instances	Original benchmark reference
NL4Opt	213	Ramamonjison et al. (2023)
MAMO-EasyLP	545	Huang et al. (2024)
MAMO-ComplexLP	111	Huang et al. (2024)
NLP4LP	178	AhmadiTeshnizi et al. (2024)
IndustryOR (IndOR)	42	Huang et al. (2025)
ReSocratic	403	Yang et al. (2025b)

Appendix

Table 4: Cleaned test subsets used in the six-benchmark evaluation.

Component	Wall-clock hours
Proposer training	22.9
Problem generation and Solver pre-scoring	5.2
Solver training	43.3
Itemized subtotal	71.4
Approximate end-to-end total	73.0
Difference between total and subtotal (derived)	$\approx 1.6$

Appendix

Table 5: Wall-clock training time for the main Qwen3-8B run on four H100 GPUs.

Quantity	Value
Test problems	1,492
Candidates per problem	16
Total generated candidates	23,872
Generation hardware	$1\times$ H100
Generation time	3,813 s
Verification time	172 s

Appendix

Table 6: Timing for the complete

N=16

evaluation workload, with per-problem time amortized over batched generation and parallel verification.

Iter.	Vars	Cons.	Full Story
0	2	4	A manufacturing company produces two products, A and B. The company has two limited resources: raw materials and labor hours. Product A yields a profit of $5 per unit and requires 2 units of raw material and 4 labor hours. Product B yields a profit of$ 4 per unit and requires 3 units of raw material and 2 labor hours. The company has 120 units of raw material and 100 labor hours available. The goal is to determine the number of units of each product to produce to maximize profit while not exceeding the available resources.
4	9	15	A logistics company needs to allocate delivery routes for nine different product categories: Electronics, Clothing, Furniture, Books, Food, Appliances, Toys, Sports Equipment, and Home Decor. Each category has a different profit margin and resource consumption rate. The company has limited resources, including delivery vehicles and staff, which must be managed efficiently to maximize profits. Electronics require 2 units of delivery resource, Clothing require 3 units, Furniture require 4 units, Books require 1 unit, Food require 2 units, Appliances require 1 unit, Toys require 3 units, Sports Equipment require 2 units, and Home Decor require 1 unit. The company has a total of 100 units of delivery resource. Additionally, the total number of delivery routes across all categories must not exceed 20 due to staff limitations. There are specific constraints: Electronics and Clothing combined can have at most 10 routes, Furniture and Books combined can have at most 8 routes, Food and Appliances combined can have at most 6 routes, and Toys, Sports Equipment, and Home Decor combined can have at most 12 routes. The profit margins for each category are as follows: Electronics generate a profit of $50,000, Clothing generate$ 40,000, Furniture generate $60,000, Books generate$ 30,000, Food generate $20,000, Appliances generate$ 50,000, Toys generate $45,000, Sports Equipment generate$ 35,000, and Home Decor generate $25,000. The company wants to determine the optimal number of delivery routes for each product category to maximize total profit while staying within the resource and capacity constraints.
9	8	13	A city planner is tasked with optimizing the allocation of public resources for a major city event. The event has eight candidate resource types, represented by binary variables $x_{1},\ldots,x_{8}$ , where $x_{i}=1$ means that resource type $i$ is selected. Each resource type provides a different benefit: Resource 1 provides 10 units of impact, Resource 2 provides 15, Resource 3 provides 12, Resource 4 provides 8, Resource 5 provides 11, Resource 6 provides 9, Resource 7 provides 7, and Resource 8 provides 14. The planner must satisfy several operational constraints. At most five resource types can be selected in total. The resource usage coefficients for the eight resource types are $2,3,1,4,2,3,1,$ and $2$ , respectively, and the total weighted usage cannot exceed 10 units. In addition, at most two resources can be selected from the first group $\{1,2\}$ , at most three from the second group $\{3,4,5,6\}$ , and at most two from the third group $\{7,8\}$ . The goal is to select the subset of resource types that maximizes the total event impact while satisfying all usage and grouping constraints.
14	9	10	In a bustling city, a tech startup named InnovateTech is planning its annual hackathon. The event requires the allocation of resources across nine different project categories, each with varying levels of complexity and potential impact. The project categories are Algorithms, Machine Learning, Web Development, Mobile App Development, Data Visualization, Blockchain, Cybersecurity, IoT, and AI Ethics. The startup has a total of 8 slots available for project presentations. The sum of projects in Algorithms, Machine Learning, and Web Development cannot exceed 5; the sum of projects in Mobile App Development, Data Visualization, and Blockchain cannot exceed 3; and the sum of projects in Cybersecurity, IoT, and AI Ethics cannot exceed 2. The allocation must also satisfy several coverage constraints. At least one project must be allocated across Algorithms, Mobile App Development, and Cybersecurity; at least one project must be allocated across Machine Learning, Data Visualization, and IoT; and at least one project must be allocated across Web Development, Blockchain, and AI Ethics. In addition, the total number of projects across Algorithms, Data Visualization, and AI Ethics cannot exceed 4; the total number of projects across Machine Learning, Mobile App Development, and Cybersecurity cannot exceed 3; and the total number of projects across Web Development, Data Visualization, and IoT cannot exceed 2. The per-project costs for the nine categories are 2, 3, 4, 5, 6, 7, 8, 9, and 10 units, respectively. The startup wants to determine the optimal integer number of projects in each category to minimize total resource allocation cost while satisfying all slot, coverage, and grouping constraints.

Appendix

Table 7: Selected Proposer-generated stories from different self-play iterations.

Model	GSM8K Acc.	GPQA Acc.	HumanEval Pass@1	MBPP Pass@1
StepORLM-8B	82.1	33.8	18.9	47.4
LLMOPT-14B	20.9	39.4	50.6	65.8
OPT-Zero -8B	88.3	39.9	80.5	69.6

Appendix

Table 8 : General reasoning and coding performance of optimization-tuned models.

Model	NL4Opt	EasyLP	ComplexLP	NLP4LP	IndOR	ReSocratic
CPLEX
ORLM-8B	0.0	0.0	0.0	0.0	0.0	0.0
StepORLM-8B	1.9	2.9	0.0	4.5	2.4	3.0
LLMOPT-14B	23.5	36.7	27.9	39.3	19.1	26.8
OPT-Zero -8B	92.5	94.5	34.2	94.9	57.1	80.1
SCIP

Appendix

Table 9 : Zero-shot solver transferability of different fine-tuned optimization modeling LLMs.

Panel of Figure 5 — Figure 5 : Pass@ $N$ (oracle coverage) across six benchmarks at equal candidate counts. An instance is covered if at least one of its $N$ sampled candidates matches the reference answer.

Figure 6 : OPT-Zero performance improvement with the reflection mechanism.

Figure 7 : Model performance with different sampling strategies.

Method	NL4Opt	EasyLP	ComplexLP	NLP4LP	IndOR	ReSocratic
Solve-rate-based DPO	86.8	89.4	29.7	88.8	42.9	80.9
Structural-reward GRPO	93.0	96.7	58.6	96.6	54.8	89.8

Appendix

Table 10: Comparison between our method and the solve-rate-based DPO baseline.

Model	NL4Opt	EasyLP	ComplexLP	NLP4LP	IndOR	ReSocratic	Macro Avg.	Micro Avg.
Qwen3-8B-Base	70.4	83.7	15.3	66.3	35.7	65.5	56.2	68.4
+ OPT-Zero	93.0	96.7	58.6	96.6	54.8	89.8	81.6	90.3
Absolute Gain ( $\Delta$ )	$+22.6$	$+13.0$	$+43.3$	$+30.3$	$+19.1$	$+24.3$	$+25.4$	$+21.9$
Qwen3.5-4B	12.7	24.4	14.4	11.2	7.1	12.7	13.8	16.8
+ OPT-Zero	91.6	95.2	51.4	96.1	57.1	93.1	80.8	89.9
Absolute Gain ( $\Delta$ )	$+78.9$	$+70.8$	$+37.0$	$+84.9$	$+50.0$	$+80.4$	$+67.0$	$+73.1$

Appendix

Table 11 : Performance of OPT-Zero in fine-tuning LLMs with different architectures and sizes.

Model	OptMATH-Bench	MIPLIB-NL
Gemma4-31B-IT	41.0	8.92
+ OPT-Zero	48.2	11.74
Absolute gain (percentage points)	+7.2	+2.82

Appendix

Table 12: Gemma4-31B-IT performance before and after OPT-Zero on two additional held-out benchmarks. Scores are objective-value-based Pass@1 (%); gains are in percentage points.

Successful rollouts	Share	Empirical interpretation	Treatment at the solve-rate gate
$0/8$	55.7%	No successful rollout observed	Excluded at this scoring step
$1$ – $6/8$	32.3%	Mixed outcomes; hard or medium	Eligible for retention
$7$ – $8/8$	12.0%	High observed solve rate	$7/8$ eligible (easy); $8/8$ excluded

Appendix

Table 13: Empirical solve-rate distribution of canonically verified Proposer candidates across 15 iterations. Each candidate is evaluated with eight rollouts of the contemporaneous Solver before filtering. Buffer admission further applies structural deduplication and capacity control.

Model / inference	NL4Opt	EasyLP	ComplexLP	NLP4LP	IndOR	ReSocratic	Macro	Micro
Qwen3-8B-Base / single	70.4	83.7	15.3	66.3	35.7	65.5	56.2	68.4
Qwen3-8B-Base / CoE	87.3	83.5	45.0	87.6	40.5	83.4	71.2	80.4
OPT-Zero -8B / single	93.0	96.7	58.6	96.6	54.8	89.8	81.6	90.3
OPT-Zero -8B / CoE	95.4	96.7	62.2	96.6	52.4	93.3	82.8	91.8

Appendix

Table 14 : Single-generation and CoE results on the six-benchmark suite (%). Single-generation uses one LLM call per problem, while CoE uses approximately eight calls with the same procedure for both models.

Training configuration	Micro Avg. (%)
Full OPT-Zero -8B	90.3
Without Proposer code generation	87.1
Difference (full minus ablated, percentage points)	+3.2

Appendix

Table 15: Ablation of Proposer code generation and its code-correctness reward. The Solver retains its original outputs and reward. Scores are Micro-average objective-value accuracy (%).

Corpus sample	Statements	Vendi Score $\uparrow$	NN redundancy $\downarrow$
OPT-Zero reference buffer	300	244	0.32
OptMATH-Train	300	178	0.56

Appendix

Table 16: Textual diversity of two samples of 300 problem statements, measured using word 1–2 gram TF-IDF features and cosine similarity.

Method	Task source and adaptation	Learned roles	Feedback and scope
OptMATH ( Lu et al., 2025 )	Seed formulations and generators support bidirectional synthesis of a training corpus	Downstream optimization model trained on synthesized examples	Forward reconstruction and validation in corpus construction
SIRL ( Chen et al., 2025a )	Externally constructed, filtered optimization tasks	Optimization-modeling policy updated by RL	Solver execution, objective accuracy, and instance-level modeling feedback
StepORLM ( Zhou et al., 2026 )	Teacher-synthesized training set; new solution trajectories collected during co-evolution	Policy and generative process reward model	Solver outcome feedback and learned process supervision
Absolute Zero ( Zhao et al., 2025 )	Self-generated code-reasoning tasks; no external task corpus	A shared model proposes and solves tasks	Code execution validates tasks and answers
R-Zero ( Huang et al., 2026 )	Challenger-generated reasoning tasks; no external task corpus	Separately optimized Challenger and Solver in the primary configuration	Solver-relative challenge and solution feedback
OPT-Zero	New optimization problems generated and pre-scored during training; difficulty-refreshed buffer	One shared model alternates Proposer and Solver updates	Canonical formulation solving, code checks, and adaptive structural targets

Appendix

Table 17 : Task sources, learned roles, and feedback mechanisms in the primary configurations of representative methods.

Explore similar work

Jun 24, 2026cs.LG

MiniOpt: Reasoning to Model and Solve General Optimization Problems with Limited Resources

Achieving strong optimization generalization across diverse optimization problems while requiring limited training resources remains a challenging problem for optimization-oriented large language models (LLMs). Existing approaches typically rely on large-scale supervised datasets, costly reasoning annotations, and expensive intermediate step verification, resulting in substantial training overhead. To address these challenges, we propose MiniOpt, a reinforcement learning framework that learns to solve optimization problems through an "reasoning-to-model-and-solve" paradigm. MiniOpt decomposes optimization reasoning into structured optimization modeling and executable solver generation. Building upon this paradigm, we introduce OptReward, a reward function with hierarchical score structure that jointly evaluates formulation and solution, enabling effective policy learning without expert demonstrations. We further develop an optimization-oriented policy optimization strategy that improves exploration efficiency and stabilizes reinforcement learning for compact models. Extensive experiments show that MiniOpt-3B exhibits strong optimization generalization across various optimization types, problem scenarios, and task domains. For models with fewer than 10B parameters, MiniOpt series achieves the highest average solving accuracy (SA). For models with more than 10B parameters, MiniOpt still shows competitive performance. These results suggest that optimization-oriented reward design and reinforcement learning provide an effective pathway for developing compact optimization-specialized language models with strong optimization generalization capabilities. The code is available at https://github.com/Hsiang-1/MiniOpt.

Ke Zhao, Zixiang Di, Hong Qian +9

East China Normal University, Shanghai 200062, China · AntGroup, Hangzhou 310000, China · Southern University of Science and Technology, Shenzhen 518055, China +1

Optimization Modeling Large Reasoning Models

Ask PDF

Sep 28, 2026cs.LG

Direct Self-Evolving Optimization: Evolving LLMs without Challenger Training

Self-evolving language models improve by generating tasks and learning from their own feedback, but adapting the task generator often requires a separate challenger-training loop. Can we generate tasks adapted to the current solver without explicitly training a challenger? We introduce \textbf{D}irect Self-\textbf{E}volving \textbf{O}ptimization (DEO), which replaces challenger parameter updates with solver-guided task sampling. The KL-regularized challenger objective defines an exponential tilt of a fixed base task distribution. DEO uses this distribution as a sampling target: a frozen LLM generates and mutates tasks, the solver scores them, and an approximate Metropolis selection rule refines the training pool. Only the solver is trained. Theoretically, for an idealized variant that samples exactly from the tilted distribution, and under regularity, local gradient-dominance, and initialization conditions, we show that DEO learns distributionally robust reasoning ability. In experiments, DEO achieves reasoning performance competitive with R-Zero while using over

50\%

less wall-clock training time, and improves reasoning accuracy over a no-walk ablation. Replacing the task generator with a frozen API-only LLM further improves the local solver, illustrating a capability enabled by removing challenger training.

Yuyang Deng, Yu Wang, Jiayun Wang

Accenture, Center for Advanced AI · Georgia Institute of Technology

On-Policy Self-Evolution LLM Reasoning Strategies Adaptive Sampling Data Generation Competition

Ask PDF

May 11, 2026cs.LG

G-Zero: Self-Play for Open-Ended Generation from Zero Data

Self-evolving LLMs excel in verifiable domains but struggle in open-ended tasks, where reliance on proxy LLM judges introduces capability bottlenecks and reward hacking. To overcome this, we introduce G-Zero, a verifier-free, co-evolutionary framework for autonomous self-improvement. Our core innovation is Hint-

δ

, an intrinsic reward that quantifies the predictive shift between a Generator model's unassisted response and its response conditioned on a self-generated hint. Using this signal, a Proposer model is trained via GRPO to continuously target the Generator's blind spots by synthesizing challenging queries and informative hints. The Generator is concurrently optimized via DPO to internalize these hint-guided improvements. Theoretically, we prove a best-iterate suboptimality guarantee for an idealized standard-DPO version of G-Zero, provided that the Proposer induces sufficient exploration coverage and the data filteration keeps pseudo-label score noise low. By deriving supervision entirely from internal distributional dynamics, G-Zero bypasses the capability ceilings of external judges, providing a scalable, robust pathway for continuous LLM self-evolution across unverifiable domains.

Chengsong Huang, Haolin Liu, Tong Zheng +7

Washington University in St. Louis · University of Virginia · University of Maryland +1

Self-Play Recursive Self-Improvement

Ask PDF