Cheap Thrills: Effective Amortized Optimization Using Inexpensive Labels

Khai Nguyen; Petros Ellinas; Anvita Bhagavathula; Priya L. Donti

cs.LGMar 5, 2026

Cheap Thrills: Effective Amortized Optimization Using Inexpensive Labels

Authors: Khai Nguyen, Petros Ellinas, Anvita Bhagavathula, Priya L. Donti

Organizations: Massachusetts Institute of Technology, Cambridge, MA, USA · Technical University of Denmark, Kongens Lyngby, Denmark

Abstract

To scale optimization and simulation, prior work has explored training machine-learning surrogates that map problem parameters to solutions inexpensively at inference time. Unfortunately, commonly used approaches, including supervised and self-supervised learning with either soft or hard feasibility enforcement, face inherent challenges such as reliance on expensive high-quality labels or difficult optimization landscapes. To address their trade-offs, we propose a novel framework that collects "cheap" imperfect labels, performs supervised model pretraining with a merit loss-based termination scheme, and finally refines the model through self-supervised learning to improve final performance. Empirical validation across challenging domains -- including nonconvex constrained optimization, power-grid operation, and stiff dynamical systems -- shows that this three-stage strategy yields faster convergence; improved accuracy, feasibility, and optimality; and up to 59x reductions in total offline computational cost. We further analyze why and when our framework improves surrogate model training, finding that (i) merit loss is an informative signal and (ii) only small numbers of cheap, inexact labels are needed to place the model in a favorable regime for self-supervised learning.

Figures & tables

Figure 1: Overview of our approach. We propose a simple but effective three-stage amortized optimization framework: (1) collecting cheap imperfect labels via approximate solvers, (2) supervised model pretraining using these labels alongside merit loss validation, and (3) self-supervised training.

Panel of Figure 2 — Figure 2: Training loss $\mathcal{L}_{\mathrm{SSL}}(\theta)$ (left) and merit loss $\mathcal{M}(\theta)$ (right) landscapes along two weight directions from a trained FSNet in the nonconvex constrained optimization task. The trained weights land in a smooth local minimum of $\mathcal{L}_{\mathrm{SSL}}$ . When evaluated with the more task-faithful $\mathcal{M}$ , the surroundings are extremely rugged with sharp ridges and multiple local minima. This explains the potential failure of vanilla SSL when trained directly on $\mathcal{M}$ . Here, $\rho_{\text{merit}}=10^{4}\rho_{\text{SSL}}$ .

Method	Mean Objective $\downarrow$	Equality Violation $\downarrow$		Inequality Violation $\downarrow$
	Mean Objective $\downarrow$	Mean	Max	Mean	Max
Solver	-2.40 $\pm$ 0.00	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	3.40e-09 $\pm$ 0.00	5.90e-08 $\pm$ 0.00
Supervised w/ high-quality labels	-6.16 $\pm$ 0.06	1.07e+01 $\pm$ 3.92e-02	1.85e+01 $\pm$ 6.12e-01	7.16e+01 $\pm$ 8.22e-01	1.99e+02 $\pm$ 4.76e+00
Supervised w/ cheap labels	22.49 $\pm$ 1.24	1.90e+00 $\pm$ 4.44e-02	2.72e+00 $\pm$ 8.38e-02	1.23e-01 $\pm$ 9.51e-02	2.41e+00 $\pm$ 1.54e+00
Penalty	-0.06 $\pm$ 1.26	9.63e-01 $\pm$ 5.65e-02	1.54e+00 $\pm$ 2.06e-01	2.72e-02 $\pm$ 9.68e-03	6.11e-01 $\pm$ 1.02e-01
Our Penalty	-3.29 $\pm$ 0.06	9.56e-01 $\pm$ 3.69e-02	1.52e+00 $\pm$ 5.25e-02	2.28e-02 $\pm$ 6.06e-03	5.89e-01 $\pm$ 6.59e-02

Table 1: Performance comparison on a synthetic constrained optimization benchmark, evaluated on 2,000 hold-out test instances, over four random seeds. Our variants that use cheap-label supervision consistently achieve lower objective values and improved feasibility, with reduced output variance compared to other learning-based baselines. Compared to the solver, amortized models solve 2,000 problem instances over

40{,}000\times

faster with batched GPU inference, and over

100\times

faster under sequential CPU execution (see Table A.2 ).

	Gen (s)	SL (s)	SSL (s)	Total (s)
Synthetic optimization
Supervised w/ 7,000 10.0 labels	28118	190	0	28308
Penalty	0	0	180	180
Penalty w/ warm-start via 800 0.5 labels (Ours)	400	40	40	480
FSNet	0	0	990	990
FSNet w/ warm-start via 800 0.5 labels (Ours)	400	40	495	935

Table 2: Breakdown of offline computational time across label generation, supervised learning, and self-supervised learning. Pure SL is dominated by expensive label generation, while penalty-based SSL avoids labels but requires longer training. Our method uses a small number of cheap labels, substantially reducing labeling cost and accelerating SSL convergence, achieving comparable or better solution quality with orders-of-magnitude lower total offline time. For these timing comparisons, both vanilla and our methods are trained to the same level of task performance.

Panel of Figure 5 — Figure 5: Performance of our method across imperfect label qualities and quantities. Left: Across different levels of label refinement (with 10 s max CPU time per instance allowing the oracle solver to converge on all instances), performance remains statistically indistinguishable, indicating that higher-accuracy labels provide only marginal additional benefit within our framework. Right: Performance improves rapidly with a small number of labels and then plateaus, indicating that coarse manifold structure is captured early and is sufficient for subsequent SSL.

Method	Mean Obj. $\downarrow$	Mean Eq. Vio. $\downarrow$	Mean Ineq. Vio. $\downarrow$
Vanilla Penalty	-0.06 $\pm$ 1.26	9.63e-01 $\pm$ 5.65e-02	2.72e-02 $\pm$ 9.68e-03
Penalty w/ random labels	3.89 $\pm$ 0.03	1.03e+00 $\pm$ 3.81e-02	3.26e-02 $\pm$ 9.31e-03
Penalty w/ cheap labels (Ours)	-3.29 $\pm$ 0.06	9.56e-01 $\pm$ 3.69e-02	2.28e-02 $\pm$ 6.06e-03
Vanilla FSNet	-0.73 $\pm$ 1.42	5.85e-05 $\pm$ 2.42e-05	7.89e-07 $\pm$ 5.03e-07
FSNet w/ random labels	1.46 $\pm$ 0.68	5.15e-05 $\pm$ 2.39e-05	3.78e-06 $\pm$ 5.51e-06
FSNet w/ cheap labels (Ours)	-3.28 $\pm$ 0.14	4.06e-05 $\pm$ 8.54e-06	5.01e-07 $\pm$ 5.91e-08

Table 3: Assessment of random-label warm-start on synthetic constrained optimization. We compare SSL performance with no warm-start, random-label warm-start, and cheap-label warm-start. Random solution labels are sampled uniformly over the cheap label ranges in each dimension.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Mean Objective $\downarrow$	Equality Violation $\downarrow$	Inequality Violation $\downarrow$
	Mean Objective $\downarrow$	Mean (Max)	Mean (Max)
FSNet	-0.73 $\pm$ 1.42	5.85e-05 $\pm$ 2.42e-05 (1.93e-03 $\pm$ 1.16e-03)	7.89e-07 $\pm$ 5.03e-07 (1.31e-04 $\pm$ 1.62e-04)
Hybrid FSNet w/ cheap labels	2.68 $\pm$ 4.55	7.17e-05 $\pm$ 9.49e-06 (1.11e-03 $\pm$ 1.85e-04)	4.12e-07 $\pm$ 1.00e-07 (3.35e-05 $\pm$ 1.42e-05)
Semi-Supervised FSNet w/ cheap labels	-0.33 $\pm$ 0.50	8.84e-05 $\pm$ 2.36e-05 (1.00e-03 $\pm$ 2.94e-04)	5.43e-07 $\pm$ 1.23e-07 (3.48e-05 $\pm$ 1.17e-05)
FSNet w/ no-label warm-start (feas.)	-2.99 $\pm$ 0.01	9.97e-05 $\pm$ 2.87e-06 (8.39e-04 $\pm$ 7.14e-05)	5.05e-07 $\pm$ 4.04e-08 (2.14e-05 $\pm$ 1.06e-06)
FSNet w/ no-label warm-start (both)	0.30 $\pm$ 0.03	8.35e-05 $\pm$ 6.81e-06 (1.13e-03 $\pm$ 1.16e-04)	4.86e-07 $\pm$ 6.31e-08 (3.62e-05 $\pm$ 4.31e-06)

Appendix

Table A.1 : Performance comparison of FSNet with different training schemes on a synthetic constrained optimization benchmark (800 labels, max CPU time 0.5s). We report the mean and maximum objective value,

\ell_{1}

equality violation, and

\ell_{1}

inequality violation for vanilla FSNet and FSNet with different warm-start strategies. The pretraining loss in (feas.) only includes feasibility and (both) additionally includes objective. Lower is better.

Method	Solution Time (s) $\downarrow$
	Batched GPU	Sequential CPU
Solver	–	8033.60 $\pm$ 0.00
Supervised w/ high-quality data	0.0003 $\pm$ 0.0000	0.9479 $\pm$ 0.0351
Supervised w/ cheap data	0.0003 $\pm$ 0.0000	0.9479 $\pm$ 0.0351
Penalty	0.0003 $\pm$ 0.0000	0.9479 $\pm$ 0.0351
Our Penalty	0.0003 $\pm$ 0.0000	0.9479 $\pm$ 0.0351

Appendix

Table A.2 : Comparison of solution time at inference on the synthetic constrained optimization benchmark. Compared to the solver (IPOPT), amortized models solve 2,000 instances over

40{,}000\times

faster with batched GPU inference, and over

100\times

faster under sequential CPU execution. Our framework preserves such runtime benefits.

Method	Mean Objective $\downarrow$	Equality Violation $\downarrow$		Inequality Violation $\downarrow$
	Mean Objective $\downarrow$	Mean	Max	Mean	Max
Vanilla SSL	-0.73 $\pm$ 1.42	5.85e-05 $\pm$ 2.42e-05	1.93e-03 $\pm$ 1.16e-03	7.89e-07 $\pm$ 5.03e-07	1.31e-04 $\pm$ 1.62e-04
SSL from SL step 50	-1.32 $\pm$ 0.04	6.69e-05 $\pm$ 1.39e-05	1.35e-03 $\pm$ 1.43e-04	4.60e-07 $\pm$ 1.47e-07	4.89e-05 $\pm$ 1.54e-05
SSL from SL step 100	-1.28 $\pm$ 1.12	2.17e-05 $\pm$ 6.56e-06	2.32e-03 $\pm$ 6.21e-04	4.79e-07 $\pm$ 2.70e-07	9.71e-05 $\pm$ 6.03e-05
SSL from SL step 250*	-2.60 $\pm$ 0.16	2.16e-05 $\pm$ 4.51e-06	1.89e-03 $\pm$ 1.77e-04	3.32e-07 $\pm$ 3.16e-08	6.30e-05 $\pm$ 1.41e-05
SSL from SL step 500	-0.05 $\pm$ 0.19	4.70e-05 $\pm$ 3.59e-05	7.85e-03 $\pm$ 8.98e-03	6.35e-06 $\pm$ 9.02e-06	1.55e-03 $\pm$ 2.28e-03

Appendix

Table A.3 : Performance comparison on a synthetic constrained optimization benchmark of SSL warm-started from different SL pretraining steps. We use

7,000

inexact labels (with a maximum CPU solve time of 2.0 s per label) for supervised warm-starting. SL achieves lowest validation merit at around epoch 250. We then perform FSNet SSL for 300 epochs from different SL pretraining steps. Results are reported over 4 random seeds.

Figure A.4 : Analysis of pre-training strategy and computational efficiency for RAR physics-informed learning. (a) Impact of warm-starting steps (left & center): The learning curves (left) show the evolution of validation MSE, where the vertical lines mark the transition from warm-starting to self-supervised learning. The optimal cutoff at epoch 5,000 (green) leads to better convergence. The comparative bar chart (center) highlights that this optimal duration yields the lowest final error ( $\approx 1.69\times 10^{-6}$ ). (b) Label generation runtime (right): Our cheap labeling strategy achieves an $\sim$ 100 $\times$ speedup compared to ground-truth generation.

Max CPU Time	Obj.	Eq. Vio.	Ineq. Vio.	Success Rate	Total Time (s)	# Iters	Merit
0.5	1.06	3.77	122.23	0.00	4957	14	$1.26\times 10^{7}$
1.5	11.54	1.31	39.53	0.00	14712	46	$4.08\times 10^{6}$
2.0	19.33	0.37	11.68	0.00	20265	64	$1.21\times 10^{6}$
3.0	10.32	0.01	1.05	0.03	30229	95	$1.06\times 10^{5}$
4.0	-1.67	0.00	0.21	0.56	37845	122	$2.10\times 10^{4}$
10.0*	-2.44	0.00	0.00	1.00	40168	128	$-2.44\times 10^{0}$

Appendix

Table A.4 : Label generation cost and quality for synthetic constrained optimization. The maximum CPU time allowed per instance controls both the number of solver iterations and the total wall-clock time required to generate 10,000 labels, which we use to vary label quality. Allowing 10.0 s per instance ensures solver convergence for all problems under tight tolerances, yielding the highest-quality labels collected.

Method	Normalized Obj.	Eq. Violation	Ineq. Violation	Total Time (s)	Merit
ACOPF ( $10^{-6}$ )	1.000	$5.0\times 10^{-4}$	$1.743\times 10^{-6}$	827	$5.12\times 10^{1}$
DCOPF	0.957	$4.978\times 10^{1}$	$8.265\times 10^{-7}$	34	$4.98\times 10^{6}$

Appendix

Table A.5 : Comparison of ACOPF and DCOPF data generation. We report normalized average objective (ACOPF being 1.00),

\ell_{2}

constraint violation, merit value and total computation time of producing 10,000 labels. We solve both formulations to a tight tolerance without any budget constraints.

Table C.6 : Hyperparameter settings for synthetic optimization experiments.

Table C.7 : Hyperparameter settings for AC-OPF experiments.

Table C.8 : Hyperparameter settings for flow-map PINN and local-minima experiments.

Learning to Optimize

Learning to Optimize uses data to adapt algorithms, solver decisions, and solution maps for faster or better optimization.

See all 138 papers

Oct 20, 2026

Self-Supervised Tuning of Low-Fidelity Optimization Models via High-Fidelity Feedback

Low-fidelity (LF) optimization models are widely used in large-scale operations because they can be solved efficiently and reliably, but they often exhibit systematic errors relative to high-fidelity (HF) formulations that better capture physical and operational constraints. Existing learning-based approaches typically rely on supervised or semi-supervised training with HF solutions, which are expensive to compute and may not align with HF operational objectives. We propose a self-supervised framework that parameterizes LF optimization models directly from HF feedback. The low-fidelity problem is embedded as a differentiable optimization layer and trained end-to-end, while a HF correction problem evaluates its solution and provides gradient information through its dual variables, without requiring HF solution labels. We evaluate the framework on optimal power flow, optimal gas flow, and pooling problems, spanning different sources of low- and high-fidelity mismatch. Across all three applications, the self-supervised approach improves objective quality and feasibility over untuned LF models and generally outperforms semi-supervised training. The learned parameterizations also generalize to unseen operating conditions and, for network applications with up to 30,000 nodes, to systems not encountered during training.

Gonzalo E. Constante-Flores, André H. Quisaguano-Paredes, Asha Ramanujam +2

Department of Electrical, Computer & Energy Engineering, University of Colorado Boulder · Davidson School of Chemical Engineering, Purdue University · ExxonMobil Technology and Engineering Company

Learning to Optimize Multi-Fidelity Optimization Self-Supervised Learning Differentiable Optimization

Ask PDF

Jun 17, 2026

A Solver-Free Training Method for Predict-then-Optimize

We propose a scalable method for training prediction (machine learning) models in the predict-then-optimize paradigm, where model outputs serve as coefficients for a subsequent linear optimization task. Directly minimizing the empirical decision regret is intractable for linear programming and combinatorial optimization since the decision mapping is piecewise constant, and the gradients are zero almost everywhere. While existing methods address this by smoothing the differentiation process, they suffer from scalability issues, since a computationally expensive solver call is required for every gradient evaluation. To address this, we propose a decision-focused learning pipeline based on a measure transformation principle, which yields a new surrogate loss that is completely optimization-solver-free during training. We establish theoretical guarantees, including Fisher consistency and excess risk bounds. Empirically, our method achieves decision quality competitive with state-of-the-art methods while reducing training time by orders of magnitude.

Beichen Wan, Mo Liu

Department of Statistics and Operations Research, University of North Carolina at Chapel Hill, NC, USA.

Predict-Then-Optimize Learning to Optimize Decision-Focused Learning

Ask PDF

Aug 16, 2026

Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modeling

Agents that learn from experience improve at optimization modeling by storing solved trajectories and reusing them as skills. A wrong trajectory that enters the library can be retrieved again and again, and on a stream of new problems there is no ground-truth answer to decide with. Existing learners admit trajectories by matching known optima or labels, and label-free substitutes such as execution success or agreement at one instance can admit wrong models. We introduce ADMITOR, a label-free admission gate. It generates models from three model families, runs each on the stated problem and on instances with resampled parameters, keeps the largest group of models whose optimal values agree on every instance across families, and applies a threshold fitted on solver-verified problems to accept, abstain, or escalate, with a finite-sample bound on the false-discovery rate among accepted values. Inside a state-of-the-art skill learner, ADMITOR raises candidate-level admission precision to 0.927, against 0.871 for majority vote over the host's own samples and 0.726 for execution success, and its library, the smallest of the four, reaches the highest macro accuracy over five public benchmarks, 58.4 against 54.8 for majority vote. An ablation on the same records shows that the gain comes from the accepted value being external to the learner and unanimous across families; on this stream, resampling never changed an accepted value and only reduced coverage. The false-discovery bound holds on the calibration set but not on the benchmark stream: an audit of every false certificate traces most of them to benchmark texts that omit or round the numbers needed to reproduce the labeled answer, and a label-free check of the extracted numbers against the text flags most of these cases.

Junbo Jacob Lian, Huiling Chen, Hanzhang Qin +1

Institute of Operations Research and Analytics National University of Singapore, Singapore · Wenzhou Buyi Pharmacy, Wenzhou, China · College of Computer Science and Artificial Intelligence Wenzhou University, Wenzhou, China

Large Language Model-Guided Optimization Learning to Optimize

Ask PDF

Abstract

Learning to Optimize

Explore similar work

Self-Supervised Tuning of Low-Fidelity Optimization Models via High-Fidelity Feedback

A Solver-Free Training Method for Predict-then-Optimize

Admission Without Answers: Label-Free Certification and Experience Learning for LLM-Based Optimization Modeling