A tale of perfect fit and phantom optima: how data-driven models can fail in real-time optimization

Prithvi Dake; Rahul Bindlish; James B. Rawlings

eess.SYAug 24, 2026

A tale of perfect fit and phantom optima: how data-driven models can fail in real-time optimization

Authors: Prithvi Dake, Rahul Bindlish, James B. Rawlings

Organizations: Department of Chemical Engineering, University of California, Santa Barbara, CA 93106, United States · Dow Chemical Company, TX, United States

Abstract

Real-time optimization (RTO) relies on process models to locate economically optimal operating conditions. Because developing first-principles models requires significant process knowledge, data-driven alternatives are increasingly attractive. Modern machine-learning models can fit historical plant data accurately and often pass standard validation tests. Whether such models can be trusted for economic optimization, however, remains unclear. We investigate this question using a vinyl acetate monomer benchmark process with a unique, well-conditioned economic optimum. We train a structured hybrid model that combines known mass balances and thermodynamics with a neural-network closure for unknown kinetics, and a fully data-driven neural ordinary differential equation (ODE) model. Both models reproduce plant measurements accurately and exhibit little variation in predictions across random initializations. Yet their economic optima differ substantially from that of the plant. Where the plant returns a single optimum on multistart search, the trained models return many phantom optima. We further show that the training optimizer alone can be yet another source of error. Even with noise-free data and initialization at weights that recover the plant optimum, stochastic gradient training can drift to weights that yield substantially worse RTO solutions. The identified model is thus an artifact of the training optimizer as well as the data. These results demonstrate that a good predictive fit of all available measurements does not guarantee reliable economic performance. A data-driven model for RTO should at least be required to recover the optimum on a decision-oriented benchmark like the one developed here before being considered for plant testing and application.

Figures & tables

Panel of Figure 1 — Figure 1: (A) Historical plant datasets are used to train (B) a structured model and a black-box ML/AI closure model, each deployed in real-time optimization (RTO). (C) Near-identical fits map to very different optima, the worst-case structured model loses close to 30% profit and the black-box model over 50%. A reliable data-driven model must recover the optimum, not be judged solely on the validation dataset.

Figure 3: A snapshot of training data from the plant model with no measurement noise. The measurements are shown in red and the decision variables in black. The dotted grey lines represent $x_{s}^{\star}$ and $u_{s}^{\star}$ from section 2.2 .

Model	Case	Training problem and measurements	NN initialization	Noise $n_{l}$
Structured	rmeas	eq. 46 , rates	random	$0$
	cmeas_rinit	eq. 45 , concentrations	rmeas solution	$0$
	cmeas	eq. 45 , concentrations	random	$0$
	cmeas_noise	eq. 45 , concentrations	random	$10^{-2}$
Black-box	bbox	eq. 47 , concentrations	random	$0$
Black-box	bbox_noise	eq. 47 , concentrations	random	$10^{-2}$

Table 1: Summary of the six data-driven cases.

n_{l}

is the sensor-noise level in eq. 50 . A random NN initialization uses the LeCun normal initialization popular for tanh -based neural networks ( LeCun et al., 2002 , pg. 20) .

Parameter	$n_{l}=0$		$n_{l}=10^{-3}$		$n_{l}=10^{-2}$
$\log{\tilde{k}_{1}}$	$-4.836$ $\pm$	$2.7\times 10^{-11}$	$-4.840$ $\pm$	$4.4\times 10^{-3}$	$-4.876$ $\pm$	$3.9\times 10^{-2}$
$\log{\tilde{E}_{1}}$	$10.61$ $\pm$	$4.3\times 10^{-11}$	$10.60$ $\pm$	$7.1\times 10^{-3}$	$10.55$ $\pm$	$6.7\times 10^{-2}$
$a_{1}$	$1.100$ $\pm$	$5.2\times 10^{-11}$	$1.092$ $\pm$	$8.6\times 10^{-3}$	$1.021$ $\pm$	$7.7\times 10^{-2}$
$b_{1}$	$1.100$ $\pm$	$5.5\times 10^{-11}$	$1.091$ $\pm$	$9.0\times 10^{-3}$	$1.014$ $\pm$	$8.1\times 10^{-2}$
$c_{1}$	$1.000$ $\pm$	$4.4\times 10^{-11}$	$0.993$ $\pm$	$7.2\times 10^{-3}$	$0.934$ $\pm$	$6.5\times 10^{-2}$
$\log{\tilde{k}_{2}}$	$-7.992$ $\pm$	$1.1\times 10^{-11}$	$-7.993$ $\pm$	$1.8\times 10^{-3}$	$-7.994$ $\pm$	$1.8\times 10^{-2}$

Table 2: Parameter estimation results and profit loss for the control experiment with parametric models for different levels of measurement noise.

Figure 4: Concentration predictions on the validation set for (a) rmeas , (b) cmeas_rinit , (c) cmeas and (d) cmeas_noise . Measurements are shown in red. The shaded bands are the ensemble spread, taken as the 5–95% quantiles of the 20 random initializations.

Figure 5: Parity plots of the reaction rates $[r_{1},r_{2}]^{\mathrm{T}}$ on the validation set for (a) rmeas , (b) cmeas_rinit , (c) cmeas and (d) cmeas_noise . Top row shows $r_{1}$ and bottom row $r_{2}$ . The dotted line is ideal parity. The bands are the ensemble spread, taken as the 5–95% quantiles across the 20 random initializations.

Figure 6: Concentration predictions on the validation set for the black-box model, (a) bbox and (b) bbox_noise . Measurements are shown in red. The bands are the ensemble spread, taken as the 5–95% quantiles of the 20 random initializations.

Model	Case	NN architecture	rmse meas	rmse val	Train time (hr)
Structured	rmeas	$r_{\mathrm{NN},r_{1}}=[4,64,1]$ $r_{\mathrm{NN},r_{2}}=[3,64,1]$	$0.00067\pm 0.00012$	$0.0069\pm 0.0021$	$0.22\pm 0.08$
	cmeas_rinit		$0.00435\pm 0.00060$	$0.0146\pm 0.0024$	$11.97\pm 0.94$
	cmeas		$0.00966\pm 0.00131$	$0.0219\pm 0.0035$	$12.27\pm 1.17$
	cmeas_noise		$0.01472\pm 0.00064$	$0.0217\pm 0.0030$	$12.45\pm 0.99$
Black-box	bbox	$f_{\mathrm{NN}}=[16,128,7]$	$0.03224\pm 0.00187$	$0.0484\pm 0.0051$	$12.01\pm 1.06$
Black-box	bbox_noise	$f_{\mathrm{NN}}=[16,128,7]$	$0.03410\pm 0.00177$	$0.0481\pm 0.0057$	$12.04\pm 0.90$

Table 3: Structured- and black-box-model results across cases. Values are reported as mean

\pm

1.96\,\sigma

over 20 random initializations (95% interval). rmse meas and rmse val are the mean-normalized errors of eq. 66 on the measurement and validation sets; training time is wall-clock.

Figure 7: Plant profit landscape $\ell(T,\alpha)$ with the remaining degrees of freedom fixed at the plant optimum. The star marks the plant optimum $(T^{\star},\alpha^{\star})$ . The surface is unimodal and well-conditioned.

Figure 8: Model-predicted profit landscape $\hat{\ell}(T,\alpha)$ that each trained model presents to the RTO optimizer. Panels, in order of degrading structure and data, are (a) rmeas , (b) cmeas_rinit , (c) cmeas , (d) cmeas_noise , (e) bbox , and (f) bbox_noise . The star marks the plant optimum as a fixed reference and the circles each model’s own optimum. Each panel shows one of the worst-performing members of the 20 initializations.

Figure 9: Profit loss

\Delta\ell

for (a) rmeas , (b) cmeas_rinit , (c) cmeas and (d) cmeas_noise . Columns are cases and rows are the 3 multi-start RTO seeds; each panel is the distribution over the 20 ensemble members, plotted as relative frequency. The dotted line marks the best achievable profit loss,

0\%

Figure 10: Profit loss

\Delta\ell

for (a) bbox and (b) bbox_noise . Columns are cases and rows are the 3 multi-start RTO seeds; each panel is the distribution over the 20 ensemble members, plotted as relative frequency. The dotted line marks the best achievable profit loss,

0\%

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Unit	In	Split/Rem	Split fraction
Flash drum	$N^{3}$	$N^{5}/N^{4}$	$\phi_{j}=1$ : Ee,Ea,O,C; $0$ : A,V,W
Distillation column	$N^{4}$	$N^{8}/N^{10}$	$\gamma_{j}=1$ : A; $0$ : otherwise
Absorber	$N^{5}$	$N^{11}/N^{6}$	$\mu_{j}=0.03$ : C; $0$ : otherwise
Splitter I	$N^{6}$	$N^{7}/N^{9}$	$\alpha$ (all $j$ )
Splitter II	$N^{8}$	$N^{13}/N^{12}$	$\beta$ (all $j$ )

Appendix

Table 4: Linear split for each separation unit and splitter. Stream superscripts follow the flowsheet; the last column gives each unit’s split fraction, the fraction of

N_{j}^{\rm in}

routed to the split-off stream.

Parameter	Value	Units
$k_{1}$	$1.312\times 10^{-2}$	$\nicefrac{\mathrm{mol}}{\mathrm{kg\,s}}\left(\nicefrac{\mathrm{m^{3}}}{\mathrm{mol}}\right)^{3.2}$
$E_{1}$	40500	J/mol
$k_{2}$	$2.363$	$\nicefrac{\mathrm{mol}}{\mathrm{kg\,s}}\left(\nicefrac{\mathrm{m^{3}}}{\mathrm{mol}}\right)^{2}$
$E_{2}$	52500	J/mol
R	8.314	$\nicefrac{{J}}{{molK}}$
$P$	882.529	kPa

Appendix

Table 5: Parameter values used for the VAc model.

Figure S1: cmeas_rinit : drift away from the initialization. The model is initialized at the weights at which it recovers the plant optimum, marked by the dash-dotted line (learning rate

0

, no training step taken). Under ADAM (left) the loss moves away from that line as training proceeds, so the optimizer leaves the basin it was handed. After the switch to L-BFGS at epoch 250 (right) the loss returns toward the initialization, so L-BFGS recovers part of what ADAM gave up. As the learning rate decreases, ADAM takes smaller steps and the curves stay at the initialization; this preserves the optimum only because it was supplied, not because training located it.

Figure S2: Learning-rate sweep for cmeas . The chosen learning rate is

0.05

Figure S3: Learning-rate sweep for cmeas_noise . The chosen learning rate is

0.05

Figure S4: Learning-rate sweep for bbox . The chosen learning rate is

0.01

Figure S5: Learning-rate sweep for bbox_noise . The chosen learning rate is

0.01

Neural Surrogate Modeling

Neural surrogate modeling uses learned models to approximate costly simulations, enabling faster prediction, design, control, and inference.

See all 418 papers

Jun 7, 2026

Bayesian Optimization of a Multi-Product Chemical Reactor Using Composite Models and Partial Physics Knowledge

We study data-driven real-time economic optimization of a multi-product chemical reactor when no reliable first-principles model is available beyond a steady-state energy balance. Instead of learning the economic objective directly as a black-box function, we use a composite formulation in which Gaussian process (GP) models predict physically meaningful outputs, including product concentrations and reactor temperature, while profit is computed analytically from these predictions together with raw-material, product, and utility prices. This preserves the structure of the economic objective, makes it parametric in changing prices without needing retraining, and allows candidate operating points to be checked against the available energy balance through a physics residual. The GPs also provide predictive uncertainty, which is exploited in a Bayesian optimization (BO) framework both for data-efficient exploration and for conservative enforcement of the reactor temperature constraint through an upper confidence bound. The acquisition function additionally penalizes large energy-balance mismatch obtained by substituting the GP-predicted outputs and candidate inputs into the available steady-state energy balance. The approach is demonstrated on a benchmark simulation of a non-isothermal multi-product reactor. Relative to a trust-region safe BO implementation, the proposed method achieves better simulated economic performance within the available iteration budget. Relative to a purely data-driven BO approach that does not use the available physics information, it avoids reactor temperature constraint violations.

Liqiu Dong, Marta Zagórowska, Mehmet Mercangöz

Department of Chemical Engineering, Imperial College London, London, UK · DCSC, Delft University of Technology, Delft, Netherlands

Bayesian Optimization Safe Bayesian Optimization Constrained Optimization

Ask PDF

Mar 1, 2025

Dissecting the Impact of Model Misspecification in Data-driven Optimization

Data-driven optimization aims to translate a machine learning model into decision-making by optimizing decisions on estimated costs. Such a pipeline can be conducted by fitting a distributional model which is then plugged into the target optimization problem. While this fitting can utilize traditional methods such as maximum likelihood, a more recent approach uses estimation-optimization integration that minimizes decision error instead of estimation error. Although intuitive, the statistical benefit of the latter approach is not well understood yet is important to guide the prescriptive usage of machine learning. In this paper, we dissect the performance comparisons between these approaches in terms of the amount of model misspecification. In particular, we show how the integrated approach offers a ``universal double benefit'' on the top two dominating terms of regret when the underlying model is misspecified, while the traditional approach can be advantageous when the model is nearly well-specified. Our comparison is powered by finite-sample tail regret bounds that are derived via new higher-order expansions of regrets and the leveraging of a recent Berry-Esseen theorem.

Adam N. Elmachtoub, Henry Lam, Haixiang Lan +1

Columbia University · Morgan Stanley

Model Misspecification Decision-Focused Learning Statistical Learning Theory

Ask PDF

Apr 1, 2026

Learning to Model Optimization Problems from Solver Outcomes

Automating optimization modeling with LLMs is a promising path toward scalable decision intelligence, but existing approaches either rely on agentic pipelines built on closed-source LLMs with high inference latency or fine-tune smaller LLMs using costly process supervision that often overfits to a single solver API. We study how reinforcement learning from inexpensive outcome labels can improve optimization modeling and support reuse across heterogeneous solver environments. To this end, we propose Optimization Modeling from Solver Outcomes (OMSO), which uses reinforcement learning (RL) with a mathematical programming solver as a deterministic, interactive environment. Given a natural-language problem and a target solver, OMSO generates solver-specific code, executes it in a sandboxed harness, and converts execution outcomes into scalar rewards in a closed-loop generate-execute-feedback-update process. This RL formulation learns optimization modeling without process-level supervision and supports cross-solver generalization by switching the solver environment rather than reconstructing solver-specific datasets. Experiments on NL4OPT, MAMO, IndustryOR, and OptiBench across Gurobi, OR-Tools, and COPT show that OMSO matches or outperforms process-supervised SFT, supports zero-shot training-free transfer, direct RL training, and SFT-cold-started RL, and improves modeling reasoning and semantic correctness beyond solver API usage.

Runda Guan, Xiangqing Shen, Jiajun Zhang +3

School of Computer Science and Engineering, Nanjing University of Science and Technology · School of Intelligence Science and Technology, Nanjing University · Institute of Automation, Chinese Academy of Sciences

Natural Language to Optimization Modeling Learning to Optimize LLM-Based Program Synthesis Reinforcement Learning with Verifiable Rewards

Ask PDF

Abstract

Neural Surrogate Modeling

Explore similar work

Bayesian Optimization of a Multi-Product Chemical Reactor Using Composite Models and Partial Physics Knowledge

Dissecting the Impact of Model Misspecification in Data-driven Optimization

Learning to Model Optimization Problems from Solver Outcomes