A tale of perfect fit and phantom optima: how data-driven models can fail in real-time optimization

Prithvi Dake; Rahul Bindlish; James B. Rawlings

eess.SYAug 24, 2026

A tale of perfect fit and phantom optima: how data-driven models can fail in real-time optimization

Authors: Prithvi Dake, Rahul Bindlish, James B. Rawlings

Organizations: Department of Chemical Engineering, University of California, Santa Barbara, CA 93106, United States · Dow Chemical Company, TX, United States

Abstract

Real-time optimization (RTO) relies on process models to locate economically optimal operating conditions. Because developing first-principles models requires significant process knowledge, data-driven alternatives are increasingly attractive. Modern machine-learning models can fit historical plant data accurately and often pass standard validation tests. Whether such models can be trusted for economic optimization, however, remains unclear. We investigate this question using a vinyl acetate monomer benchmark process with a unique, well-conditioned economic optimum. We train a structured hybrid model that combines known mass balances and thermodynamics with a neural-network closure for unknown kinetics, and a fully data-driven neural ordinary differential equation (ODE) model. Both models reproduce plant measurements accurately and exhibit little variation in predictions across random initializations. Yet their economic optima differ substantially from that of the plant. Where the plant returns a single optimum on multistart search, the trained models return many phantom optima. We further show that the training optimizer alone can be yet another source of error. Even with noise-free data and initialization at weights that recover the plant optimum, stochastic gradient training can drift to weights that yield substantially worse RTO solutions. The identified model is thus an artifact of the training optimizer as well as the data. These results demonstrate that a good predictive fit of all available measurements does not guarantee reliable economic performance. A data-driven model for RTO should at least be required to recover the optimum on a decision-oriented benchmark like the one developed here before being considered for plant testing and application.

Figures & tables

Panel of Figure 1 — Figure 1: (A) Historical plant datasets are used to train (B) a structured model and a black-box ML/AI closure model, each deployed in real-time optimization (RTO). (C) Near-identical fits map to very different optima, the worst-case structured model loses close to 30% profit and the black-box model over 50%. A reliable data-driven model must recover the optimum, not be judged solely on the validation dataset.

Figure 3: A snapshot of training data from the plant model with no measurement noise. The measurements are shown in red and the decision variables in black. The dotted grey lines represent $x_{s}^{\star}$ and $u_{s}^{\star}$ from section 2.2 .

Model	Case	Training problem and measurements	NN initialization	Noise $n_{l}$
Structured	rmeas	eq. 46 , rates	random	$0$
	cmeas_rinit	eq. 45 , concentrations	rmeas solution	$0$
	cmeas	eq. 45 , concentrations	random	$0$
	cmeas_noise	eq. 45 , concentrations	random	$10^{-2}$
Black-box	bbox	eq. 47 , concentrations	random	$0$
Black-box	bbox_noise	eq. 47 , concentrations	random	$10^{-2}$

Table 1: Summary of the six data-driven cases.

n_{l}

is the sensor-noise level in eq. 50 . A random NN initialization uses the LeCun normal initialization popular for tanh -based neural networks ( LeCun et al., 2002 , pg. 20) .

Parameter	$n_{l}=0$		$n_{l}=10^{-3}$		$n_{l}=10^{-2}$
$\log{\tilde{k}_{1}}$	$-4.836$ $\pm$	$2.7\times 10^{-11}$	$-4.840$ $\pm$	$4.4\times 10^{-3}$	$-4.876$ $\pm$	$3.9\times 10^{-2}$
$\log{\tilde{E}_{1}}$	$10.61$ $\pm$	$4.3\times 10^{-11}$	$10.60$ $\pm$	$7.1\times 10^{-3}$	$10.55$ $\pm$	$6.7\times 10^{-2}$
$a_{1}$	$1.100$ $\pm$	$5.2\times 10^{-11}$	$1.092$ $\pm$	$8.6\times 10^{-3}$	$1.021$ $\pm$	$7.7\times 10^{-2}$
$b_{1}$	$1.100$ $\pm$	$5.5\times 10^{-11}$	$1.091$ $\pm$	$9.0\times 10^{-3}$	$1.014$ $\pm$	$8.1\times 10^{-2}$
$c_{1}$	$1.000$ $\pm$	$4.4\times 10^{-11}$	$0.993$ $\pm$	$7.2\times 10^{-3}$	$0.934$ $\pm$	$6.5\times 10^{-2}$
$\log{\tilde{k}_{2}}$	$-7.992$ $\pm$	$1.1\times 10^{-11}$	$-7.993$ $\pm$	$1.8\times 10^{-3}$	$-7.994$ $\pm$	$1.8\times 10^{-2}$

Table 2: Parameter estimation results and profit loss for the control experiment with parametric models for different levels of measurement noise.

Figure 4: Concentration predictions on the validation set for (a) rmeas , (b) cmeas_rinit , (c) cmeas and (d) cmeas_noise . Measurements are shown in red. The shaded bands are the ensemble spread, taken as the 5–95% quantiles of the 20 random initializations.

Figure 5: Parity plots of the reaction rates $[r_{1},r_{2}]^{\mathrm{T}}$ on the validation set for (a) rmeas , (b) cmeas_rinit , (c) cmeas and (d) cmeas_noise . Top row shows $r_{1}$ and bottom row $r_{2}$ . The dotted line is ideal parity. The bands are the ensemble spread, taken as the 5–95% quantiles across the 20 random initializations.

Figure 6: Concentration predictions on the validation set for the black-box model, (a) bbox and (b) bbox_noise . Measurements are shown in red. The bands are the ensemble spread, taken as the 5–95% quantiles of the 20 random initializations.

Model	Case	NN architecture	rmse meas	rmse val	Train time (hr)
Structured	rmeas	$r_{\mathrm{NN},r_{1}}=[4,64,1]$ $r_{\mathrm{NN},r_{2}}=[3,64,1]$	$0.00067\pm 0.00012$	$0.0069\pm 0.0021$	$0.22\pm 0.08$
	cmeas_rinit		$0.00435\pm 0.00060$	$0.0146\pm 0.0024$	$11.97\pm 0.94$
	cmeas		$0.00966\pm 0.00131$	$0.0219\pm 0.0035$	$12.27\pm 1.17$
	cmeas_noise		$0.01472\pm 0.00064$	$0.0217\pm 0.0030$	$12.45\pm 0.99$
Black-box	bbox	$f_{\mathrm{NN}}=[16,128,7]$	$0.03224\pm 0.00187$	$0.0484\pm 0.0051$	$12.01\pm 1.06$
Black-box	bbox_noise	$f_{\mathrm{NN}}=[16,128,7]$	$0.03410\pm 0.00177$	$0.0481\pm 0.0057$	$12.04\pm 0.90$

Table 3: Structured- and black-box-model results across cases. Values are reported as mean

\pm

1.96\,\sigma

over 20 random initializations (95% interval). rmse meas and rmse val are the mean-normalized errors of eq. 66 on the measurement and validation sets; training time is wall-clock.

Figure 7: Plant profit landscape $\ell(T,\alpha)$ with the remaining degrees of freedom fixed at the plant optimum. The star marks the plant optimum $(T^{\star},\alpha^{\star})$ . The surface is unimodal and well-conditioned.

Figure 8: Model-predicted profit landscape $\hat{\ell}(T,\alpha)$ that each trained model presents to the RTO optimizer. Panels, in order of degrading structure and data, are (a) rmeas , (b) cmeas_rinit , (c) cmeas , (d) cmeas_noise , (e) bbox , and (f) bbox_noise . The star marks the plant optimum as a fixed reference and the circles each model’s own optimum. Each panel shows one of the worst-performing members of the 20 initializations.

Figure 9: Profit loss

\Delta\ell

for (a) rmeas , (b) cmeas_rinit , (c) cmeas and (d) cmeas_noise . Columns are cases and rows are the 3 multi-start RTO seeds; each panel is the distribution over the 20 ensemble members, plotted as relative frequency. The dotted line marks the best achievable profit loss,

0\%

Figure 10: Profit loss

\Delta\ell

for (a) bbox and (b) bbox_noise . Columns are cases and rows are the 3 multi-start RTO seeds; each panel is the distribution over the 20 ensemble members, plotted as relative frequency. The dotted line marks the best achievable profit loss,

0\%

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Unit	In	Split/Rem	Split fraction
Flash drum	$N^{3}$	$N^{5}/N^{4}$	$\phi_{j}=1$ : Ee,Ea,O,C; $0$ : A,V,W
Distillation column	$N^{4}$	$N^{8}/N^{10}$	$\gamma_{j}=1$ : A; $0$ : otherwise
Absorber	$N^{5}$	$N^{11}/N^{6}$	$\mu_{j}=0.03$ : C; $0$ : otherwise
Splitter I	$N^{6}$	$N^{7}/N^{9}$	$\alpha$ (all $j$ )
Splitter II	$N^{8}$	$N^{13}/N^{12}$	$\beta$ (all $j$ )

Appendix

Table 4: Linear split for each separation unit and splitter. Stream superscripts follow the flowsheet; the last column gives each unit’s split fraction, the fraction of

N_{j}^{\rm in}

routed to the split-off stream.

Parameter	Value	Units
$k_{1}$	$1.312\times 10^{-2}$	$\nicefrac{\mathrm{mol}}{\mathrm{kg\,s}}\left(\nicefrac{\mathrm{m^{3}}}{\mathrm{mol}}\right)^{3.2}$
$E_{1}$	40500	J/mol
$k_{2}$	$2.363$	$\nicefrac{\mathrm{mol}}{\mathrm{kg\,s}}\left(\nicefrac{\mathrm{m^{3}}}{\mathrm{mol}}\right)^{2}$
$E_{2}$	52500	J/mol
R	8.314	$\nicefrac{{J}}{{molK}}$
$P$	882.529	kPa

Appendix

Table 5: Parameter values used for the VAc model.

Figure S1: cmeas_rinit : drift away from the initialization. The model is initialized at the weights at which it recovers the plant optimum, marked by the dash-dotted line (learning rate

0

, no training step taken). Under ADAM (left) the loss moves away from that line as training proceeds, so the optimizer leaves the basin it was handed. After the switch to L-BFGS at epoch 250 (right) the loss returns toward the initialization, so L-BFGS recovers part of what ADAM gave up. As the learning rate decreases, ADAM takes smaller steps and the curves stay at the initialization; this preserves the optimum only because it was supplied, not because training located it.

Figure S2: Learning-rate sweep for cmeas . The chosen learning rate is

0.05

Figure S3: Learning-rate sweep for cmeas_noise . The chosen learning rate is

0.05

Figure S4: Learning-rate sweep for bbox . The chosen learning rate is

0.01

Figure S5: Learning-rate sweep for bbox_noise . The chosen learning rate is

0.01

Neural Surrogate Modeling

Neural surrogate modeling uses learned models to approximate costly simulations, enabling faster prediction, design, control, and inference.

See all 418 papers

Bayesian Optimization of a Multi-Product Chemical Reactor Using Composite Models and Partial Physics Knowledge
Jun 7, 2026Liqiu Dong, Marta Zagórowska, Mehmet MercangözBayesian Optimization Safe Bayesian Optimization
Department of Chemical Engineering, Imperial College London, London, UK · DCSC, Delft University of Technology, Delft, Netherlands
Dissecting the Impact of Model Misspecification in Data-driven Optimization
Mar 1, 2025Adam N. Elmachtoub, Henry Lam, Haixiang Lan +1Model Misspecification Decision-Focused Learning
Columbia University · Morgan Stanley
Learning to Model Optimization Problems from Solver Outcomes
Apr 1, 2026Runda Guan, Xiangqing Shen, Jiajun Zhang +3Natural Language to Optimization Modeling Learning to Optimize
School of Computer Science and Engineering, Nanjing University of Science and Technology · School of Intelligence Science and Technology, Nanjing University · Institute of Automation, Chinese Academy of Sciences

Jun 7, 2026J/K move · Enter open · S save

Bayesian Optimization of a Multi-Product Chemical Reactor Using Composite Models and Partial Physics Knowledge

Liqiu Dong, Marta Zagórowska, Mehmet Mercangöz

Department of Chemical Engineering, Imperial College London, London, UK · DCSC, Delft University of Technology, Delft, Netherlands

Bayesian Optimization Safe Bayesian Optimization Constrained Optimization

Open paper AskPDF arXiv

We study data-driven real-time economic optimization of a multi-product chemical reactor when no reliable first-principles model is available beyond a steady-state energy balance. Instead of learning the economic objective directly as a black-box function, we use a composite formulation in which Gaussian process (GP) models predict physically meaningful outputs, including product concentrations and reactor temperature, while profit is computed analytically from these predictions together with raw-material, product, and utility prices. This preserves the structure of the economic objective, makes it parametric in changing prices without needing retraining, and allows candidate operating points to be checked against the available energy balance through a physics residual. The GPs also provide predictive uncertainty, which is exploited in a Bayesian optimization (BO) framework both for data-efficient exploration and for conservative enforcement of the reactor temperature constraint through an upper confidence bound. The acquisition function additionally penalizes large energy-balance mismatch obtained by substituting the GP-predicted outputs and candidate inputs into the available steady-state energy balance. The approach is demonstrated on a benchmark simulation of a non-isothermal multi-product reactor. Relative to a trust-region safe BO implementation, the proposed method achieves better simulated economic performance within the available iteration budget. Relative to a purely data-driven BO approach that does not use the available physics information, it avoids reactor temperature constraint violations.

Abstract

Neural Surrogate Modeling

Explore similar work

Bayesian Optimization of a Multi-Product Chemical Reactor Using Composite Models and Partial Physics Knowledge

Dissecting the Impact of Model Misspecification in Data-driven Optimization

Learning to Model Optimization Problems from Solver Outcomes