math.OCOct 26, 2026
SaveConstraint Learning by Optimizing over Trained Neural Networks: A Pragmatic Tutorial
Organizations: University of Iowa
Abstract
Constraint learning is an emerging paradigm of modeling optimization problems with the help of machine learning. In this paradigm, trained machine learning models are embedded into the optimization model. For example, we can learn constraints empirically from available data, or we can avoid using a nonlinear objective function by learning a simpler approximation for it. Neural networks become a compelling option if the machine learning models must capture more elaborate and nuanced relationships. However, optimization models embedding neural networks may become too large or too slow to solve. In this tutorial, we present practical considerations for an effective and efficient use of constraint learning with neural networks and validate it with a case study highlighting the benefit of smaller and sparser networks.
Figures & tables
Figure 1: Left: Structure of a feedforward neural network with input , hidden layers to with neurons each, and output . Right: Formula for output of neuron in layer from activation function and preactivation output , which uses previous layer output , neuron weights , and bias . For details, see Appendix C Neural Network Structure and Mapping (in Appendix).
Figure 2: Top: Visual description of neural network used as example. Bottom left: Lines partitioning the space based on what inputs produce a positive output for each neuron, with the arrow pointing to the positive side, the length of the arrow proportional to the magnitude of the weights, and the arrow label denoting the influence on . Bottom right: Contour plots of over the lines associated with neural activations. The example and plots are adapted from Tong et al. (2026) .
| Reference | Type of problem(s) | Prediction(s) involved |
|---|---|---|
| Huang et al. (2003) | Catalyst composition | Catalytic performance |
| Fernandes (2006) | Synthesis of a specific oil product | Outcome of a catalytic process |
| Fahmi and Cremaschi (2012) | Biodiesel production plant design | Unit operation, thermodynamics, mixing |
| Sant Anna et al. (2017) | PSA operation for N 2 /CH 4 separation | Nitrogen/methane purity and recovery |
| Lombardi et al. (2017) | Computational workload dispatch | Processor efficiency for assignment |
| Say et al. (2017) | HVAC, navigation, reservoir control | State transition in sensors |
Table 1: Applications from constraint learning studies in which neural networks are used.
| Type of predictive model | References |
|---|---|
| Linear regression | Bertsimas et al. (2016) , Verwer et al. (2017) |
| Logistic regression | Bergman et al. (2022) |
| Support vector machine | Hooshmand and Nikoomanesh (2025) |
| Multivariate adaptive regression spline | Martinez et al. (2017) |
| Regression tree | Verwer et al. (2017) , Ammari et al. (2023) |
| Decision tree | Bonfietti et al. (2015) , Lombardi et al. (2017) , Kudła and Pawlak (2018) |
Table 2: Constraint learning studies in which other types of predictive models are embedded.
Figure 3: Comparison of predictive quality with number of weights of configurations (as a scatter plot), contrasted by pruning rate (as a color gradient). The configurations that are not dominated by any other in both predictive quality and number of weights define the efficient frontier (as a line chart).
Figure 4: Comparison of predictive quality with pruning rate (as a line chart for each width ) for neural networks with width 10, 50, 100, and 1,000. The pruning rate with smallest validation error for each width is circled. The configurations at the efficient frontier are diamond-shaped.
Figure 5: Comparison of decision quality with number of weights for configurations at the efficient frontier (as line charts with error bars). Using the solution obtained with the optimization model, we evaluate it as an input of the embedded network and compare it with the ground truth.
Figure 6: Comparison of decision quality with number of weights for configurations at the efficient frontier (as a line chart; the same ones from Figure 5 ) vs. other configurations (as a scatter plot), contrasted by pruning rate (as a color gradient). For ground truth value, 104 configurations are below the curve defined by the efficient frontier, 15 are above, and 37 are outside the range. For optimization bias, 84 of those configurations are above the curve, 35 are below, and 37 are outside the range.
Figure 7: Comparison of solution time with number of weights at the efficient frontier (as a line chart) vs. other configurations (as a scatter plot), contrasted by ground truth values (as a color gradient).
Figure 8: Comparison of ground truth value with predictive accuracy at the efficient frontier (as a line chart) vs. other configurations (as a scatter plot), contrasted on the left by the number of weights and on the right by solution time (as a color gradient in both; the plots are otherwise identical). The plots exclude configurations with mean validation MSE above . The correlation across for the points shown at the efficient frontier is -0.9535, dropping to -0.7885 for all points.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Geometric illustration of how defining half-spaces with hyperplanes in more ways than the number of dimensions does not imply that every intersection of half-spaces is feasible. Left: two partitionings in one dimension lead to a maximum of 3 regions instead of 4. Right: three partitionings in two dimensions lead to a maximum of 7 regions instead of 8.
Figure 10: Comparison of predictive quality with pruning rate (as a line chart for each width ) in small (left) and large (right) neural networks. The pruning rate with smallest validation error for each width is circled. The configurations at the efficient frontier are diamond-shaped.
More in this topic
Constrained Optimization
Constrained optimization finds the best decisions while satisfying equality, inequality, resource, safety, or fairness requirements.
See all 337 papers