Decision-Driven Regularization: A Blended Model for Learning and Optimization

Gar Goei Loke; Qinshen Tang; Yangge Xiao; Xun Zhang

cs.LGAug 15, 2026

Decision-Driven Regularization: A Blended Model for Learning and Optimization

Authors: Gar Goei Loke, Qinshen Tang, Yangge Xiao, Xun Zhang

Organizations: Department of Management and Marketing, Durham University Business School, Waterside, Durham DH1 1SL, United Kingdom · Nanyang Business School, Nanyang Technological University, Singapore · Faculty of Business and Economics, The University of Melbourne · International Institute of Finance, School of Management, University of Science and Technology of China, Hefei 230036, China

Abstract

In contextual optimization, the decision-maker seeks optimal decisions to minimize a cost function, that varies based on observed features. This context is common in many business applications ranging from on-demand delivery and retail operations to portfolio optimization and inventory management. In this paper, we study the learning and optimization approach, which first learns how outcomes result from the features, and then selects optimal decisions based on these outcomes. We focus on the integrated learning and optimization literature, and identify that a lack of control for prediction accuracy can lead to overfitting and a loss of decision effectiveness against simple separate learning and optimization models. Instead, we propose a bi-objective formulation that balances prediction accuracy and cost minimization, termed decision-driven regularization. It also addresses ambiguity in the definition of the cost function via a surrogate that depends on a new hyperparameter. We additionally show that alternative perspectives for formulating the problem, namely robust optimization and regret minimization, lead to models that are closely related to our proposed model. As a consequence, our framework generalizes models such as SPO+. Our model is shown to be numerically superior to other benchmarks, such as OLS, Random Forest, XGBoost, SPO+, Perturbation Gradient, and Learning and Rank, in our synthetic studies.

Figures & tables

Panel of Figure 1 — Figure 1: Visualization of decision for OLS (left) and ILO (right). The green line represents the Ground Truth ( $\hat{x}_{1}=1.5\hat{x}_{2}$ ).

Grid Size	$(\mu,\lambda)$
Grid Size	(0.5,0.25)	(0.5,0.5)	(0.5,0.75)	(0.25,0.5)	(0.75,0.5)
(2,2)	0.02	0.02	0.02	0.02	0.02
(3,3)	0.11	0.11	0.12	0.12	0.11
(4,4)	0.18	0.17	0.18	0.18	0.17
(5,5)	0.31	0.30	0.30	0.31	0.32

Table 1: Average solution time (in seconds) required to compute DDR model.

Figure 2: Performance of DDR against OLS for different parameters of

\mu

and

\lambda

Figure 3: Performance of DDR against SLO models: OLS (left), RF (middle), and XGBoost (right)

Models	In-sample RMSE			Out-of-sample RMSE
Models	$5^{\scriptscriptstyle{\text{th}}}\%$ -tile	Mean	$95^{\scriptscriptstyle{\text{th}}}\%$ -tile	$5^{\scriptscriptstyle{\text{th}}}\%$ -tile	Mean	$95^{\scriptscriptstyle{\text{th}}}\%$ -tile
OLS	3.8393	3.9416	4.0682	4.1279	4.1846	4.2395
RF	1.5872	1.6531	1.7039	4.3479	4.4255	4.5155
XGBoost	2.9835	3.1140	3.2322	4.4822	4.5833	4.6913
DDR	3.8418	3.9440	4.0705	4.1286	4.1830	4.2352

Table 2: Prediction accuracy of SLO models in terms of root mean square error (RMSE) across multiple datasets

Figure 4: Performance of DDR against ILO models: SPO+ (left), PG (centre) and LTR (right)

Figure 5: Regret reduction of the DDR against OLS under different network structures (left panel) and comparison of DDR against SPO+ for a

5\times 5

grid network (right panel)

Figure 6: Performance of DDR against ILO models: SPO+ (left), PG (centre) and LTR (right) in the misspecified setting

Figure 7: Performance of DDR against SPO+ in the misspecified setting

\beta=8.0

and

N=500

without (left) and with (right) quadratic terms

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure B.1: One-dimensional illustrative of centred (dark blue shaded area) and non-centred (broken orange line) confidence intervals

Angles to DDR	Lasso	Ridge
Mean	82°	72°
10th Percentile	80°	72°
90th Percentile	85°	73°

Appendix

Table B.1: Angles of local directions of Lasso and Ridge to DDR in

100

simulations

Models	In-sample RMSE			Out-of-sample RMSE
Models	$5^{\scriptscriptstyle{\text{th}}}\%$ -tile	Mean	$95^{\scriptscriptstyle{\text{th}}}\%$ -tile	$5^{\scriptscriptstyle{\text{th}}}\%$ -tile	Mean	$95^{\scriptscriptstyle{\text{th}}}\%$ -tile
OLS	3.8393	3.9416	4.0682	4.1279	4.1846	4.2395
DDR	3.8418	3.9440	4.0705	4.1286	4.1830	4.2352
RF(Maximum depth = 2)	4.0068	4.1499	4.2974	4.3693	4.4779	4.5875
RF(Maximum depth = 4)	3.36	3.4682	3.5785	4.3047	4.3904	4.4816
RF(Maximum depth = 6)	2.4572	2.6123	2.782	4.3191	4.3939	4.4766

Appendix

Table D.2: Prediction accuracy of SLO models in terms of root mean square error (RMSE) across multiple datasets

Panel of Figure 13 — Figure D.2: Performance of DDR against OLS under $3\times 3$ grid and $\mu=0.75$ and $\lambda=0.8$ under various settings

Panel of Figure 14 — Figure D.3: Performance of DDR against SPO+ under $3\times 3$ grid and $\mu=0.75$ and $\lambda=0.8$ under various settings

Predict-Then-Optimize

Predict-then-optimize trains or evaluates predictors by how well their outputs support downstream decisions under constraints.

See all 36 papers

Contextual Decision Making: Predictive or Generative
Oct 23, 2026L. Jeff Hong, Weihuan Huang, Xinyao LiPredict-Then-Optimize Decision-Making under Uncertainty
Department of Industrial and Systems Engineering, University of Minnesota, Minneapolis, MN 55455, U.S. · School of Management & Engineering, Nanjing University, Nanjing 210093, China
Scalable Decision-Focused Learning through Cost-Sensitive Regression
May 18, 2026Noah Schutte, Senne Berden, Tias Guns +2Cost-Sensitive Learning Decision-Focused Learning
Delft University of Technology · KU Leuven · Independent Researcher
Decision-Focused On-Policy Learning for Contextual Linear Optimization with Partial Feedback
May 31, 2026Wyame Benslimane, Tinghan Ye, Pascal Van Hentenryck +1Predict-Then-Optimize Decision-Focused Learning
Department of Industrial Engineering and Operations Research University of California, Berkeley · H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology

Oct 23, 2026J/K move · Enter open · S save

Contextual Decision Making: Predictive or Generative

L. Jeff Hong, Weihuan Huang, Xinyao Li

Department of Industrial and Systems Engineering, University of Minnesota, Minneapolis, MN 55455, U.S. · School of Management & Engineering, Nanjing University, Nanjing 210093, China

Predict-Then-Optimize Decision-Making under Uncertainty

Open paper AskPDF arXiv

Contextual decision making uses observed covariates to guide decisions under uncertainty. Two natural approaches are predict-then-optimize (PtO), which learns the conditional expected objective, and generate-then-optimize (GtO), which learns conditional uncertainty and optimizes using generated scenarios. We study when each approach is preferable by examining how the choice of learning target affects statistical accuracy and optimization tractability. Although PtO learns a narrower target, our regret bounds show how differences in smoothness can offset GtO's additional dimensional burden. Exploiting smoothness also raises an optimization tradeoff: within the class of nonnegative-weight PtO methods studied, a lower bound limits the gains from additional smoothness, while more flexible methods may introduce signed weights and lose the guarantee of preserving convexity. GtO preserves scenario-wise convexity, but requires learning a richer object and generating scenarios for optimization. Synthetic experiments and a real case study illustrate how these tradeoffs vary with data availability and problem structure. Together with the scope for reusing a model across tasks, our findings provide practical guidance for choosing between the approaches without identifying a universal winner.

Abstract

Predict-Then-Optimize

Explore similar work

Contextual Decision Making: Predictive or Generative

Scalable Decision-Focused Learning through Cost-Sensitive Regression

Decision-Focused On-Policy Learning for Contextual Linear Optimization with Partial Feedback