cs.LGSep 29, 2026

RL-PaO: Prediction as Action in Decision Making under Uncertainty

Authors: Jiahui Feng, Dafang Zhao, Zheng Chen, Zhengmao Li, Lingwei Zhu

Organizations: Great Bay University, School of Computing and Information Technology, China · The University of Osaka, School of Information Science and Technology, Japan · The University of Osaka, SANKEN, Japan · Aalto University, School of Electrical Engineering, Finland

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Decision-making under uncertainty often relies on predicted parameters, yet accurate prediction does not necessarily lead to good operational decisions. Aligning prediction with downstream optimization requires learning from the consequences of the decisions those predictions induce. We introduce RL-PaO, a reinforcement learning framework that integrates system formulation, optimization, and decision execution into a single environment. This yields a Markov decision process in which prediction is regarded as action: it shifts the environment to produce subsequent context and reward that explicitly aligns prediction error with realized cost, and learning the optimal policy does not require differentiating through the black-box solver. We evaluate RL-PaO on day-ahead energy scheduling using real historical data. On the test year, RL-PaO achieves the lowest annual cost among the non-oracle baselines, achieving on average 10%10\% cost reduction. Moreover, RL-PaO is capable of further analyses to provide strong interpretability both from the policy evolution perspective and the cost-accuracy trade-off.

Figures & tables

 Prior-based OptimizationPredict-Optimize Paradigm
    PtOPaO
     Supervised PaORL-PaO
 DOROSO  Ours
Uncertainty modeling×Robust uncertainty setsProbability distributionsOpen-loop predictionClosed-loop predictionPrediction-optimization alignment
Placement-free uncertainty–✓✓✓×✓
Decision-focused training–––×✓✓
Table 1: Comparison of methods for optimization under uncertainty. Ours is the first RL-based PaO method that aims at improving the alignment between prediction and optimization. We emphasize the importance of Decision‑focused training that optimizes by downstream decision cost instead of prediction error; and Model‑agnostic optimization that operates without full knowledge of the underlying optimization formulation.
Figure 1: Overview of the proposed RL-PaO framework. (A) We explicitly include system modeling, optimization, scheduling and real-world operation into one environment. The RL agent observes current context to predict the unknown constraint in the optimization objective to be solved by an MILP solver. The solution influences execution process, yielding reward and next context. (B) our RL formulation is under the PaO category and different from the conventional supervised PaO methods. (C) Architectural comparison of different methods. PtO uses an open-loop predictor. Supervised PaO closes the training loop by incorporating downstream loss. By contrast, the proposed method aligns prediction and optimization by a novel MDP and RL training.
SymbolDescription
c\bm{c}Electricity price
PtpvP_{t}^{\text{pv}}Actual PV output
PtgridP_{t}^{\text{grid}}Grid import power
PtloadP_{t}^{\text{load}}Fixed load demand
pt⋅\bm{p}_{t}^{\cdot}ESS (dis)charging power
UtpvU_{t}^{\text{pv}}PV upper bound †
Table 2: Notations for the problem.
Figure 2: Cumulative rewards (upper row) and downstream operation costs (lower) across different learning rates. Transparency of lines set according to β\beta value. RL-PaO is effective in (1) establishing correspondence between high rewards and lower costs; (2) the performance curves change smoothly along gradual increase of hyperparameters.
Figure 3: Comparison on annual cost. Blue solid line and shade are mean and standard deviation over 5 independent runs. Left : RL-PaO compared against the conventional PbO baselines that require model knowledge in uncertainty (DO, RO, SO). Right : comparison against PtO, PaO methods. In both cases the model-free RL-PaO achieves the best cost.
CategoryMethodAnnual Cost ↓\downarrowCost Gap (%) ↓\downarrow
PbOOracle963.760.00
DO1496.2055.25
RO1145.2018.83
SO1133.9017.65
PtO/PaOPtO1233.4027.98
Supervised PaO1207.9025.33
Table 3: Overall performance comparison on the 2019 test set. Annual cost is measured in the unit of ×104\times 10^{4} . ↓\downarrow indicates lower is better.
Figure 4: Policy evolution plot of the learned Gaussian distribution across 12 action dimensions at training day 0, 100, and 200. Recall that prediction of the uncertain parameter is sampled from the policy. As learning proceeds, the policies gradually narrow and shift toward the optimal decision region that aligns prediction with downstream cost optimization.
Figure 5: Prediction error versus operation cost, daily level. The first three windows show scatter plots on the test set. The last window shows a joint-distribution plot. Transparency distinguishes weekdays and weekends. PtO and PaO points are concentrated in a low estimation error region but the average cost is high. Outliers with extremely high costs are identified by red dots with purple background. By contrast, RL-PaO error distribution is more dispersed, but the average cost is lower, as can be seen from the 80%80\% quantile line from the last window.
Figure 6: Prediction error versus operation cost, annual level. (a) Pareto front of annual cost against prediction MAE, recorded for lr=5×10−5\texttt{lr}=5\times 10^{-5} . The reward weight β\beta spans a continuous trade-off curve and sits on the Pareto frontier, indicating that no competing methods can simultaneously achieve lower prediction error and lower operation cost. (b) Overall correlation across all β\beta and learning rate configurations. The blue points scattered in the lower left corner supports the finding that lower prediction error does not equal lower cost.
Figure 7: Mahanttan plot of the proposed method across learning rates and reward weights β\beta . Transparent dots show final operational costs, with deep blue dots indicating the mean. (Upper) Results of fixing β\beta and varying lr . (Lower) Fixing lr and varying β\beta . The superior performance of RL can be attained from various combinations.
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

HyperparameterValue
Reward weight β\betaSwept in {0.0,0.1,0.2,…,1.0}\{0.0,0.1,0.2,\dots,1.0\}
Learning rateSwept in {10−7, 5×10−7, 10−6, 5×10−6,\{10^{-7},\,5{\times}10^{-7},\,10^{-6},\,5{\times}10^{-6}, 10−5, 5×10−5, 10−4, 5×10−4, 10−3}10^{-5},\,5{\times}10^{-5},\,10^{-4},\,5{\times}10^{-4},\,10^{-3}\}
Discount factor γ\gamma0.99
PPO clip range0.2
GAE parameter λ\lambda0.95
Value function coefficient vfv_{f}0.5
Appendix
Table 4: Hyperparameters and configurations.
Figure C1: Daily operation cost by month compared with conventional optimization baselines on the 2019 test set. Each subplot corresponds to one calendar month, comparing Oracle, DO, RO, SO and the proposed RL-PaO.
Figure C2: Daily operation cost by month compared with predict-and-optimize baselines on the 2019 test set. Each subplot corresponds to one calendar month, comparing Oracle, predict-then-optimize (PtO), supervised PaO, improved PaO and the proposed RL-PaO.
Figure C3: Convergence behavior of RL-PaO across different reward weights β\beta and learning rates. Top: mean reward convergence across training episodes. Bottom: annual operation cost convergence across training episodes. Each column corresponds to a fixed β\beta value, and each colored curve corresponds to a different learning rate.
Figure C4: Left: Reward and operation cost convergence under different learning rates with β\beta fixed at 0.90. Each colored band corresponds to one learning rate configuration. Right: Reward convergence under different reward weights β\beta with learning rate fixed at 5×10−55\times 10^{-5} . Each colored band corresponds to one β\beta configuration.
Figure C5: Left: Annual operation cost convergence under different reward weights β\beta with learning rate fixed at 5×10−55\times 10^{-5} . Each colored band corresponds to one β\beta configuration. Right: Annual operation cost convergence under different learning rates with β\beta fixed at 0.90. Each colored band corresponds to one learning rate configuration.
Figure C6: Final annual operation cost with mean and standard deviation across different learning rates, with β\beta fixed at 0.90.
Figure C7: Final annual operation cost with mean and standard deviation across different reward weights β\beta , with learning rate fixed at 5×10−55\times 10^{-5} . The dashed line marks the β\beta value adopted in the main experiment.

Explore similar work

CardsList
Sep 26, 2026

PolyStepOR: Learning to Decide Without Optimal Decisions

Decision-focused learning (DFL) trains predictors for downstream decision quality, but often relies on optimal reference decisions that are expensive to obtain. We present PolyStepOR, which trains directly from realized decision costs without pre-computed optima and extends to in-constraint predictions through repair or infeasibility penalties. To handle piecewise-constant losses, PolyStepOR perturbs predictor parameters, evaluates the resulting decisions, and uses optimal transport to favor lower-cost directions, requiring no derivatives. Without task-specific tuning, PolyStepOR performs strongly on classical optimization benchmarks and competitively on predicted-constraint and real-world problems. Theoretically, we characterize decision-preserving perturbations and boundary detection, bound sensitivity to cost errors, and establish stationarity guarantees for a smoothed objective. PolyStepOR thus replaces optimal reference decisions and derivatives with forward evaluations.

Viet The Nguyen, Gunther Gust, An Thai Le

University of Würzburg, Germany · Center for Artificial Intelligence and Data Science (CAIDAS), Germany · Center for AI Research, VinUniversity, Vietnam +1

Decision-Focused LearningPredict-Then-OptimizeZeroth-Order OptimizationConstrained Optimization
AskPDF
Jun 19, 2026

Decision-Focused Learning: When and Why Traditional Prediction Models Fail

Plugging predictions of unknown parameters into downstream optimization problems, often referred to as the ``predict-then-optimize'' paradigm, has long been a standard approach in decision-making under uncertainty. However, improved predictive accuracy does not, in general, translate into improved decision quality. This disconnect has motivated growing interest in decision-focused learning (DFL) within the operations research community. This tutorial reviews recent developments in DFL and highlights key methodological insights, with a particular focus on stochastic linear programming as the downstream decision-making problem. We discuss why several widely used tools in traditional statistical learning are not directly suited to decision-focused settings and must be rethought, including (i) data collection strategies driven purely by predictive uncertainty and (ii) distributional distance measures such as the Wasserstein distance. We summarize properties of DFL that distinguish it from conventional predictive modeling and provide insights into the development of new decision-focused tools.

Mo Liu

Department of Statistics and Operations Research, University of North Carolina at Chapel Hill

Predict-Then-OptimizeDecision-Focused LearningDecision-Making under Uncertainty
AskPDF
Jun 29, 2026

Decision-Value Attribution in Predict-then-Optimize Systems

Predictive models are increasingly embedded in operational decision-making, yet standard explanation methods typically explain forecasts rather than the decisions those forecasts induce. This distinction is important in predict-then-optimize systems: large forecast changes may leave the optimizer's action unchanged, while small changes can alter the selected decision and its realized value. We propose Decision Value Attribution (DVA), a Shapley-based framework for attributing the value of a fixed prediction--optimization pipeline. The framework defines cooperative games whose payoff is the downstream decision value, allowing the players to be information sources, optimization or design parameters, or both. We present three variants: InfoDVA attributes value to features, DesignDVA attributes value to operational configurations, and Decision-Value Interactions (DVI) quantifies how information and design jointly create value. We further distinguish post-DVA, which evaluates decisions using realized outcomes, from pre-DVA, which evaluates decisions under the model's full prediction. This separation turns attribution into a decision-level diagnostic of whether the model's operational beliefs align with realized performance. The resulting attributions are expressed in the units of the operational objective and decompose the gain or loss relative to a baseline. Case studies in electricity storage arbitrage and emergency medical service coverage show that predictive explanations can be poor proxies for operational value, that DVA can guide targeted information-control interventions, and that optimization configurations determine when predictive information is decision-relevant.

Konstantinos Ziliaskopoulos, Alexander Vinel, Alice E. Smith

Auburn University, Industrial & Systems Engineering, 36849, Auburn, AL, USA · University of Alabama, Mechanical Engineering & Computer Science, 35487, Tuscaloosa, AL, USA

Feature AttributionPredict-Then-OptimizeShapley Value Attribution
AskPDF