cs.LGSep 29, 2026
SaveRL-PaO: Prediction as Action in Decision Making under Uncertainty
Organizations: Great Bay University, School of Computing and Information Technology, China · The University of Osaka, School of Information Science and Technology, Japan · The University of Osaka, SANKEN, Japan · Aalto University, School of Electrical Engineering, Finland
Abstract
Decision-making under uncertainty often relies on predicted parameters, yet accurate prediction does not necessarily lead to good operational decisions. Aligning prediction with downstream optimization requires learning from the consequences of the decisions those predictions induce. We introduce RL-PaO, a reinforcement learning framework that integrates system formulation, optimization, and decision execution into a single environment. This yields a Markov decision process in which prediction is regarded as action: it shifts the environment to produce subsequent context and reward that explicitly aligns prediction error with realized cost, and learning the optimal policy does not require differentiating through the black-box solver. We evaluate RL-PaO on day-ahead energy scheduling using real historical data. On the test year, RL-PaO achieves the lowest annual cost among the non-oracle baselines, achieving on average cost reduction. Moreover, RL-PaO is capable of further analyses to provide strong interpretability both from the policy evolution perspective and the cost-accuracy trade-off.
Figures & tables
| Prior-based Optimization | Predict-Optimize Paradigm | |||||
| PtO | PaO | |||||
| Supervised PaO | RL-PaO | |||||
| DO | RO | SO | Ours | |||
| Uncertainty modeling | × | Robust uncertainty sets | Probability distributions | Open-loop prediction | Closed-loop prediction | Prediction-optimization alignment |
| Placement-free uncertainty | – | ✓ | ✓ | ✓ | × | ✓ |
| Decision-focused training | – | – | – | × | ✓ | ✓ |
Table 1: Comparison of methods for optimization under uncertainty. Ours is the first RL-based PaO method that aims at improving the alignment between prediction and optimization. We emphasize the importance of Decision‑focused training that optimizes by downstream decision cost instead of prediction error; and Model‑agnostic optimization that operates without full knowledge of the underlying optimization formulation.
Figure 1: Overview of the proposed RL-PaO framework. (A) We explicitly include system modeling, optimization, scheduling and real-world operation into one environment. The RL agent observes current context to predict the unknown constraint in the optimization objective to be solved by an MILP solver. The solution influences execution process, yielding reward and next context. (B) our RL formulation is under the PaO category and different from the conventional supervised PaO methods. (C) Architectural comparison of different methods. PtO uses an open-loop predictor. Supervised PaO closes the training loop by incorporating downstream loss. By contrast, the proposed method aligns prediction and optimization by a novel MDP and RL training.
| Symbol | Description |
| Electricity price | |
| Actual PV output | |
| Grid import power | |
| Fixed load demand | |
| ESS (dis)charging power | |
| PV upper bound † |
Table 2: Notations for the problem.
Figure 2: Cumulative rewards (upper row) and downstream operation costs (lower) across different learning rates. Transparency of lines set according to value. RL-PaO is effective in (1) establishing correspondence between high rewards and lower costs; (2) the performance curves change smoothly along gradual increase of hyperparameters.
Figure 3: Comparison on annual cost. Blue solid line and shade are mean and standard deviation over 5 independent runs. Left : RL-PaO compared against the conventional PbO baselines that require model knowledge in uncertainty (DO, RO, SO). Right : comparison against PtO, PaO methods. In both cases the model-free RL-PaO achieves the best cost.
| Category | Method | Annual Cost | Cost Gap (%) |
| PbO | Oracle | 963.76 | 0.00 |
| DO | 1496.20 | 55.25 | |
| RO | 1145.20 | 18.83 | |
| SO | 1133.90 | 17.65 | |
| PtO/PaO | PtO | 1233.40 | 27.98 |
| Supervised PaO | 1207.90 | 25.33 |
Table 3: Overall performance comparison on the 2019 test set. Annual cost is measured in the unit of . indicates lower is better.
Figure 4: Policy evolution plot of the learned Gaussian distribution across 12 action dimensions at training day 0, 100, and 200. Recall that prediction of the uncertain parameter is sampled from the policy. As learning proceeds, the policies gradually narrow and shift toward the optimal decision region that aligns prediction with downstream cost optimization.
Figure 5: Prediction error versus operation cost, daily level. The first three windows show scatter plots on the test set. The last window shows a joint-distribution plot. Transparency distinguishes weekdays and weekends. PtO and PaO points are concentrated in a low estimation error region but the average cost is high. Outliers with extremely high costs are identified by red dots with purple background. By contrast, RL-PaO error distribution is more dispersed, but the average cost is lower, as can be seen from the quantile line from the last window.
Figure 6: Prediction error versus operation cost, annual level. (a) Pareto front of annual cost against prediction MAE, recorded for . The reward weight spans a continuous trade-off curve and sits on the Pareto frontier, indicating that no competing methods can simultaneously achieve lower prediction error and lower operation cost. (b) Overall correlation across all and learning rate configurations. The blue points scattered in the lower left corner supports the finding that lower prediction error does not equal lower cost.
Figure 7: Mahanttan plot of the proposed method across learning rates and reward weights . Transparent dots show final operational costs, with deep blue dots indicating the mean. (Upper) Results of fixing and varying lr . (Lower) Fixing lr and varying . The superior performance of RL can be attained from various combinations.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
| Reward weight | Swept in |
| Learning rate | Swept in |
| Discount factor | 0.99 |
| PPO clip range | 0.2 |
| GAE parameter | 0.95 |
| Value function coefficient | 0.5 |
Table 4: Hyperparameters and configurations.
Figure C1: Daily operation cost by month compared with conventional optimization baselines on the 2019 test set. Each subplot corresponds to one calendar month, comparing Oracle, DO, RO, SO and the proposed RL-PaO.
Figure C2: Daily operation cost by month compared with predict-and-optimize baselines on the 2019 test set. Each subplot corresponds to one calendar month, comparing Oracle, predict-then-optimize (PtO), supervised PaO, improved PaO and the proposed RL-PaO.
Figure C3: Convergence behavior of RL-PaO across different reward weights and learning rates. Top: mean reward convergence across training episodes. Bottom: annual operation cost convergence across training episodes. Each column corresponds to a fixed value, and each colored curve corresponds to a different learning rate.
Figure C4: Left: Reward and operation cost convergence under different learning rates with fixed at 0.90. Each colored band corresponds to one learning rate configuration. Right: Reward convergence under different reward weights with learning rate fixed at . Each colored band corresponds to one configuration.
Figure C5: Left: Annual operation cost convergence under different reward weights with learning rate fixed at . Each colored band corresponds to one configuration. Right: Annual operation cost convergence under different learning rates with fixed at 0.90. Each colored band corresponds to one learning rate configuration.
Figure C6: Final annual operation cost with mean and standard deviation across different learning rates, with fixed at 0.90.
Figure C7: Final annual operation cost with mean and standard deviation across different reward weights , with learning rate fixed at . The dashed line marks the value adopted in the main experiment.