CVaR Deep Optimal Stopping
Organizations: Seoul National University
Abstract
Risk-sensitive optimal stopping is useful when poor stopping outcomes carry substantial costs, as in option exercise, sequential medical treatment, and safety-critical control. A natural objective is Conditional Value at Risk (CVaR) of the stopped reward. We develop a theoretical framework and algorithm for CVaR deep optimal stopping. The framework and algorithm address four difficulties in applying risk-sensitive reinforcement learning to optimal stopping: state augmentation, score-function pathology, blindness to success, and dependence on auxiliary networks. We isolate three structural properties of optimal stopping: single-point reward, action-independent dynamics, and sum-zero gradient. We exploit these properties to establish Markov optimality without state augmentation and derive a pathwise gradient estimator. Under bounded policy derivatives, the pathwise estimator has bounded second moments near deterministic policies, avoiding the divergence that can occur with score-function estimators. Building on this framework, we propose DIOS (Distribution-Informed Optimal Stopping) to address the remaining two difficulties. A per-time-step soft penalty combined with tail-level annealing mitigates blindness to success. Estimating the reward quantile directly from each batch removes the need for an auxiliary critic. For a fixed tail level, we establish a finite-time stationarity guarantee under regularity assumptions along the training iterates. Across the 24 main Bermudan max-call configurations, DIOS achieves the highest mean CVaR and the shortest recorded training time among the tested CVaR baselines.
Figures & tables
| DIOS (ours) | CVaR-VaR | CVaR-PG | MIX | RET-CAP | |||
|---|---|---|---|---|---|---|---|
| CVaR | |||||||
| wall (coll) | (0/10) | (0/10) | (2/10) | (2/10) | (0/10) | ||
| CVaR | |||||||
| wall (coll) | (0/10) | (0/10) | (0/10) | (0/10) | (0/10) | ||
| CVaR | |||||||
| wall (coll) | (0/10) | (0/10) | (3/10) | (5/10) | (0/10) |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Value |
|---|---|
| Strike | |
| Risk-free rate | |
| Dividend yield | |
| Volatility | (shared across all underlyings) |
| Asset correlation | (independent) |
| Maturity | years |
| Item | Value |
|---|---|
| Trajectories per update | |
| Outer iterations | |
| Risk level | |
| Discount | |
| Seeds | |
| Optimizer | Adam |
| Method | Policy LR | Critic LR mult. | Norm. adv. | Method-specific |
|---|---|---|---|---|
| DIOS (ours) | — | — | 3-phase -schedule with warmup fraction , anneal fraction , hold fraction : for the first iterations, linear over the next , thereafter | |
| CVaR-VaR | no | atoms in the quantile critic; tradeoff weight throughout training, selected from the three -schedules of Luo and Delage [17] (Maze: ; LunarLander: for first 40% then linear decay to 0; Pendulum: for first 25% then 0) | ||
| CVaR-PG | — | no | trajectory-level score-function estimator with no value critic [ 25 ] | |
| RET-CAP (cap-anchor) | yes | minimum cap computed from DIOS evaluation quantiles as in Appendix D.2 ; advantage-shaping weight | ||
| RET-CAP (RN-anchor) | yes | risk-neutral anchor variant of Appendix D.3 ; advantage-shaping weight | ||
| MIX | — | — | IQL component: LR , update interval , expectile , temperature , batch size per update, replay buffer capacity transitions |
| 95% CI | W | |||||
| Symmetric | ||||||
| 20 | 90 | [-0.064, +0.271] | 0.197 | 0.625 | no | |
| 20 | 100 | [-0.086, +0.311] | 0.23 | 0.492 | no | |
| 20 | 110 | [-0.204, +0.350] | 0.565 | 1 | no | |
| 50 | 90 | [-0.081, +0.095] | 0.857 | 1 | no | |
| 50 | 100 | [-0.010, +0.186] | 0.0736 | 0.232 | no | |
| CVaR 0.2 | wall (s) | |||
|---|---|---|---|---|
| 20 | 90 | |||
| 100 | ||||
| 110 | ||||
| 50 | 90 | |||
| 100 | ||||
| 110 |
| cap CVaR 0.2 | RN CVaR 0.2 | ||||
|---|---|---|---|---|---|
| 20 | 90 | ||||
| 100 | |||||
| 110 | |||||
| 50 | 90 | ||||
| 100 | |||||
| 110 |
| DIOS | RN | |||
|---|---|---|---|---|
| 20 | 90 | |||
| 100 | ||||
| 110 | ||||
| 50 | 90 | |||
| 100 | ||||
| 110 |
| DIOS | No schedule | coll | ||
|---|---|---|---|---|
| 20 | 90 | |||
| 100 | ||||
| 110 | ||||
| 50 | 90 | |||
| 100 | ||||
| 110 |
| full CVaR | wall | no-critic CVaR | wall | speedup | |||
|---|---|---|---|---|---|---|---|
| 20 | 90 | ||||||
| 100 | |||||||
| 110 | |||||||
| 50 | 90 | ||||||
| 100 | |||||||
| 110 |
| Method | Policy LR | Critic LR mult. | Norm. adv. | Notes |
|---|---|---|---|---|
| DIOS (ours) | — | — | same as symmetric (Table 4 ) | |
| CVaR-VaR | no | Maze schedule ( ), as in Table 4 | ||
| CVaR-PG | — | no | selected by our learning-rate sweep | |
| MIX | — | — | same as symmetric (Table 4 ) | |
| RET-CAP (cap-anchor) | yes | DIOS-derived anchor as in § 5 | ||
| RET-CAP (RN-anchor) | yes | risk-neutral anchor variant of Appendix E.3 |
| DIOS (ours) | CVaR-VaR | CVaR-PG | MIX | RET-CAP | |||
|---|---|---|---|---|---|---|---|
| CVaR | |||||||
| wall (coll) | (0/10) | (0/10) | (1/10) | (2/10) | (0/10) | ||
| CVaR | |||||||
| wall (coll) | (0/10) | (0/10) | (0/10) | (1/10) | (0/10) | ||
| CVaR | |||||||
| wall (coll) | (0/10) | (0/10) | (3/10) | (4/10) | (0/10) |
| CVaR 0.2 | wall (s) | |||
|---|---|---|---|---|
| 20 | 90 | |||
| 100 | ||||
| 110 | ||||
| 50 | 90 | |||
| 100 | ||||
| 110 |
| cap CVaR 0.2 | RN CVaR 0.2 | ||||
|---|---|---|---|---|---|
| 20 | 90 | ||||
| 100 | |||||
| 110 | |||||
| 50 | 90 | ||||
| 100 | |||||
| 110 |
| DIOS | RN | |||
|---|---|---|---|---|
| 20 | 90 | |||
| 100 | ||||
| 110 | ||||
| 50 | 90 | |||
| 100 | ||||
| 110 |
| DIOS | No schedule | coll | ||
|---|---|---|---|---|
| 20 | 90 | |||
| 100 | ||||
| 110 | ||||
| 50 | 90 | |||
| 100 | ||||
| 110 |
| full CVaR | wall | no-critic CVaR | wall | speedup | |||
|---|---|---|---|---|---|---|---|
| 20 | 90 | ||||||
| 100 | |||||||
| 110 | |||||||
| 50 | 90 | ||||||
| 100 | |||||||
| 110 |
| DIOS (raw) | DIOS ( -EMA) | CVaR-VaR | CVaR-PG | ||
| ( -EMA ) | |||||
| 20 | 90 | ||||
| 20 | 100 | ||||
| 20 | 110 | ||||
| 50 | 90 | ||||
| 50 | 100 | ||||
| cell | DIOS | CVaR-VaR | CVaR-PG | RET-CAP | MIX | |
|---|---|---|---|---|---|---|
| ln0.3_c1 | CVaR | |||||
| wall | 12.3 | 49.3 | 15.4 | 23.5 | 23.3 | |
| ln0.3_c2 | CVaR | |||||
| wall | 12.4 | 48.6 | 15.1 | 23.2 | 23.4 | |
| ln0.3_c3 | CVaR | |||||
| wall | 12.0 | 55.1 | 15.2 | 23.0 | 23.3 |
| cell | DIOS | CVaR-VaR | CVaR-PG | RET-CAP | MIX | |
|---|---|---|---|---|---|---|
| pa6_c1 | CVaR | |||||
| wall | 12.0 | 58.3 | 15.3 | 22.2 | 23.2 | |
| pa6_c2 | CVaR | |||||
| wall | 12.4 | 54.7 | 15.0 | 22.8 | 23.3 | |
| pa6_c3 | CVaR | |||||
| wall | 12.0 | 63.8 | 14.9 | 21.2 | 21.4 |
| hidden width | parameters | CVaR 0.2 | wall (s) |
|---|---|---|---|
| 32 | 5,025 | 13.5 | |
| 64 | 13,121 | 13.0 | |
| 128 | 38,529 | 12.7 | |
| 256 | 126,209 | 13.1 |
Optimal Stopping
Optimal stopping studies when to act on sequential information rather than wait, balancing expected reward against delay, cost, and risk.
See all 43 papers