An agent still learning its environment should be cautious while ignorant and bold once confident. The entropic value-at-risk captures this through a robust-optimization identity---a confidence level fixes the radius of a relative-entropy ball of alternative models---but that ball cannot reach catastrophes the nominal deems impossible, precisely what a safe agent must hedge. We instead use an optimal-transport ball and study the coherent risk measure it induces, the Wasserstein entropic value-at-risk. It has a variational dual mirroring the entropic formula (an inverse temperature becomes a transport price), occupies a definite place in the risk hierarchy, and provably accounts for the reachable catastrophes the entropic measure ignores; we verify both dualities numerically. Driving the transport radius by belief entropy then yields a closed-form robust dynamic-programming operator whose caution contracts as the belief sharpens, with a certified safety sandwich and a sharp safety switch.
Figures & tables
Figure 1: Why Wasserstein, geometrically. On the next-state simplex at a confident belief—the nominal Pˉb places zero mass on Fail —the relative-entropy ball (red) is trapped on the support edge, blind to the catastrophe, while the 1 -Wasserstein ball (blue) reaches the Fail vertex; the worst-case adversary ( × ) sits on its boundary. This is Thm. 2 (iii) drawn on Δ(S) .
Figure 2: The entropy-modulated ambiguity set U(b) on the next-state simplex. As the belief sharpens the radius ε=βH(b) contracts ( 1.04→0.20 ) and the nominal Pˉb drifts toward the goal; the worst-case adversary ( × ) loses its reach toward the failure vertex—robustness that vanishes with uncertainty (the geometric content of Thm. 3 ).
Figure 3: Computed guarantees on a five-state corridor with a hidden slip type ( γ=0.9 , Gurobi-checked). (a) The adaptive value V⋆(s0,b) stays inside the safety sandwich between the always-maximally-cautious floor Vwc and the type-aware oracle ceiling; the price of robustness shrinks as the belief sharpens. (b) From arbitrary initializations, value iteration with the closed-form operator collapses geometrically onto V⋆ along the γk envelope, certifying the bound at every iterate.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Gurobi-verified comparison on a five-state metric space. (a) hierarchy and the Pinsker sandwich; (b) the zero-probability catastrophe: the entropic value-at-risk is invariant to the disaster’s magnitude, WEVaR is not.
Figure 5: Manifolds over the belief simplex Δ(Z) , ∣Z∣=3 (calm/breezy/storm): ambiguity radius, robust value, and policy region (cruise only near the calm vertex).
Figure 6: Robust action values vs. belief. Left: the Ambiguous Bridge ( Sprint vs. Crawl , switch at α⋆≈0.983 ). Right: the stormy-drone canyon ( Cruise vs. Creep , switch at b(calm)≈0.63 ).
Figure 7: Catastrophe rate vs. time-to-goal on the stormy-drone canyon ( 4000 episodes). WEVaR dominates the static-robust Maximin baseline; the Bayesian planner is already cautious here (see caveat).
How cautious should an agent be while it is still learning its environment? We propose RATTL (Risk-Adversarial Total-Reward Learning), which ties caution to epistemic uncertainty: the agent holds a Bayesian posterior over unknown dynamics and plans against a Wasserstein ambiguity set whose radius is a monotone function of that posterior. The radius contracts with evidence, so behaviour interpolates continuously between worst-case robustness and risk-neutral total-reward maximization. The design follows the duality underlying the Entropic Value-at-Risk, which converts the choice of a risk level into the choice of an ambiguity radius. We show the resulting planning problem is well posed under transience and compactness conditions, and prove a Safety Sandwich: the RATTL value lies between the uninformed robust value and the full- knowledge optimum, with a gap that vanishes as the posterior concentrates. In a canonical binary-hazard instance, the induced criterion reduces to Conditional Value-at-Risk at a level set by the posterior entropy. A worked example shows the agent deferring the efficient action until a sharp identification threshold. RATTL targets runtime safety for agents, including LLM-based systems, acting under uncertainty.
Deep Kumar Ganguly, Jan Kretinsky
Technical University of Munich, Germany · Masaryk University, Brno, Czech Republic
We propose a distributionally robust approach to risk-sensitive estimation of an unknown signal x from an observed signal y. The observation and unknown signal are modeled as random vectors whose joint probability distribution is unknown, but assumed to belong to a given type-2 Wasserstein ball of distributions, termed the ambiguity set. The performance of an estimator is measured according to the conditional value-at-risk (CVaR) of the squared estimation error. Within this framework, we study the problem of computing affine estimators that minimize the worst-case CVaR over all distributions in the given ambiguity set. As our main result, we show that, when the nominal distribution at the center of the Wasserstein ball is finitely supported, such estimators can be exactly computed by solving a tractable semidefinite program. We evaluate the proposed estimators on a wholesale electricity price forecasting task using real market data and show that they deliver lower out-of-sample CVaR of squared error compared to existing methods.
Feras Al Taha, Eilyan Bitar
School of Electrical and Computer Engineering, Cornell University, Ithaca, NY 14853 USA
Tail-end risk measures such as static conditional value-at-risk (CVaR) are used in safety-critical applications to prevent rare, yet catastrophic events. Unlike risk-neutral objectives, the static CVaR of the return depends on entire trajectories without admitting a recursive Bellman decomposition in the underlying Markov decision process. A classical resolution relies on state augmentation with a continuous variable. However, unless restricted to a specialized class of admissible value functions, this formulation induces sparse rewards and degenerate fixed points. In this work, we propose a novel formulation of the static CVaR objective based on augmentation. Our alternative approach leads to a Bellman operator with: (1) dense per-step rewards; (2) contracting properties on the full space of bounded value functions. Building on this theoretical foundation, we develop risk-averse value iteration and model-free Q-learning algorithms that rely on discretized augmented states. We further provide convergence guarantees and approximation error bounds due to discretization. Empirical results demonstrate that our algorithms successfully learn CVaR-sensitive policies and achieve effective performance-safety trade-offs.
Aneri Muni, Vincent Taboga, Esther Derman +2
Universit´e de Montr´eal, Quebec, Canada · Mila - Quebec AI Institute, Quebec, Canada · GERAD & Department of Decision Sciences, HEC Montr´eal, Quebec, Canada