How cautious should an agent be while it is still learning its environment? We propose RATTL (Risk-Adversarial Total-Reward Learning), which ties caution to epistemic uncertainty: the agent holds a Bayesian posterior over unknown dynamics and plans against a Wasserstein ambiguity set whose radius is a monotone function of that posterior. The radius contracts with evidence, so behaviour interpolates continuously between worst-case robustness and risk-neutral total-reward maximization. The design follows the duality underlying the Entropic Value-at-Risk, which converts the choice of a risk level into the choice of an ambiguity radius. We show the resulting planning problem is well posed under transience and compactness conditions, and prove a Safety Sandwich: the RATTL value lies between the uninformed robust value and the full- knowledge optimum, with a gap that vanishes as the posterior concentrates. In a canonical binary-hazard instance, the induced criterion reduces to Conditional Value-at-Risk at a level set by the posterior entropy. A worked example shows the agent deferring the efficient action until a sharp identification threshold. RATTL targets runtime safety for agents, including LLM-based systems, acting under uncertainty.
Figures & tables
Figure 1: The coherent-risk hierarchy: EVaR is the single-parameter coherent family sweeping E to esssup . RATTL uses belief entropy H(b) to slide along it.
Figure 2: Wasserstein ambiguity balls U(b) on the next-state simplex for three beliefs of decreasing entropy. The ball shrinks (and its center Pˉb moves) as the belief sharpens: high entropy lets the adversary push mass toward catastrophic states; low entropy renders it nearly powerless.
W1
TV
KL
Worst-case EQ[V] , ε=0.25
−345.0
−217.5
−3.4
Mass moved onto catastrophe
0.40
0.25
0.00
Table 1: Inner adversary at radius ε=0.25 on a cliff transition whose catastrophe carries zero nominal mass (nominal value +57.5 ). KL cannot place any mass on the off-support catastrophe ( DKL=+∞ ), so its mass there stays 0 and it cannot price the tail even as it reweights on-support mass; only Wasserstein reaches the catastrophe.
Figure 3: The KL support catastrophe, visualized (radius ε=0.25 ). The KL adversary cannot reach the off-support catastrophe (its mass there stays 0 ), so it barely departs from the non-robust value (dashed line) despite reweighting on-support mass; only Wasserstein transports mass onto the catastrophe and reports the true tail risk ( −345 ).
Figure 4: Entropy is a CVaR dial (Proposition 1 ). As the belief sharpens ( α→1 , H→0 ), the induced CVaR tail level θ(ε)=q/(q+βH/D) rises from a deep tail (worst-case-like caution) toward 1 (risk-neutrality), for catastrophe masses q∈{0.05,0.1,0.2} . This is the risk spectrum of Figure 1 , now traversed automatically by learning.
α
ε(α)
Q(\textscSprint)
Q(\textscCrawl)
π∗
0.50
0.6931
−1001.00
−682.46
Crawl
0.85
0.4227
−530.98
−384.98
Crawl
0.99
0.0560
+26.40
+18.40
Sprint
α∗=1081/1100
0.0872
−15.95
−15.95
switch
Table 2: Robust value iteration on the Ambiguous Bridge ( β=1 , D=1 ). The safety floor is Vmaximin(0.5)=80−1100ln2=−682.46 ; the best-response ceiling is VBR=99 as α→1 . The threshold α∗=1081/1100 is β -independent here.
Figure 5: The safety switch. Crawl dominates while uncertain; at α∗≈0.983 the risky action’s worst-case value overtakes the safe one and the agent switches to Sprint .
Figure 6: The Safety Sandwich, instantiated. RATTL’s value rises from the worst-case floor ( Vmaximin , never-shrinking ambiguity) toward the full-information ceiling ( VBR ) as the belief sharpens—never less safe than maximin, never more reckless than the informed optimum. The two reference lines are constant bounds ( −682.5 and 99 ); V∗ provably stays between them at every belief.