How Risky Is Your Data-Driven Policy?

Omar Besbes; Alon Rieger

stat.MLNov 3, 2026

How Risky Is Your Data-Driven Policy?

Authors: Omar Besbes, Alon Rieger

Abstract

Data-driven algorithms are ubiquitous across industries, and there is an extensive literature on analyzing their average performance. The risk profile of such algorithms, while critical, is much less understood. This paper quantifies the risk of data-driven policies and isolates its sources. We evaluate a policy through the conditional value-at-risk (CVaR) of its loss over the training sample, any policy randomization, and the future outcome, at a fixed unknown distribution. Regret against an oracle that knows the distribution decomposes into expected post-sample regret (PS), the average outcome-only CVaR gap of the selected action, and a sample-risk premium (SR), the additional tail risk omitted by averaging those evaluations. The analogous premium vanishes under expected loss. We characterize when the premium vanishes and how large it is relative to post-sample regret. For losses with a continuous density near the tail cutoff, we identify conditions under which both components decay at the same rate, yet the limiting ratio of the premium to post-sample regret can reach the thousands. When a probability mass at the cutoff moves with the action, the premium can decay more slowly and dominate post-sample regret. We establish conditions under which sample average approximation learns the oracle action at rate $n^{-1/2}$ in both settings, but its combined-risk regret is of order $n^{-1}$ in the first and $n^{-1/2}$ in the second. In finite-sample numerical examples of SAA in both settings, the premium accounts for approximately $74\%$ -- $97\%$ of regret.

Figures & tables

Family	$\boldsymbol{\delta=0.80}$	$\boldsymbol{\delta=0.90}$	$\boldsymbol{\delta=0.95}$	$\boldsymbol{\delta=0.99}$	$\boldsymbol{\delta=0.999}$
Exponential	$1.56$	$3.93$	$8.76$	$47.9$	$492$
Gamma	$0.228$	$0.413$	$0.637$	$1.33$	$2.93$
Lognormal	$34.7$	$136$	$428$	$3900$	$49700$
Pareto	$4.46$	$13.4$	$36.6$	$330$	$7200$
Weibull	$6.21$	$22.1$	$70.5$	$884$	$28700$

Table 1: The relative variance penalty

K

at the oracle action for

c(x,\xi)=(\xi-x)^{2}/2

, from ( A.19 ), with the oracle action and cutoff located numerically. For the limiting-ratio interpretation, take a compact action interval with the oracle in its interior and a policy satisfying Theorem 3.4 with centered limiting fluctuations. Then the entries are the limit of

\rho_{n}=\mathrm{SR}_{\delta,n}\!\left(\pi,F\right)/\mathrm{PS}_{\delta,n}\!\left(\pi,F\right)

. The coefficient is invariant to location–scale changes of the outcome for this squared loss. Exponential has rate

1

; gamma has shape

4

and scale

1/8

; lognormal has

\log\xi\sim\mathcal{N}(0,1.8^{2})

; Pareto is Type I with tail index

3

and scale

1

; Weibull has shape

0.7

and scale

1

Regime	Entered when	Sample-risk premium	Expected post-sample regret
Smooth	Unique optimum; the cutoff moves to first order and the loss has a positive finite density there; the policy concentrates at rate $r_{n}\searrow 0$ .	$\Theta(r_{n}^{2})$	$\Theta(r_{n}^{2})$
Atomic	Unique optimum; an atom at the cutoff moves with the action; the policy concentrates at rate $r_{n}\searrow 0$ .	$\Theta(r_{n})$	$\Theta(r_{n}^{2})$
Mixing	Several actions are optimal, and the policy keeps mixing over them, with no common minimizing threshold.	$\Theta(1)$	$o(1)$
Displaced	Unique well-separated optimum, and the policy is persistently bounded away from it.	unrestricted	$\Omega(1)$

Table 2: Structural regimes governing the relative magnitude of the sample-risk premium and expected post-sample regret. In the displaced row, unrestricted means that Proposition 3.10 does not constrain the premium. Each row abbreviates the hypotheses of the corresponding result. The smooth and atomic rows also use positive finite curvature, the moment and nondegeneracy conditions, and, in the atomic row, strict straddling and the isolation limits. Concentration at a rate

r_{n}\searrow 0

implies consistency. Persistent displacement is strictly stronger than failure of consistency: an inconsistent policy need only sit away from

x_{F}^{*}

along a subsequence, which yields

\limsup_{n}\mathrm{PS}_{\delta,n}\!\left(\pi,F\right)>0

rather than the

\mathrm{PS}_{\delta,n}\!\left(\pi,F\right)=\Omega(1)

recorded in the row.

Outcome	$K$	$n$	$\mathrm{SR}/\mathrm{PS}$	Premium share (%)
Exponential	$8.76$	$100$	$2.90\ (0.05)$	$74.4\ (0.31)$
		$400$	$3.88\ (0.04)$	$79.5\ (0.16)$
		$1600$	$5.59\ (0.04)$	$84.8\ (0.09)$
Lognormal	$418.42$	$100$	$14.08\ (0.20)$	$93.4\ (0.09)$
		$400$	$12.52\ (0.14)$	$92.6\ (0.08)$
		$1600$	$14.38\ (0.15)$	$93.5\ (0.06)$

Table 3: Smooth-regime SAA with the truncated outcome distributions specified in the text. Premium share is

\mathrm{SR}/(\mathrm{PS}+\mathrm{SR})

. Estimates use

12{,}000

independent training samples per row. Parentheses give approximate Monte Carlo standard errors, in percentage points for shares. The limiting coefficient

K

is computed from the population distribution. Appendix C.2 describes the computation.

$n$	$n\,\mathrm{PS}$	$\sqrt{n}\,\mathrm{SR}$	$\mathrm{SR}/\mathrm{PS}$	Premium share
$110$	$0.2548$	$0.0781$	$3.21$	$76.3\%$
$430$	$0.2434$	$0.0962$	$8.19$	$89.1\%$
$1630$	$0.2391$	$0.1041$	$17.58$	$94.6\%$
$6430$	$0.2379$	$0.1078$	$36.34$	$97.3\%$
Limit	$0.2375$	$0.1112$	$\sim 0.4683\sqrt{n}$	$\to 100\%$

Table 4: Finite-sample SAA regret for uniform demand. Premium share is

\mathrm{SR}/(\mathrm{PS}+\mathrm{SR})

. Values are evaluated numerically from the exact Beta distribution using incomplete Beta moments and a scalar quantile equation, without simulation.

Risky Decision-Making

See all 18 papers

Tail-Influence Sampling for CVaR Policy Evaluation
Sep 29, 2026Pauline Bourigault, Xiaotong Ji, Matthieu Zimmer +2Policy Evaluation Adaptive Sampling
Imperial College London · Huawei Noah’s Ark Lab · UCL Centre for AI
On the Generalization and Robustness in Conditional Value-at-Risk
Feb 20, 2026Dinesh Karthik Mulumudi, Piyushi Manupriya, Gholamali Aminian +1Conditional Value-at-Risk Empirical Risk Minimization
Dept. of Mathematics, IISER, Pune. · Dept. of Computer Science and Automation, Indian Institute of Sciecne, Bangalore. · The Alan Turing Institute, Greater London, England, UK.
Risk-averse learning with delayed feedback
Sep 25, 2024Siyi Wang, Zifan Wang, Karl Henrik Johansson +1Dynamic Regret Minimization Zeroth-Order Optimization
Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, 10044 Stockholm, Sweden · Chair of Information-oriented Control (ITR), Technical University of Munich, Germany

Sep 29, 2026J/K move · Enter open · S save

Tail-Influence Sampling for CVaR Policy Evaluation

Pauline Bourigault, Xiaotong Ji, Matthieu Zimmer, Rasul Tutunov+1

Imperial College London · Huawei Noah’s Ark Lab · UCL Centre for AI

Policy Evaluation Adaptive Sampling Conditional Value-at-Risk Risk-Sensitive RL Variance Reduction

Open paper AskPDF arXiv

Policies with similar mean returns can differ sharply in rare failures, yet estimating lower-tail conditional value-at-risk (CVaR) accurately can require many costly rollouts. When different conditional components of a stochastic workflow can be queried separately, we ask how to allocate a fixed evaluation budget to estimate a fixed policy's CVaR most accurately. We derive a tail influence for each queryable conditional law that aggregates how its uncertainty affects CVaR across every Bellman reuse. Its variance yields the fixed-design efficiency bound and the oracle Neyman allocation. Tail-Influence Sampling (TIS) estimates these influence scales from a pilot model and reallocates fresh queries toward kernels that matter most for the tail; a visitation-anchored variant protects against pilot underallocation. Under fixed dimension and a positive quantile margin, TIS attains oracle asymptotic variance and first-order MSE including pilot cost, while the anchored variant is within a factor two of the oracle. We also characterize an exact-grid regime in which tail- and mean-optimal allocations coincide. On CliffWalking, TIS reduces MSE by 41% versus learned occupancy and 76% versus complete rollouts at the same charged transition budget. In frozen language-model review workflows, anchored TIS beats an equally regularized mean-influence blend in 23 of 24 MMLU-Pro settings and reaches 2.4-3.4

\times

lower MSE than rollouts on six-call FinQA reviews.

Abstract

Risky Decision-Making

Explore similar work

Tail-Influence Sampling for CVaR Policy Evaluation

On the Generalization and Robustness in Conditional Value-at-Risk

Risk-averse learning with delayed feedback