How Risky Is Your Data-Driven Policy?

Omar Besbes; Alon Rieger

stat.MLNov 3, 2026

How Risky Is Your Data-Driven Policy?

Authors: Omar Besbes, Alon Rieger

Abstract

Data-driven algorithms are ubiquitous across industries, and there is an extensive literature on analyzing their average performance. The risk profile of such algorithms, while critical, is much less understood. This paper quantifies the risk of data-driven policies and isolates its sources. We evaluate a policy through the conditional value-at-risk (CVaR) of its loss over the training sample, any policy randomization, and the future outcome, at a fixed unknown distribution. Regret against an oracle that knows the distribution decomposes into expected post-sample regret (PS), the average outcome-only CVaR gap of the selected action, and a sample-risk premium (SR), the additional tail risk omitted by averaging those evaluations. The analogous premium vanishes under expected loss. We characterize when the premium vanishes and how large it is relative to post-sample regret. For losses with a continuous density near the tail cutoff, we identify conditions under which both components decay at the same rate, yet the limiting ratio of the premium to post-sample regret can reach the thousands. When a probability mass at the cutoff moves with the action, the premium can decay more slowly and dominate post-sample regret. We establish conditions under which sample average approximation learns the oracle action at rate $n^{-1/2}$ in both settings, but its combined-risk regret is of order $n^{-1}$ in the first and $n^{-1/2}$ in the second. In finite-sample numerical examples of SAA in both settings, the premium accounts for approximately $74\%$ -- $97\%$ of regret.

Figures & tables

Family	$\boldsymbol{\delta=0.80}$	$\boldsymbol{\delta=0.90}$	$\boldsymbol{\delta=0.95}$	$\boldsymbol{\delta=0.99}$	$\boldsymbol{\delta=0.999}$
Exponential	$1.56$	$3.93$	$8.76$	$47.9$	$492$
Gamma	$0.228$	$0.413$	$0.637$	$1.33$	$2.93$
Lognormal	$34.7$	$136$	$428$	$3900$	$49700$
Pareto	$4.46$	$13.4$	$36.6$	$330$	$7200$
Weibull	$6.21$	$22.1$	$70.5$	$884$	$28700$

Table 1: The relative variance penalty

K

at the oracle action for

c(x,\xi)=(\xi-x)^{2}/2

, from ( A.19 ), with the oracle action and cutoff located numerically. For the limiting-ratio interpretation, take a compact action interval with the oracle in its interior and a policy satisfying Theorem 3.4 with centered limiting fluctuations. Then the entries are the limit of

\rho_{n}=\mathrm{SR}_{\delta,n}\!\left(\pi,F\right)/\mathrm{PS}_{\delta,n}\!\left(\pi,F\right)

. The coefficient is invariant to location–scale changes of the outcome for this squared loss. Exponential has rate

1

; gamma has shape

4

and scale

1/8

; lognormal has

\log\xi\sim\mathcal{N}(0,1.8^{2})

; Pareto is Type I with tail index

3

and scale

1

; Weibull has shape

0.7

and scale

1

Regime	Entered when	Sample-risk premium	Expected post-sample regret
Smooth	Unique optimum; the cutoff moves to first order and the loss has a positive finite density there; the policy concentrates at rate $r_{n}\searrow 0$ .	$\Theta(r_{n}^{2})$	$\Theta(r_{n}^{2})$
Atomic	Unique optimum; an atom at the cutoff moves with the action; the policy concentrates at rate $r_{n}\searrow 0$ .	$\Theta(r_{n})$	$\Theta(r_{n}^{2})$
Mixing	Several actions are optimal, and the policy keeps mixing over them, with no common minimizing threshold.	$\Theta(1)$	$o(1)$
Displaced	Unique well-separated optimum, and the policy is persistently bounded away from it.	unrestricted	$\Omega(1)$

Table 2: Structural regimes governing the relative magnitude of the sample-risk premium and expected post-sample regret. In the displaced row, unrestricted means that Proposition 3.10 does not constrain the premium. Each row abbreviates the hypotheses of the corresponding result. The smooth and atomic rows also use positive finite curvature, the moment and nondegeneracy conditions, and, in the atomic row, strict straddling and the isolation limits. Concentration at a rate

r_{n}\searrow 0

implies consistency. Persistent displacement is strictly stronger than failure of consistency: an inconsistent policy need only sit away from

x_{F}^{*}

along a subsequence, which yields

\limsup_{n}\mathrm{PS}_{\delta,n}\!\left(\pi,F\right)>0

rather than the

\mathrm{PS}_{\delta,n}\!\left(\pi,F\right)=\Omega(1)

recorded in the row.

Outcome	$K$	$n$	$\mathrm{SR}/\mathrm{PS}$	Premium share (%)
Exponential	$8.76$	$100$	$2.90\ (0.05)$	$74.4\ (0.31)$
		$400$	$3.88\ (0.04)$	$79.5\ (0.16)$
		$1600$	$5.59\ (0.04)$	$84.8\ (0.09)$
Lognormal	$418.42$	$100$	$14.08\ (0.20)$	$93.4\ (0.09)$
		$400$	$12.52\ (0.14)$	$92.6\ (0.08)$
		$1600$	$14.38\ (0.15)$	$93.5\ (0.06)$

Table 3: Smooth-regime SAA with the truncated outcome distributions specified in the text. Premium share is

\mathrm{SR}/(\mathrm{PS}+\mathrm{SR})

. Estimates use

12{,}000

independent training samples per row. Parentheses give approximate Monte Carlo standard errors, in percentage points for shares. The limiting coefficient

K

is computed from the population distribution. Appendix C.2 describes the computation.

$n$	$n\,\mathrm{PS}$	$\sqrt{n}\,\mathrm{SR}$	$\mathrm{SR}/\mathrm{PS}$	Premium share
$110$	$0.2548$	$0.0781$	$3.21$	$76.3\%$
$430$	$0.2434$	$0.0962$	$8.19$	$89.1\%$
$1630$	$0.2391$	$0.1041$	$17.58$	$94.6\%$
$6430$	$0.2379$	$0.1078$	$36.34$	$97.3\%$
Limit	$0.2375$	$0.1112$	$\sim 0.4683\sqrt{n}$	$\to 100\%$

Table 4: Finite-sample SAA regret for uniform demand. Premium share is

\mathrm{SR}/(\mathrm{PS}+\mathrm{SR})

. Values are evaluated numerically from the exact Beta distribution using incomplete Beta moments and a scalar quantile equation, without simulation.

Risky Decision-Making

See all 18 papers

Sep 29, 2026

Tail-Influence Sampling for CVaR Policy Evaluation

Policies with similar mean returns can differ sharply in rare failures, yet estimating lower-tail conditional value-at-risk (CVaR) accurately can require many costly rollouts. When different conditional components of a stochastic workflow can be queried separately, we ask how to allocate a fixed evaluation budget to estimate a fixed policy's CVaR most accurately. We derive a tail influence for each queryable conditional law that aggregates how its uncertainty affects CVaR across every Bellman reuse. Its variance yields the fixed-design efficiency bound and the oracle Neyman allocation. Tail-Influence Sampling (TIS) estimates these influence scales from a pilot model and reallocates fresh queries toward kernels that matter most for the tail; a visitation-anchored variant protects against pilot underallocation. Under fixed dimension and a positive quantile margin, TIS attains oracle asymptotic variance and first-order MSE including pilot cost, while the anchored variant is within a factor two of the oracle. We also characterize an exact-grid regime in which tail- and mean-optimal allocations coincide. On CliffWalking, TIS reduces MSE by 41% versus learned occupancy and 76% versus complete rollouts at the same charged transition budget. In frozen language-model review workflows, anchored TIS beats an equally regularized mean-influence blend in 23 of 24 MMLU-Pro settings and reaches 2.4-3.4

\times

lower MSE than rollouts on six-call FinQA reviews.

Pauline Bourigault, Xiaotong Ji, Matthieu Zimmer +2

Imperial College London · Huawei Noah’s Ark Lab · UCL Centre for AI

Policy Evaluation Adaptive Sampling Conditional Value-at-Risk Risk-Sensitive RL Variance Reduction

Ask PDF

Feb 20, 2026

On the Generalization and Robustness in Conditional Value-at-Risk

Conditional Value-at-Risk (CVaR) is a widely used risk-sensitive objective for learning under rare but high-impact losses, yet its statistical behavior under heavy-tailed data remains poorly understood. Unlike expectation-based risk, CVaR depends on an endogenous, data-dependent quantile, which couples tail averaging with threshold estimation and fundamentally alters both generalization and robustness properties. In this work, we develop a learning-theoretic analysis of CVaR-based empirical risk minimization under heavy-tailed and contaminated data. We establish sharp, high-probability generalization and excess risk bounds under minimal moment assumptions, covering fixed hypotheses, finite and infinite classes, and extending to

β

-mixing dependent data; we further show that these rates are minimax optimal. To capture the intrinsic quantile sensitivity of CVaR, we derive a uniform Bahadur-Kiefer type expansion that isolates a threshold-driven error term absent in mean-risk ERM and essential in heavy-tailed regimes. We complement these results with robustness guarantees by proposing a truncated median-of-means CVaR estimator that achieves optimal rates under adversarial contamination. Finally, we show that CVaR decisions themselves can be intrinsically unstable under heavy tails, establishing a fundamental limitation on decision robustness even when the population optimum is well separated. Together, our results provide a principled characterization of when CVaR learning generalizes and is robust, and when instability is unavoidable due to tail scarcity.

Dinesh Karthik Mulumudi, Piyushi Manupriya, Gholamali Aminian +1

Dept. of Mathematics, IISER, Pune. · Dept. of Computer Science and Automation, Indian Institute of Sciecne, Bangalore. · The Alan Turing Institute, Greater London, England, UK.

Conditional Value-at-Risk Empirical Risk Minimization Statistical Learning Theory Robust Estimation Heavy-Tailed Distributions

Ask PDF

Sep 25, 2024

Risk-averse learning with delayed feedback

In real-world scenarios, risk-averse learning is valuable for mitigating potential adverse outcomes. However, the delayed feedback makes it challenging to assess and manage risk effectively. In this paper, we investigate risk-averse learning using Conditional Value at Risk (CVaR) as risk measure, while incorporating feedback with random but bounded delays. We develop two risk-averse learning algorithms that rely on one-point and two-point zeroth-order optimization approaches, respectively. The dynamic regrets of the algorithms are analyzed in terms of the cumulative delay and the number of total samplings. In the absence of delay, the regret bounds match the established bounds of zeroth-order stochastic gradient methods for risk-averse learning. Furthermore, the two-point risk-averse learning outperforms the one-point algorithm by achieving a smaller regret bound. We provide numerical experiments on a dynamic pricing problem to demonstrate the performance of the algorithms.

Siyi Wang, Zifan Wang, Karl Henrik Johansson +1

Division of Decision and Control Systems, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, 10044 Stockholm, Sweden · Chair of Information-oriented Control (ITR), Technical University of Munich, Germany

Dynamic Regret Minimization Zeroth-Order Optimization Conditional Value-at-Risk

Ask PDF

Abstract

Risky Decision-Making

Explore similar work

Tail-Influence Sampling for CVaR Policy Evaluation

On the Generalization and Robustness in Conditional Value-at-Risk

Risk-averse learning with delayed feedback