Safe Meta-Policy Design with Risk Control

Wenbin Zhou; Michael Lingzhi Li; Shixiang Zhu

stat.MLOct 7, 2026

Safe Meta-Policy Design with Risk Control

Authors: Wenbin Zhou, Michael Lingzhi Li, Shixiang Zhu

Organizations: Heinz College of Information Systems and Public Policy, Carnegie Mellon University · Technology and Operations Management, Harvard Business School

Abstract

Models can be retrained as new data arrive, but deploying every new version risks replacing a good policy with a worse one. We study how to plan policy updates (i.e., meta-policy) before future candidates are trained, balancing the benefits of improvement against the risk of performance regression. Our offline meta-policy maximizes expected cumulative value subject to a budget on the expected number of updates that perform worse than the policies they replace. We estimate the value and risk of possible switches from historical learning trajectories, represent an update schedule as a path in a directed acyclic graph, and select a schedule using dynamic programming. A leading-order analysis identifies the signal-to-noise ratio of policy improvement as a key driver of update frequency, waiting times, and risk allocation: clearer improvements support earlier, more frequent updates, while noisier improvements call for longer waits or greater risk expenditure. Their asymptotic rates also reveal a diminishing marginal cost of achieving greater safety over time. Experiments on synthetic and clinical trial data illustrate the performance--risk tradeoff and compare our method with alternative baselines.

Figures & tables

Figure 1: The core tradeoff in deciding how often to update a policy. Updating slowly (left) leaves an outdated policy in place and loses value, while updating quickly (right) tracks improvements closely but risks deploying a policy worse than the one it replaces (a safety violation).

Figure 2: A deployment schedule and its path representation for

T=4

. Left : The meta-policy updates at times

2

and

4

, retaining

\pi_{2}

at time

3

. Right : The corresponding path

1\to 2\to 4\to 5

deploys

\pi_{1},\pi_{2},\pi_{4}

, with node

5=T+1

serving as the terminal sink.

Figure 3: Phase diagrams of the schedules returned by Algorithm 1 on homogeneous value processes with constant drift $\mu$ and volatility $\sigma$ ( $T=512$ , $\epsilon=0.35$ , $\Delta=1\times 10^{-3}$ ). From left to right, the panels report the number of updates $K$ , the mean per-update risk $\bar{p}$ , the mean normalized pre-update interval preceding an update $\bar{h}$ , and the expected deployed value $J$ . Each cell is evaluated under the population rewards and safety costs and averaged over three simulated datasets. White-labeled contours show the leading-order theoretical predictions established in Section 4 .

Figure 4: Safety–value Pareto frontiers of the proposed meta-policy (Ours) and three baseline update rules, in three synthetic settings with low, medium, and high value-process SNR and on the International Stroke Trial (IST) data. The horizontal axis is the safety budget

\epsilon

(log scale). The vertical axis is the objective of ( 1 ), normalized so that

0

and

1

correspond to never updating and to updating every period. All methods use the same estimated rewards and safety costs.

Figure 5: Update times in the optimal schedule selected by the proposed meta-policy as the safety budget

\epsilon

varies, in the same four settings as Figure 4 . Each black dot indicates a scheduled normalized update time. The left dashed lines mark the smallest budget that admits one update.

Explore similar work

Conformal Policy Control
Mar 2, 2026Drew Prinster, Clara Fannjiang, Ji Won Park +4Conformal Risk Control Exploration
Prescient Design, Genentech, U.S.A. · Johns Hopkins University, Baltimore, U.S.A. · New York University, NYC, U.S.A. +1
Risk-Controlled Post-Processing of Decision Policies
May 7, 2026Sunay Joshi, Tao Wang, Hamed Hassani +1Conformal Risk Control Post-Processing
University of Pennsylvania
Safe Meta-Reinforcement Learning via Information Space Reachability
Sep 14, 2026Zeyang Li, Sunbochen Tang, Navid AzizanSafety Constraints Offline Reinforcement Learning
Laboratory for Information and Decision Systems (LIDS), Massachusetts Institute of Technology, Cambridge, MA 02139, USA

Mar 2, 2026, cs.AIJ/K move · Enter open · S save

Conformal Policy Control

Drew Prinster, Clara Fannjiang, Ji Won Park, Kyunghyun Cho+3

Prescient Design, Genentech, U.S.A. · Johns Hopkins University, Baltimore, U.S.A. · New York University, NYC, U.S.A.+1

Conformal Risk Control Exploration

Open paper AskPDF arXiv

An agent must try new behaviors to explore and improve. In high-stakes environments, an agent that violates safety constraints may cause harm and must be taken offline, curtailing any future interaction. Imitating old behavior is safe, but excessive conservatism discourages exploration. How much behavior change is too much? We show how to use any safe reference policy as a probabilistic regulator for any optimized but untested policy. Conformal calibration on data from the safe policy determines how aggressively the new policy can act, while provably enforcing the user's declared risk tolerance. Unlike conservative optimization methods, we do not assume the user has identified the correct model class nor tuned any hyperparameters. Unlike previous conformal methods, our theory provides finite-sample guarantees even for non-monotonic bounded loss functions, and it introduces a new policy control setting. Our experiments on applications ranging from natural language question answering to biomolecular engineering show that safe exploration is not only possible from the first moment of deployment, but can also improve performance.

Abstract

Explore similar work

Conformal Policy Control

Risk-Controlled Post-Processing of Decision Policies

Safe Meta-Reinforcement Learning via Information Space Reachability