stat.MLAug 29, 2026

Deciding When to Decide: Testing Operational Suboptimality Under Distributional Shift

Authors: Minxing Zheng, Holly Wiberg, Shixiang Zhu

Organizations: Carnegie Mellon University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deployed decisions are often optimized once and retained because updates impose operational, regulatory, or switching costs. As operating conditions change, when should such decisions be re-optimized? We study this question for stochastic optimization when the objective's functional form is known but the decision maker's trade-offs are encoded by an unknown preference parameter. Standard distribution-shift tests are poorly aligned with this goal: they can flag detectable yet decision-irrelevant changes without determining whether the incumbent decision has become materially suboptimal. We propose RADAR (Regret-based Assessment of Decision Adequacy and Risk), a decision-focused framework that uses inverse optimization to infer latent preferences and tests the deployed decision's optimality gap under the current distribution. By targeting regret, RADAR ignores decision-irrelevant shifts while detecting changes that warrant re-optimization. We develop two-sample and sequential changepoint procedures and establish asymptotic guarantees for Type-I error and power. Across synthetic optimization problems, a semi-synthetic capacity allocation task, and police-zone planning, RADAR more reliably distinguishes harmful from harmless shifts than decision-agnostic alternatives.

Figures & tables

Figure 1: Stylized two-zone redistricting, where dots denote demand locations and boundaries are chosen to balance demand. Left: The baseline-optimal boundary. Middle: Uniform demand growth shifts the distribution but leaves the boundary optimal. Right: Localized demand relocation makes the old boundary (red dashed) suboptimal and favors an L-shaped boundary (black solid).
Figure 2: Overview of the proposed RADAR framework.
Figure 3: Empirical rejection rates on synthetic LP and QP examples across different decision-irrelevant ( H0H_{0} ) and decision-relevant ( H1H_{1} ) shift settings. Each panel reports the rejection rate at level α=0.05\alpha=0.05 as shift magnitude δ\delta varies. RADAR stays near the nominal level under H0H_{0} , demonstrating Type-I error control, and gains competitive power under H1H_{1} . In contrast, baselines often reject in both regimes, leading to inflated Type-I error under H0H_{0} .
Figure 4: Fashion-MNIST capacity-allocation regimes. Each panel shows four representative images of the regime, and the induced coarse-group shares bar. The stacked bars report, from left to right, the proportions of clothing, footwear, and accessories. (a) Baseline, share dominated by clothing. (b) A decision-irrelevant H0H_{0} shift replaces categories within the clothing group, producing visually different images while preserving the coarse-group shares. (c) A decision-relevant H1H_{1} shift changes the group shares, warranting re-optimization. Panels (a) and (b) are therefore statistically distinguishable but operationally equivalent.
ShiftLevel δ\deltaRADARX-MeanX-DistrRisk-Value
H0H_{0}0.00.00.02±0.040.02\pm 0.040.06±0.060.06\pm 0.060.02±0.040.02\pm 0.040.06±0.060.06\pm 0.06
H0H_{0}0.30.30.04±0.050.04\pm 0.050.44±0.110.44\pm 0.110.90±0.070.90\pm 0.070.04±0.050.04\pm 0.05
H0H_{0}0.60.60.04±0.050.04\pm 0.050.90±0.070.90\pm 0.071.00±0.031.00\pm 0.030.12±0.080.12\pm 0.08
H0H_{0}0.90.90.06±0.060.06\pm 0.061.00±0.031.00\pm 0.031.00±0.031.00\pm 0.030.06±0.060.06\pm 0.06
H1H_{1}0.00.00.00±0.030.00\pm 0.030.04±0.050.04\pm 0.050.02±0.040.02\pm 0.040.04±0.050.04\pm 0.05
H1H_{1}0.30.30.14±0.080.14\pm 0.080.04±0.050.04\pm 0.050.06±0.060.06\pm 0.060.02±0.040.02\pm 0.04
Table 1: Representative Fashion-MNIST results for the image-only H0H_{0} shift and the orthogonal harmful H1H_{1} shift. Entries report the empirical rejection rate ±\pm the half-width of the 90% Wilson confidence interval across 5050 independent repetitions at level α=0.05\alpha=0.05 .
Figure 5: Analysis of the Atlanta police districting task. (a)–(f) Spatial distributions of annual mean response times from 2014 to 2019; darker colors indicate longer response times, and black outlines delineate patrol zones. (g) Decision-focused sequential monitoring of the incumbent districting plan, with the scan statistic peaking around June 2016.
Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Decision-relevant sequential monitoring with two changes: the shift at γ1=1000\gamma_{1}=1000 is decision-irrelevant, whereas the shift at γ2=2000\gamma_{2}=2000 is decision-relevant. Every statistic at monitoring time tt uses the w=200w=200 observations ending at tt , so neither method can respond before t=γ+wt=\gamma+w . (a) Standard CPD flags both changes and then forgets each one, returning to its nominal level; RADAR ignores γ1\gamma_{1} and, once the incumbent becomes inadequate at γ2\gamma_{2} , keeps rejecting for the remainder of the record. (b) The alarm statistic TtT_{t} of ( 21 ) against the threshold z1−αz_{1-\alpha} . (c) The estimated and oracle window optimality gaps against the tolerance τ=0\tau=0 . Shading denotes pointwise 95%95\% intervals in (a) and (c) and the interquartile range across replications in (b) .
Shift familyδ\delta[Δ‾inv,Δ‾inv][\underline{\Delta}_{\rm inv},\overline{\Delta}_{\rm inv}]Decision
Decision-irrelevant ( H0H_{0} )all[0.000,0.000][0.000,0.000]Adequate
Incumbent worsens ( H1H_{1} )0.2500.250[0.000,0.088][0.000,0.088]Indeterminate
Incumbent worsens ( H1H_{1} )0.9500.950[0.084,0.335][0.084,0.335]Re-optimize
Rival improves ( H1H_{1} )0.2900.290[0.000,0.099][0.000,0.099]Indeterminate
Rival improves ( H1H_{1} )0.7620.762[0.127,0.267][0.127,0.267]Re-optimize
Appendix
Table 2: Population identified ranges [Δ‾inv,Δ‾inv][\underline{\Delta}_{\rm inv},\overline{\Delta}_{\rm inv}] over Θinv\Theta_{\rm inv} , with τ=0.005\tau=0.005 . The incumbent is adequate if Δ‾inv≤τ\overline{\Delta}_{\rm inv}\leq\tau , re-optimization is warranted if Δ‾inv>τ\underline{\Delta}_{\rm inv}>\tau , and the conclusion is indeterminate if Δ‾inv≤τ<Δ‾inv\underline{\Delta}_{\rm inv}\leq\tau<\overline{\Delta}_{\rm inv} .
Shift typeFormulationDescription
Global scalingα′=(1+δ)α\boldsymbol{\alpha}^{\prime}=(1+\delta)\boldsymbol{\alpha}Scale all rows uniformly
Row scalingαi,:′=(1+δ)αi,:\boldsymbol{\alpha}^{\prime}_{i,:}=(1+\delta)\boldsymbol{\alpha}_{i,:}Scale one selected row
Orthogonal shiftαi,:′=αi,:+δv, ⟨v,θ⟩=0\boldsymbol{\alpha}^{\prime}_{i,:}=\boldsymbol{\alpha}_{i,:}+\delta v,\ \langle v,\theta\rangle=0Perturb along a direction orthogonal to θ\theta
Localized meanαi,j′=αi,j+δ\boldsymbol{\alpha}^{\prime}_{i,j}=\boldsymbol{\alpha}_{i,j}+\deltaIncrease one coordinate in a selected row
Row redistributionαi,j1′=αi,j1+δ, αi,j2′=αi,j2−δ\boldsymbol{\alpha}^{\prime}_{i,j_{1}}=\boldsymbol{\alpha}_{i,j_{1}}+\delta,\ \boldsymbol{\alpha}^{\prime}_{i,j_{2}}=\boldsymbol{\alpha}_{i,j_{2}}-\deltaMove mass within a selected row
Tail-sensitiveselective perturbation on tail rowsModify rows in the worst α\alpha -tail under CVaR
Appendix
Table 3: Structured distributional shifts considered in the synthetic experiments.
ShiftMagnitude δ\deltaRADARX-MeanX-DistrRisk-Value
H0H_{0}0.00.00.02±0.040.02\pm 0.040.06±0.060.06\pm 0.060.02±0.040.02\pm 0.040.06±0.060.06\pm 0.06
H0H_{0}0.10.10.02±0.040.02\pm 0.040.06±0.060.06\pm 0.060.14±0.080.14\pm 0.080.02±0.040.02\pm 0.04
H0H_{0}0.20.20.10±0.070.10\pm 0.070.16±0.080.16\pm 0.080.36±0.110.36\pm 0.110.06±0.060.06\pm 0.06
H0H_{0}0.30.30.04±0.050.04\pm 0.050.44±0.110.44\pm 0.110.90±0.070.90\pm 0.070.04±0.050.04\pm 0.05
H0H_{0}0.40.40.06±0.060.06\pm 0.060.64±0.110.64\pm 0.110.96±0.050.96\pm 0.050.04±0.050.04\pm 0.05
H0H_{0}0.50.50.04±0.050.04\pm 0.050.68±0.110.68\pm 0.111.00±0.031.00\pm 0.030.02±0.040.02\pm 0.04
Appendix
Table 4: Full Fashion-MNIST results for the image-only H0H_{0} path and two harmful H1H_{1} constructions: the balanced group-share shift and the orthogonal objective-blind shift. Entries report the empirical rejection rate ±\pm the half-width of the 90% Wilson confidence interval across 5050 independent repetitions at level α=0.05\alpha=0.05 .
More in this topic

Distribution Shift Detection

Distribution shift detection identifies when data or model behavior changes between reference and deployment settings.

See all 62 papers

Explore similar work

CardsList

  1. Monitoring for Shifts at Unknown Times: Parameter-Free Conditional Conformal Martingales

    Oct 12, 2026Tuo Liu, Drew Prinster, Francesco OrabonaDistribution Shift DetectionChange-Point Detection

    King Abdullah University of Science and Technology, Saudi Arabia · Johns Hopkins University, USA

  2. Change Detection in Probability Flow ODE: Online Testing in Diffusion Latent Spaces

    Aug 24, 2026Artem Kraevskiy, Artem ProkhorovDistribution Shift DetectionChange-Point Detection

    National Research University Higher School of Economics Moscow, Russia · The University of Sydney Business School & CIREQ & CEBDA Sydney, Australia

  3. Contextual Preference Distribution Learning

    Mar 17, 2026Benjamin Hudson, Laurent Charlin, Emma FrejingerPreference LearningDecision-Making under Uncertainty

    Mila – Quebec Artificial Intelligence Institute, Montreal, Quebec, Canada · Département d’informatique et de recherche opérationnelle (DIRO), Université de Montréal, Montreal, Quebec, Canada · HEC Montréal, Montreal, Quebec, Canada