Post-Selection Distributional Model Evaluation

Amirmohammad Farzaneh; Osvaldo Simeone

stat.MLMar 24, 2026

Post-Selection Distributional Model Evaluation

Authors: Amirmohammad Farzaneh, Osvaldo Simeone

Organizations: Institute for Intelligent Networked Systems (INSI), Northeastern University London, London, UK

Abstract

Formal model evaluation methods typically certify that a model satisfies a prescribed target key performance indicator (KPI) level. However, in many applications, the relevant target KPI level may not be known a priori, and the user may instead wish to compare candidate models by analyzing the full trade-offs between performance and reliability achievable at test time by the models. This task, requiring the reliable estimate of the test-time KPI distributions, is made more complicated by the fact that the same data must often be used both to pre-select a subset of candidate models and to estimate their KPI distributions, causing a potential post-selection bias. In this work, we introduce post-selection distributional model evaluation (PS-DME), a general framework for statistically valid distributional model assessment after arbitrary data-dependent model pre-selection. Building on e-values, PS-DME controls post-selection false coverage rate (FCR) for the distributional KPI estimates and we establish explicit conditions under which it is provably more sample efficient than a baseline method based on sample splitting. Experiments on synthetic data, text-to-SQL decoding with large language models, and telecom network performance evaluation demonstrate that PS-DME enables reliable comparison of candidate configurations across a range of reliability levels, supporting the statistically reliable exploration of performance--reliability trade-offs.

Figures & tables

Figure 1: (a) CDF of a negatively-oriented key performance indicator (KPI), such as the prefill latency for an LLM agent to be deployed on a device ( Huang et al., 2026 ) , for two model configurations. For a test-time reliability level

1-\gamma

, knowing the CDF for each configuration allows the user to compare achievable performance across different configurations. Furthermore, it also supports the exploration of different reliability levels

1-\gamma

. (b) Two strategies for handling model pre-selection, namely data splitting and in-sample inference. (c)-(d) Illustration of the proposed post-selection distributional model evaluation (PS-DME), which applies in-sample pre-selection: (c) A user-defined selection strategy identifies a subset

\mathcal{K}

of promising configurations among the

K

candidates; (d) For all the configurations in set

\mathcal{K}

, PS-DME constructs confidence bands for the CDFs of the KPI of interest with guaranteed false coverage rate (FCR), enabling reliable comparison of performance–reliability trade-offs across configurations.

Figure 2: For a candidate hyperparameter

\lambda_{k}=0.0231

, we show the true CDF

F_{k}(x)

, the empirical CDF

\widehat{F}_{k}(x)

, and the post-selection confidence band

[L_{k}(x),U_{k}(x)]

for the data splitting, naive in-sample, and post-hoc in-sample benchmarks.

Panel of Figure 3 — Figure 3: Best guaranteed KPI for the synthetic data experiment as a function of the target probability $1-\gamma$ for SS-DME and PS-DME. For SS-DME, a fraction $n^{\text{sel}}/n\in\{0.5,0.6,0.7\}$ of the available data is used for model pre-selection, while the remaining samples are used to construct the CDF confidence bands. (a) Results with 20 calibration samples. (b) Results with 100 calibration samples. The vertical error bars show the standard error of the mean across 100 random splits of the dataset.

Figure 4: Best guaranteed KPI (loss) for the Spider text-to-SQL experiment as a function of the target probability

1-\gamma

for SS-DME and PS-DME. For SS-DME, a fraction

n^{\text{sel}}/n\in\{0.6,0.7,0.8\}

of the available data is used for decoding strategy pre-selection, while the remaining samples are used to construct the CDF confidence bands. Results are shown for

n=50

calibration samples. The vertical error bars show the standard error of the mean across 100 random splits of the dataset.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Illustration of the miscoverage event $\mathcal{O}_{k}$ defined in ( 5 ). (a) A scenario in which the miscoverage event $\mathcal{O}_{k}$ in ( 5 ) does not occur, as the true CDF lies entirely within the reported confidence band. (b) A scenario in which the miscoverage event $\mathcal{O}_{k}$ in ( 5 ) occurs, since the true CDF exits the band over the region of input $x$ marked in red.

Figure 6: Comparison of the band widths produced by SS-DME and by PS-DME with the e-calibrator family ( 12 ). The parameter

\delta|\mathcal{K}|/K

is fixed to 0.01.

Panel of Figure 7 — Figure 7: Comparison between the DKW-based and Berk–Jones-based implementations of PS-DME. (a) CDF confidence bands for a representative hyperparameter; (b) Best guaranteed KPI as a function of the outage probability $\gamma$ .

$n$	Method	Empirical FCR
20	SS-DME	0.0401
20	PS-DME	0.0116
100	SS-DME	0.0923
100	PS-DME	0

Appendix

Table 1: Empirical FCR for the synthetic experiment at

\delta=0.1

, averaged over 100 replications.

Figure 8: Representative post-selection CDF bands for the O-RAN buffer occupancy experiment for a selected configuration for PS-DME, and SS-DME with dataset split ratio

|\mathcal{D}^{\text{sel}}|/|\mathcal{D}|=0.5

Figure 9: Best guaranteed buffer occupancy for the O-RAN experiment as a function of the target probability

1-\gamma

for PS-DME and SS-DME with dataset split ratio

n^{\text{sel}}/n\in\{0.2,0.3,0.4\}

. The vertical error bars show the standard error of the mean across 100 random splits of the dataset.

Panel of Figure 10 — Figure 10: Effect of the SS-DME split ratio $n^{\text{sel}}/n$ on the best guaranteed buffer occupancy in the O-RAN experiment, for two target probabilities $1-\gamma\in\{0.7,0.8\}$ . Each panel corresponds to a different calibration regime, with (a) $30\%$ and (b) $100\%$ of the available data used for calibration. The vertical error bars show the standard error of the mean across 1,000 random splits of the calibration dataset into pre-selection and evaluation datasets.

Figure 11: Representative post-selection CDF bands for a selected server configuration using three different e-calibrators. The empirical CDF is common across all curves, while the shaded regions correspond to different values of parameter

\tau

in ( 12 ).

Benchmark	Setting	Empirical FCR
Spider (text-to-SQL)	$M=5$ selected	$0.031\pm 0.007$
	$M=10$ selected	$0.046\pm 0.007$
	$M=20$ selected	$0.070\pm 0.008$
O-RAN (telecom)	$M=6$ selected	$0.007\pm 0.001$

Appendix

Table 2: Empirical FCR of PS-DME on the Spider text-to-SQL and O-RAN telecom benchmarks at target

\delta=0.1

. Mean

\pm

one standard error of the mean across random splits.

$M$	$M/K$	$\varepsilon_{\mathrm{PS}}$	$\varepsilon_{\mathrm{SS}}$	$X^{\star}_{\mathrm{PS}}$	$X^{\star}_{\mathrm{SS}}$
2	0.06	0.149	0.179	0.793	0.814
5	0.16	0.139	0.168	0.791	0.807
10	0.32	0.132	0.158	0.790	0.804
15	0.48	0.127	0.153	0.789	0.800
20	0.65	0.124	0.148	0.789	0.799
25	0.81	0.121	0.145	0.789	0.796

Appendix

Table 3: Band width

\varepsilon

and best guaranteed KPI

X^{\star}

1-\gamma=0.75

for PS-DME and SS-DME on synthetic data, as a function of the selected-set size

M

(with

K=31

). Lower is better for both

\varepsilon

and

X^{\star}

Target $M$	Mean $\|\mathcal{K}\|$	Empirical FCR	$\varepsilon_{\mathrm{PS}}$
5	$5.0\pm 4.1$	$0.004\pm 0.003$	0.203
10	$9.2\pm 4.1$	$0.018\pm 0.006$	0.191
20	$21.3\pm 6.1$	$0.078\pm 0.012$	0.175

Appendix

Table 4: PS-DME under data-dependent selection with target size

M

. Mean realized

|\mathcal{K}|

, empirical FCR, and PS-DME band width

\varepsilon_{\mathrm{PS}}

across 500 repetitions.

$M$	$\tau^{\star}$	$\varepsilon(\tau^{\star})$	$\varepsilon(\tau=1/2)$	$X^{\star}(\tau^{\star})$	$X^{\star}(\tau=1/2)$
5	0.859	0.139	0.161	0.791	0.808
10	0.841	0.132	0.150	0.790	0.793
20	0.816	0.124	0.138	0.789	0.791

Appendix

Table 5: Band width

\varepsilon

and best guaranteed KPI

X^{\star}

1-\gamma=0.75

for the power e-calibrator with

\tau=\tau^{\star}

from Corollary 2 and with

\tau=1/2

(square-root e-calibrator).

$n$	$\varepsilon_{\mathrm{PS}}$	$\varepsilon_{\mathrm{Bonf}}$	$\varepsilon_{\mathrm{DKW}1}$	$\mathrm{FCR}_{\mathrm{PS}}$	$\mathrm{FCR}_{\mathrm{DKW}1}$
20	0.401	0.498	0.274	0.005	0.102
100	0.179	0.223	0.122	0.000	0.080
200	0.127	0.157	0.087	0.000	0.106
500	0.080	0.100	0.055	0.000	0.078

Appendix

Table 6: Band widths and empirical FCRs for PS-DME, Bonferroni, and single-DKW baselines on synthetic data with

K=2000

M=1000

\delta=0.1

$M$	$\mathrm{FCR}_{\mathrm{DKW}}$	$\mathrm{FCR}_{\mathrm{BJ}}$	$X^{\star}_{\mathrm{DKW}}$ (p=0.9)	$X^{\star}_{\mathrm{BJ}}$ (p=0.9)
5	$0.038\pm 0.008$	$0.043\pm 0.008$	0.66	0.74
10	$0.039\pm 0.007$	$0.055\pm 0.008$	0.69	0.75
20	$0.080\pm 0.009$	$0.096\pm 0.009$	0.72	0.75

Appendix

Table 7: Empirical FCR and best guaranteed KPI at

1-\gamma=0.9

for the DKW and BJ implementations of PS-DME on Spider text-to-SQL.

Method	Empirical FCR	$X^{\star}$ (p=0.90)	$X^{\star}$ (p=0.99)
PS-DME	$0.087\pm 0.011$	0.751	0.772
N-PS-DME	$0.372\pm 0.017$	0.762	0.781

Appendix

Table 8: Empirical FCR and best guaranteed KPI for PS-DME and the naive in-sample baseline (N-PS-DME) on the BIRD text-to-SQL benchmark.

Method	Empirical FCR	$\varepsilon$	$X^{\star}$ (p=0.75)
PS-DME	$0.003$	0.183	4.92 / 10
SS-DME	$-$	0.220	0.93 / 10

Appendix

Table 9: PS-DME and SS-DME under an LLM-as-judge KPI on Spider. SS-DME uses

n^{\mathrm{eval}}=31

samples (set to

70\%

of the Corollary 2 crossover threshold).

May 8, 2026

Query-efficient model evaluation using cached responses

Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment. For modern evaluation frameworks, generating and evaluating a response for all queries can be prohibitively expensive. In practice, responses from previously-evaluated models are often cached -- creating a potential opportunity to use this additional information to decrease the number of queries required to accurately evaluate a new model. In this paper, we introduce an approach for predicting benchmark performance that leverages cached model responses based on the Data Kernel Perspective Space (DKPS), a method for quantifying the relationship between models in the black-box setting. Theoretically, we show that DKPS-based methods are query-efficient under certain conditions. Empirically, we demonstrate that DKPS-based methods achieve the same mean absolute error as baselines with a substantially decreased query budget. We conclude by proposing an offline method for selecting a set of queries that maximizes the goodness-of-fit on reference models, improving prediction accuracy over random query selection.

Hayden Helm, Ben Johnson, Carey Priebe

Helivan · Jataware · Johns Hopkins University

Kernel Methods

Ask PDF

Jun 12, 2026

Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias

Understanding how a prediction model will perform in a new environment before deployment is essential to preventing harm when algorithms inform decision-making. Two common sources of model performance degradation are (i) covariate shift, where the target covariate distribution differs from the source, and (ii) selective labels, where the observability of outcomes depends on historical decisions. We study pre-deployment model evaluation under the joint presence of covariate shift and labeling of outcomes selectively based on observed features. In particular, we present a double machine learning procedure for estimating the target risk of an arbitrary black-box prediction model under a general loss function. We show identification of this estimand under standard assumptions and derive a bias-corrected estimator based on the influence function of the target risk. Finally, we evaluate our estimator through experiments using the eICU electronic health records database, showing that it tracks the true target risk more accurately than methods that address either selective labels or covariate shift alone, as well as baselines that combine standard plug-in approaches.

Annie Ulichney, Amanda Coston

Department of Statistics, University of California, Berkeley

Double ML Distribution Shift Robustness Distribution Shift Covariate Shift Selection Bias Doubly Robust Estimation+1

Ask PDF

Jul 5, 2026

evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations

The dominant practice in language model evaluation is to report a single accuracy number per model and declare the higher one better, without testing whether the gap could plausibly be sampling noise. On benchmarks of a few thousand items, and under temperature sampling where a model can differ from itself run to run by more than the reported gap between models, this practice routinely overstates confidence in headline claims. The statistical machinery to fix this -- confidence intervals, paired significance tests, power analysis, clustered standard errors, multiple-comparison correction -- is well established, but no standard, pip-installable tool packages it in the shape an evaluation actually takes: a per-item results table. We present evalci, a pure-Python library (numpy/scipy/pandas only) that turns a per-item results table into a publication-ready claim -- e.g., "Model A beats Model B,

Δ=3.1

pts, 95% CI [1.2, 5.0], paired permutation

p=0.002

n=1{,}319

" -- in one function call, with adapters for lm-evaluation-harness and HELM output. Every routine is validated against an independent reference (statsmodels, or brute-force exact enumeration) rather than only against itself. As a case study, we re-analyze a public comparison of nine language models' MMLU accuracy and find that 3 of the 8 adjacent leaderboard-rank gaps are not statistically significant after correcting for the 36 pairwise comparisons the ranking implies. evalci is available at https://pypi.org/project/evalci/ (source: https://github.com/Shreyaskc/evalci, DOI: https://doi.org/10.5281/zenodo.21201815)

Shreyas K Chandrahas

LLM Evaluation Pairwise Comparison Statistical Power Analysis Statistical Hypothesis Testing

Ask PDF

Abstract

Explore similar work

Query-efficient model evaluation using cached responses

Beyond the Training Distribution: Evaluating Predictions Under Distribution Shift and Selection Bias

evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations