Oracle Gaps in Reliability Coverage: Sampling Noise or Policy Specialization?

Mert Onur Cakiroglu; Mehmet Dalkilic; Hasan Kurban

cs.CVSep 26, 2026

Oracle Gaps in Reliability Coverage: Sampling Noise or Policy Specialization?

Authors: Mert Onur Cakiroglu, Mehmet Dalkilic, Hasan Kurban

Organizations: Luddy School of Informatics, Computing, and Engineering, Indiana University Bloomington, Bloomington, IN, USA · College of Science and Engineering, Hamad Bin Khalifa University, Doha, Qatar

Abstract

Policies trained from the same base model can appear to solve different problems. An oracle that chooses the best policy for each problem may therefore appear much stronger than any single policy. Selecting the largest estimated success rate also selects favorable sampling errors. We study this effect through reliability coverage, the fraction of problems whose success probability reaches a chosen threshold. Our first test redistributes stored correctness outcomes across policies within each problem. A second also preserves each policy's total successes, accounting for overall quality differences under a specified statistical model. For five training seeds of a seven-billion-parameter vision-language model, redistribution reproduces 0.096 of an estimated 0.113 oracle gap at threshold 0.10. Neither test finds significant evidence at this threshold. Small advantages remain unresolved. Mixtures, routers, voting, and weight averaging show no detectable improvement over their corresponding single-policy baselines. Training policies on different datasets shows little detectable specialization under light post-training, and no router gain. A stronger recipe does create it, both tests detect it, and a router gain appears only at the high thresholds where the specialists separate. In a control with predictable specialization, a router recovers about half the oracle gap. Coverage bounds explain why even a genuine oracle advantage need not yield a deployment gain. The tests assess apparent specialization from stored responses before investment in routing. Code: https://github.com/KurbanIntelligenceLab/oracle-gaps

Figures & tables

Figure 1: Constructing the re-deal null for one problem. (a) Each row contains

16

binary correctness outcomes from one of five policies. Averaging all

80

outcomes estimates the success probability of choosing a policy uniformly for each response. (b) A re-deal randomly redistributes these outcomes into five groups of

16

, preserving all

10

successes but removing the original policy assignments. (c) Distribution of the largest group success count over

2{,}000

re-deals. The permutation

p

-value is the fraction with a maximum at least as large as the observed four successes. The dashed line marks the reliability threshold

\tau=0.2

, equivalent to

3.2

successes out of

16

. An integer count must therefore reach four to clear it.

$\tau$	obs.	null	obs.	null	null $90\%$	$p$	$p_{\mathrm{m}}$	split-half	obs.	null	$p$	obs.	null	$p$
	$D(\tau)$		$L(\tau)$						Oracle $-$ mixture			Rise in $L$ , $M{=}2$ to $5$
$0.05$	$0.296$	$0.277$	$0.142$	$0.125$	$[0.108,0.137]$	$0.036$	$0.027$	$0.021$	$0.150$	$0.150$	$1.00$	$0.076$	$0.069$	$0.11$
$0.10$	$0.233$	$0.220$	$0.113$	$0.096$	$[0.079,0.113]$	$0.074$	$0.063$	$0.020$	$0.113$	$0.104$	$0.22$	$0.058$	$0.051$	$0.19$
$0.20$	$0.208$	$0.196$	$0.075$	$0.078$	$[0.062,0.096]$	$0.67$	$0.651$	$0.001$	$0.071$	$0.059$	$0.14$	$0.034$	$0.038$	$0.69$
$0.30$	$0.196$	$0.193$	$0.075$	$0.080$	$[0.062,0.096]$	$0.75$	$0.714$	$-0.002$	$0.104$	$0.104$	$0.60$	$0.037$	$0.040$	$0.67$

Table 1: Oracle-gap tests on

240

Geometry3K test prompts. Each policy has

16

responses per prompt. Estimates use raw frequencies. Obs. is the observed statistic and null the plain-null mean. Null

90\%

is the

5

th–

95

th percentile range of randomized statistics, not an interval for the population gap. The upper-tail fractions

p

and

p_{\mathrm{m}}

use the plain and margin-preserving nulls, respectively. Split-half selects both per-prompt members and the fixed comparator on eight responses and scores them on the other eight. Re-dealing preserves the uniform mixture’s raw rate.

$\sigma$	$L$	excess	power, re-deal	power, margin
$0.25$	$0.023$	$0.003$	$0.00$	$0.05$
$0.50$	$0.052$	$0.013$	$0.50$	$0.50$
$1.00$	$0.108$	$0.052$	$1.00$	$1.00$

Table 2: Power against diffuse rate differences at

k=16

. Independent Gaussian perturbations with standard deviation

\sigma

are added to every member’s log-odds of success, centered on the observed prompt mean. No sole specialist is designated. The table reports the population gap, excess over the plain-null mean, and rejection fractions at level

0.05

over

20

replicates.

Rule	System	Geometry3K $\mathrm{rAUC}$	$\mathrm{Pass@1}$	MathVista $\mathrm{rAUC}$	$\mathrm{Pass@1}$
single	base $B$	$0.579$	$0.354$	$0.498$	$0.610$
single	best member by test rAUC	$0.607$	$0.376$	$0.543$	$0.632$
oracle	best member per prompt	$0.702$	$0.451$	$0.612$	$0.698$
mixture	uniform mixture of the seeds	$0.592$	$0.368$	$0.534$	$0.628$
soup	mean of the five updates	$0.592$	$0.369$	$0.536$	$0.622$
routing	best family on validation	$0.603$	$0.374$	$0.533$	$0.631$

Table 3: Policy selection and mixing on the two test sets. Both evaluations use

k=16

responses per policy and prompt, and rates are posterior means (Appendix B ). Geometry3K has

240

test prompts and reliability band

[0.05,0.30]

. MathVista has

184

prompts and band

[0.50,0.90]

. The best member maximizes test rAUC. The plug-in oracle chooses a member separately for each prompt using estimated success rates. The mixture is scored by subsampling

16

responses from the pool. The soup is one policy formed by averaging the five parameter updates. The routing row uses the family selected on validation.

Figure 2: Cross-dataset pools under light and stronger training. (a,b) Two-member pools, one seed trained on each dataset, on the

424

union test prompts with

16

responses per member and prompt. Solid curves are the plug-in oracle gap, dashed and dotted curves the plain and margin-preserving null means, shaded over the central

90\%

of each null distribution. Diamonds mark

\tau=0.10

, where

p_{\mathrm{m}}

reports the margin-preserving test. (c) The stronger two-member pool at every reported threshold: oracle gap (purple bars) and router gains over the best member (blue points,

95\%

prompt-bootstrap intervals) for the dataset-identity router and the embedding nearest-neighbor (NN) router, with each dataset’s reliability band shaded and the grey region marking a router below the best member. All vertical values are percentage points.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

$\tau$	Family	gain	$95\%$ interval	gain (val.)	$e_{D}$	$L/D$
Geometry3K test split, $240$ prompts
$0.05$	linear	$-0.008$	$[-0.062,+0.008]$	$-0.008$	$0.507$	$0.479$
$0.05$	gradient-boosted	$0.004$	$[-0.046,+0.021]$	$0.004$	$0.465$	$0.479$
$0.05$	nearest-neighbor	$0.004$	$[-0.042,+0.025]$	$0.004$	$0.465$	$0.479$
$0.05$	embedding, linear	$-0.017$	$[-0.054,+0.021]$	$-0.017$	$0.535$	$0.479$
$0.05$	embedding, nearest-neighbor	$-0.025$	$[-0.058,+0.008]$	$-0.025$	$0.563$	$0.479$

Appendix

Table A1 : Prompt-based routing at each reliability threshold. Routers are fitted on validation. Embedding families use image and text representations from the base model. Gain compares coverage with the threshold-specific best member on the evaluated split. Gain (val.) uses the validation-selected member. Intervals are

95\%

prompt-bootstrap intervals. The disagreement error

e_{D}

counts selections below threshold on disagreement prompts, and

L/D

is its break-even value.

$\delta$	target	power, re-deal	power, margin	excess
$0.00$	$0$	$0.10$	$0.15$	$0.017$
$0.05$	$1$	$0.65$	$0.70$	$0.024$
$0.10$	$2$	$1.00$	$1.00$	$0.049$
$0.15$	$2$	$1.00$	$1.00$	$0.052$
$0.20$	$3$	$1.00$	$1.00$	$0.070$
$0.25$	$4$	$1.00$	$1.00$	$0.076$

Appendix

Table A2 : Sensitivity to concentrated specialization on Geometry3K test. The perturbation strength

\delta

sets the transfer count reported as target , the maximum integer count. Power is the reported rejection fraction across

20

trials at level

0.05

. Excess is the planted

L(0.10)

minus the plain-null mean. The

\delta=0

row repeats the tests on the unchanged observed pool. False-positive rates are assessed separately using null-generated datasets.

$f$	$s$	$L$	$\hat{L}$	excess	ratio
$0.1$	$0.05$	$0.003$	$0.088$	$0.000$	$0.01$
$0.1$	$0.10$	$0.021$	$0.091$	$0.003$	$0.14$
$0.1$	$0.20$	$0.023$	$0.098$	$0.004$	$0.19$
$0.3$	$0.05$	$0.012$	$0.099$	$0.006$	$0.53$
$0.3$	$0.10$	$0.070$	$0.113$	$0.013$	$0.19$
$0.3$	$0.20$	$0.076$	$0.135$	$0.024$	$0.32$

Appendix

Table A3 : Finite-sample excess versus the population oracle gap. Simulated prompt rates use the fitted Geometry3K Beta distribution. A fraction

f

of prompts receives a specialist perturbation of strength

s

, followed by

k=16

sampled responses per member. Entries average

30

replicates.

L

is the known population gap at

\tau=0.10

, and

\hat{L}

is the plug-in gap. Excess subtracts the plain-null mean from

\hat{L}

, and ratio is the excess divided by the population gap.

Link	gap	weak mean	$\tau{=}0.05$	$0.10$	$0.20$	$0.05$	$0.10$	$0.20$
			reject, margin-preserving			$L$
logit (the null’s model)	$0.13$	$0.24$	$0.03$	$0.13$	$0.03$	$0.140$	$0.103$	$0.079$
	$0.20$	$0.17$	$0.03$	$0.00$	$0.07$	$0.133$	$0.095$	$0.083$
probit	$0.13$	$0.24$	$0.00$	$0.07$	$0.10$	$0.135$	$0.098$	$0.083$
	$0.20$	$0.17$	$0.00$	$0.00$	$0.00$	$0.130$	$0.091$	$0.080$
additive probability	$0.13$	$0.24$	$0.00$	$0.00$	$0.33$	$0.124$	$0.096$	$0.077$

Appendix

Table A4 : Quality differences that are constant on different scales. Two groups of five members share Geometry3K prompt difficulties. The stronger group has mean rate

0.37

. The table gives the weaker group’s mean and the gap between them. Each row uses

k=16

30

replicates, and

400

permutations per test. Reject is the margin-preserving rejection fraction at nominal level

0.05

. For non-logit rows, rejection can reflect misspecification of the null’s link. The plain re-deal never rejects in these simulations.

L

is the mean plug-in oracle gap.

						nearest-neighbor		linear		size rule / specialist
pool	$L$	$D$	$p$	$p_{\mathrm{m}}$	$L/D$	gain	$e_{D}$	gain	$e_{D}$	gain	$e_{D}$
7B+3B, $\tau=0.05$	$0.158$	$0.442$	$0.223$	$<$ 0.001	$0.36$	$-0.029$	$0.42$	$-0.029$	$0.42$	$0.000$	$0.36$
7B+3B, $\tau=0.10$	$0.150$	$0.421$	$0.539$	$<$ 0.001	$0.36$	$0.000$	$0.36$	$-0.029$	$0.43$	$0.000$	$0.36$
7B+3B, $\tau=0.20$	$0.121$	$0.454$	$0.886$	$0.009$	$0.27$	$-0.033$	$0.34$	$-0.042$	$0.36$	$-0.029$	$0.33$
planted $1$ of $16$ , $\tau=0.05$	$0.150$	$0.317$	$0.002$	$<$ 0.001	$0.47$	$0.075$	$0.24$	$0.021$	$0.41$	$0.150$	$0.00$
planted $1$ of $16$ , $\tau=0.10$	$0.121$	$0.263$	$0.011$	$0.008$	$0.46$	$0.037$	$0.32$	$0.008$	$0.43$	$0.075$	$0.17$

Appendix

Table A5 : Routing on positive controls. The mixed-size pool contains five 7B and five 3B policies evaluated on shared Geometry3K prompts. The planted pools use five 7B policies with the specialist assigned by the quintile of the first principal component of the prompt embedding. All statistics shown are test estimates. Each router block gives coverage gain over the threshold-specific best member and disagreement error

e_{D}

. The break-even error is

L/D

. Routers are trained on validation with the same planting rule. The final block uses the validation-selected 7B member for the mixed-size pool and the known specialist assignment for planted pools.

Set	$n$	deg.	$\tau$	$L$	null	$p$	$p_{\downarrow}$	margin	$p_{\mathrm{m}}$	$p_{\mathrm{m}\downarrow}$	split-half
Geometry3K val	$240$	$39$	$0.05$	$0.088$	$0.103$	$0.959$	$0.083$	$0.101$	$0.954$	$0.104$	$-0.021$
	$240$	$39$	$0.10$	$0.067$	$0.105$	$1.000$	$0.001$	$0.103$	$0.999$	$0.005$	$-0.019$
	$240$	$39$	$0.20$	$0.108$	$0.100$	$0.303$	$0.814$	$0.097$	$0.230$	$0.855$	$-0.000$
	$240$	$39$	$0.30$	$0.096$	$0.090$	$0.374$	$0.755$	$0.088$	$0.312$	$0.802$	$-0.008$
Geometry3K test	$240$	$37$	$0.05$	$0.142$	$0.126$	$0.042$	$0.992$	$0.125$	$0.034$	$0.992$	$0.021$
	$240$	$37$	$0.10$	$0.113$	$0.096$	$0.062$	$0.975$	$0.095$	$0.068$	$0.969$	$0.020$

Appendix

Table A6 : Both nulls across evaluation sets and thresholds. Each set has

n

prompts. The deg. column counts prompts with all outcomes correct or all incorrect across the pool. These prompts contribute no permutation variance. The table reports observed gap

L

, each null mean, and upper- and lower-tail fractions. Upper tails count null statistics at least as large as observed, and lower tails count statistics at most as large. The split-half estimate selects both per-prompt members and the fixed comparator on

k/2

responses and scores them on the other half, averaging

200

splits. Each null uses

2{,}000

samples. This is an independent run from Table 1 , so permutation fractions differ slightly. The final block combines the two model sizes.

Dataset	Split	$b$	Pooled vote	Best member’s vote	Gain over single	Pooled $-$ member
Geometry3K	val	1	$0.553$	$0.566$	$-0.013$	$-0.013$
Geometry3K	val	3	$0.549$	$0.556$	$-0.017$	$-0.007$
Geometry3K	val	5	$0.557$	$0.553$	$-0.009$	$+0.004$
Geometry3K	val	7	$0.561$	$0.549$	$-0.005$	$+0.012$
Geometry3K	val	9	$0.560$	$0.546$	$-0.006$	$+0.014$
Geometry3K	test	1	$0.584$	$0.595$	$-0.011$	$-0.011$

Appendix

Table A10 : Voting within and across policies. Entries are rAUC on

[0.05,0.30]

for Geometry3K and

[0.50,0.90]

for MathVista. A pooled vote samples a policy uniformly for each of its

b

responses. The last two columns compare pooled voting with the best member using one response and that same member using

b

responses, respectively.

Candidate	Items	Mean rate	0/16 correct	16/16 correct	Mixed outcomes	Band share
MathVerse	$240$	$0.271$	$0.446$	$0.067$	$0.487$	$0.496$
MathVista	$137$	$0.638$	$0.117$	$0.255$	$\mathbf{0.628}$	$0.174$

Appendix

Table A11 : Dataset selection using base-model validation responses. Each prompt has

16

sampled responses. The two extreme-count columns give fractions of all prompts. Mixed outcomes is the fraction with both correct and incorrect responses. Band share is the fraction within that mixed-outcome group whose estimated success rate lies in

[0.05,0.30]

System or quantity	Geometry3K	MathVista	Geometry3K	Geometry3K	3B backbone	3B backbone
	test, $k{=}64$	test, $k{=}64$	$361$ new, test	$601$ pooled, test	validation	test
	$240$ prompts	$184$ prompts	$361$ prompts	$601$ prompts	$240$ prompts	$240$ prompts
base $B$	$0.562$	$0.509$	$0.531$	$0.550$	$0.382$	$0.371$
best member by rAUC	$0.598$	$0.549$	$0.550$	$0.567$	$0.446$	$0.482$
oracle, best member per prompt	$0.644$	$0.581$	$0.666$	$0.681$	$0.576$	$0.601$
mixture, uniform over the seeds	$0.584$	$0.539$	$0.546$	$0.564$	$0.436$	$0.459$

Appendix

Table A12 : Replication results for mixtures and routers. Entries are rAUC on each dataset’s selected band, with

k=16

unless stated otherwise. Pools contain the trained policies. The base is reported separately. The mixture uses the same response budget as a member. The best member maximizes rAUC on the evaluated split. Dashes omit routing results on 3B validation, which is used for router fitting. The final two rows summarize threshold-specific routing intervals and the member-spread diagnostic.

Explore similar work

Opportunity Is Not Realizability: Selection-Valid Diagnostics for Multi-LLM Routing
Aug 8, 2026Ibne Farabi Shihab, Abu Sa-Adat Mohamed Moon-Im Al Ahsan, Md Najmus SwaqeebModel Selection LLM Evaluation
Department of Computer Science, Iowa State University · Department of Computer Science & Engineering, BRAC University
What Does a Routing Oracle Measure Under Stochastic Decoding? Coupling, Scorer Choice, and Single-Commit Ceilings
Jul 3, 2026Teng-Ruei ChenLLM Evaluation Language Model Decoding
Institute of Bioinformatics and Systems Biology, National Yang Ming Chiao Tung University, Hsinchu 300, Taiwan · Krixvon AI, Taiwan
Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity
May 1, 2026Anamika Lochab, Bolian Li, Ruqi ZhangRL for Language Model Reasoning Group Relative Policy Optimization
Department of Computer Science Purdue University, West Lafayette

Aug 8, 2026J/K move · Enter open · S save

Opportunity Is Not Realizability: Selection-Valid Diagnostics for Multi-LLM Routing

Ibne Farabi Shihab, Abu Sa-Adat Mohamed Moon-Im Al Ahsan, Md Najmus Swaqeeb

Department of Computer Science, Iowa State University · Department of Computer Science & Engineering, BRAC University

Model Selection LLM Evaluation LLM Routing Language Model Ensembles

Open paper AskPDF arXiv

Oracle routing measures how much a pool of language models could gain from per-query selection, but the diagnostic has two flaws: testing against a best fixed model selected on the same examples invalidates paired inference, and a full-information oracle sees outcomes no deployable router observes. We separate three estimands (outcome-oracle opportunity, the Bayes-optimal gain from a declared pre-answer signal, and the held-out gain of a learned router) and prove selection-valid confidence intervals that survive choosing the best fixed model or the best member of a router family, a signal-information sandwich, and a

(1-1/e)

greedy guarantee for building compact pools from submodular complementary coverage. On eight checkpoints from six families over four benchmarks, selection-valid intervals certify a population oracle gap of

9.7

30.7

points on every task, yet the strongest deployable prompt router recovers only

7.5

14.4\%

of it, and the simultaneous interval for the best of eleven tested policies has lower limit zero throughout. The realizable share of oracle opportunity is small and certifiable: strong routers beat the best fixed model, and most of the gap remains.

Abstract

Explore similar work

Opportunity Is Not Realizability: Selection-Valid Diagnostics for Multi-LLM Routing

What Does a Routing Oracle Measure Under Stochastic Decoding? Coupling, Scorer Choice, and Single-Commit Ceilings

Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity