POOL: Propagated Uncertainty Over Lookalikes

Rounak Sharma; Ananya B. Sai; Soumyabrata Pal

cs.AIAug 24, 2026

POOL: Propagated Uncertainty Over Lookalikes

Authors: Rounak Sharma, Ananya B. Sai, Soumyabrata Pal

Organizations: Adobe Research, India

Abstract

Black-box large language models need confidence scores that can separate likely-correct from likely-incorrect outputs, enabling systems to prioritize human review, route uncertain cases to stronger models, or choose abstention thresholds on development data. Yet existing confidence estimators face a cost-quality trade-off: verbal confidence is cheap but is often overconfident, while sampling-based uncertainty is more informative but scales linearly with the number of samples per query. We propose POOL (Propagated Uncertainty Over Lookalikes),a cost-efficient framework that addresses this trade-off taking inspiration from group-testing.POOL clusters query stems with overlaps, evaluates a base estimator on representative medoids, softly propagates confidence scores to nearby queries, and selectively evaluates high-disagreement cases. We instantiate this framework with Hy@ $p$ , a hybrid estimator that combines verbal confidence with spectral answer diversity computed from the negative von Neumann entropy of sampled answer embeddings.Across six domains from three datasets and five black-box LLMs, Hy@5 achieves higher average AUROC than verbal confidence and Vn@10 sampling while using half as many samples as Vn@10. POOL-Hy@5 retains 93.5--97.9% of its AUROC while saving 19.3--39.3% of generations. On paraphrase-dense workloads, generation savings rise to 73-76%, showing that semantic redundancy can be leveraged to lower confidence-estimation costs.

Figures & tables

Figure 1: Soft-cluster size distribution (members per medoid under POOL’s attention rule) for all benchmarks. Heavier right tails (Law, Physics, MATH-500) reflect denser topical redundancy and translate directly into higher POOL compression (Table 5 ). Psychology has the lightest tail and correspondingly the smallest savings.

Figure 2: The two faces of the rescue fraction

q

, averaged across all 30 (model, dataset) cells. Left: POOL-Hy@5 AUROC rises smoothly with

q

. Right: cost savings vs. unpooled Hy@ 5 fall linearly with

q

. Open circles mark the three operating points

q\in\{0,\,0.10,\,0.20\}

used in Table 1 ;

q

is a single, predictable cost-quality dial.

Figure 3: The two faces of the cluster-cutoff knob

\tau

. Top: AUROC of POOL-Hy@5 (

q\!=\!0

) rises monotonically with

\tau

. Bottom: cost savings vs. un-pooled Hy@ 5 fall monotonically with

\tau

. Each line is the mean across all five models with shaded SEM. The mirror-image behaviour confirms the cost equation

(c+q)\,g_{M}

\tau

trades quality against compute on a smooth, predictable curve.

		Baselines			POOL - Vn@ 10			POOL - Hy@ 5
Model	Dataset	Vc	Vn@ 10	Hy@ 5	$q\!=\!0$	$q\!=\!0.10$	$q\!=\!0.20$	$q\!=\!0$	$q\!=\!0.10$	$q\!=\!0.20$
GPT-4.1-nano	Biology	0.679	0.754	0.758	0.698	0.723	0.744	0.713	0.732	0.762
	Law	0.530	0.641	0.599	0.596	0.605	0.616	0.567	0.569	0.572
	Physics	0.746	0.777	0.798	0.707	0.729	0.747	0.718	0.744	0.773
	Psychology	0.630	0.744	0.716	0.730	0.743	0.753	0.700	0.702	0.726
	MATH-500	0.700	0.896	0.869	0.822	0.837	0.843	0.804	0.813	0.812

Table 1: AUROC across all (model, dataset) cells for the three per-query baselines ( Vc , Vn@ 10, Hy@ 5) and six POOL variants (two base methods

\times

three rescue budgets). POOL configuration

(\tau,\theta,T,k_{\max})=(0.55,\,0.40,\,0.1,\,10)

is fixed across all cells, except TriviaQA , whose dispersed open-domain stems use

\tau\!=\!0.35

(

\theta\!=\!0.20

) to reach a comparable

{\sim}3

gen/q operating point (

\tau

is the cost dial, §3.4); only the rescue fraction

q

varies. GPT-5-nano and GPT-5-mini use reasoning effort low ; the remaining backbones are non-reasoning models.

Estimator	AUROC	ECE
Hy@ 5 (calibrated)	0.752	0.044
POOL - Hy@ 5 ( $q\!=\!0$ )	0.702	0.037
POOL - Hy@ 5 ( $q\!=\!0.20$ )	0.736	0.038

Table 2: Calibration of Hy@ 5 vs. POOL - Hy@ 5, averaged over all 30 (model, dataset) cells. ECE uses 15 equal-width bins; scores are Platt-scaled via 5-fold CV and evaluated out-of-fold.

Method	Paradigm	AUROC
		$q\!=\!0$	$q\!=\!0.2$
P(IK)-LR	supervised	0.618
P(IK)-MLP (q $+$ a)	supervised	0.586
APRICOT	supervised	0.576
rand $+$ kNN	zero-label	0.685	0.713
POOL - Hy@ 5	zero-label	0.702	0.736

Table 3: End-to-end baselines vs. POOL , avg AUROC over 30 cells. Supervised probes train on labels (no sampling). Zero-label propagation methods are shown budget-matched at

q\!=\!0

and

q\!=\!0.2

; the kNN baselines use a random rescue (no disagreement signal), POOL uses its disagreement rescue.

Rescue strategy	AUROC	% oracle
Tier-1 only ( $q\!=\!0$ )	0.702	0 0%
Tier1 + random	0.724	52%
Tier1 + boundary	0.726	57%
Tier1 + high-conf	0.725	55%
Tier1 + low-conf	0.728	61%
Tier1 + disagreement ( POOL )	0.736	80%

Table 4: Rescue-selection strategies at fixed Tier-1 base

=

Hy@ 5 and budget

q\!=\!0.20

, avg over 30 cells. % oracle is the fraction of the oracle rescue’s AUROC gain over Tier-1 that each rule recovers. POOL ’s disagreement signal captures the most.

Dataset	$q\!=\!0$	$q\!=\!0.10$	$q\!=\!0.20$
Biology	30.6%	20.6%	10.6%
Law	53.5%	43.5%	33.5%
Physics	43.2%	33.2%	23.2%
Psychology	20.6%	10.6%	0.6%
MATH-500	48.7%	38.7%	28.7%
TriviaQA	39.4 %	29.4 %	19.4 %

Table 5: Cost savings of POOL relative to its base estimator, per dataset, averaged across all models. Each cell reports

1-\text{gen/q}_{\text{POOL}}/\text{gen/q}_{\text{base}}

. Savings are identical for POOL - Vn@ 10 and POOL - Hy@ 5 at any fixed

q

		Biology		Psychology
$\tau$	$q$	AUROC	sav.	AUROC	sav.
Base Hy@ 5		0.808	–	0.819	–
0.55	0.00	0.669	54%	0.636	47%
0.55	0.20	0.714	34%	0.687	27%
0.75	0.00	0.705	42%	0.674	30%
0.75	0.20	0.757	22%	0.786	10%

Table 6: Adversarial twins (GPT-5-nano). Twin-subset AUROC and cost savings of POOL - Hy@ 5. Raising the cutoff

\tau

and rescue budget

q

recovers most of the base AUROC while retaining savings.

Figure 4: Mean member accuracy of clusters whose medoid lies in the top vs. bottom half of Hy@ 5 scores, binned by cluster radius and averaged over all (model, dataset) cells (

\tau=0.55

4{,}442

multi-member clusters). Members of confident-medoid clusters are correct

\sim 79\%

of the time; members of un-confident-medoid clusters only

\sim 63\%

— a

\sim 16

-point gap that persists across every radius bin and motivates weighting medoid contributions by their reliability during propagation.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

		Biology-Ext		Psychology-Ext
Method	$q$	AUROC	sav.	AUROC	sav.
Vn@ 10 (unpooled)		0.773	–	0.736	–
Hy@ 5 (unpooled)		0.814	–	0.753	–
POOL - Vn@ 10	0.00	0.763	76%	0.716	73%
	0.10	0.767	66%	0.718	63%
	0.20	0.780	56%	0.724	53%

Appendix

Table 7: Paraphrase-augmented datasets (GPT-5-nano). AUROC and cost savings on the

3\times

-density Biology/Psychology splits; POOL at

(\tau,\theta,T,k_{\max})=(0.55,0.40,0.1,10)

. Savings depend only on the compression ratio and

q

, so they coincide for both POOL variants.

		Baselines			POOL - Vn@ 5			POOL - Hy@ 10
Model	Dataset	Vc	Vn@ 5	Hy@ 10	$q\!=\!0$	$q\!=\!0.10$	$q\!=\!0.20$	$q\!=\!0$	$q\!=\!0.10$	$q\!=\!0.20$
GPT-4.1-nano	Biology	0.679	0.734	0.777	0.680	0.702	0.727	0.720	0.736	0.768
	Law	0.530	0.618	0.611	0.576	0.587	0.601	0.577	0.585	0.591
	Physics	0.746	0.736	0.818	0.680	0.695	0.723	0.730	0.760	0.784
	Psychology	0.630	0.671	0.746	0.674	0.669	0.695	0.718	0.724	0.749
	MATH-500	0.700	0.858	0.882	0.804	0.807	0.828	0.816	0.832	0.829

Appendix

Table 8: Appendix AUROC table – same setup as Table 1 but with Vn@ 5 (instead of Vn@ 10) and Hy@ 10 (instead of Hy@ 5). POOL configuration

(\theta,T,k_{\max})\!=\!(0.40,0.1,10)

with

\tau\!=\!0.55

(MMLU-Pro/MATH-500) and

\tau\!=\!0.35

(TriviaQA); only

q

varies.

Dataset	$q\!=\!0$	$q\!=\!0.10$	$q\!=\!0.20$
Biology	30.6%	20.6%	10.6%
Law	53.5%	43.5%	33.5%
Physics	43.2%	33.2%	23.2%
Psychology	20.6%	10.6%	0.6%
MATH-500	48.7%	38.7%	28.7%
TriviaQA	39.4 %	29.4 %	19.4 %

Appendix

Table 9: Cost savings of POOL over its base estimator, per dataset, averaged across all five models. Savings are identical for every POOL -

M

variant (

M\in\{

Vn@ 5, Vn@ 10, Hy@ 5, Hy@ 10, Maj@ 5, Maj@ 10, SE@ 5, SE@ 10

\}

) at any fixed

q

, because the per-query cost formula

(c+q)\,g_{M}

makes the base-method factor

g_{M}

cancel out in the ratio, the relative reduction depends only on the clustering compression

c\!=\!K/N

and the rescue fraction

q

		Baselines		POOL - Maj@ 5			POOL - Maj@ 10
Model	Dataset	Maj@ 5	Maj@ 10	$q\!=\!0$	$q\!=\!0.10$	$q\!=\!0.20$	$q\!=\!0$	$q\!=\!0.10$	$q\!=\!0.20$
GPT-4.1-nano	Biology	0.742	0.772	0.710	0.730	0.759	0.724	0.748	0.772
	Law	0.633	0.658	0.590	0.610	0.621	0.603	0.620	0.623
	Physics	0.761	0.802	0.703	0.723	0.728	0.724	0.746	0.765
	Psychology	0.674	0.735	0.665	0.664	0.688	0.723	0.737	0.744
	MATH-500	0.866	0.896	0.812	0.828	0.827	0.831	0.847	0.854

Appendix

Table 10: Appendix AUROC table for Maj@ 5 and Maj@ 10 baselines and their POOL variants. Same setup as Table 1 ; POOL configuration

(\theta,T,k_{\max})\!=\!(0.40,0.1,10)

with

\tau\!=\!0.55

(MMLU-Pro/MATH-500) and

\tau\!=\!0.35

(TriviaQA); only

q

varies.

		Baselines		POOL - SE@ 5			POOL - SE@ 10
Model	Dataset	SE@ 5	SE@ 10	$q\!=\!0$	$q\!=\!0.10$	$q\!=\!0.20$	$q\!=\!0$	$q\!=\!0.10$	$q\!=\!0.20$
GPT-4.1-nano	Biology	0.685	0.699	0.670	0.661	0.690	0.678	0.666	0.683
	Law	0.619	0.638	0.583	0.587	0.603	0.599	0.604	0.604
	Physics	0.666	0.692	0.644	0.648	0.655	0.656	0.669	0.680
	Psychology	0.651	0.712	0.651	0.651	0.670	0.711	0.725	0.730
	MATH-500	0.839	0.863	0.797	0.793	0.818	0.818	0.821	0.830

Appendix

Table 11: Appendix AUROC table for SE@ 5 and SE@ 10 baselines and their POOL variants. Same setup as Table 1 ; POOL configuration

(\theta,T,k_{\max})\!=\!(0.40,0.1,10)

with

\tau\!=\!0.55

(MMLU-Pro/MATH-500) and

\tau\!=\!0.35

(TriviaQA); only

q

varies.

Model	Dataset	Vc	Vn@ 5	Vn@ 10	Hy@ 5	Hy@ 10	Maj@ 5	Maj@ 10	SE@ 5	SE@ 10
GPT-4.1-nano	Biology	0.679	0.734	0.754	0.758	0.777	0.742	0.772	0.685	0.699
	Law	0.530	0.618	0.641	0.599	0.611	0.633	0.658	0.619	0.638
	Physics	0.746	0.736	0.777	0.798	0.818	0.761	0.802	0.666	0.692
	Psychology	0.630	0.671	0.744	0.716	0.746	0.674	0.735	0.651	0.712
	MATH-500	0.700	0.858	0.896	0.869	0.882	0.866	0.896	0.839	0.863
	TriviaQA	0.686	0.714	0.761	0.748	0.770	0.717	0.758	0.716	0.752

Appendix

Table 12: All nine per-query (point-based) confidence estimators evaluated across five models and six datasets. Vc = verbal confidence; Vn@

p

= spectral sample-diversity (negative von Neumann entropy of

p

answer embeddings); Hy@

p

= z-scored sum of Vc and Vn@

p

; Maj@

p

= self-consistency / majority-vote fraction; SE@

p

= discrete semantic entropy (Shannon entropy over union-find clusters of answer embeddings,

\eta\!=\!0.85

). GPT-5-nano and GPT-5-mini use reasoning effort low ; the rest are non-reasoning. Each method costs

p

generations per query ( Vc costs

1

Combination	AUROC
Hy@5 (default, z(Vc)+z(Vn@5))	0.752
pure Vc	0.689
pure Vn@5	0.696
$\alpha\!=\!0.25$ ( $0.25\,z(Vc)+0.75\,z(Vn))$	0.750
$\alpha\!=\!0.75$ ( $0.75\,z(Vc)+0.25\,z(Vn))$	0.745
max(z(Vc), z(Vn@5))	0.714

Appendix

Table 13: Hybrid score combination ablation for Hy@ 5. AUROC averaged across all 30 (model, dataset) cells.

z(\cdot)

denotes benchmark-wise standardisation. All variants cost

5

generations per query.

	Vc	Vn@ 5	Vn@ 10	Hy@ 5	Hy@ 10
Vc	1.000	–	–	–	–
Vn@5	0.242	1.000	–	–	–
Vn@10	0.278	0.650	1.000	–	–
Hy@5	0.762	0.710	0.587	1.000	–
Hy@10	0.757	0.552	0.748	0.892	1.000
$\rho$ vs. $y$	0.289	0.285	0.336	0.356	0.378

Appendix

Table 14: Pairwise Spearman rank correlations between the five per-query confidence signals, averaged across all 30 (model, dataset) cells. The lower triangle is shown. Weakly-correlated signals ( Vc vs.

\textsc{Vn@}p

) are precisely the ones whose

z

-scored sum produces the strongest hybrid Hy@

p

; strongly-correlated signals are different sample budgets of the same estimator family. The bottom row reports each method’s Spearman correlation with the binary correctness label

y

the strength of the underlying confidence signal.

Dataset	gen/q	emb-calls/q	LLM $/q	embed $/q	emb / (LLM+emb)
Biology	4.47	2.37	$\$ 2.67!\times!10^{-4}$	$\$ 0.02!\times!10^{-4}$	0.79%
Law	3.32	2.97	$\$ 2.49!\times!10^{-4}$	$\$ 0.06!\times!10^{-4}$	2.27%
Physics	3.84	2.79	$\$ 2.35!\times!10^{-4}$	$\$ 0.03!\times!10^{-4}$	1.19%
Psychology	4.97	2.40	$\$ 2.94!\times!10^{-4}$	$\$ 0.02!\times!10^{-4}$	0.69%
MATH-500	3.56	2.15	$\$ 2.17!\times!10^{-4}$	$\$ 0.02!\times!10^{-4}$	0.97%
Average			$\boldsymbol{\$ 2.52!\times!10^{-4}}$	$\boldsymbol{\$ 0.03!\times!10^{-4}}$	1.18%

Appendix

Table 15: Full cost breakdown for POOL - Hy@ 5 at

q\!=\!0.20

(the most expensive operating point reported in the paper), per dataset. Per-query LLM cost is

\bigl((K/N)+q\bigr)\cdot 5

generations. Per-query embedding cost is

1

stem embedding plus the unique answer-text embeddings needed for the sampled generations. Prices are 2026 OpenAI public list for the cheapest LLM ( gpt-4.1-nano ,

0.10

/

0.40

per 1M input/output tokens) and text-embedding-3-small (

0.02

per 1M tokens); the cheaper LLM choice makes the embedding fraction look as large as possible (conservative defense). Token counts use

\sim

0.25

tokens/character for question stems, an 80-token prompt overhead per LLM call, an average

120

-token generation, and a

50$ -token average answer-embedding input.

		POOL - Hy@ 5
Embedding	Hy@ 5 (base)	$q\!=\!0$	$q\!=\!0.10$	$q\!=\!0.20$
text-embedding-3-small	0.752	0.702	0.721	0.736
bge-large-en-v1.5	0.753	0.699	0.717	0.729
e5-large-v2	0.755	0.685	0.704	0.717

Appendix

Table 16: Average AUROC across embedding encoders, with

\tau

recalibrated per encoder to a matched mean compression

K/N

(

\theta=\tau-0.15

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 229 papers

Improved Confidence Estimates for Black-Box Large Language Models
Aug 19, 2026Sokhna Diarra Mbacke, Mouloud Belbahri, Gabriel Loaiza-GanemConfidence Estimation in Language Models Confidence Estimation
Layer 6 AI · TD Insurance
How do LLMs Compute Verbal Confidence
Mar 18, 2026Dharshan Kumaran, Arthur Conmy, Federico Barbero +3Verbalized Confidence Estimation Confidence Estimation in Language Models
Google DeepMind
Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection
Feb 18, 2026Xie Xiaohu, Liu Xiaohu, Yao BenjaminLanguage Model Error Detection Confidence Estimation in Language Models
Alexa AI, Amazon, Seattle, WA, USA

Aug 19, 2026J/K move · Enter open · S save

Improved Confidence Estimates for Black-Box Large Language Models

Sokhna Diarra Mbacke, Mouloud Belbahri, Gabriel Loaiza-Ganem

Layer 6 AI · TD Insurance

Confidence Estimation in Language Models Confidence Estimation LLM Uncertainty Estimation Language Model Calibration

Open paper AskPDF arXiv

Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs). Existing methods, from verbalized confidence to ones requiring multiple generations, are often zero-shot and produce scores quantifying uncertainty without the need for labelled data. Nonetheless, in practice one must always evaluate their performance on a dataset of interest before deployment. In this work we show that, by leveraging this dataset, we consistently outperform these existing scores. Specifically, we build simple classifiers that predict LLM response correctness by using these scores and the correctness of similar queries as features. Our method produces minimal computational overhead, making it a cheap and straightforward enhancement for UQ in LLMs for real-world applications.

Abstract

Confidence Estimation in Language Models

Explore similar work

Improved Confidence Estimates for Black-Box Large Language Models

How do LLMs Compute Verbal Confidence

Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection