Neighborhood Elites: Quality-Diversity Search for Uncertainty Quantification in Question Answering

Mesut Ceylan; Arda Baris Basaran; Elgar Fleisch; Filipe Barata

cs.AIOct 29, 2026

Neighborhood Elites: Quality-Diversity Search for Uncertainty Quantification in Question Answering

Authors: Mesut Ceylan, Arda Baris Basaran, Elgar Fleisch, Filipe Barata

Organizations: ETH Zurich, Zurich, Switzerland · University of St.Gallen, St.Gallen, Switzerland

Abstract

Large language models (LLMs) are increasingly capable, yet they produce wrong answers indistinguishable from correct ones. In question answering (QA), such an error propagates unnoticed into downstream tasks. Perturbation-based uncertainty quantification (UQ) addresses this by probing a model over a set of input rewrites and reading uncertainty from how much its predictions vary. Yet this set is usually built generically, by free paraphrasing, token-level perturbation, or adversarial search. Such rewrites leave two things uncontrolled that matter in QA: what kind of change is made to the context, and how explicitly the context supports the answer. When neither is controlled, the composition of the set is determined by the generator rather than by design, and the estimate inherits whatever that generator happens to produce. Quality-diversity (QD) search is built to cover a predefined exploration space, so we recast input perturbation as a QD problem and propose neighborhood elites (NE), a neighborhood constructor that spreads answer-preserving rewrites across two axes, the kind of edit and the explicitness of support. We compare NE against three other neighborhood constructors, holding the editor, QA model, uncertainty measure and search objective fixed, and against three scores read from the original context. Across five extractive-QA datasets and five instruction-tuned LLMs, NE improves failure detection against every other constructor in at least 24 of the 25 dataset-model pairs, reaching 0.726 AUROC and 0.211 AUPRC lift over the base rate. Additionally, NE detects failure better than semantic entropy computed over decoder samples, by 0.074 AUROC and 0.062 AUPRC lift, and this advantage widens as the neighborhood grows.

Figures & tables

Figure 1: A MAP-Elites search builds a per-instance archive of answer-preserving context rewrites spread over the descriptor space $\mathcal{D}$ (Section 2.2 ). The frozen QA model $f_{\theta}$ is queried on $K$ of them, and the semantic entropy of its answers is the uncertainty signal (Section 2.3 ).

Figure 2: A SQuAD v1.1 instance: question, gold answer (highlighted), context [S1] – [S4] . B three rewrites, distractor (1), reorder (2) and style rewrite (3), each preserving the gold answer (red deleted, green inserted). C the instance archive over $\mathcal{D}$ , one elite per cell, shaded by search-objective value (darker higher, grey empty). Edits (1)–(3) are the marked paraphrastic-support cells; $6/12$ cells are filled.

Neighborhood constructed
		AUROC		AUPRC lift
Method	$n$	Value	$\Delta$	Value	$\Delta$
NE	$21{,}916$	$\mathbf{0.726}$	—	$\mathbf{0.211}$	—
U-ME	$23{,}911$	$0.661$	$+0.062$ $[+0.053,+0.070]$	$0.148$	$+0.060$ $[+0.048,+0.071]$
SBS	$16{,}288$	$0.642$	$+0.096$ $[+0.085,+0.106]$	$0.122$	$+0.101$ $[+0.090,+0.116]$
FP	$17{,}862$	$0.621$	$+0.114$ $[+0.105,+0.123]$	$0.113$	$+0.110$ $[+0.097,+0.123]$

Table 1: Failure detection, macro-averaged over

25

dataset–model pairs. Upper block: construction varies, measure fixed. Lower block: scores read from the original context. A method is scored where it has at least two predictions, the smallest count at which semantic entropy is defined, so

n

differs by method, and the NLL needs only the greedy answer. AUPRC is lift over the base rate (

0.315

–

0.361

\Delta

is the difference from NE on shared instances, with

95\%

intervals from

2000

bootstrap replicates resampled within each pair.

Figure 3: Per-pair differences, NE minus each comparator on the instances they share, in AUROC (a) and AUPRC lift (b). Each point is one of the $25$ dataset–model pairs, with colour and vertical offset both encoding the dataset. The upper three rows are neighborhood constructors, the lower three are scores read from the original context. Points right of zero favour NE, and the count beside each row gives how many of the $25$ do. Table 9 and Table 10 in Appendix B.2 list all $150$ values.

	FP	SBS	U-ME	NE
Cells occupied, of twelve	$0.98$ $(0.66)$	$2.00$ $(1.01)$	$2.33$ $(1.07)$	$\mathbf{4.43}$ $(2.15)$
Rewrites kept	$18439$	$10024$	$29506$	$\mathbf{22163}$
Rewrites per instance	$3.69$	$2.01$	$5.90$	$\mathbf{4.43}$
Rewrites per occupied cell	$3.77$	$1.00$	$2.53$	$\mathbf{1.00}$
Instances reaching $\geq 4$ cells	$0.001$	$0.110$	$0.143$	$\mathbf{0.658}$

Table 2: What a single neighborhood covers of the descriptor space

\mathcal{D}

, the grid of twelve cells spanned by edit type and answer support. The first row is the mean number of distinct cells a neighborhood occupies, with the standard deviation in brackets. The lower block pools all 5000 instances. Per-dataset values are in Appendix B.3 , Table 12 .

Figure 4: For each cell, the fraction of the $5000$ instances whose neighborhood places at least one rewrite in that cell. The twelve cells of a panel therefore sum to that constructor’s entry in Table 2 . A grey cell marked with a dash was never reached, and $<\!0.01$ marks a cell reached by fewer than one instance in a hundred. Rewrite totals by column are in Appendix B.3 , Table 11 .

	$K{=}3$		$K{=}4$		$K{=}5$		$K{=}6$		$K{=}7$
Method	AUROC	AUPRC	AUROC	AUPRC	AUROC	AUPRC	AUROC	AUPRC	AUROC	AUPRC
NE	0.705	0.215	0.746	0.262	0.765	0.282	0.780	0.301	0.805	0.329
U-ME	0.637	0.150	0.667	0.184	0.683	0.206	0.685	0.212	0.713	0.252
FP	0.607	0.130	0.622	0.150	0.626	0.151	0.633	0.155	0.654	0.202
PE	0.677	0.133	0.677	0.139	0.674	0.137	0.684	0.156	0.686	0.166
NLL	0.672	0.127	0.670	0.129	0.666	0.124	0.678	0.144	0.684	0.170

Table 3: Failure detection as neighborhood size

K

increases, on the matched instance pool for each

K

(evaluation method described in Appendix B.1 ). SBS cannot supply

K

rewrites and is omitted.

Appendix figures & tables40 assets

Supplementary material from the paper’s appendix.

Appendix

$K$	SQuAD v1.1	Quoref	AdversarialQA	Nat. Questions	NewsQA	Total	Per pair	Base rate
3	3980	3587	3673	2566	1924	15730	629	0.300
4	2866	2637	2751	1822	1281	11357	454	0.284
5	2476	1924	2475	1257	830	8962	358	0.276
6	1786	1209	1819	675	441	5930	237	0.276
7	705	435	747	210	176	2273	91	0.300

Appendix

Table 4: Matched evaluation sets by

K

. Counts are dataset–model–example observations out of a possible 25000. “Per pair” is the mean over the 25 dataset–model pairs. “Base rate” is the mean of the per-pair failure rates.

SQuAD
	NE $-$ FP		NE $-$ U-ME		U-ME $-$ FP
QA model	$\Delta$	$P$	$\Delta$	$P$	$\Delta$	$P$
Falcon3-10B	$0.090$	$1.00$	$0.085$	$1.00$	$0.005$	$0.64$
Mistral-Nemo-12B	$0.111$	$1.00$	$0.058$	$1.00$	$0.053$	$1.00$
Gemma3-12B	$0.108$	$1.00$	$0.053$	$0.99$	$0.055$	$1.00$
OLMo2-13B	$0.132$	$1.00$	$0.086$	$1.00$	$0.047$	$1.00$

Appendix

Table 5: Per dataset–model paired

\Delta

AUROC (semantic entropy,

K=3

) for the three pairwise contrasts among the constructors FP, U-ME, and NE.

\Delta

is the mean

\Delta

AUROC and

P=P(\Delta>0)

over

2000

repeats; a positive

\Delta

favors the first-named constructor. A difference is reliably positive when

P\geq 0.975

SQuAD
	NE $-$ FP		NE $-$ U-ME		U-ME $-$ FP
QA model	$\Delta$	$P$	$\Delta$	$P$	$\Delta$	$P$
Falcon3-10B	$0.048$	$0.98$	$0.073$	$1.00$	$-0.025$	$0.15$
Mistral-Nemo-12B	$0.124$	$1.00$	$0.062$	$0.99$	$0.062$	$1.00$
Gemma3-12B	$0.078$	$1.00$	$0.049$	$0.98$	$0.029$	$0.94$
OLMo2-13B	$0.133$	$1.00$	$0.120$	$1.00$	$0.012$	$0.71$

Appendix

Table 6: Per dataset–model paired

\Delta

AUPRC (semantic entropy,

K=3

) for the three pairwise contrasts among the constructors FP, U-ME, and NE.

\Delta

is the mean

\Delta

AUPRC and

P=P(\Delta>0)

over

2000

repeats; a positive

\Delta

favors the first-named constructor, and a difference is reliably positive when

P\geq 0.975

. AUPRC is base-rate dependent, so per-pair

\Delta

AUPRC is a noisier statistic than the rank-based

\Delta

AUROC of Table 5 .

Constructor	$K{=}3$	$K{=}4$	$K{=}5$	$K{=}6$	$K{=}7$
Semantic entropy
FP	$0.607$ $[.603,.611]$	$0.622$ $[.619,.626]$	$0.626$ $[.623,.629]$	$0.633$ $[.629,.636]$	$0.654$ $[.648,.658]$
U-ME	$0.637$ $[.633,.642]$	$0.667$ $[.662,.672]$	$0.683$ $[.679,.687]$	$0.685$ $[.681,.689]$	$0.713$ $[.707,.718]$
NE	$\mathbf{0.705}$ $[.701,.710]$	$\mathbf{0.746}$ $[.742,.750]$	$\mathbf{0.765}$ $[.761,.768]$	$\mathbf{0.780}$ $[.776,.784]$	$\mathbf{0.805}$ $[.801,.809]$

Appendix

Table 7: Failure-detection AUROC across

K

on semantic entropy, macro over the

25

dataset–model pairs (

95\%

subselection percentile intervals).

Constructor	$K{=}3$	$K{=}4$	$K{=}5$	$K{=}6$	$K{=}7$
Semantic entropy
FP	$0.430$ $[.424,.435]$	$0.434$ $[.429,.440]$	$0.427$ $[.422,.432]$	$0.431$ $[.426,.436]$	$0.502$ $[.496,.507]$
U-ME	$0.450$ $[.443,.457]$	$0.468$ $[.461,.476]$	$0.482$ $[.475,.490]$	$0.488$ $[.481,.495]$	$0.552$ $[.543,.561]$
NE	$\mathbf{0.515}$ $[.508,.522]$	$\mathbf{0.546}$ $[.538,.553]$	$\mathbf{0.559}$ $[.552,.565]$	$\mathbf{0.577}$ $[.571,.583]$	$\mathbf{0.629}$ $[.623,.636]$

Appendix

Table 8: Failure-detection AUPRC across

K

, on semantic entropy, macro over the

25

dataset–model pairs (

95\%

subselection percentile intervals).

			Neighborhood constructed			Original context
Dataset	Model	NE	U-ME	SBS	FP	PE	NLL	DSE
SQuAD v1.1	Falcon3-10B	$0.759$	$+0.086$	$+0.092$	$+0.104$	$+0.067$	$+0.078$	$+0.058$
	Gemma3-12B	$0.720$	$+0.068$	$+0.072$	$+0.135$	$+0.082$	$+0.077$	$+0.201$
	Mistral-NeMo-12B	$0.727$	$+0.055$	$+0.082$	$+0.134$	$-0.011$	$-0.004$	$+0.021$
	OLMo2-13B	$0.768$	$+0.077$	$+0.085$	$+0.158$	$+0.072$	$+0.089$	$+0.046$
	Phi-4-14B	$0.786$	$+0.098$	$+0.108$	$+0.138$	$+0.162$	$+0.165$	$+0.053$

Appendix

Table 9: Per-pair differences in AUROC between NE and each comparator, for the

25

dataset–model pairs. Positive values favor NE; the

19

150

values against NE are underlined. Each difference is computed on the instances the two methods share within that pair under the

\min K=2

requirement, so a difference is a within-pair contrast and is not the difference of two separately pooled levels. The NE column gives its AUROC on its own pool for scale and should not be used to recover a comparator level by subtraction. Column means equal the paired macro differences reported in Table 1 . Figure 3 plots these values.

			Neighborhood constructed			Original context
Dataset	Model	NE	U-ME	SBS	FP	PE	NLL	DSE
SQuAD v1.1	Falcon3-10B	$0.239$	$+0.077$	$+0.085$	$+0.074$	$+0.143$	$+0.153$	$+0.024$
	Gemma3-12B	$0.190$	$+0.077$	$+0.070$	$+0.113$	$+0.149$	$+0.148$	$+0.168$
	Mistral-NeMo-12B	$0.242$	$+0.054$	$+0.095$	$+0.165$	$+0.047$	$+0.058$	$+0.033$
	OLMo2-13B	$0.291$	$+0.114$	$+0.118$	$+0.186$	$+0.167$	$+0.167$	$+0.074$
	Phi-4-14B	$0.345$	$+0.150$	$+0.154$	$+0.196$	$+0.274$	$+0.285$	$+0.069$

Appendix

Table 10: Per-pair differences in AUPRC, same protocol and layout as Table 9 , with

14

150

values against NE. The NE column gives AUPRC lift over the base rate of that pair, as in Table 1 , while the differences are differences in AUPRC. A difference in AUPRC and the corresponding difference in AUPRC lift are identical, because both methods face the same base rate on the instances they share. Figure 3 plots these comparisons.

Per instance, mean over 5000 instances
	FP	SBS	U-ME	NE
Editor calls	15.43	25.00	24.26	24.21
Proposals passing the gates	3.69	10.35	21.67	12.12
Retained in the archive	3.69	2.01	5.90	4.43
Distinct	3.69	2.01	4.94	4.43
Duplicate rate	0.00	0.00	0.17	0.00

Appendix

Table 11: Archive construction diagnostics, per instance and pooled over the 5000 instances. Rejection reasons are fractions of editor calls and do not sum to the total, since a proposal may fail for more than one reason and only the principal ones are listed.

Dataset	FP	SBS	U-ME	NE
SQuAD v1.1	$1.19$ $(0.68)$	$2.87$ $(0.95)$	$2.44$ $(1.05)$	$\mathbf{5.40}$ $(2.26)$
Quoref	$0.99$ $(0.55)$	$1.85$ $(1.04)$	$2.56$ $(1.03)$	$\mathbf{4.67}$ $(2.06)$
AdversarialQA	$1.04$ $(0.66)$	$2.87$ $(0.90)$	$2.38$ $(1.10)$	$\mathbf{5.69}$ $(2.06)$
Natural Questions	$0.92$ $(0.69)$	$1.17$ $(0.85)$	$2.17$ $(1.10)$	$\mathbf{3.33}$ $(2.25)$
NewsQA	$0.76$ $(0.70)$	$1.27$ $(1.25)$	$2.12$ $(1.06)$	$\mathbf{3.07}$ $(2.08)$
Macro	$0.98$ $(0.66)$	$2.00$ $(1.01)$	$2.33$ $(1.07)$	$\mathbf{4.43}$ $(2.15)$

Appendix

Table 12: Distinct cells of

\mathcal{D}

occupied per neighborhood, by dataset. Each entry is a mean over that dataset’s 1000 instances, with the standard deviation in brackets.

Figure 5: Descriptor space across datasets. For each cell, the proportion of the $5000$ instances whose neighborhood places at least one rewrite in that cell.

	Method	AUROC	AUPRC	Lift	Base	$n$	Cov.	sd	min – max
1	NE	0.702	0.546	0.189	0.357	23739	0.950	0.053	0.602–0.776
2	PE	0.676	0.488	0.131	0.357	23739	0.950	0.047	0.582–0.772
3	NLL	0.674	0.492	0.131	0.361	24984	0.999	0.050	0.563–0.761
4	U-ME	0.658	0.496	0.146	0.351	24060	0.962	0.020	0.624–0.686
5	DSE	0.652	0.498	0.149	0.349	21920	0.877	0.066	0.518–0.734
6	FP	0.605	0.421	0.097	0.323	19569	0.783	0.031	0.546–0.651

Appendix

Table 13: Failure detection on the full archives , with every method scoring each instance from all of the rewrites or samples it has and no minimum imposed.

	Method	AUROC	AUPRC	Lift	Base	$n$	Cov.	sd	min – max
1	NE	0.726	0.561	0.211	0.349	21920	0.877	0.038	0.647–0.786
2	PE	0.676	0.480	0.131	0.349	21920	0.877	0.046	0.580–0.765
3	NLL	0.674	0.492	0.131	0.361	24984	0.999	0.050	0.563–0.761
4	U-ME	0.661	0.497	0.148	0.349	23920	0.957	0.019	0.627–0.689
5	DSE	0.652	0.498	0.149	0.349	21920	0.877	0.066	0.518–0.734
6	FP	0.621	0.428	0.113	0.315	17864	0.715	0.032	0.563–0.679

Appendix

Table 14: Failure detection restricted to instances holding at least two rewrites or samples. Semantic entropy is identically zero on a single prediction, since there is no partition to measure, so two is the smallest count at which the score is defined. NE, PE and DSE sit on an identical population of 21920 instances.

AUROC
Dataset	NE	U-ME	FP	DSE	PE	NLL
SQuAD v1.1	0.742	0.671	0.616	0.676	0.673	0.666
Quoref	0.740	0.674	0.619	0.614	0.663	0.662
NewsQA	0.647	0.636	0.581	0.668	0.686	0.691
Natural Questions	0.638	0.652	0.594	0.643	0.728	0.731
AdversarialQA	0.741	0.659	0.613	0.658	0.628	0.620

Appendix

Table 15: Failure detection per dataset on the full archives, macro-averaged over the five QA models. Best value per row in bold.

AUROC
Dataset	NE	U-ME	FP	DSE	PE	NLL
SQuAD v1.1	0.752	0.672	0.628	0.676	0.678	0.666
Quoref	0.756	0.674	0.626	0.614	0.665	0.662
NewsQA	0.687	0.641	0.611	0.668	0.687	0.691
Natural Questions	0.687	0.654	0.617	0.643	0.722	0.731
AdversarialQA	0.746	0.662	0.623	0.658	0.628	0.620

Appendix

Table 16: Failure detection per dataset at two or more rewrites, macro-averaged over the five QA models. Best value per row in bold.

AUROC
QA model	NE	U-ME	FP	DSE	PE	NLL
Gemma3-12B	0.698	0.650	0.588	0.538	0.648	0.646
OLMo2-13B-Instruct	0.708	0.661	0.594	0.685	0.693	0.693
Phi4-14B	0.713	0.668	0.641	0.707	0.656	0.654
Mistral-Nemo-12B	0.692	0.648	0.578	0.670	0.706	0.705
Falcon3-10B-Instruct	0.698	0.665	0.621	0.660	0.676	0.672

Appendix

Table 17: Failure detection per QA model on the full archives, macro-averaged over the five datasets. Best value per row in bold.

AUROC
QA model	NE	U-ME	FP	DSE	PE	NLL
Gemma3-12B	0.721	0.652	0.601	0.538	0.647	0.646
OLMo2-13B-Instruct	0.730	0.664	0.609	0.685	0.696	0.693
Phi4-14B	0.738	0.670	0.662	0.707	0.659	0.654
Mistral-Nemo-12B	0.712	0.650	0.589	0.670	0.704	0.705
Falcon3-10B-Instruct	0.726	0.668	0.644	0.660	0.674	0.672

Appendix

Table 18: Failure detection per QA model at two or more rewrites, macro-averaged over the five datasets. Best value per row in bold.

NE versus	Pairs won	Mean $\Delta$	Pairs won	Mean $\Delta$
	AUROC		AUPRC
U-ME	25/25	$+0.065$	24/25	$+0.064$
FP	25/25	$+0.104$	25/25	$+0.133$
DSE	21/25	$+0.074$	19/25	$+0.062$
PE	18/25	$+0.050$	21/25	$+0.080$
NLL	18/25	$+0.052$	19/25	$+0.069$

Appendix

Table 19: Per-pair comparison of NE against each other method at two or more rewrites, over the 25 dataset–model pairs. Counts are pairs won by NE against pairs lost, on the instances the two methods share, with the mean difference alongside.

Method	Scored	$N{=}1$	$N{=}2$	$N{\geq}3$	$N{<}3$	Unsc.	$N{=}1$	$N{=}2$	$N{\geq}3$
	Instances				Share		Failure rate
NE	23739	1819	2582	19338	18.5%	1261	0.462	0.460	0.330
U-ME	24060	140	505	23415	2.7%	940	0.679	0.677	0.341
FP	19569	1705	1915	15949	18.5%	5431	0.407	0.405	0.293
DSE	21920	0	2582	19338	11.8%	3080	–	0.460	0.330
PE	23739	1819	2582	19338	18.5%	1261	0.462	0.460	0.330

Appendix

Table 20: Rewrites available per instance over each method’s full archive, with the failure rate at each count. Unscored counts instances the method produces no score for at all, against 1000 per dataset–model pair. Predictive entropy is defined at a single sample and is listed for its counts only, which match NE’s because it is computed on the same instances.

Dataset	Scored	$N{<}3$	Full	$N{\geq}3$	$\Delta$	Full	$N{\geq}3$	$\Delta$
			AUROC			AUPRC
SQuAD v1.1	4980	8.2%	0.742	0.759	$-0.017$	0.480	0.485	$-0.006$
Quoref	4943	14.1%	0.740	0.767	$-0.027$	0.510	0.527	$-0.017$
NewsQA	4438	37.7%	0.647	0.712	$-0.065$	0.656	0.682	$-0.025$
Natural Questions	4385	30.7%	0.638	0.709	$-0.071$	0.432	0.468	$-0.036$
AdversarialQA	4993	5.6%	0.741	0.748	$-0.007$	0.654	0.656	$-0.002$

Appendix

Table 21: Neighborhood elites (NE) only. Detection per dataset on the full archive against the same computation restricted to instances holding at least three rewrites, macro-averaged over the five QA models.

\Delta

is the full-archive value minus the restricted one.

Figure 6: NE AUROC at three editor sizes, on the 21031 instances for which NE retains at least two rewrites at all three.

	Method	AUROC	AUPRC	Lift	Base	$n$	Cov.	sd	min – max
1	NE	0.731	0.579	0.222	0.357	24133	0.965	0.038	0.653–0.784
2	NLL	0.674	0.491	0.130	0.361	24993	1.000	0.050	0.563–0.760
3	PE	0.673	0.486	0.129	0.357	24133	0.965	0.047	0.584–0.770
4	DSE	0.658	0.511	0.158	0.353	23224	0.929	0.066	0.527–0.753
5	FP	0.638	0.455	0.127	0.328	21584	0.863	0.031	0.564–0.685

Appendix

Table 22: Failure detection with the Qwen3-14B editor on the full archives, macro-averaged over the 25 dataset–model pairs.

	Method	AUROC	AUPRC	Lift	Base	$n$	Cov.	sd	min – max
1	NE	0.744	0.588	0.235	0.353	23224	0.929	0.030	0.676–0.786
2	NLL	0.674	0.491	0.130	0.361	24993	1.000	0.050	0.563–0.760
3	PE	0.673	0.483	0.129	0.353	23224	0.929	0.047	0.584–0.768
4	DSE	0.658	0.511	0.158	0.353	23224	0.929	0.066	0.527–0.753
5	FP	0.649	0.460	0.137	0.323	20600	0.824	0.031	0.568–0.693

Appendix

Table 23: Failure detection with the Qwen3-14B editor, restricted to instances holding at least two rewrites or samples. NE, PE and DSE sit on an identical population of 23224 instances.

AUROC
Dataset	NE	FP	DSE	PE	NLL
SQuAD v1.1	0.750	0.649	0.673	0.670	0.666
Quoref	0.749	0.651	0.616	0.660	0.662
NewsQA	0.677	0.619	0.673	0.683	0.691
Natural Questions	0.713	0.623	0.666	0.728	0.730
AdversarialQA	0.766	0.647	0.660	0.625	0.620

Appendix

Table 24: Failure detection per dataset with the Qwen3-14B editor on the full archives, macro-averaged over the five QA models. Best per row in bold.

AUROC
Dataset	NE	FP	DSE	PE	NLL
SQuAD v1.1	0.758	0.659	0.673	0.672	0.666
Quoref	0.760	0.658	0.616	0.659	0.662
NewsQA	0.709	0.643	0.673	0.684	0.691
Natural Questions	0.729	0.631	0.666	0.724	0.730
AdversarialQA	0.766	0.657	0.660	0.626	0.620

Appendix

Table 25: Failure detection per dataset with the Qwen3-14B editor at two or more rewrites, macro-averaged over the five QA models. Best per row in bold.

AUROC
QA model	NE	FP	DSE	PE	NLL
Gemma3-12B	0.745	0.634	0.543	0.648	0.646
OLMo2-13B-Instruct	0.738	0.639	0.691	0.689	0.693
Phi4-14B	0.736	0.663	0.711	0.658	0.653
Mistral-Nemo-12B	0.708	0.607	0.678	0.704	0.705
Falcon3-10B-Instruct	0.727	0.645	0.664	0.668	0.671

Appendix

Table 26: Failure detection per QA model with the Qwen3-14B editor on the full archives, macro-averaged over the five datasets. Best per row in bold.

AUROC
QA model	NE	FP	DSE	PE	NLL
Gemma3-12B	0.757	0.643	0.543	0.647	0.646
OLMo2-13B-Instruct	0.753	0.653	0.691	0.690	0.693
Phi4-14B	0.750	0.674	0.711	0.657	0.653
Mistral-Nemo-12B	0.719	0.617	0.678	0.705	0.705
Falcon3-10B-Instruct	0.743	0.660	0.664	0.666	0.671

Appendix

Table 27: Failure detection per QA model with the Qwen3-14B editor at two or more rewrites, macro-averaged over the five datasets. Best per row in bold.

NE versus	Pairs won	Mean $\Delta$	Pairs won	Mean $\Delta$
	AUROC		AUPRC
FP	25/25	$+0.093$	25/25	$+0.124$
DSE	19/25	$+0.073$	22/25	$+0.068$
PE	18/25	$+0.057$	21/25	$+0.093$
NLL	17/25	$+0.057$	20/25	$+0.088$

Appendix

Table 28: Per-pair comparison of NE against each other method with the Qwen3-14B editor on the full archives, over the 25 dataset–model pairs. Counts are pairs won by NE.

NE versus	Pairs won	Mean $\Delta$	Pairs won	Mean $\Delta$
	AUROC		AUPRC
FP	25/25	$+0.095$	25/25	$+0.128$
DSE	23/25	$+0.087$	22/25	$+0.077$
PE	21/25	$+0.072$	23/25	$+0.105$
NLL	21/25	$+0.071$	22/25	$+0.097$

Appendix

Table 29: Per-pair comparison of NE against each other method with the Qwen3-14B editor at two or more rewrites, over the 25 dataset–model pairs. Counts are pairs won by NE.

	Method	AUROC	AUPRC	Lift	Base	$n$	Cov.	sd	min – max
1	NE	0.724	0.568	0.211	0.356	24475	0.979	0.045	0.634–0.790
2	PE	0.674	0.486	0.129	0.356	24475	0.979	0.046	0.582–0.762
3	NLL	0.674	0.491	0.130	0.361	25000	1.000	0.049	0.563–0.762
4	DSE	0.662	0.512	0.162	0.350	23865	0.955	0.068	0.523–0.743

Appendix

Table 30: Failure detection with the Qwen3-32B editor on the full archives, macro-averaged over the 25 dataset–model pairs. FP and U-ME have no 32B run.

AUROC
Dataset	NE	DSE	PE	NLL
SQuAD v1.1	0.743	0.681	0.669	0.667
Quoref	0.754	0.616	0.660	0.661
NewsQA	0.660	0.677	0.683	0.691
Natural Questions	0.701	0.667	0.727	0.730
AdversarialQA	0.765	0.668	0.630	0.620

Appendix

Table 31: Failure detection per dataset with the Qwen3-32B editor on the full archives, macro-averaged over the five QA models. Best per row in bold.

AUROC
QA model	NE	DSE	PE	NLL
Gemma3-12B	0.727	0.544	0.644	0.646
OLMo2-13B-Instruct	0.723	0.698	0.689	0.693
Phi4-14B	0.733	0.718	0.654	0.654
Mistral-Nemo-12B	0.716	0.682	0.710	0.705
Falcon3-10B-Instruct	0.723	0.666	0.673	0.670

Appendix

Table 32: Failure detection per QA model with the Qwen3-32B editor on the full archives, macro-averaged over the five datasets. Best per row in bold.

NE versus	Pairs won	Mean $\Delta$	Pairs won	Mean $\Delta$
	AUROC		AUPRC
DSE	19/25	$+0.063$	20/25	$+0.056$
PE	16/25	$+0.051$	21/25	$+0.082$
NLL	15/25	$+0.051$	18/25	$+0.076$

Appendix

Table 33: Per-pair comparison of NE against each other method with the Qwen3-32B editor on the full archives, over the 25 dataset–model pairs. Counts are pairs won by NE.

No 32B run
	AUROC			Coverage
Method	8B	14B	32B	8B	14B	32B
NE	0.702	0.731	0.724	0.950	0.965	0.979
PE	0.676	0.673	0.674	0.950	0.965	0.979
NLL	0.674	0.674	0.674	0.999	1.000	1.000
DSE	0.652	0.658	0.662	0.877	0.929	0.955

Appendix

Table 34: Detection and coverage as the editor is scaled, on the full archives, macro-averaged over the 25 dataset–model pairs. Values are each editor’s own full archive, not the matched population of Figure 6 .

Figure 7: Where rewrite variation lives, over the full archives and macro-averaged over the $25$ dataset–model pairs. (a)–(c) the share of each quality component’s variance that falls within a neighborhood. (d)–(f) the within-neighborhood range of the same component at a fixed count of three rewrites ( $m=3$ per neighborhood, averaged over $200$ random subsets, with boxes spanning $p_{25}$ – $p_{75}$ and whiskers $p_{10}$ – $p_{90}$ across neighborhoods. Each panel in (d)–(f) has its own scale.

Model	SQuAD v1.1	Quoref	NewsQA	Natural Questions	AdversarialQA	Mean
Accuracy
Gemma3-12B	$0.843$	$0.781$	$0.540$	$0.755$	$0.636$	$0.711$
OLMo2-13B-Instruct	$0.750$	$0.698$	$0.420$	$0.692$	$0.531$	$0.618$
Phi4-14B	$0.778$	$0.764$	$0.405$	$0.689$	$0.554$	$0.638$
Mistral-Nemo-12B	$0.729$	$0.685$	$0.415$	$0.657$	$0.528$	$0.603$
Falcon3-10B-Instruct	$0.759$	$0.676$	$0.467$	$0.699$	$0.520$	$0.624$

Appendix

Table 35: QA performance of each model on each dataset, over the full evaluation population of

1{,}000

instances per dataset–model pair and

25{,}000

in total. Accuracy is the fraction of instances answered correctly under greedy decoding, using the stored correctness label that defines the failure target throughout the paper. Token F1 is the token overlap between the model’s answer and the reference, which is sensitive to answer length in a way the correctness label is not: Phi4-14B answers at

11.5

words on average on AdversarialQA and

20.7

on NewsQA, against

2.3

–

4.7

for Gemma3-12B, and its token F1 falls correspondingly further below its accuracy. Row means are unweighted over the five datasets.

How candidates are generated
	FP	SBS	U-ME	NE
Archive over $\mathcal{D}$	—	yes	yes	yes
Mutates an archived parent	—	—	yes	yes
Parent binds the offspring	—	—	no	yes
Targets empty cells	—	no	no	yes
Edit types requested	style only	all three	all three	all three

Appendix

Table 36: The four neighborhood constructors. All four use the same editor, the same answer-preservation gate and the same search objective, and they differ in whether they search, in whether they verify the edit, and in whether the rewrite-type axis carries a verified label or a requested one. Static bucket sampling verifies without evolving and the untargeted search evolves without verifying, so between them they bracket the two mechanisms NE combines.

\mathcal{D}

is the twelve-cell descriptor space spanned by edit type and answer support. A dash marks a property the constructor has no machinery for.

Rewrite type	Prompt strategies
distractor	distractor_support_proximal
distractor	distractor_question_overlap
reorder	reorder_adjacent_swap
	reorder_support_back
	reorder_global
style_rewrite	support_direct_consolidation

Appendix

Table 37: The ten prompt strategies grouped by rewrite type. Candidates from different strategies can occupy the same cell in descriptor space

\mathcal{D}

, where they compete on quality. We detail the prompt templates in Appendix G .

Rewrite type	Prompt intention	Deterministic realization condition
distractor	Add plausible but answer-irrelevant material while preserving the evidence.	The child differs from the reference, adds at least one sentence, and retains every reference sentence as an exact ordered subsequence .
reorder	Change sentence order without adding, deleting, or paraphrasing sentences.	The reference has at least two sentences; the child differs from it and has the same normalized sentence multiset in a different order.
style_rewrite	Rewrite substantive wording or syntax.	The child’s token sequence changes, and the child is not merely a sentence permutation and not merely a sentence insertion.

Appendix

Table 38: The three edit types: the prompt intention (what the editor is asked to do) and the deterministic condition verified between the child and its realization reference (the parent

P

for offspring, the original context

x

for seeds).

Editor (Appendix C.3 )
Setting	Value
Random seed	$7$
Model	Qwen3-8B (local MLX, 8-bit)
Decoding parameters	temperature $0.7$ ; sampling on; max new tokens $1200$
Support-explicitness rule (Appendix C.2.2 )
Sentence-split / token patterns	[.!?]+ / [A-Za-z0-9]+(’[A-Za-z0-9]+)?

Appendix

Table 39: Configuration used throughout, held fixed across all datasets and QA models.

Dataset	Context (mean/max)	Question (mean)	Answer (mean)	Multi-gold
SQuAD v1.1	$120.9/486$	$10.0$	$3.1$	$11.2$
Quoref	$334.9/781$	$15.4$	$1.7$	$9.0$
NewsQA	$704.0/1980$	$7.8$	$5.5$	$0.0$
NaturalQuestions (MRQA)	$161.9/800$	$9.1$	$4.4$	$9.8$
AdversarialQA	$116.0/327$	$9.8$	$3.1$	$0.0$

Appendix

Table 40: The five evaluation datasets, with statistics over the sampled

1000

examples each. All lengths are whitespace-word counts; answer length is that of the primary (first) gold answer. “Multi-gold” is the fraction of questions with more than one gold answer.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 663 papers

Semantic Uncertainty Quantification Needs Factual Equivalence
Sep 28, 2026Joseph Hoche, Quentin Guimard, Gianni FranchiConfidence Estimation in Language Models Factual Consistency Evaluation
AMIAD, Pˆole Recherche Palaiseau, France
Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees
Jul 5, 2026Sijin Dong, Hiroyuki ShinnouSelective Prediction Confidence Estimation in Language Models
1Ibaraki University, Japan.
Asymptotic Risk Calibration for Selective Question Answering
Aug 12, 2026Shufan Lin, Sijin DongQuestion Answering Selective QA
Zhangjiang University · Ibaraki University

Sep 28, 2026J/K move · Enter open · S save

Semantic Uncertainty Quantification Needs Factual Equivalence

Joseph Hoche, Quentin Guimard, Gianni Franchi

AMIAD, Pˆole Recherche Palaiseau, France

Confidence Estimation in Language Models Factual Consistency Evaluation LLM Reliability LLM Uncertainty Estimation Semantic Uncertainty Estimation

Open paper AskPDF arXiv

Semantic uncertainty quantification for large language models rests on a common template: sample several answers, measure how much they agree, and treat disagreement as uncertainty. We first formalize this template as two separate roles: an operator that compares two answers, and an aggregator that combines all pairwise comparisons into a scalar. Existing methods differ almost entirely in how they aggregate, while taking the operator off the shelf, typically an NLI model or a generic sentence encoder. We show that this reliance on off-the-shelf operators is the primary bottleneck of semantic UQ: they do not accurately measure factual equivalence of multiple answers to the same question. We resolve this with a deliberately simple recipe: a single encoder trained contrastively to isolate the targeted fact, utilizing synthetic data generated by an LLM and dataset both disjoint from all evaluation settings. Integrating the resulting operator into existing methods improves performance on 120 of 126 evaluation settings (95%) spanning 18 model dataset combinations across language and vision-language models. The best variant reaches 0.76 mean AUROC against 0.68 for the strongest baseline, while replacing the quadratic cross-encoder comparisons of entailment-based operators with one encoder pass per answer. The uniformity of the improvement supports the view that the operator, not the aggregator, is the limiting factor. The same operator also improves single generation token-level estimators: the norm it assigns to each token measures how much that token bears on the answer, and reweighting token log-likelihoods accordingly sharpens the estimate.

Abstract

Uncertainty Quantification

Explore similar work

Semantic Uncertainty Quantification Needs Factual Equivalence

Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guarantees

Asymptotic Risk Calibration for Selective Question Answering