Evolutionary Search for Automated Design of Uncertainty Quantification Methods

Mikhail Seleznyov; Daniil Korbut; Viktor Moskvoretskii; Oleg Somov; Alexander Panchenko; Elena Tutubalina

cs.CLApr 3, 2026

Evolutionary Search for Automated Design of Uncertainty Quantification Methods

Authors: Mikhail Seleznyov, Daniil Korbut, Viktor Moskvoretskii, Oleg Somov, Alexander Panchenko, Elena Tutubalina

Organizations: AIRI · Skoltech · Amazon · EPFL ·

Abstract

Uncertainty quantification (UQ) methods for large language models are predominantly designed by hand based on domain knowledge and heuristics, limiting their scalability and generality. We apply LLM-powered evolutionary search to automatically discover unsupervised UQ methods represented as Python programs. On the task of atomic claim verification, our evolved methods outperform strong manually-designed baselines, achieving up to 6.7% relative ROC-AUC improvement across 9 datasets while generalizing robustly out-of-distribution. Qualitative analysis reveals that different LLMs employ qualitatively distinct evolutionary strategies: Claude models consistently design high-feature-count linear estimators, while Gpt-oss-120B gravitates toward simpler and more interpretable positional weighting schemes. Surprisingly, only Sonnet 4.5 and Opus 4.5 reliably leverage increased method complexity to improve performance -- Opus 4.6 shows an unexpected regression relative to its predecessor. Overall, our results indicate that LLM-powered evolutionary search is a promising paradigm for automated, interpretable hallucination detector design.

Figures & tables

Figure 1: A visualization of LLM-powered evolutionary search pipeline, used to autonomously design uncertainty quantification (UQ) methods for hallucination detection tasks. At the start the candidate pool is initialized with a single selected baseline method.

Figure 2: Average ROC-AUC across 9 hallucination detection datasets (atomic factual claims). For Claude-generated candidates, top-30 methods were selected by validation performance on PopQA dataset. We report 5 of them with test-performance ranks 1, 8, 15, 22, and 30 respectively. For Gpt-oss-generated methods, we show every second method among top-16 by validation performance on PopQA.

	Avg PRR	BaBI	CoQA	MMLU	Truth fulQA	GSM8K	WMT 14 de	WMT 14 fr	WMT 19 de	WMT 19 ru	SAM Sum
RAUQ	54.6	79.2	80.2	75.9	37.2	88.5	34.7	38.8	45.8	33.4	32.4
SP $\times$ ExpectedRank	54.6	78.7	80.6	75.9	39.1	90.2	34.0	37.6	44.7	32.7	32.8
SP $\times$ MeanEntropy	54.6	78.0	79.8	75.8	39.0	90.2	34.2	38.0	45.4	33.0	32.9
Evo-Anth-30	54.5	78.1	79.0	75.8	39.3	90.1	34.0	37.8	45.1	32.7	32.8
Evo-Gpt-9	54.4	79.2	80.5	75.8	39.4	89.9	33.3	37.0	44.0	32.2	32.7
Evo-Gpt-7	54.4	79.1	80.5	75.8	39.4	89.9	33.3	37.0	44.0	32.2	32.7

Table 1: PRR scores across different methods and datasets (sorted by average rank). Blue and teal : Gpt-generated methods, orange : Claude-generated methods. Teal methods are evolved specifically for selective prediction, using training subsets of CoQA and SAMSum.

Figure 3: Left: average and median token lengths of atomic claims. Right: visualization of exponential and linear weighting for several typical claim lengths. Exponential weighting puts more emphasis on last tokens than linear weighting, especially for shorter sequences.

	AUC	95% CI
Sequence Probability	74.1	(72.7, 75.4)
Evolution (attention maps)	75.1	(73.8, 76.7)
Evolution (logits)	78.3	(76.9, 79.6)
Evolution (hidden states)	74.1	(72.7, 75.4)
Evolution, T = 0.7	78.3	(76.9, 79.6)
Evolution, T = 1.0	78.7	(77.4, 80.0)

Table 2: Ablation study, PopQA test split (candidates chosen by validation performance).

	PopQA (train)	PopQA (test)	AVeriTeC	X-Fact	UHead	Cities	Companies	Common Claim	Counter fact.	Wild Hall.
Evo-1	$0.84$	$0.83$	$0.68$	$0.53$	$0.68$	$0.99$	$0.89$	$0.69$	$0.78$	$0.65$
LogReg-1	$\underset{{\color[rgb]{0,0.4688,0}+0.02}}{0.86}$	$\underset{{\color[rgb]{0,0.4688,0}+0.01}}{0.84}$	$\underset{{\color[rgb]{1,0,0}-0.04}}{0.65}$	$\underset{{\color[rgb]{0,0.4688,0}+0.04}}{0.57}$	$\underset{{\color[rgb]{0,0.4688,0}+0.01}}{0.69}$	$\underset{{\color[rgb]{1,0,0}-0.01}}{0.98}$	$\underset{{\color[rgb]{1,0,0}-0.02}}{0.87}$	$\underset{{\color[rgb]{1,0,0}-0.03}}{0.67}$	$\underset{{\color[rgb]{1,0,0}-0.01}}{0.77}$	$\underset{{\color[rgb]{0.5,0.5,0.5}0.00}}{0.65}$
Evo-15	0.82	0.81	0.68	0.51	0.67	0.99	0.84	0.67	0.76	0.66
LogReg-15	$\underset{{\color[rgb]{0,0.4688,0}+0.03}}{0.85}$	$\underset{{\color[rgb]{0,0.4688,0}+0.02}}{0.83}$	$\underset{{\color[rgb]{1,0,0}-0.03}}{0.65}$	$\underset{{\color[rgb]{0.5,0.5,0.5}0.00}}{0.51}$	$\underset{{\color[rgb]{0,0.4688,0}+0.02}}{0.68}$	$\underset{{\color[rgb]{1,0,0}-0.01}}{0.99}$	$\underset{{\color[rgb]{0,0.4688,0}+0.06}}{0.90}$	$\underset{{\color[rgb]{0,0.4688,0}+0.01}}{0.68}$	$\underset{{\color[rgb]{0,0.4688,0}+0.03}}{0.80}$	$\underset{{\color[rgb]{1,0,0}-0.02}}{0.63}$

Table 3: Results of training logistic regressions on feature sets of Evo-Anth-1 and Evo-Anth-15 canditates. LogReg-X corresponds to a logistic regression employing Evo-X features. Notable, weights found via evolutionary search generalize slightly better to OOD datasets, despite both approaches only used PopQA train set. Displayed deltas sometimes disagree with AUC values due to rounding; in these cases deltas are more precise.

Figure 4: Correlation between method complexity and performance for different LLMs. Each dot corresponds to one evolution run.

Figure 5: Evolution dynamics in (complexity, performance) coordinates for 6 different models. We use line count as the simplest proxy for method complexity; other proxies such as amount of AST nodes, number of binary and unary operators, or Halstead volume follow the same pattern.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Methods’ similarity to SequenceProbability (measured as Spearman correlation between predictions of method and SequenceProbability) versus test performance. Notably, there exist methods which have accuracy comparable to SequenceProbability, but producing significantly different sample rankings (with Spearman correlation even less than 0).

Figure 7: Best autonomously designed methods deliver statistically significant improvement over the strongest baseline (SequenceLogProb) on 6/9 datasets, and tie on one. We use paired bootstrap difference test with Bonferroni correction. Orange methods are designed using Claude models, and blue methods are designed using Gpt-oss-120B.

Uncertainty Quantification

Uncertainty quantification estimates how reliable predictions and scientific inferences are, and how uncertainty propagates into decisions.

See all 649 papers

The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models
Jun 22, 2026Xiang-Jun Ou, Shuang Liang, Xin-Yu Hu +3LLM Evaluation Uncertainty Quantification
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210063, China. · School of Intelligent Science and Technology, Nanjing University, Suzhou 215163, China.
Position: Uncertainty Quantification in LLMs is Just Unsupervised Clustering
May 19, 2026Tiejin Chen, Longchao Da, Xiaoou Liu +1LLM Evaluation Uncertainty Quantification
School of Computing and Augmented Intelligence, Arizona State University
When Do Supervised UQ Ensembles Improve LLM Hallucination Detection? A Robustness Study
Aug 25, 2026Mohit Singh Chauhan, Vipin Gyanchandani, Dylan BouchardLLM Hallucination Detection LLM Uncertainty Estimation
CVS Health®, Wellesley, MA, USA

Jun 22, 2026J/K move · Enter open · S save

The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models

Xiang-Jun Ou, Shuang Liang, Xin-Yu Hu, Rong-Hao Huang+2

National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing 210063, China. · School of Intelligent Science and Technology, Nanjing University, Suzhou 215163, China.

LLM Evaluation Uncertainty Quantification LLM Uncertainty Estimation

Open paper AskPDF arXiv

Recent advancements in Large Language Models (LLMs) have enabled sophisticated reasoning and content generation, yet their inherent stochasticity poses significant challenges for ensuring predictive credibility. While traditional uncertainty taxonomy paradigms, such as the dichotomy of aleatoric and epistemic uncertainties, provide conceptual foundations, they often fail to capture the multi-component and multi-stage nature of LLM generation and struggle to evaluate the effectiveness of various Uncertainty Quantification (UQ) methods. In this paper, we propose a granular uncertainty taxonomy that systematically attributes LLM uncertainty into input-level, parameter-level, token-level, and decoding-process sources. Correspondingly, we categorize existing UQ methods into Bayesian, ensemble, consensus-based, and single-pass approaches. Furthermore, we introduce a comprehensive evaluation framework covering diverse generation settings and metrics. We empirically evaluate 21 typical UQ methods across three prominent LLM families, including Qwen3, Llama 3.2, and DeepSeek-V3, on benchmarks such as TriviaQA, GSM8K, and HumanEval. Our experimental results demonstrate that (i) the effectiveness of UQ methods is sensitive to task types and generation settings; (ii) consensus-based methods, typed Deg and EigV, consistently outperform other UQ approaches; and (iii) larger model scales correlate with lower uncertainty estimates, suggesting an empirical scaling law for LLM uncertainty. This work bridges the gap between theoretical origins and practical deployment, providing a versatile diagnostic tool for systematically quantifying uncertainty in LLM applications.

Abstract

Uncertainty Quantification

Explore similar work

The Origins of Stochasticity: Comprehensive Investigations on Uncertainty Quantification for Large Language Models

Position: Uncertainty Quantification in LLMs is Just Unsupervised Clustering

When Do Supervised UQ Ensembles Improve LLM Hallucination Detection? A Robustness Study