S-EDL: Eliciting Self-Evidence from Sequence Likelihoods for Semantic Calibration of LLMs

Yawei Li; Jiazheng Li; David Rügamer; Bernd Bischl; Yulan He; Mina Rezaei

cs.CLOct 21, 2026

S-EDL: Eliciting Self-Evidence from Sequence Likelihoods for Semantic Calibration of LLMs

Authors: Yawei Li, Jiazheng Li, David Rügamer, Bernd Bischl, Yulan He, Mina Rezaei

Organizations: LMU Munich · Munich Center for Machine Learning (MCML) · King’s College London

Abstract

Calibrating large language models (LLMs) in open-ended generation is uniquely challenging because many distinct token sequences can express the same underlying meaning. Prior calibration techniques target either fixed-set classification confidence or the binary correctness of individual strings, making them inherently ill-suited for the dynamic, multi-string nature of open-ended semantic outcomes. To bridge this gap, we introduce S-EDL, a novel evidential method that natively optimizes LLMs for semantic calibration. By eliciting evidence from the sequence likelihoods of the model itself, S-EDL constructs a differentiable Dirichlet prior over prompt-specific semantic classes. Optimizing the evidential loss under this prior yields a calibration-aware training signal over variable semantic classes, completely bypassing the need for fixed multiple-choice options. Evaluations across four open-ended benchmarks and three models demonstrate that S-EDL substantially reduces semantic calibration error while improving accuracy. By aligning generative likelihoods with semantic correctness, this work establishes a framework for deploying trustworthy LLMs in complex, open-ended applications. Code is available at: https://github.com/sandylaker/s-edl.

Figures & tables

Method	GSM8K			MATH-500			ChemistryQA			FormulaReasoning			Average
Method	Pass@1 $\uparrow$	S-Acc. $\uparrow$	smECE $\downarrow$	Pass@1 $\uparrow$	S-Acc. $\uparrow$	smECE $\downarrow$	Pass@1 $\uparrow$	S-Acc. $\uparrow$	smECE $\downarrow$	Pass@1 $\uparrow$	S-Acc. $\uparrow$	smECE $\downarrow$	Pass@1 $\uparrow$	S-Acc. $\uparrow$	smECE $\downarrow$
Fine-tuned model: Qwen3-8B-Base
SFT	70.8 1.2	93.4 0.4	20.2 0.4	49.4 2.8	68.7 2.7	16.1 0.6	47.9 0.7	52.9 1.4	20.7 0.6	75.6 0.8	87.5 0.3	8.9 0.3	60.9	75.6	16.5
RCFT	79.1 0.2	92.8 0.1	12.9 0.2	50.1 0.3	71.1 1.4	17.5 1.6	49.7 3.9	55.4 4.3	15.1 1.5	78.4 0.3	89.5 0.2	8.9 0.1	64.3	77.6	13.6
LS	79.1 1.0	92.7 0.2	12.5 0.6	49.6 0.2	70.7 0.7	17.5 0.7	50.7 0.7	58.1 0.3	11.8 0.7	76.8 1.0	88.9 0.4	9.4 0.8	64.0	77.6	12.8
BLoB	79.1 0.1	92.5 0.9	12.4 0.3	49.9 0.3	69.5 1.5	15.9 1.6	52.4 0.9	57.9 0.2	14.9 1.3	78.4 1.3	90.3 0.6	10.6 0.1	64.9	77.6	13.4

Table 1: Calibration and accuracy across four open-ended question answering datasets. We report the single-sequence metric Pass@1 (

\uparrow

) and two semantic-level metrics: Semantic Accuracy (S-Acc.,

\uparrow

) and Smooth ECE (smECE,

\downarrow

). Entries show means over three seeds with standard deviations as smaller subscripts (

\text{mean}_{\text{std}}

). The Average columns report means across the four datasets. The best and second-best results are highlighted.

Method	MATH $\rightarrow$ GSM8K			MATH $\rightarrow$ ChemistryQA			MATH $\rightarrow$ FormulaReasoning			Average
Method	Pass@1 $\uparrow$	S-Acc. $\uparrow$	smECE $\downarrow$	Pass@1 $\uparrow$	S-Acc. $\uparrow$	smECE $\downarrow$	Pass@1 $\uparrow$	S-Acc. $\uparrow$	smECE $\downarrow$	Pass@1 $\uparrow$	S-Acc. $\uparrow$	smECE $\downarrow$
Fine-tuned model: Qwen3-8B-Base
SFT	79.1 1.9	92.8 1.7	14.7 2.0	38.9 0.5	49.5 0.6	7.5 0.3	50.4 0.4	72.0 1.3	16.8 1.9	56.1	71.4	13.0
RCFT	77.2 0.4	93.0 1.6	15.6 1.2	38.2 1.7	49.8 1.3	6.6 0.8	47.6 1.7	76.5 0.2	22.1 0.8	54.3	73.1	14.8
LS	78.4 1.7	93.7 0.5	14.6 0.2	38.9 0.5	51.3 0.7	4.9 1.0	45.2 1.2	74.3 1.4	22.7 1.7	54.2	73.1	14.1
BLoB	75.5 0.8	94.8 0.3	16.5 1.0	38.6 1.2	51.3 0.7	5.9 0.4	49.4 0.6	77.6 0.9	21.4 0.7	54.5	74.6	14.6

Table 2: Cross-domain evaluation. A

\rightarrow

B denotes training on dataset A and evaluating on dataset B.

Figure 1: Violin plots of per-token probability profiles on MATH-500 generated answers. We record the probabilities of the sampled tokens. Solid and dotted black ticks mark the mean and median, respectively. Logarithm is applied to the violin widths to improve readability.

Panel of Figure 2 — Figure 2: Ablation study on $\beta$ and $\lambda_{\text{MSE}}$ with Qwen3-8B-Base trained and evaluated on GSM8K. The bottom $x$ -axis varies $\beta$ with $\lambda_{\text{MSE}}=5.0$ held fixed; the top $x$ -axis varies $\lambda_{\text{MSE}}$ with $\beta=1.2$ fixed.

Panel of Figure 3 — Figure 3: Ablation study on the number of Monte-Carlo samples $N_{\text{train}}$ with Qwen3-8B-Base trained and evaluated on GSM8K. The bottom $x$ -axis varies $N_{\text{train}}$ with $\beta=1.2$ and $\lambda_{\text{MSE}}=5.0$ held fixed.

Panel of Figure 4 — Figure 4: Empirical cost of find-first semantic clustering on math tasks. Bars show the mean number of symbolic verifier calls per question when clustering $N=50$ generated sequences, with error bars denoting one standard deviation. The dashed red line marks the naive pairwise baseline of $N(N-1)/2=1225$ calls per question. The find-first algorithm achieves a substantial reduction in computational cost, bringing the empirical cost much closer to $O(N)$ . Specifically, it averages roughly $1.3N$ calls on GSM8K and $3.3N$ calls on MATH. The reduction is smaller on MATH, where harder problems produce more distinct semantic classes and therefore require more comparisons.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Model & LoRA
Hyperparameter	GSM8K	MATH	ChemistryQA	FormulaReasoning
Model max_seq_length	2048	4096	4096	4096
LoRA $r$ / $\alpha$ / dropout	16 / 32 / 0.0	16 / 32 / 0.0	16 / 32 / 0.0	16 / 32 / 0.0
LoRA target modules	Q,K,V,O,gate,up,down	Q,K,V,O,gate,up,down	Q,K,V,O,gate,up,down	Q,K,V,O,gate,up,down
LoRA bias	none	none	none	none
Precision	bf16	bf16	bf16	bf16

Appendix

Table 3: EDL fine-tuning hyperparameters for Qwen3-8B-Base across the four evaluation datasets.

Model & LoRA
Hyperparameter	GSM8K	MATH	ChemistryQA	FormulaReasoning
Model max_seq_length	2048	4096	4096	4096
LoRA $r$ / $\alpha$ / dropout	16 / 32 / 0.0	16 / 32 / 0.0	16 / 32 / 0.0	16 / 32 / 0.0
LoRA target modules	Q,K,V,O,gate,up,down	Q,K,V,O,gate,up,down	Q,K,V,O,gate,up,down	Q,K,V,O,gate,up,down
LoRA bias	none	none	none	none
Precision	bf16	bf16	bf16	bf16

Appendix

Table 4: EDL fine-tuning hyperparameters for Ministral-3-8B-Base across the four evaluation datasets.

Model & LoRA
Hyperparameter	GSM8K	MATH	ChemistryQA	FormulaReasoning
Model max_seq_length	2048	4096	4096	4096
LoRA $r$ / $\alpha$ / dropout	16 / 32 / 0.0	16 / 32 / 0.0	16 / 32 / 0.0	16 / 32 / 0.0
LoRA target modules	Q,K,V,O,gate,up,down	Q,K,V,O,gate,up,down	Q,K,V,O,gate,up,down	Q,K,V,O,gate,up,down
LoRA bias	none	none	none	none
Precision	bf16	bf16	bf16	bf16

Appendix

Table 5: EDL fine-tuning hyperparameters for MiMo-7B-Base across the four evaluation datasets.

Panel of Figure 5 — Figure 5: Average in-distribution performance across varying temperatures. Experiment settings are consistent with Section 5.2 , except for the various temperatures. Note that certain data points, such as those for LS, may fall outside the plotted area because the y-axis limits are clamped to ensure better readability and visual clarity.

Panel of Figure 6 — Figure 6: Accuracy–calibration plane, averaged across the four datasets. Bottom-right is the best corner (high S-Acc, low smECE).

Panel of Figure 7 — Figure 7: Token probability distributions on GSM8K, ChemistryQA, and FormulaReasoning benchmarks.

Panel of Figure 8 — Figure 8: Ablation study on $\beta$ and $\lambda_{\text{MSE}}$ with Ministral3-8B-Base trained on MATH and evaluated on MATH-500. The bottom $x$ -axis varies $\beta$ with $\lambda_{\text{MSE}}=5.0$ held fixed; the top $x$ -axis varies $\lambda_{\text{MSE}}$ with $\beta=1.2$ fixed. The default values $\beta=1.2$ and $\lambda_{\text{MSE}}=5.0$ demonstrate strong performance and are used throughout all experiments.

Language Model Calibration

Language model calibration studies whether models’ confidence estimates match their actual accuracy across tasks, prompts, and deployment settings.

See all 386 papers

A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering
May 8, 2026Zhanliang Wang, Jiancong Xiao, Ruochen Jin +3LLM Evaluation Confidence Estimation in Language Models
University of Pennsylvania, Philadelphia, PA · Dartmouth College, Hanover, NH
Self-Calibrating Language Models via Test-Time Discriminative Distillation
Mar 18, 2026Mohamed Rissal Hedna, Jan Strich, Martin Semmann +1Inference-Time Adaptation of Language Models Language Model Calibration
Hub of Computing and Data Science (HCDS), University of Hamburg, Germany
Calibrating LLMs with Semantic-level Reward
May 15, 2026Fengfei Yu, Ruijia Niu, Dongxia Wu +2Language Model Calibration LLM Reliability
Department of Computer Science and Engineering, University of California San Diego, La Jolla, California, USA · Halıcıoğlu Data Science Institute, University of California San Diego, La Jolla, California, USA · Department of Statistics, Stanford University, Stanford, California, USA

May 8, 2026J/K move · Enter open · S save

A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering

Zhanliang Wang, Jiancong Xiao, Ruochen Jin, Shu Yang+2

University of Pennsylvania, Philadelphia, PA · Dartmouth College, Hanover, NH

LLM Evaluation Confidence Estimation in Language Models Language Model Calibration Model Calibration LLM Reliability LLM Uncertainty Estimation

Open paper AskPDF arXiv

Calibration measures whether a model's predicted confidence aligns with its empirical accuracy, and is central to the reliable deployment of large language models (LLMs) in high-stakes domains such as medicine and law. While much recent work focuses on improving LLM calibration, the equally important question of how to evaluate it in realistic settings remains underdeveloped. Open-ended question answering (QA), the most common deployment setting for modern LLMs, is where existing evaluation methods fall short: logit-based metrics need restricted output formats and internal probabilities; verbalized confidence is self-reported and often overconfident; and sampling-based methods rely on task-specific extraction rules without a clear finite-sample target. We introduce Sem-ECE (Semantic-Sampling Expected Calibration Error), a calibration evaluation framework for open-ended QA that samples answers from the model, groups them into semantic classes, and uses the resulting frequencies as confidence. We study two estimators within this framework: Sem

_1

-ECE, the same-sample self-consistency score, and Sem

_2

-ECE, a held-out variant that separates answer selection from confidence evaluation. We prove both are asymptotically unbiased, and further show that they agree on easy questions but diverge on hard ones with Sem

_2

achieving strictly smaller calibration error, so their gap also serves as a diagnostic for question difficulty. Experiments on three open-ended QA benchmarks across five leading commercial LLMs match our theoretical predictions and show that Sem-ECE outperforms verbalized confidence and existing sampling-based methods, while complementing logit-based evaluation when internal probabilities are unavailable.

Abstract

Language Model Calibration

Explore similar work

A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering

Self-Calibrating Language Models via Test-Time Discriminative Distillation

Calibrating LLMs with Semantic-level Reward