Conformal Prediction with Paraphrase-Aware Scoring for LLM Uncertainty Quantification

Jiayi Xin; Evan Qiang; Zihan Zhu; Xiang Li; Weijie J. Su; Qi Long

cs.CLOct 3, 2026

Conformal Prediction with Paraphrase-Aware Scoring for LLM Uncertainty Quantification

Authors: Jiayi Xin, Evan Qiang, Zihan Zhu, Xiang Li, Weijie J. Su, Qi Long

Organizations: University of Pennsylvania

Abstract

Uncertainty quantification (UQ) for large language models (LLMs) aims to provide reliable measures of predictive confidence, yet current methods are often unstable under meaning-preserving perturbations. Semantically equivalent paraphrases can induce substantial variability in predictive confidence, even for methods with formal guarantees, such as conformal prediction. To address this issue, we propose a paraphrase-aware UQ framework robust to semantic rewordings. Our approach trains a lightweight proxy model on LLM hidden states and aggregates its predictions across paraphrases to construct label-wise nonconformity scores. Under score exchangeability, conformal calibration retains marginal coverage. This guarantee can also hold under test-only rewording, provided that the paraphrase pipeline satisfies an additional distributional alignment condition. We evaluate three settings (normal, fully reworded, and semi-reworded) which apply rewording to neither dataset, both calibration and test datasets, or only the test dataset, respectively. Across seven multiple-choice QA benchmarks and multiple model families, our method produces compact prediction sets with empirical coverage generally near the nominal target, even in the semi-reworded setting. Ablation studies show that the learned proxy accounts for most of the reduction in set size, while paraphrase-augmented training and inference-time aggregation improve stability under rewording. Code is available at https://github.com/Raina-Xin/PA_Score.

Figures & tables

Figure 1: Prior scores are sensitive to rewording or paraphrasing of input question (denoted as $\mathcal{T}$ ), producing larger prediction sets (Panel A and B). Our PA framework aggregates multiple paraphrases with a proxy model to reduce score and prediction-set sensitivity to rewording (Panel C and D).

Figure 2: Schematic illustration of our proposed PA score integrated with CP.

Score	Definition	Interpretation	Properties
Mean	$\frac{1}{m}\sum_{x^{\prime}\in\mathcal{B}(x)}S_{\mathrm{prob}}(x^{\prime},y)$	Captures average semantic stability	Most efficient; may miss rare paraphrases
Weighted	$\frac{\sum_{x^{\prime}\in\mathcal{B}(x)}w(\mathcal{T}(x),x^{\prime})\,S_{\mathrm{prob}}(x^{\prime},y)}{\sum_{x^{\prime}\in\mathcal{B}(x)}w(\mathcal{T}(x),x^{\prime})}$	Emphasizes semantically similar paraphrases	Balances compactness and robustness
Worst-case	$\max_{x^{\prime}\in\mathcal{B}(x)}S_{\mathrm{prob}}(x^{\prime},y)$	Sensitive to hardest paraphrase	Robust but conservative; sensitive to low-quality paraphrases

Table 1: Paraphrase-aware nonconformity scores. The weight is defined as

w(x,x^{\prime})=1/(1+\|E(x)-E(x^{\prime})\|_{2})

, an inverse-distance weighting in the embedding space, so paraphrases closer to the original input receive larger weights. We use

\mathcal{T}(\cdot)

to denote the paraphrase operator.

Setting.	Calibration Data	Test Sample	Exchangeable
Normal	$\{(X_{i},Y_{i})\}_{i=1}^{n}$	$(X_{n+1},Y_{n+1})$	✓
Full	$\{(\mathcal{T}(X_{i}),Y_{i})\}_{i=1}^{n}$	$(\mathcal{T}(X_{n+1}),Y_{n+1})$	✓
Semi	$\{(X_{i},Y_{i})\}_{i=1}^{n}$	$(\mathcal{T}(X_{n+1}),Y_{n+1})$	✓ (*)

Table 2: Summary of score exchangeability in Lemma 5.1 where

\{(X_{i},Y_{i})\}_{i=1}^{n+1}

is i.i.d. Here, (*) requires Asm. 5.1 .

Coverage Rate (CR, %) Better if closer to $90\%$ — Split CP
	QA		RC		CI		DRS		DS
Method	Normal	Full	Normal	Full	Normal	Full	Normal	Full	Normal	Full
LAC	$89.60$	$98.83$	90.53	$90.47$	$89.73$	90.03	$99.00$	$99.10$	$89.27$	$89.10$
APS	$99.73$	$98.90$	$99.07$	$99.20$	$98.90$	$99.00$	$100.00$	$99.97$	$100.00$	$98.50$
SPUQ-adapt.	$90.37$	$89.77$	$91.27$	90.20	89.87	$89.77$	90.90	$91.13$	89.80	89.73
TRON-adapt.	89.77	$90.40$	90.53	$90.67$	$89.80$	$90.20$	$90.93$	$91.53$	$89.53$	$89.70$

Table 3: Performance under split CP and QCCP with Qwen3-8B on normal and fully reworded (Full) test sets across five benchmarks. For coverage rate (CR, %), bold indicates values closest to the nominal

90\%

target. For set size (SS,

\downarrow

), bold indicates the smallest value.

	Method	QA	RC	CI	DRS	DS	QA	RC	CI	DRS	DS
	Semi.	Coverage Rate (CR, %) Better if closer to $90\%$					Set Size (SS) $\downarrow$
Split CP	LAC	89.83	88.60	89.20	99.10	88.23	2.54	1.62	1.69	5.72	2.61
	APS	99.47	98.90	99.00	100.00	100.00	5.74	5.57	5.61	6.00	5.97
	SPUQ-adapt.	84.83	87.93	87.30	86.23	88.20	2.21	1.53	1.60	2.72	2.49
	TRON-adapt.	90.10	88.60	89.23	91.40	88.70	2.55	1.62	1.69	3.36	2.66
	PA	92.07	92.20	93.23	95.90	94.23	2.26	1.31	1.56	1.70	1.00

Table 4: Performance under the semi-reworded (Semi.) setting. Bold numbers indicate the best.

		Llama-3.1-8B						Phi-3-small
		CR (%)			SS			CR (%)			SS
	Method	Normal	Semi.	Full	Normal	Semi.	Full	Normal	Semi.	Full	Normal	Semi.	Full
Split CP	LAC	88.97	86.50	88.87	2.63	2.75	2.95	91.03	88.67	89.03	5.65	5.55	5.57
	APS	95.87	94.90	94.93	3.81	3.97	3.98	99.53	99.67	99.67	5.98	5.98	5.98
	SPUQ-adapt.	88.50	85.40	89.63	2.68	2.69	3.04	90.33	84.47	90.13	5.61	5.38	5.61
	TRON-adapt.	89.00	86.53	88.93	2.63	2.75	2.95	90.00	87.17	89.37	5.60	5.49	5.58

Table 5: Cross-LLM generalization on MMLU. Coverage Rate (CR, %) closer to

90\%

is better; Set Size (SS)

\downarrow

is better. Results are shown for normal, semi-reworded (Semi.), and fully reworded (Full).

Figure 3: The proxy improves both task performance and calibration consistently. Evaluation of prediction accuracy and calibration metric using proxy model vs. raw LLM logits across datasets. Left: accuracy (higher is better). Middle: Brier (lower is better). Right: NLL (lower is better).

Score and proxy training	Inference	Rewrite Acc. $\uparrow$	Semi CR	Semi SS $\downarrow$	Set change $\downarrow$
Raw probabilities; no proxy	Single view	65.74	91.36	2.8929	0.3561
Raw-probability proxy; Orig-only	Single view	70.16	89.74	2.2032	0.2369
Hidden-state proxy; Orig-only	Single view	79.80	89.58	1.3725	0.2009
Hidden-state proxy; augmented	Single view	80.38	89.96	1.3587	0.2029
Full PA; same augmented proxy	Six-view $S_{\rm mean}$	80.34	89.81	1.3331	0.1708
Raw probabilities; no proxy	Six-view mean	65.53	86.11	2.0394	0.5785

Table 6: Factorized attribution on Qwen3-8B (five datasets, three seeds; equal-weight mean over 15 dataset–seed results). Rewrite Acc. is label accuracy on rewritten inputs. Semi CR and SS are empirical coverage (%) and average set size under split CP in the semi-reworded setting at

\alpha=0.1

. Set change is the mean number of labels whose membership differs between the prediction sets for an original input and its rewrite,

|\widehat{C}(x)\triangle\widehat{C}(\mathcal{T}(x))|

. The Full PA and augmented-proxy single-view rows use the same trained proxy checkpoint; only inference-time aggregation differs.

Figure 4: Evaluation of class-conditional coverage with QCCP on the CosmosQA dataset (left) and HaluDial dataset (right). Results on additional datasets are provided in Figure 9 .

Panel of Figure 6 — Figure 6: Comparison of different scores under QCCP.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

Setting / Hyperparameter		QA, RC, CI, DRS, DS
General
	Base LLM (for reps/logits)	Qwen3-8B
	Random seed	42
	Samples per question ( $n$ )	7 (1 base + 6 paraphrases)
	Options per item ( $\|\mathcal{Y}\|$ )	6 (A–F)
Proxy model (training)

Appendix

Table 8: Hyperparameters for QA, RC, CI, DRS, DS. We train an MLP proxy with ECE regularization and compute paraphrase-aware scores using 6 paraphrases per question. Evaluation includes QCP with SBERT-clustered

\Phi

and plain CP with intercept

\Phi

, using the same manifest-based splits and

\alpha{=}0.1

General
Setting / Hyperparameter		MedMCQA-10k	MedQA-10k
	Base LLM (for reps/logits)	Qwen3-8B
	Random seed	42
	Samples per question ( $n$ )	7 (1 base + 6 paraphrases)
	Options per item ( $K$ )	6 (A–F)
Proxy model (training)

Appendix

Table 9: Hyperparameters for MedMCQA-10k and MedQA-10k. We train an MLP proxy with ECE regularization and compute paraphrase-aware scores using 6 paraphrases per question. Evaluation includes QCP with SBERT-clustered

\Phi

and plain CP with intercept

\Phi

, using the same manifest-based splits and

\alpha{=}0.1

Dataset	Hidden dimension $h$	Batch size	Learning rate	Weight decay	$\lambda_{\text{ECE}}$
MMLU	256	64	1e-3	0.0	0.5
CosmosQA	512	128	1e-3	0.0001	0.5
HellaSwag	256	128	1e-3	0.0001	0.5
HaluDial	256	64	1e-3	0.0	0.5
HaluSum	256	128	1e-4	0.0	0.5

Appendix

Table 10: Proxy model specific hyperparameters for the 5 general datasets

Dataset	Hidden dimension $h$	Batch size	Learning rate	Weight decay	$\lambda_{\text{ECE}}$
MedMCQA	256	64	1e-3	0.0	0.5
MedQA	256	64	1e-3	0.0	0.5

Appendix

Table 11: Proxy model specific hyperparameters for MedMCQA and MedQA.

		MedMCQA						MedQA
		CR (%)			SS			CR (%)			SS
	Method	Normal	Semi.	Full	Normal	Semi.	Full	Normal	Semi.	Full	Normal	Semi.	Full
Split CP	LAC	99.03	98.90	90.20	5.64	5.63	2.74	98.97	98.73	99.40	5.81	5.82	5.90
	APS	98.97	98.93	98.90	5.64	5.65	5.65	99.97	100.00	100.00	6.00	6.00	6.00
	SPUQ-adapt.	90.10	86.37	91.27	2.59	2.34	2.86	89.97	81.83	88.80	4.05	3.72	4.43
	TRON-adapt.	90.17	87.73	91.03	2.46	2.50	2.81	90.47	84.97	89.00	3.97	3.99	4.35

Appendix

Table 12: Results on two medical QA benchmarks with Qwen3-8B . Coverage rate (CR, %) closer to the nominal

90\%

is better, while smaller set size (SS,

\downarrow

) denotes better. Results are shown for normal, semi-reworded (Semi.), and fully reworded (Full) inputs.

Dataset	LAC	APS	PA
QA (MMLU)	0.0555	0.0687	0.0461
CI (HellaSwag)	0.0402	0.0860	0.0293
DRS (HaluDial)	0.0787	0.0476	0.0262

Appendix

Table 13: Cluster-size-weighted coverage deviation (WCD,

\downarrow

) under QCCP in the semi-reworded setting on Qwen3-8B . Lower values indicate more uniform alignment with the nominal

1-\alpha=0.9

target. Bold marks the best method per dataset.

	Dataset	RC	CI	RC	CI
	Semi.	Coverage Rate (CR)		Set Size (SS)
Split CP	LAC	87.17	86.97	3.51	3.91
	APS	99.33	90.90	5.85	4.79
	PA	89.97	90.27	1.41	2.03
QCCP	LAC	87.00	87.77	3.51	3.98
	APS	97.40	95.10	5.41	5.33

Appendix

Table 14: Performance of nonconformity scores under the semi-reworded (Semi.) setting on Llama-3.1-8B as paraphrase generator. Coverage Rate (CR, %) closer to

90\%

is better; Set Size (SS)

\downarrow

is better. Results are reported on RC (Cosmos) and CI (HellaSwag). Bold numbers indicate the best performance.

Dataset	Semi CR (%)	Error from 90% (pp)	$\Delta_{\hat{q}}$ (pp)	KS
MMLU	85.5	$-4.5$	$-4.5$	0.059
CosmosQA	89.1	$-0.9$	$-1.1$	0.052
HellaSwag	89.5	$-0.5$	$-0.5$	0.084
HaluDialogue	87.6	$-2.4$	$-2.7$	0.135
HaluSummarization	93.1	$+3.1$	$+2.9$	0.085

Appendix

Table 15: Paired fixed-clean-cutoff diagnostics on Qwen3-8B with

S_{\mathrm{mean}}

and split CP at

\alpha=0.1

. Each row is the mean over seeds 7, 42, and 123. The proxy and clean calibration cutoff are held fixed while the same test questions are evaluated before and after rewriting. Coverage error is relative to the nominal

90\%

target;

\Delta_{\hat{q}}

is relative to clean empirical coverage.

RC (CosmosQA)
Scenario / Model	Cosine	BLEU	ROUGE-L
$\mathcal{T}^{*}(x)$ — paraphrases for human-written inputs
Llama	0.82 $\pm$ 0.10	16.68 $\pm$ 8.57	45.18 $\pm$ 12.02
Qwen3	0.81 $\pm$ 0.10	16.06 $\pm$ 8.60	42.19 $\pm$ 11.64
$\mathcal{T}(x)$ — LLM-reworded inputs
Llama	0.70 $\pm$ 0.21	11.94 $\pm$ 13.24	35.84 $\pm$ 19.98

Appendix

Table 16: Paraphrase quality evaluation using Llama-3.1-8B and Qwen3-8B as paraphrase generators. We report SBERT cosine similarity, BLEU, and ROUGE-L (mean

\pm

std) for three paraphrase scenarios:

\mathcal{T}^{*}(x)

\mathcal{T}(x)

, and

\mathcal{T}^{*}(\mathcal{T}(x))

Dataset	Batch	Raw LAC	Aug-Proxy Single	Full PA $m{=}2$	Full PA $m{=}6$	Cached $m{=}6$
MMLU	1	19.51	18.73	1599.76	1626.58	0.200
CosmosQA	1	24.14	23.52	1565.11	1611.55	0.195
MMLU	16	9.63	9.57	187.54	357.37	0.056
CosmosQA	16	10.78	10.74	150.32	245.09	0.056

Appendix

Table 17: End-to-end latency (ms/question) on one NVIDIA B200. Batch denotes the inference batch size;

m

is the number of paraphrases. Cached Full PA excludes paraphrase generation and backbone representation extraction.

Figure 7: Effect of removing the proxy model under the split CP setting on Qwen3-8B . Coverage remains close to

90\%

(dashed line), but set size increases notably without the proxy model.

Figure 8: Effect of removing the proxy model under the quasi-conditional CP setting. Coverage remains close to

90\%

(dashed line), but set size increases notably without the proxy model.

Semi-reworded, Split CP
	Variant	QA (MMLU)	RC (Cosmos)	CI (HellaSwag)	DRS (HaluDial)	DS (HaluSum)
	Para + Proxy (Full)	87.10 / 1.85	90.10 / 1.12	89.13 / 1.28	89.67 / 1.36	94.13 / 1.00
	Para only	84.83 / 2.21	87.93 / 1.53	87.30 / 1.60	86.23 / 2.72	88.20 / 2.49
	Aug-Proxy Single	89.53 / 2.02	90.47 / 1.14	89.37 / 1.29	90.87 / 1.43	95.20 / 1.00
Semi-reworded, QCCP
	Para + Proxy (Full)	89.50 / 2.01	90.67 / 1.39	89.53 / 1.36	91.10 / 1.70	94.20 / 1.11

Appendix

Table 18: Single-run attribution of the PA score on Qwen3-8B . Each cell reports CR / SS (coverage rate, %, closer to

90

is better; set size

\downarrow

is better). Para+Proxy denotes the full method, Para only removes the proxy, and Aug-Proxy Single removes inference-time aggregation while retaining paraphrase-augmented training. Bold entries highlight severe under- or over-coverage exhibited by single-component variants. These single-run rows retain the paraphrase-trained proxy in the single-view arm and are kept separate from the three-seed controls in Table 6 and Table 22 ; they are not pooled with those later three-seed controls.

Proxy training	Rewrite Acc. $\uparrow$	Semi CR	Semi SS $\downarrow$	Score drift $\downarrow$	Set change $\downarrow$
Standard Orig-only	79.80	89.58	1.3725	0.0596	0.2009
Repeat-Orig	79.64	89.27	1.3717	0.0635	0.2188
Paraphrase-augmented	80.38	89.96	1.3587	0.0574	0.2029

Appendix

Table 19: Training-view control for the hidden-state proxy on Qwen3-8B , averaged equally over five datasets and three seeds. All rows use single-view inference. Score drift is the mean absolute clean/rewrite change in the true-label nonconformity score. Repeat-Orig controls for repeated training updates; standard Orig-only is the primary no-paraphrase training baseline.

Dataset	Single CR (%)	Full PA CR (%)	Full PA $-$ Single SS
MMLU	89.81	89.43	$-0.0022$
CosmosQA	91.27	91.03	$-0.0233$
HellaSwag	89.90	89.43	$-0.0404$
HaluDialogue	90.12	90.18	$-0.0178$
HaluSummarization	89.27	89.36	$+0.0002$

Appendix

Table 20: Held-out, approximately coverage-matched comparison in the Semi setting. The coverage columns report achieved coverage on the held-out evaluation half. Negative Full PA minus Single set-size differences favor Full PA. This is a separate operating-point analysis, not the fixed-

\alpha

experiment.

Proxy and cutoff fitted on	Aug-Proxy Single	Full PA	Relative change
Original questions	$0.1994\pm 0.0082$	$0.1737\pm 0.0089$	$-12.9\%$
Reworded questions	$0.2213\pm 0.0004$	$0.2098\pm 0.0043$	$-5.2\%$

Appendix

Table 21: Paired prediction-set stability in the same separate study. Set change is the mean symmetric-difference size

|\widehat{C}(x)\triangle\widehat{C}(\mathcal{T}(x))|

, the number of labels entering or leaving a set under rewriting. Lower is better. Values are five-dataset macro means

\pm

sample standard deviation across seeds 7, 42, and 123.

Setting	Aug-Proxy Single: CR / SS	Full PA: CR / SS
Normal	$90.58\pm 0.16\;/\;1.293\pm 0.007$	$90.50\pm 0.06\;/\;1.299\pm 0.005$
Semi	$90.40\pm 0.18\;/\;1.329\pm 0.008$	$89.56\pm 0.31\;/\;1.286\pm 0.006$
Fully reworded	$90.59\pm 0.37\;/\;1.352\pm 0.012$	$90.17\pm 0.31\;/\;1.338\pm 0.010$

Appendix

Table 22: Separate same-checkpoint study of inference-time aggregation under split CP at

\alpha=0.1

. Both variants use the same paraphrase-augmented proxy. CR is empirical coverage in percent and SS is mean prediction-set size. Values are five-dataset macro means

\pm

sample standard deviation across seeds 7, 42, and 123.

Figure 9: Evaluation of class-conditional coverage with QCCP on the MMLU (left) and HaluSum (right) datasets under Qwen3-8B .

Panel of Figure 10 — Figure 10: Coverage rate (left) and prediction set size (right) vs. $\alpha$ on MMLU under Split CP, with Qwen3-8B as the backbone.

Coverage Rate (CR, %) Better if closer to $90\%$ — Split CP
	RC (Cosmos)		CI (HellaSwag)		MedMCQA
Method	Normal	Full	Normal	Full	Normal	Full
LAC	90.40	99.00	89.83	89.87	89.42	99.20
APS	99.27	99.10	99.63	98.90	90.92	99.97
PA	90.13	90.60	90.13	90.67	91.03	89.83
Coverage Rate (CR, %) Better if closer to $90\%$ — QCCP

Appendix

Table 23: Performance of nonconformity scores under split CP and QCCP with Llama-3.1-8B as the paraphrase generator and Qwen3-8B as the embedding model. Results are reported on normal and fully reworded test sets across three benchmarks. Bold numbers indicate the best performance.

		Last Hidden (Paper)		Attention Weighted		Intermediate Layer		Mean Pooling
CP Type	Method	Normal	Full	Normal	Full	Normal	Full	Normal	Full
Split CP	LAC	90.53	90.47	$90.60$	$98.87$	90.47	$98.87$	90.47	$98.87$
	APS	$99.07$	$99.20$	$99.40$	$99.50$	$99.60$	$100.00$	$99.60$	$100.00$
	PA	$91.00$	$90.97$	90.00	90.10	$89.27$	89.43	$88.83$	90.30
QCCP	LAC	$92.23$	$93.67$	$91.63$	$93.37$	$91.73$	$93.13$	$91.73$	$93.13$
	APS	$98.63$	$97.50$	$98.63$	$97.77$	$98.40$	$97.73$	$98.40$	$97.73$

Appendix

Table 24: Coverage Rate (CR, %) under different pooling and layer choices for Qwen3-8B on CosmosQA. CR is better if closer to 90%. Results are reported for both the normal and fully reworded (Full) settings. Best results are in bold.

		Last Hidden (Paper)		Attention Weighted		Intermediate Layer		Mean Pooling
CP Type	Method	Normal	Full	Normal	Full	Normal	Full	Normal	Full
Split CP	LAC	$1.56$	$1.80$	$1.61$	$5.56$	$1.57$	$5.56$	1.57	$5.56$
	APS	$5.57$	$5.61$	$5.64$	$5.66$	$5.77$	$6.00$	$5.77$	$6.00$
	PA	1.13	1.17	1.15	1.12	1.16	1.12	$2.58$	2.52
QCCP	LAC	$2.47$	$3.11$	$2.35$	$2.91$	$2.39$	$2.87$	2.39	$2.87$
	APS	$4.79$	$4.73$	$4.80$	$4.68$	$4.61$	$4.59$	$4.61$	$4.59$

Appendix

Table 25: Prediction set size (SS,

\downarrow

) under different pooling and layer choices for Qwen3-8B on CosmosQA. Lower SS indicates sharper uncertainty in Normal and fully reworded (Full) settings. Best results are in bold.

		RC (CosmosQA)		CI (HellaSwag)
tf		cf $=$ 0.05	cf $=$ 1.00	cf $=$ 0.05	cf $=$ 1.00
0.01	CR	$91.16\pm 0.73$	$91.17\pm 0.46$	$92.51\pm 1.57$	$90.31\pm 0.10$
	SS	$1.859\pm 0.147$	$1.602\pm 0.116$	$2.309\pm 0.030$	$1.848\pm 0.128$
0.05	CR	$91.00\pm 2.37$	$91.31\pm 0.25$	$92.41\pm 0.94$	$91.14\pm 0.77$
	SS	$1.640\pm 0.182$	$1.408\pm 0.050$	$2.205\pm 0.037$	$1.802\pm 0.097$
0.10	CR	$90.79\pm 1.81$	$91.01\pm 0.16$	$91.06\pm 2.04$	$90.18\pm 0.95$

Appendix

Table 26: Low-label sensitivity of PA on Qwen3-8B in the normal setting under PA-QCCP. tf is the proxy-training fraction; cf is the calibration fraction. Each cell is CR / SS, mean

\pm

std over 3 random seeds. Bold rows mark the full-budget (

\mathrm{tf}=1

) reference.

Dataset	Mean Score	Weighted Score	Worst Score
MMLU	0.385 $\pm$ 0.326	0.375 $\pm$ 0.332	0.534 $\pm$ 0.368
CosmosQA	0.190 $\pm$ 0.284	0.182 $\pm$ 0.287	0.286 $\pm$ 0.356
HellaSwag	0.258 $\pm$ 0.313	0.249 $\pm$ 0.317	0.371 $\pm$ 0.373
HaluDial	0.299 $\pm$ 0.310	0.288 $\pm$ 0.316	0.445 $\pm$ 0.363
HaluSum	0.068 $\pm$ 0.174	0.060 $\pm$ 0.182	0.151 $\pm$ 0.297

Appendix

Table 27: Score statistics for Qwen3-8B displayed as mean

\pm

std for the correct answer choice on each dataset. These statistics are calculated using scores from both the calibration and test sets. A lower score indicates higher confidence (higher predicted probability) from the proxy model.

Method	QA	RC	CI	DRS	DS
LAC	93.17 / 3.72	90.47 / 2.50	91.07 / 2.63	97.87 / 5.02	94.23 / 4.30
APS	95.83 / 4.68	98.43 / 4.83	98.60 / 4.92	93.37 / 4.45	96.57 / 5.03
PA (API-only)	87.83 / 2.66	88.43 / 1.76	89.37 / 2.33	93.97 / 4.38	91.97 / 3.77

Appendix

Table 28: Black-box / API-only PA variant on Qwen3-8B in the semi-reworded QCCP setting. Each cell is CR / SS. The API-only variant uses option logprobs only—no hidden states and no proxy classifier—and is provided as evidence that the paraphrase-aware idea is not restricted to white-box access.

	Method	Qwen3-14B			Qwen3-32B
Dataset		Normal	Semi	Full	Normal	Semi	Full
RC	LAC	89.87 / 1.71	88.40 / 1.78	90.90 / 1.99	88.90 / 1.49	86.13 / 1.57	90.20 / 1.86
	APS	100.00 / 6.00	100.00 / 6.00	100.00 / 6.00	99.77 / 5.74	99.77 / 5.76	97.93 / 3.49
	PA	89.60 / 1.01	88.90 / 1.01	88.80 / 1.07	90.23 / 1.00	89.60 / 1.00	90.40 / 1.01
CI	LAC	89.40 / 1.28	87.97 / 1.32	89.93 / 1.46	91.30 / 1.30	90.63 / 1.34	91.30 / 1.37
	APS	99.33 / 5.56	99.40 / 5.57	99.83 / 5.72	99.87 / 5.69	99.90 / 5.72	99.87 / 5.69

Appendix

Table 29: Performance of nonconformity scores with larger backbone LLMs under standard split CP . Results are reported on RC ( CosmosQA ) and CI ( HellaSwag ) using Qwen3-14B and Qwen3-32B . Each entry is CR / SS (coverage rate / set size), where coverage closer to

90\%

is better and set size

\downarrow

is better. Normal / Semi / Full denote normal / semi-reworded / fully reworded . Boldface highlights the smallest set size in each column.

Method	Clean		Semi.		Full
	CR	SS	CR	SS	CR	SS
LAC	93.7	1.000	90.5	1.000	89.7	1.000
APS	100.0	1.910	100.0	1.965	100.0	1.993
TRON-adapt.	93.7	1.000	90.5	1.000	89.7	1.000
SPUQ-adapt.	93.7	1.000	90.7	1.000	89.4	1.000
PA	94.0	1.000	91.2	1.000	90.5	1.000

Appendix

Table 30: Preliminary extension to open-ended generation on FEVER under Split CP at

\alpha=0.1

, with Qwen3-8B as the backbone. Coverage Rate (CR, %) closer to

90\%

is better; Set Size (SS)

\downarrow

is better. Results are reported under the clean , semi-reworded (Semi.) , and Fully-reworded (Full) settings. TRON-adapt. and SPUQ-adapt. denote the per-option probability-score controls here; the probability-score adaptation equals LAC for any label count, so these two rows match by construction.

Method	Clean		Semi.		Full
	CR	SS	CR	SS	CR	SS
LAC	93.7	1.000	90.5	1.000	89.9	1.002
APS	100.0	1.947	99.8	1.982	100.0	1.987
TRON-adapt.	93.7	1.000	90.5	1.000	89.9	1.002
SPUQ-adapt.	93.5	1.000	90.5	1.000	89.7	1.005
PA	94.5	1.000	90.7	1.000	90.3	1.000

Appendix

Table 31: Preliminary extension to open-ended generation on FEVER under QCCP at

\alpha=0.1

, with Qwen3-8B as the backbone. Coverage Rate (CR, %) closer to

90\%

is better; Set Size (SS)

\downarrow

Explore similar work

May 5, 2026

LLMs Uncertainty Quantification via Adaptive Conformal Semantic Entropy

LLMs' overconfidence, particularly when hallucinating, poses a significant challenge for the deployment of the models in safety-critical settings and makes a reliable estimation of uncertainty necessary. Existing approaches for uncertainty quantification typically prioritize lexical or probabilistic measures; however, these techniques often ignore the semantic variance of different responses with similar meaning. In this paper, we propose Adaptive Conformal Semantic Entropy (ACSE), a method for estimating prompt-level uncertainty by adaptively measuring semantic dispersion in LLMs outputs. Our uncertainty scoring function is based on clustering semantic entropy of multiple diverse responses to the same prompt. The function adaptively adjusts the uncertainty score based on semantic features of each cluster. To ensure statistical reliability of our score, we use conformal calibration to apply a decision rule to accept/abstain the prompts, providing a finite-sample, distribution-free guarantee such that the error rate among the accepted responses remains bounded by a user-specified tolerance. Our extensive experimental evaluations using different LLMs and datasets, demonstrate that our approach consistently outperforms state-of-the-art uncertainty quantification baselines using discriminative performance, conformal guarantees, and probabilistic calibration indicators. As a highlight, for TriviaQA dataset, AUROC of our approach is 0.88 compared to 0.65 produced by the token entropy approach.

Hamed Karimi, Vaishali Meyappan, Reza Samavi

Toronto Metropolitan University, Toronto, Ontario, Canada · Vector Institute, Toronto, Ontario, Canada

Selective Prediction Semantic Entropy LLM Uncertainty Estimation Uncertainty Calibration

Ask PDF

Apr 17, 2026

Beyond Surface Statistics: Robust Conformal Prediction for LLMs via Internal Representations

Large language models are increasingly deployed in settings where reliability matters, yet output-level uncertainty signals such as token probabilities, entropy, and self-consistency can become brittle under calibration--deployment mismatch. Conformal prediction provides finite-sample validity under exchangeability, but its practical usefulness depends on the quality of the nonconformity score. We propose a conformal framework for LLM question answering that uses internal representations rather than output-facing statistics: specifically, we introduce Layer-Wise Information (LI) scores, which measure how conditioning on the input reshapes predictive entropy across model depth, and use them as nonconformity scores within a standard split conformal pipeline. Across closed-ended and open-domain QA benchmarks, with the clearest gains under cross-domain shift, our method achieves a better validity--efficiency trade-off than strong text-level baselines while maintaining competitive in-domain reliability at the same nominal risk level. These results suggest that internal representations can provide more informative conformal scores when surface-level uncertainty is unstable under distribution shift.

Yanli Wang, Peng Kuang, Xiaoyu Han +2

Imperial College London · Zhejiang University · University of Illinois Urbana-Champaign +1

Confidence Estimation in Language Models Conformal Prediction Language Model Calibration LLM Reliability Conformal Risk Control LLM Uncertainty Estimation+1

Ask PDF

Aug 19, 2026

Improved Confidence Estimates for Black-Box Large Language Models

Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs). Existing methods, from verbalized confidence to ones requiring multiple generations, are often zero-shot and produce scores quantifying uncertainty without the need for labelled data. Nonetheless, in practice one must always evaluate their performance on a dataset of interest before deployment. In this work we show that, by leveraging this dataset, we consistently outperform these existing scores. Specifically, we build simple classifiers that predict LLM response correctness by using these scores and the correctness of similar queries as features. Our method produces minimal computational overhead, making it a cheap and straightforward enhancement for UQ in LLMs for real-world applications.

Sokhna Diarra Mbacke, Mouloud Belbahri, Gabriel Loaiza-Ganem

Layer 6 AI · TD Insurance

Confidence Estimation in Language Models LLM Uncertainty Estimation Language Model Calibration

Ask PDF