Certainty Is Not Just Correctness: Rethinking Token-Level Certainty in LLM Reasoning

Yunfan Zhou; Ye Zhu; Zhihai Wang; Jianguo Yao; Haibing Guan; Xijun Li

cs.CLSep 26, 2026

Certainty Is Not Just Correctness: Rethinking Token-Level Certainty in LLM Reasoning

Authors: Yunfan Zhou, Ye Zhu, Zhihai Wang, Jianguo Yao, Haibing Guan, Xijun Li

Organizations: School of Computer Science, Shanghai Jiao Tong University · Qwen Team, Alibaba Group

Abstract

Token-level certainty is widely used as a proxy for correctness in LLM training and inference. However, the performance of certainty-based methods depends both on the information in certainty scores and on how those scores are used. We therefore directly assess certainty's predictive ability through controlled empirical evaluations across models and tasks. We distinguish two prediction targets: identifying questions a model is more likely to answer correctly and distinguishing correct from incorrect responses to the same question. In our experiments, certainty is generally better at identifying questions a model is likely to answer correctly than at distinguishing correct from incorrect responses to the same question. Certainty also varies systematically across token types and positions within words, reflecting local properties of words and text form. Information about question difficulty appears early in generation, while the weaker information about answer correctness is more concentrated near the end. These findings show that the information certainty provides for decisions depends on the prediction target, the model, the certainty metric, and which token positions in the response are included in aggregation. We further demonstrate the practical value of these findings for test-time compute. We allocate the number of responses using certainty early in generation and weight answer votes using certainty near the end of each response. Compared with a fixed-sampling majority-voting baseline, this approach increases overall accuracy from 78.71% to 79.54% while reducing generated-token cost by 82.4%.

Figures & tables

Figure 1: From token-level certainty to two prediction targets. A metric scores each next-token distribution; averaging scores over selected positions yields a response (or trace) score. Within-question comparisons test whether scores distinguish correct from incorrect responses to the same question. Cross-question comparisons test whether a question’s mean response score reflects its proportion of correct responses.

Figure 2: Within-question and cross-question discrimination across five models and four certainty metrics. Each panel shows one metric, with rows linking each model’s within-question AUROC (blue circles) and cross-question C-Index (red diamonds). Points average results equally over AIME 2024, AIME 2025, and GPQA-Diamond.

Figure 3: Structural, Symbolic, and Digit tokens have higher mean certainty than Semantic tokens across four metrics and five models. The left panel shows each category’s share of all tokens, including Others (below 1% for each model); the four radar panels show category means. Radar axes denote models; colors denote token categories. We average shares and means equally over the three datasets. The model abbreviations G4-E4B and G4-26B denote Gemma 4-E4B and Gemma4-26B, respectively.

Figure 4: Later subwords have higher mean certainty than the first across the four metrics and five models shown. Colors distinguish words split into 2, 3, or 4 tokens. The left panel shows these detected words’ share of all token occurrences. Model symbols in the four right panels show each position’s mean minus the first-position mean for the same word length. Positive differences show higher certainty after the first subword across metrics and word lengths. G4-E4B denotes Gemma 4-E4B; G4-26B denotes Gemma4-26B.

Figure 5: Segment selection changes the target captured by certainty. Cells show each model’s logit-gap ranking scores for reasoning, summary, and answer tokens, averaged equally across the three datasets. Panels share a color scale; boldface marks each model’s highest score per target. Answer tokens more often favor within-question ranking, whereas reasoning or summary tokens more often favor cross-question ranking.

Figure 6: Temporal windows locate certainty signals within a trace. Panels (a) and (b) show absolute prefix and suffix windows, respectively. Averaging logit gap over early prefixes quickly recovers cross-question accuracy ranking, while early within-question AUROC remains near chance. Short suffix windows already provide a stronger within-question trace-correctness signal. Lines show averages across five models and three datasets; error bars show the standard deviation across model-dataset settings.

	Fixed Budget					Early allocation
	Accuracy (%) $\uparrow$				Cost $\downarrow$	Accuracy (%) $\uparrow$				Cost $\downarrow$
Model	Majority	Full	Suffix 1k	Suffix 500		Majority	Full	Suffix 1k	Suffix 500
Qwen3-8B	75.43	76.15	76.36	76.45	1.000	74.96	75.73	76.21	76.33	0.142
Qwen3-32B	79.63	81.11	80.78	81.31	1.000	80.68	81.38	81.37	81.75	0.145
Gemma4-E4B	62.11	63.72	63.64	64.38	1.000	61.22	62.92	63.10	63.80	0.186
Gemma4-26B	91.39	91.51	91.37	91.49	1.000	90.87	91.13	90.71	90.86	0.192

Table 1: Answer aggregation under fixed-budget sampling and early allocation. We compare majority voting with rank-weighted voting using certainty computed over the full trace or the final 1,000 or 500 tokens. Within each sampling run, all voting methods share candidate responses and thus generated-token cost. Rank-weighted voting based on the final 500 tokens has the highest overall accuracy among the four methods under both sampling conditions. With early allocation, it reaches 79.54% accuracy at

0.176\times

the token cost of fixed 256-trace sampling.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Model	within-question AUROC (W)	cross-question C-Index (C)	C $-$ W 95% CI	cross-question $>$ within-question?
A. Logit gap ( $\Delta_{19}$ )
Qwen3-8B	0.674	0.764	+0.0899 [0.0396, 0.1424]	Yes
Qwen3-32B	0.647	0.758	+0.1109 [0.0549, 0.1631]	Yes
Gemma4-4B	0.656	0.776	+0.1202 [0.0565, 0.1860]	Yes
Gemma4-26B	0.568	0.732	+0.1645 [0.0603, 0.2741]	Yes
GPT-OSS-20B	0.529	0.668	+0.1385 [0.0686, 0.2041]	Yes

Appendix

Table 2: Full-trace within-question AUROC and cross-question C-Index by model. Cross-question discrimination is higher for every model–metric average. Model rows average three datasets equally; Overall averages all 15 settings equally. Evaluation follows Section 3.1 . Gemma4-4B denotes Gemma 4-E4B. Blue “Yes” denotes a positive difference, not statistical significance. Brackets give pointwise 95% paired question-bootstrap confidence intervals (Appendix E.2 ).

Dataset	within-question AUROC (W)	cross-question C-Index (C)	C $-$ W 95% CI	cross-question $>$ within-question?
A. Logit gap ( $\Delta_{19}$ )
AIME 2024	0.624	0.759	+0.1348 [0.0548, 0.2138]	Yes
AIME 2025	0.637	0.783	+0.1458 [0.0500, 0.2373]	Yes
GPQA-Diamond	0.583	0.677	+0.0938 [0.0550, 0.1309]	Yes
Overall	0.615	0.740	+0.1248 [0.0829, 0.1677]	Yes
B. Self-certainty

Appendix

Table 3: Full-trace within-question AUROC and cross-question C-Index by dataset. Cross-question discrimination is higher for every dataset–metric average. Dataset rows average five models equally; Overall averages all 15 settings equally. Evaluation follows Section 3.1 . Blue “Yes” denotes a positive difference, not statistical significance. Brackets give pointwise 95% paired question-bootstrap intervals (Appendix E.2 ).

Model	Same-question concordance ( $A_{\mathrm{same}}$ )	Different-question concordance ( $A_{\mathrm{diff}}$ )	Pooled AUROC	Difference $A_{\mathrm{diff}}-A_{\mathrm{same}}$ [95% CI]	Different $>$ Same? Point estimate
A. Logit gap ( $\Delta_{19}$ )
Qwen3-8B	0.633	0.869	0.866	+0.2360 [0.1674, 0.3033]	Yes
Qwen3-32B	0.642	0.848	0.846	+0.2061 [0.1406, 0.2757]	Yes
Gemma4-4B	0.619	0.839	0.836	+0.2195 [0.1319, 0.3088]	Yes
Gemma4-26B	0.498	0.766	0.763	+0.2682 [0.1169, 0.3454]	Yes
GPT-OSS-20B	0.526	0.686	0.684	+0.1601 [0.0887, 0.2390]	Yes

Appendix

Table 4: Correctness discrimination by model under equal trace-pair weighting. Different-question concordance exceeds same-question concordance for every model–metric average. Model rows average three datasets equally; Overall averages all 15 settings equally.

A_{\mathrm{same}}

and

A_{\mathrm{diff}}

measure correctness concordance for equally weighted trace pairs from the same and different questions, respectively. Pooled AUROC is computed within each setting using Equation 7 before averaging. Blue “Yes” denotes a positive difference, not statistical significance. Brackets give pointwise 95% paired question-bootstrap intervals (Appendix E.2 ).

Dataset	Same-question concordance ( $A_{\mathrm{same}}$ )	Different-question concordance ( $A_{\mathrm{diff}}$ )	Pooled AUROC	Difference $A_{\mathrm{diff}}-A_{\mathrm{same}}$ [95% CI]	Different $>$ Same? Point estimate
A. Logit gap ( $\Delta_{19}$ )
AIME 2024	0.598	0.854	0.852	+0.2563 [0.1435, 0.3337]	Yes
AIME 2025	0.587	0.837	0.833	+0.2508 [0.1208, 0.3512]	Yes
GPQA-Diamond	0.565	0.712	0.712	+0.1468 [0.0925, 0.1956]	Yes
Overall	0.583	0.801	0.799	+0.2180 [0.1584, 0.2638]	Yes
B. Self-certainty

Appendix

Table 5: Correctness discrimination by dataset under equal trace-pair weighting. Different-question concordance exceeds same-question concordance for every dataset–metric average. Dataset rows average five models equally; Overall averages all 15 settings equally.

A_{\mathrm{same}}

and

A_{\mathrm{diff}}

Model	within-question AUROC (W)	cross-question C-Index (C)	C $-$ W 95% CI	cross-question $>$ within-question?
A. Logit gap ( $\Delta_{19}$ )
Qwen3-8B	0.674	0.730	+0.0557 [−0.0106, 0.1220]	Yes
Qwen3-32B	0.647	0.734	+0.0866 [0.0279, 0.1422]	Yes
Gemma4-4B	0.656	0.745	+0.0896 [0.0045, 0.1729]	Yes
Gemma4-26B	0.568	0.498	−0.0697 [−0.2793, 0.1676]	No
GPT-OSS-20B	0.529	0.621	+0.0921 [0.0165, 0.1648]	Yes

Appendix

Table 6: Discrimination on questions with both correct and incorrect responses, by model. Gemma4-26B reverses the cross-question advantage for all four metrics; difference intervals cross zero. Model rows average three datasets equally; Overall averages all 15 settings equally. Both targets use questions with both correct and incorrect responses; within-question AUROC is unchanged. Blue “Yes” denotes a positive difference, not statistical significance. Brackets give pointwise 95% paired question-bootstrap intervals (Appendix E.2 ).

Dataset	within-question AUROC (W)	cross-question C-Index (C)	C $-$ W 95% CI	cross-question $>$ within-question?
A. Logit gap ( $\Delta_{19}$ )
AIME 2024	0.624	0.614	−0.0106 [−0.1451, 0.1278]	No
AIME 2025	0.637	0.765	+0.1279 [0.0033, 0.2353]	Yes
GPQA-Diamond	0.583	0.618	+0.0353 [−0.0017, 0.0690]	Yes
Overall	0.615	0.666	+0.0509 [−0.0066, 0.1115]	Yes
B. Self-certainty

Appendix

Table 7: Discrimination on questions with both correct and incorrect responses, by dataset. AIME 2024 reverses the cross-question advantage for logit gap and self-certainty; difference intervals cross zero. Dataset rows average five models equally; Overall averages all 15 settings equally. Both targets use questions with both correct and incorrect responses; within-question AUROC is unchanged. Blue “Yes” denotes a positive difference, not statistical significance. Brackets give pointwise 95% paired question-bootstrap intervals (Appendix E.2 ).

Figure 7: Rank sensitivity varies by model and target. Panels show each model’s change in within-question AUROC (left) or cross-question C-Index (right) relative to

k=2

. Curves show averages across three datasets at each of the 21 evaluated values of

k

; markers highlight

k\in\{2,10,19,99\}

. Table 8 gives absolute scores.

Index $k$	Per-model average (W/C)					Overall
Index $k$	Qwen3-8B	Qwen3-32B	Gemma4-4B	Gemma4-26B	GPT-OSS-20B	W / C
2	0.635 / 0.745	0.642 / 0.721	0.699 / 0.751	0.590 / 0.790	0.506 / 0.685	0.614 / 0.738
6	0.659 / 0.775	0.664 / 0.755	0.701 / 0.747	0.595 / 0.789	0.502 / 0.697	0.624 / 0.753
10	0.661 / 0.786	0.669 / 0.760	0.698 / 0.749	0.595 / 0.789	0.500 / 0.703	0.625 / 0.757
14	0.660 / 0.791	0.669 / 0.766	0.699 / 0.752	0.594 / 0.789	0.500 / 0.703	0.625 / 0.760
18	0.662 / 0.791	0.671 / 0.768	0.699 / 0.753	0.594 / 0.788	0.500 / 0.702	0.625 / 0.761

Appendix

Table 8: Absolute logit-gap discrimination at selected competitor ranks. Cells report mean within-question AUROC (W) and cross-question C-Index (C) over three datasets per model; Overall averages all 15 settings. Light blue identifies

k=19

, corresponding to the 20th most probable token used in the main comparison. Bold marks the highest W and C separately within each column across displayed indices, including ties at three decimal places. The response collection differs from that used in the main four-metric comparison in Figure 2 .

Group	Occurrences	Traces	Model–questions	Settings
Semantic	2,758,980,303	660,480	1,290	15/15
Structural	1,535,677,651	660,480	1,290	15/15
Symbolic	1,311,805,321	660,480	1,290	15/15
Digit	994,217,932	652,510	1,288	15/15
Others	25,618,508	328,951	873	11/15
2-piece words	136,824,127	655,934	1,290	15/15

Appendix

Table 9: Coverage by token category and word length for all sampled responses (five models, three datasets). Occurrences count tokens by category and words by position group. Symbolic excludes digits; Digit contains digits. Traces and model–question pairs can belong to multiple groups; positions in a word share coverage.

Model	cross-question C-Index $\uparrow$		within-question AUROC $\uparrow$
Model	Original	Equal	Original	Equal
Qwen3-8B	0.7673	0.8070	0.6739	0.6842
Qwen3-32B	0.7720	0.7926	0.6467	0.6552
Gemma 4-E4B	0.7717	0.7962	0.6556	0.6481
Gemma4-26B	0.7236	0.7240	0.5675	0.5907
GPT-OSS-20B	0.6546	0.6819	0.5307	0.5498

Appendix

Table 10: Equal category weighting improves discrimination on both targets for most models. Original averages all token scores; Equal assigns 25% to each of the Semantic, Structural, Symbolic, and Digit means, excluding Others. Both use logit gap

\Delta_{19}

with the scoring and sample-selection rules in Appendix B.1 . Entries are equally weighted averages across the three datasets. Bold marks the higher value, including ties.

Model	Token category
	Semantic	Structural	Symbolic	Digit	Others
A. Logit gap ( $\Delta_{19}$ ) Color scale: 8.159 29.115
Qwen3-8B	16.794	21.585	22.441	27.802	25.962
Qwen3-32B	17.439	22.178	22.772	26.863	24.600
Gemma 4-E4B	12.946	17.217	18.775	21.679	NA
Gemma4-26B	16.147	18.811	19.616	21.501	NA

Appendix

Table 11: Token-category means for four certainty metrics. For each model shown, Structural, Symbolic, and Digit means exceed the Semantic mean. Means use all sampled responses, with equal weight for token occurrences within each setting. Model rows average three datasets equally; Overall averages 15 settings. Symbolic excludes digits; Digit contains digits. NA denotes a missing category in at least one required setting. Darker shading denotes larger raw means; each metric shares a color scale with Table 12 .

Model	2-piece words		3-piece words			4-piece words
	Pos. 1	Pos. 2	Pos. 1	Pos. 2	Pos. 3	Pos. 1	Pos. 2	Pos. 3	Pos. 4
A. Logit gap ( $\Delta_{19}$ ) Color scale: 8.159 29.115
Qwen3-8B	17.231	25.294	16.626	24.103	26.091	17.233	25.640	25.841	26.407
Qwen3-32B	17.471	23.601	16.301	22.008	23.497	16.878	22.479	22.248	22.988
Gemma 4-E4B	12.247	18.966	11.141	18.288	20.339	10.886	17.691	19.377	19.690
Gemma4-26B	17.397	21.277	17.512	20.619	22.103	17.116	19.651	20.999	21.404

Appendix

Table 12: Later subwords have higher mean certainty than the first across metrics and models, although certainty need not increase monotonically with position. Cells report raw means over word occurrences in all sampled responses, separately for two-, three-, and four-piece words. Pos. denotes the position of a subword within a word. Model rows average three datasets equally; Overall averages 15 settings. Darker shading denotes larger raw means; each metric shares a color scale with Table 11 .

Model	within-question			cross-question
	AUROC			C-Index
	All	$\Delta$ First-piece	$\Delta$ Exclude Digit and Others	All	$\Delta$ First-piece	$\Delta$ Exclude Digit and Others
A. Logit gap ( $\Delta_{19}$ )
Qwen3-8B	0.674	+0.0024 [−0.0030,0.0086]	+0.0074 [−0.0005,0.0172]	0.764	+0.0175 [0.0010,0.0400]	+0.0294 [0.0102,0.0539]
Qwen3-32B	0.647	−0.0020 [−0.0078,0.0035]	−0.0033 [−0.0133,0.0056]	0.758	+0.0024 [−0.0161,0.0206]	−0.0023 [−0.0319,0.0248]
Gemma 4-E4B	0.656	+0.0015 [−0.0028,0.0063]	+0.0003 [−0.0095,0.0102]	0.776	+0.0034 [−0.0096,0.0181]	+0.0123 [−0.0022,0.0286]

Appendix

Table 13: Masking effects by model. Overall gains are small, and effects vary with the metric, model, and prediction target. Model rows average the three datasets equally; Overall averages all 15 settings. All denotes the all-token baseline;

\Delta

is the paired change from All. Brackets show 95% paired question-bootstrap intervals. Bold marks the best condition within each row and target, including ties. Definitions of the token masks and details of the bootstrap procedure are given in Appendices B.2 and E.2 .

Dataset	within-question			cross-question
	AUROC			C-Index
	All	$\Delta$ First-piece	$\Delta$ Exclude Digit and Others	All	$\Delta$ First-piece	$\Delta$ Exclude Digit and Others
A. Logit gap ( $\Delta_{19}$ )
AIME 2024	0.624	+0.0075 [0.0015,0.0139]	+0.0247 [0.0076,0.0426]	0.759	+0.0044 [−0.0106,0.0228]	+0.0161 [−0.0121,0.0461]
AIME 2025	0.637	+0.0052 [−0.0006,0.0116]	+0.0006 [−0.0096,0.0109]	0.783	+0.0058 [−0.0104,0.0252]	+0.0057 [−0.0147,0.0275]
GPQA-Diamond	0.583	−0.0034 [−0.0072,0.0002]	−0.0004 [−0.0040,0.0037]	0.677	+0.0034 [−0.0029,0.0101]	+0.0092 [0.0017,0.0166]

Appendix

Table 14: Masking effects by dataset. First-piece improves within-question AUROC on both AIME datasets but reduces it on GPQA-Diamond for all four metrics. Rows average the five models equally. All denotes the all-token baseline;

\Delta

Model	cross-question C-Index			within-question AUROC
Model	Prefix (1k)	Suffix (1k)	Full Trace	Prefix (1k)	Suffix (1k)	Full Trace
Qwen3-8B	0.709	0.774	0.764	0.505	0.683	0.674
Qwen3-32B	0.728	0.769	0.758	0.502	0.652	0.647
Gemma4-4B	0.721	0.748	0.776	0.489	0.675	0.656
Gemma4-26B	0.753	0.701	0.732	0.514	0.533	0.568
GPT-OSS-20B	0.701	0.664	0.668	0.491	0.584	0.529

Appendix

Table 15: Temporal localization across models. Both prediction targets use the same three aggregation regions: the first 1,000 tokens, the final 1,000 tokens, and the full trace. The cross-question C-Index ranks empirical pass rates across questions, whereas within-question AUROC ranks correct and incorrect traces for the same question. Each model row equally averages AIME 2024, AIME 2025, and GPQA-Diamond; Overall equally averages all model–dataset combinations. Higher is better. Blue shading marks the selected windows (cross-question prefix and within-question suffix), not row maxima.

(a) 100–500 generated tokens
Readout	100	200	300	400	500
cross-question: prefix C-Index	$0.578$ [-1pt] $[0.449,0.664]$	$0.642$ [-1pt] $[0.492,0.731]$	$0.670$ [-1pt] $[0.550,0.744]$	$0.682$ [-1pt] $[0.595,0.754]$	$0.691$ [-1pt] $[0.608,0.754]$
within-question: suffix AUROC	$0.639$ [-1pt] $[0.501,0.742]$	$0.639$ [-1pt] $[0.402,0.756]$	$0.640$ [-1pt] $[0.443,0.758]$	$0.633$ [-1pt] $[0.414,0.771]$	$0.638$ [-1pt] $[0.429,0.791]$

Appendix

Table 16: Effect of absolute window length on prefix cross-question C-Index and suffix within-question AUROC. Entries give equal-weight averages across the 15 model–dataset combinations, with the observed minimum–maximum range in brackets. Windows count generated tokens. The 30,000-token suffix approximates the full trace, covering 98.4% of responses. Higher is better.

(a) 0.1%–5% of the trace
Readout	0.1%	0.5%	1%	2%	5%
cross-question: prefix C-Index	$0.508$ [-1pt] $[0.210,0.851]$	$0.627$ [-1pt] $[0.493,0.831]$	$0.672$ [-1pt] $[0.542,0.794]$	$0.693$ [-1pt] $[0.565,0.797]$	$0.688$ [-1pt] $[0.605,0.789]$
within-question: suffix AUROC	$0.611$ [-1pt] $[0.447,0.780]$	$0.621$ [-1pt] $[0.457,0.753]$	$0.628$ [-1pt] $[0.497,0.736]$	$0.636$ [-1pt] $[0.501,0.755]$	$0.651$ [-1pt] $[0.567,0.743]$

Appendix

Table 17: Effect of relative window length on prefix cross-question C-Index and suffix within-question AUROC. Entries average the 15 model–dataset combinations equally, with the observed minimum–maximum range in brackets. Windows are fractions of each trace. Higher is better.

Control dimension	Aggregation region	within-question AUROC	cross-question C-Index
Answer-span content	Suffix (1k)	0.6260	0.7310
Answer-span content	Suffix (1k), answers removed	0.6279	0.7334
Summary extent	Full summary	0.6100	0.7414
Summary extent	Suffix (1k)	0.6260	0.7310
Within-summary position	Summary Q1	0.5371	0.6950
	Summary Q2	0.5353	0.7021

Appendix

Table 18: Scores for the final 1,000-token suffix before and after removing final-answer spans, the full summary, and its four consecutive quarters (Q1–Q4). Quarters contain approximately equal token counts and are ordered from first to last. Values average the 15 model–dataset settings equally; higher is better.

(a) cross-question C-Index
Metric	Full trace	Prefix (1k)	Prefix (1k) $-$ full trace	Combinations with prefix $>$ full trace
Logit gap	0.740	0.723	$-0.017$	8/15
Self-certainty	0.733	0.717	$-0.016$	7/15
Negative entropy	0.664	0.708	$+0.045$	12/15
Maximum log probability	0.659	0.709	$+0.050$	12/15

Appendix

Table 19: Full-trace, prefix, and suffix scores across four certainty metrics. Both panels report equal-weight averages over the same 15 model–dataset combinations. For each metric, panel (a) compares the cross-question C-Index of the full trace with that of the 1,000-token prefix, and panel (b) compares the within-question AUROC of the full trace with that of the 1,000-token suffix. Each difference is the prefix or suffix score minus the full-trace score; the final column counts combinations on which that prefix or suffix score is higher. No exact ties occur. Blue shading identifies the prefix in panel (a) and the suffix in panel (b) and does not denote the largest value in a row. Higher is better.

(a) Fixed-budget sampling: 256 traces
		Accuracy (%) $\uparrow$				Relative token cost $\downarrow$
Model	Dataset	Majority voting	Full trace	Final 1,000 tokens	Final 500 tokens
Qwen3-8B	AIME 2024	80.00	83.02	82.94	82.76	1.000
	AIME 2025	79.77	79.35	79.61	79.79	1.000
	GPQA-Diamond	66.52	66.07	66.54	66.81	1.000
Qwen3-32B	AIME 2024	86.02	90.55	88.80	90.60	1.000

Appendix

Table 20: Voting by model and dataset. Majority voting gives one vote per response; other columns use rank-weighted voting (

p=1

) with mean logit gap over the indicated region. Methods within each panel share traces and token cost. Overall averages the 15 settings equally; relative cost is each setting’s total tokens divided by its fixed-256 total. Bold marks row maxima within each panel, including ties at displayed precision.

Figure 8: TACS allocates more traces to harder question groups. Group means are normalized by each model’s mean across all 258 questions; the horizontal dashed line marks this reference.

Model	Number of questions	Mean trace count	Median trace count	Sampling runs at the trace limit (%)
Qwen3-8B	258	41.14	32	1.94
Qwen3-32B	258	41.49	32	1.97
Gemma 4-E4B	258	71.50	32	12.82
Gemma4-26B	258	70.70	32	12.92
GPT-OSS-20B	258	43.24	32	2.71

Appendix

Table 21: Sample counts under early allocation, with a target of 32 qualifying traces and a limit of 256. Means and medians summarize trace counts per sampling run across each model’s 258 questions, weighted equally. The final column gives the percentage of runs using 256 traces. Table 23 weights datasets equally.

Method	AIME 2024	AIME 2025	GPQA- Diamond	Overall	Difference (percentage points)
Self-Consistency	83.41	81.40	71.32	78.71	+0.00
Self-Certainty Best-of- $N$	80.94	71.53	69.07	73.85	-4.86
Self-Certainty: rank-weighted voting	85.78	81.58	71.53	79.63	+0.92
Reasoning-Pruning Consistency	84.47	80.53	70.72	78.57	-0.13
DeepConf: bottom-group mean; retain 10%	83.77	78.10	70.66	77.51	-1.20
DeepConf: bottom-group mean; retain 90%	84.16	81.44	71.35	78.98	+0.27

Appendix

Table 22: Answer aggregation using the same 256 responses and token budget within each sampling run. Suffix 500 rank-weighted voting has the highest overall point estimate. Accuracy is in percent; differences are percentage points from Self-Consistency. Dataset columns average five models; Overall averages the 15 settings equally. Bold marks column maxima, including displayed ties. The rank exponents are 1.2 for Self-Certainty and 1 for Suffix 500 rank-weighted voting. DeepConf retains the indicated fraction of the highest-scoring traces. For DeepConf, bottom-group mean averages the lowest-scoring 10% of sliding windows, whereas tail mean averages token confidence over the final segment.

Method	Sampling budget	Accuracy (%)	Relative token cost	Mean trace count
Self-Consistency	32 traces (fixed)	78.27	0.125	32.0
Self-Consistency	128 traces (fixed)	78.63	0.500	128.0
Self-Consistency	256 traces (fixed)	78.71	1.000	256.0
Adaptive-Consistency	At most 256 traces	78.65	0.363	26.1
Early-Stopping Self-Consistency	Dataset-specific maximum	78.66	0.419	69.2
DeepConf-low	At most 256 traces	76.89	0.176	57.8

Appendix

Table 23: Accuracy and token cost under fixed and adaptive sampling. TACS reaches 79.54% accuracy at

0.176\times

the fixed-256 Self-Consistency cost. All three measures are averaged equally across the 15 model–dataset settings. Early-Stopping Self-Consistency allows at most 256 traces on AIME and 255 on GPQA-Diamond. Retention parameters are 10% for DeepConf-low and 90% for DeepConf-high. DeepConf counts attempted traces and tokens from truncated generations; TACS completes each observed trace.

Parameter	Setting
Temperature	$0.6$
Top- $p$	$0.95$
Maximum generation length	$32{,}768$ tokens for Qwen models; $131{,}072$ tokens for GPT-OSS and Gemma models
Batch size	$256$ traces per vLLM generation call; vLLM maximum concurrent sequences set to $512$
Number of samples	$1{,}024$ stored responses per model–question pair for the main TACS simulations
Random seed	Offline resampling uses seed $42$ ; trace generation does not pass an explicit vLLM sampling seed

Appendix

Table 24: Key decoding and inference settings used to generate the traces.

Confidence Estimation in Language Models

Confidence estimation in language models measures whether a model’s uncertainty signals predict answer correctness and support reliable decisions.

See all 249 papers

Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning
May 16, 2026Shuo Liu, Ding Liu, Shi-Ju RanConfidence Estimation in Language Models LLM Reliability
School of Computer Science and Technology, Tiangong University, Tianjin, China. · Center for Quantum Physics and Intelligent Sciences, Department of Physics, Capital Normal University, Beijing, China.
Reported Confidence in LLMs Tracks Commitment More Than Correctness
Jun 28, 2026Dharshan KumaranLLM Uncertainty Estimation LLM Interpretability
Google DeepMind
Two Axes of LLM Abstention: Answer Correctness and Question Answerability
Jul 9, 2026Benedikt J. WagnerSelective Prediction Confidence Estimation in Language Models
City St George’s, University of London

May 16, 2026J/K move · Enter open · S save

Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning

Shuo Liu, Ding Liu, Shi-Ju Ran

School of Computer Science and Technology, Tiangong University, Tianjin, China. · Center for Quantum Physics and Intelligent Sciences, Department of Physics, Capital Normal University, Beijing, China.

Confidence Estimation in Language Models LLM Reliability LLM Uncertainty Estimation LLM Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) generate not only reasoning text, but also token-level confidence trajectories that record how uncertainty evolves during inference. Whether these trajectories are relevant to reasoning correctness remains unclear. Here we show that confidence trajectories encode a content-agnostic confidence geometry associated with trace-level final-answer correctness. Using only token-level confidence values, without access to the input question, reasoning text, hidden states, or external verifiers, we find that low-dimensional representations of confidence trajectories separate correct from incorrect reasoning traces. Across GSM8K, MATH, and MMLU, this geometric separation is quantitatively linked to downstream predictability: stronger clustering of correct and incorrect traces, measured by the Davies--Bouldin index, consistently corresponds to higher correctness-discrimination AUC. We further show that correctness-related information is enriched in the tail of reasoning, suggesting that late-stage confidence dynamics carry key correctness signals. We propose NeuralConf, a lightweight estimator that learns from confidence trajectories for correctness evaluation. Under a fixed trace budget, NeuralConf-derived scores improve confidence-weighted answer aggregation over majority voting, tail confidence, and other static baselines. These results reveal that LLMs expose trace-intrinsic statistical signals of correctness through their own confidence dynamics, offering a route to improve inference using information already present within generation.

Abstract

Confidence Estimation in Language Models

Explore similar work

Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning

Reported Confidence in LLMs Tracks Commitment More Than Correctness

Two Axes of LLM Abstention: Answer Correctness and Question Answerability