Calibration-Constrained Diagnostics for Evaluating LLM Self-Evaluation

Jae Oh Woo; Mengdie Flora Wang; Rahul Ghosh; Baishali Chaudhury; Mun Young Kim

cs.AINov 1, 2026

Calibration-Constrained Diagnostics for Evaluating LLM Self-Evaluation

Authors: Jae Oh Woo, Mengdie Flora Wang, Rahul Ghosh, Baishali Chaudhury, Mun Young Kim

Organizations: AWS Generative AI Innovation Center

Abstract

We introduce a reusable evaluation artifact for diagnosing whether LLM self-reported confidence can support confidence-gated act-or-defer decisions. The artifact is designed for evaluation rather than safety certification: it provides a black-box protocol, prompts, judge-reference labels, model outputs, reproduction scripts, and calibration-constrained diagnostic metrics for auditing LLM self-evaluation across math, factual, and commonsense reasoning tasks. Our central finding is that aggregate uncertainty summaries can produce deployment-misleading rankings: a model that appears strongest under standard aggregate calibration metrics can fail under category-level calibration constraints, while a model with worse aggregate calibration can exhibit stronger calibration-constrained behavior in the positive-margin region where high self-selection accuracy and tight calibration coexist. To expose this failure mode, we define PVC and C-PVC as category-level diagnostics over an accuracy-by-calibration surface: PVC counts categories whose self-evaluation accuracy exceeds a threshold, while C-PVC additionally requires category-level calibration error below a tolerance. We aggregate these diagnostics using Volume Under Surface (VUS) and a positive-margin restriction; they are VC/fat-shattering motivated but are empirical evaluation proxies, not literal VC-dimension estimates. Across 13 models and four benchmarks, the artifact shows weak confidence-correctness separation, stable diagnostic rankings under alternative categorizations and judge-label perturbations, and persistent positive-margin failures not captured by ECE or Brier score. Confidence-gated self-evaluation therefore requires calibration-constrained, category-level diagnostics, and we provide a reusable artifact for applying them under black-box access.

Figures & tables

Figure 1: Self-evaluation diagnostic pipeline. The target model generates two candidate solutions, selects the preferred one with a confidence report, and is evaluated against a three-judge LLM reference label.

	Full Region			PM Region ( $\gamma>\tau+0.5$ )	Calibration	Accuracy
Model	PVC-VUS $\uparrow$	C-PVC-VUS $\uparrow$	Gap $\downarrow$	PM-C-PVC-VUS $\uparrow$	CalibError $\downarrow$	SEA $\uparrow$
Qwen2.5-7B (Pretrain) ( Yang et al., 2024a )	5.26	3.76	1.50	0.06	0.313	0.556
Qwen2.5-7B-Instruct (SFT+RL) ( Yang et al., 2024a )	5.61	4.08	1.53	0.17	0.291	0.599
Qwen2.5-Math-7B-Instruct ( Yang et al., 2024b )	4.70	3.20	1.50	0.002	0.335	0.506
Llama-3.1-8B-Instruct (SFT+DPO) ( Meta AI, 2024 )	4.55	2.92	1.63	0.00	0.385	0.495
OpenThinker2-7B (SFT) ( Open-Thoughts, 2025 )	5.37	3.50	1.87	0.002	0.357	0.580

Table 1: Diagnostic decomposition of LLM self-evaluation behavior (cross-dataset aggregate). PVC-VUS summarizes category-level discrimination; the PVC–C-PVC Gap quantifies the empirical calibration cost; PM-C-PVC-VUS reports calibration-constrained behavior surviving in the positive-margin region

\gamma>\tau+0.5

(Theorem 1 ). Bold : best; underline : second best. Per-dataset breakdowns (Math-360 / TruthfulQA / CSQA) and the exploratory PM-SC-VUS proxy are in Appendix Q ; MATH-500 results are in Appendix R .

		Aggregate UQ		Full Region			PM Region
Model	Scale	ECE $\downarrow$	Brier $\downarrow$	PVC-VUS $\uparrow$	C-PVC-VUS $\uparrow$	Gap $\downarrow$	PM-C-PVC-VUS $\uparrow$
JiuZhang3.0-7B	7B	0.199	0.049	4.79	3.84	0.95	0.019
Qwen2.5-7B-Instruct	7B	0.291	0.081	5.19	3.70	1.49	0.126
Llama-3.1-8B-Instruct	8B	0.385	0.140	4.40	2.86	1.55	0.000
s1.1-7B	7B	0.293	0.094	5.44	3.94	1.50	0.142
Qwen2.5-32B-Instruct	32B	0.260	0.073	5.57	4.20	1.37	0.209

Table 2: Scale extension and ranking reversal combined. The aggregate-UQ winner (JiuZhang3.0-7B by ECE/Brier) is near-zero in PM-C-PVC-VUS, while Qwen2.5-32B-Instruct wins on PM despite worse aggregate metrics. PM-C-PVC-VUS here uses the ranking-reversal aggregation across the six models shown (same reference labels and

(\gamma,\tau)

grid as Table 1 ); Table 1 reports the broader cross-dataset aggregate over eleven 7–8B open models, so absolute values differ but the qualitative ordering is preserved. Bold : best.

Panel of Figure 2 — Figure 2: (a) All models plateau near maximum PVC at lenient $\gamma$ but decline to zero near $\gamma=1$ . (b) Scatter of PVC-VUS vs. C-PVC-VUS; the diagonal is perfect calibration. JiuZhang3.0-7B lies closest; s1.1-7B achieves highest PVC-VUS at greater calibration cost.

Figure 3: Empirical C-PVC diagnostic surfaces for eleven 7–8B models ( $\tau$ : calibration tolerance, $\gamma$ : accuracy threshold, $z$ : empirical C-PVC proxy). Distinct topographical profiles give per-model signatures; C-PVC-VUS (parentheses) aggregates each surface. Extended analysis in Appendix S .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Field	Description
Artifact name	LLM Self-Evaluation Diagnostics Package (v1.0): protocol, metrics, testbed, and scripts.
Components	(i) three-stage self-evaluation protocol (two candidate-generation prompts, self-selection prompt, judge prompt, under prompts/ ); (ii) Math-360 ( data/math360/ ; $360$ original problems, $8$ domains, $5$ subcategories, difficulty-stratified); (iii) cross-domain category splits for TruthfulQA, CommonsenseQA, MATH-500 ( data/cross_domain/ ); (iv) model outputs ( model_outputs/*_self_eval.csv ); (v) three-judge reference labels ( reference_labels/ ; schema documented); (vi) selfeval package (models, judges, evaluation, experiment, utils) with YAML configs; (vii) scripts for PVC/C-PVC computation, cross-dataset aggregation, end-to-end evaluation, main-table reproduction, and reference-label rebuild; (viii) documentation (README, reviewer README, release-strategy, dataset/evaluation cards, Croissant metadata).
Intended use	Diagnosing reliability of LLM self-reported confidence under per-category accuracy and calibration constraints; comparing models under a fixed reference signal; ablating evaluation assumptions.
Not intended use	General-purpose math leaderboard; absolute human-level capability ranking; single-scalar replacement for ECE/Brier/AUROC; any claim that a model’s self-evaluation is reliable outside the declared protocol.
Math-360 source	Original problems authored for this work, stratified across $8$ domains (Arithmetic, Algebra, Calculus, Geometry, Number Theory, Combinatorics, Statistics, Linear Algebra), $5$ subcategories each, difficulty-stratified (Easy / Medium / Hard). Released in full.
Reused datasets	TruthfulQA [ Lin et al., 2022 ] , CommonsenseQA [ Talmor et al., 2019 ] , MATH-500 [ HuggingFaceH4, 2024 ] . Question text is not redistributed; we release only IDs, category mappings, and derived outputs. Sampling in Appendix Y.4 .

Appendix

Table 3: Artifact / evaluation card for the released diagnostic package.

Risk	Mitigation	Remaining limitation
Judge bias / self-preference	3-judge ensemble, randomized presentation order	Not a human oracle
Label noise	$10$ – $30\%$ synthetic label perturbation (5 seeds)	Synthetic corruption only
Ambiguous pairs (both correct / both wrong)	Unanimous-only (3-0) re-analysis	Reduces effective sample
Style-preference artifacts	Same reference applied across all compared models	May still reward judge-preferred style

Appendix

Table 4: Summary of reference-label risks, mitigations, and remaining limitations.

Claims this artifact enables	Claims this artifact does not enable
Compare whether model-reported confidence tracks judge-referenced self-selection correctness across models.	Certify absolute correctness of model reasoning.
Detect category-level calibration/discrimination failures hidden by aggregate UQ summaries.	Provide human-preference alignment without additional human validation.
Analyze whether calibration-constrained self-evaluation behavior survives in the positive-margin (PM) region.	Literally estimate VC dimension or derive distribution-free sample-complexity bounds.
Reproduce and extend PVC/C-PVC/PM-VUS surfaces across new models and datasets under the declared protocol.	Make universal claims that LLMs cannot self-evaluate outside the declared protocol.
Diagnose deployment-relevant failure modes for confidence-gated downstream decisions.	Certify deployment safety of any specific model.

Appendix

Table 5: Claim boundaries for the released evaluation package.

Symbol	Description
$\mathcal{X}$	Input space (set of problem categories)
$\mathcal{F}$	Class of probabilistic predictors $f:\mathcal{X}\rightarrow\Delta(\{0,1\})$
$\Delta(\{0,1\})$	Set of probability distributions over binary output space $\{0,1\}$
$\gamma\in(0,1]$	Accuracy threshold parameter
$\tau\in[0,1)$	Calibration error tolerance parameter
$\mathrm{VC}(\mathcal{F})$	Classical Vapnik-Chervonenkis dimension of function class $\mathcal{F}$

Appendix

Table 6: Notation used throughout the paper.

	Full Region			Positive-Margin (PM) Region ( $\gamma>\tau+0.5$ )			Calibration	Accuracy
Model	PVC-VUS $\uparrow$	C-PVC-VUS $\uparrow$	Gap $\downarrow$	PM-PVC-VUS $\uparrow$	PM-C-PVC-VUS $\uparrow$	PM-SC-VUS $\uparrow$	CalibError $\downarrow$	SEA $\uparrow$
Qwen2.5-7B (Pretrain) [ Yang et al., 2024a ]	5.26	3.76	1.50	1.25	0.06	289.77	0.313	0.556
Qwen2.5-7B (Pretrain) [ Yang et al., 2024a ]	(4.78/5.66/5.73)	(3.27/3.93/4.09)	(1.51/1.73/1.64)	(1.26/1.15/1.33)	(0.00/0.01/ 0.18 )	(283.5/284.5/ 301.3 )	(0.320/0.318/0.303)	(0.597/0.565/0.571)
Qwen2.5-7B-Instruct (SFT+RL) [ Yang et al., 2024a ]	5.61	4.08	1.53	1.78	0.17	300.30	0.291	0.599
Qwen2.5-7B-Instruct (SFT+RL) [ Yang et al., 2024a ]	(4.65/ 6.13 / 6.04 )	(3.13/ 4.73 /4.40)	(1.52/ 1.40 /1.64)	(1.24/ 2.07 / 2.03 )	(0.00/ 0.47 /0.03)	(283.5/ 330.6 /286.7)	(0.337/ 0.247 /0.290)	(0.581/ 0.613 / 0.604 )
Qwen2.5-Math-7B-Instruct (SFT+RL) [ Yang et al., 2024b ]	4.70	3.20	1.50	0.65	0.002	283.76	0.335	0.506

Appendix

Table 7: Per-dataset breakdown of Table 1 . First row per model: cross-dataset aggregate (identical to Table 1 ). Second row (parentheses): Math-360 / TruthfulQA / CSQA. PM-SC-VUS is an exploratory verification-cost proxy (baseline

283.5

; Appendix N ) and is not used in any main claim. Bold : best; underline : second best.

	Full Region			Positive-Margin (PM) Region ( $\gamma>\tau+0.5$ )			Calibration	Accuracy
Model	PVC-VUS $\uparrow$	C-PVC-VUS $\uparrow$	Gap $\downarrow$	PM-PVC-VUS $\uparrow$	PM-C-PVC-VUS $\uparrow$	PM-SC-VUS $\uparrow$	CalibError $\downarrow$	SEA $\uparrow$
Qwen2.5-7B (Pretrain) [ Yang et al., 2024a ]	3.93	2.56	1.37	0.71	0.00	283.5	0.356	0.561
Qwen2.5-7B-Instruct (SFT+RL) [ Yang et al., 2024a ]	3.94	2.54	1.40	0.69	0.00	283.5	0.358	0.562
Qwen2.5-Math-7B-Instruct (SFT+RL) [ Yang et al., 2024b ]	3.68	2.57	1.11	0.20	0.00	283.5	0.304	0.524
Llama-3.1-8B-Instruct (SFT+DPO) [ Meta AI, 2024 ]	3.97	2.67	1.30	0.75	0.00	283.5	0.331	0.565
OpenThinker2-7B (SFT) [ Open-Thoughts, 2025 ]	4.36	3.09	1.27	1.43	0.00	283.5	0.294	0.622

Appendix

Table 8: Self-evaluation performance on MATH-500 benchmark. Results are consistent with the main findings (Table 1 ): the expressivity–calibration trade-off persists, with JiuZhang3.0-7B achieving the lowest Gap (

0.63

) and best calibration. Notably, PM-C-PVC-VUS is zero for

10

11

models; only Bespoke-Stratos-7B achieves nonzero calibration-constrained behavior (

0.11

) in the positive-margin region, indicating even more severe calibration limitations on this benchmark. Bold : best; underline : second best.

Figure 4: (a) PVC scores across accuracy thresholds ( $\gamma$ ). (b) Comparison between PVC-VUS and C-PVC-VUS. JiuZhang3.0-7B shows the smallest gap, demonstrating superior calibration.

Figure 5: Empirical C-PVC diagnostic surfaces on MATH-500. The x-axis ( $\tau$ ) represents calibration tolerance, y-axis ( $\gamma$ ) shows the accuracy threshold, and z-axis displays the empirical C-PVC proxy.

Figure 6: Judge correlation analysis. Left: Pearson correlation between model self-evaluation and judge verdicts. Right: Direct agreement rates. Judges exhibit distinct evaluative patterns, justifying our ensemble approach.

LLM Judge	Correlation	Agreement
Claude 3.7 Sonnet	$-$ 0.09	0.53
Amazon Nova Premier	0.18	0.61
DeepSeek-R1	0.01	0.55

Appendix

Table 9: Overall Pearson correlation and agreement rate between judges and model self-assessments on Math-360.

Model	PM-C-PVC-VUS (full)	PM-C-PVC-VUS (unanimous)
s1.1-7B	0.142	0.996
Qwen2.5-32B-Instruct	0.209	0.597
Claude 3.5 Haiku	0.123	0.588
JiuZhang3.0-7B	0.019	0.047
Llama-3.1-8B-Instruct	0.000	0.038

Appendix

Table 10: PM-C-PVC-VUS under full-label vs. unanimous-only re-analysis. Restricting to 3-0 unanimous judge cases substantially raises PM-C-PVC-VUS for stronger models (e.g. s1.1-7B to

0.996

), but the ranking structure and the near-zero regime for weaker models (JiuZhang3.0-7B, Llama-3.1-8B-Instruct) are preserved. The central pattern is therefore not an artifact of 2-1 judge splits.

Model	Flip	PVC-VUS	PM-C-PVC-VUS
s1.1-7B	0%	20.90	0.142
s1.1-7B	10%	20.09 $\pm$ 0.30	0.103 $\pm$ 0.053
s1.1-7B	30%	18.42 $\pm$ 0.58	0.059 $\pm$ 0.064
JiuZhang3.0-7B	0%	17.40	0.019
JiuZhang3.0-7B	10%	17.22 $\pm$ 0.18	0.019 $\pm$ 0.014
JiuZhang3.0-7B	30%	17.03 $\pm$ 0.29	0.023 $\pm$ 0.026

Appendix

Table 11: Label-perturbation robustness: PVC-VUS and PM-C-PVC-VUS under independent random flips of

10

–

30\%

of reference labels (mean

\pm

one standard deviation over

5

seeds per level). PVC-VUS degrades gradually with noise as expected, but the near-zero PM-C-PVC-VUS regime for JiuZhang3.0-7B remains stable under

30\%

corruption (

0.019

–

0.023

). The central finding is therefore not primarily a judge-noise artifact.

Figure 7: Correlation between Solution Accuracy (x-axis) and Self-Evaluation Accuracy (y-axis) across four benchmarks. Each point represents a (model, category) pair. The consistently low Pearson correlation coefficients ( $r$ ) across all datasets indicate that generation and evaluation capabilities are largely independent.

Model	Math-360		TruthfulQA		CSQA		MATH-500
Model	SA	SEA	SA	SEA	SA	SEA	SA	SEA
Qwen2.5-7B	83.1	59.7	29.2	56.5	70.0	57.1	63.0	56.1
Qwen2.5-7B-Instruct	89.6	58.1	40.2	61.3	78.7	60.4	70.6	56.2
Qwen2.5-Math-7B-Instruct	90.8	53.9	9.4	46.3	49.4	51.7	79.2	52.4
Llama-3.1-8B-Instruct	63.9	60.6	30.4	41.3	68.5	46.7	41.4	56.5
OpenThinker2-7B	94.7	64.9	39.0	54.4	75.8	54.6	86.7	62.2

Appendix

Table 12: Comparison of Solution Accuracy (SA) vs. Self-Evaluation Accuracy (SEA) across four benchmarks. All values in percentages (%). Notable inverse patterns, where models with lower SA achieve higher SEA (e.g., Open-Reasoner-Zero vs. OpenThinker2 on MATH-500; Llama-3.1 vs. Qwen2.5-Instruct on Math-360), demonstrate that self-evaluation is not merely a byproduct of reasoning capability. Bold : best per metric per dataset.

Metric	Discrimination	Calibration	Generalization
AUROC	✓	✗	✗
PVC-VUS	✓	✗	Partial
C-PVC-VUS	✓	✓	✗
PM-PVC-VUS	✓	✗	✓
PM-C-PVC-VUS	✓	✓	✓

Appendix

Table 13: Hierarchy of self-evaluation metrics and their guarantees. PVC-VUS provides partial generalization bounds that depend on the accuracy threshold

\gamma

but lack calibration constraints.

Property	AUROC	ECE	C-PVC
Discrimination-aware	✓	✗	✓
Calibration-aware	✗	✓	✓
Per-category constraints	✗	✗	✓
Decomposable	✗	Partial	✓
Generalization bounds	✗	✗	✓

Appendix

Table 14: Comparison of self-evaluation metrics across key properties.

Stage	Parameter	Value
Solution Generation	Temperature	0.7
	Top-P (nucleus sampling)	0.9
	Max tokens	4096
	Solutions per problem	2
Self-Evaluation	Presentation order	Randomized
	Confidence scale	0–100%

Appendix

Table 15: Experimental configuration for solution generation and evaluation.

Category	Subcategories
Arithmetic	Basic Operations, Fractions, Percentages, Numerical Approximation, Order of Operations
Algebra	Equations, Inequalities, Polynomials, Functions, Systems of Equations
Calculus	Differentiation, Integration, Limits, Series, Applications
Geometry	Plane Geometry, Coordinate Geometry, Transformations, Mensuration, Trigonometry
Number Theory	Divisibility, Modular Arithmetic, Primes, Diophantine Equations, Number Sequences
Combinatorics	Counting Principles, Permutations, Combinations, Probability, Recursion

Appendix

Table 16: Taxonomy of mathematical reasoning categories in our benchmark.

Category	Subcategory	Example Problem
Arithmetic	Basic Operations	Calculate $238+149$ .
	Fractions	What is $\frac{2}{5}+\frac{1}{5}$ ?
	Percentages	What is $25\%$ of $80$ ?
	Numerical Approximation	Round $47.68$ to the nearest whole number.
	Order of Operations	Calculate $3+4\times 2$ .
Algebra	Equations	Solve for $x$ : $x+5=12$ .

Appendix

Table 17: Representative problems from each subcategory in our mathematical reasoning benchmark 360.

Language Model Self-Assessment

Language model self-assessment studies how models judge their own confidence, capabilities, errors, and behavior.

See all 98 papers

Jun 20, 2026

Latent Confidence Alignment for LLM Self-Assessment

Confidence calibration in large language models (LLMs) is commonly evaluated by comparing predicted confidence with observed accuracy. However, such approaches do not model item difficulty, making it difficult to interpret discrepancies and to determine whether model confidence reflects genuine self-assessment or is merely a byproduct of the response generation process. To address this, we adopt a Rasch model-based latent ability framework and a metacognitive perspective, and propose Latent Confidence Alignment Error (LCAE) to measure the consistency between model self-assessment and the latent error probability implied by model ability and item difficulty. We further incorporate item difficulty as an external signal with a reasoning mechanism. Experiments on a medical-domain dataset with 20 models show that the proposed approach improves self-assessment quality without affecting model ability, and reveals an association between reliability and inference cost.

Ting-Yu Chen, Tingting Yu, Pei-Cing Huang +3

Department of Information Management National Sun Yat-Sen University Kaohsiung, Taiwan · Kaohsiung Medical University Hospital Kaohsiung Medical University Kaohsiung, Taiwan

LLM Evaluation Confidence Estimation in Language Models Language Model Self-Assessment Language Model Calibration Metacognition in Language Models

Ask PDF

Jun 3, 2026

Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data

Large language models are increasingly evaluated by other models, raising a natural question: can a model predict how a judge will score its own output? We find that the ability is largely present before any targeted training: prompted few-shot, a base model already predicts an external judge's multi-attribute quality scores on open-ended responses well above chance across three benchmarks. We introduce Self-Evaluation Elicitation (SEE), a method that surfaces this latent ability through a short cycle comprising a calibration-coupled reinforcement learning phase that improves the answer and predicts the judge, followed by a masked distillation phase that sharpens the prediction while leaving the answer untouched. From 160 unique examples, roughly 31x fewer than a reinforcement learning baseline, SEE improves held-out calibration across three benchmarks while preserving answer quality. The elicited self-evaluation is sharply localized within the model's own token distribution and stable across judges it was never trained against, indicating a transferable notion of quality rather than a single judge's preference. These results reframe judge-aligned self-evaluation as a problem of elicitation rather than acquisition.

XiuYu Zhang, Yi Shan, Junfeng Fang +1

National University of Singapore · Beijing University of Technology

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation Language Model Self-Assessment Language Model Calibration Evaluation Awareness in Language Models

Ask PDF

May 8, 2026

Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs

Large Language Models (LLMs) are increasingly used in settings where reliable self-assessment is critical. Assessing model reliability has evolved from using probabilistic correctness estimates to, more recently, eliciting verbalized confidence. Confidence, however, has been shown to be an inconsistent and overoptimistic predictor of model correctness. Drawing on cognitive appraisal theory, a framework from human psychology that decomposes self-evaluation into multiple components, we propose a multidimensional perspective on model self-assessment. We elicit six appraisal-based dimensions of self-assessment, alongside confidence, and evaluate their utility for predicting model failure across 12 LLMs and 38 tasks spanning eight domains. We find that competence-related appraisal dimensions, particularly effort and ability, consistently match or outperform confidence across most settings. Effort additionally yields less overoptimistic estimates that remain stable across model sizes. In contrast, affective dimensions provide marginally predictive signals. Furthermore, the most informative dimension varies systematically with task characteristics: effort is most predictive for reasoning-intensive tasks, while ability and confidence dominate on retrieval-oriented tasks. Broadly, our findings indicate that structured multidimensional self-assessment is a promising approach to improving the reliability and safety of language model deployment across diverse real-world settings.

Sree Bhattacharyya, Samarth Khanna, Leona Chen +3

Department of Informatics and Intelligent Systems, College of Information Sciences, The Pennsylvania State University. · Department of Computer Science and Engineering, College of Engineering, The Pennsylvania State University.

LLM Evaluation Confidence Estimation in Language Models Language Model Self-Assessment Language Model Calibration LLM Reliability LLM Uncertainty Estimation

Ask PDF

Abstract

Language Model Self-Assessment

Explore similar work

Latent Confidence Alignment for LLM Self-Assessment

Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data

Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs