Also Small Models Can Reasonably Self-Evaluate Their Confidence

Idil Kapikiran; Thomas Decker; Thomas Runkler

cs.LGSep 30, 2026

Also Small Models Can Reasonably Self-Evaluate Their Confidence

Authors: Idil Kapikiran, Thomas Decker, Thomas Runkler

Organizations: Technical University of Munich · Siemens AG · LMU Munich · Munich Center for Machine Learning (MCML)

Abstract

This study systematically evaluates self-evaluation-based uncertainty quantification across different language models of varying sizes on question-answering tasks spanning general to specialized knowledge domains. Using various self-evaluation methods where models judge their own predictions, we examine how model scale and domain specificity affect the quality of self-assessed confidence signals. Our results reveal that while accuracy predictably declines with smaller models and more specialized domains, the reliability of self-evaluated confidence remains largely stable across both dimensions. This independence means the most capable model is not necessarily the best at self-assessing prediction reliability. These findings suggest that smaller models can achieve reasonable self-assessed confidence despite lower accuracy, making them viable for resource-constrained deployments.

Figures & tables

	TQA				MMLU				MedMCQA
	OpenAI GPT-4.1		Ministral-3-3B		OpenAI GPT-4.1		Ministral-3-3B		OpenAI GPT-4.1		Ministral-3-3B
Method	Acc.	CalAUC	Acc.	CalAUC	Acc.	CalAUC	Acc.	CalAUC	Acc.	CalAUC	Acc.	CalAUC
P(True) (single)	64.95	70.84	36.47	57.18	55.79	62.23	34.88	66.85	42.20	63.91	20.20	65.24
Seq likelihood	71.89	47.92	37.58	57.00	62.61	56.54	33.12	60.44	44.90	63.29	19.80	62.13
Seq len-norm likelihood	71.64	52.71	35.86	56.50	62.75	59.19	33.05	63.06	45.10	66.95	19.20	69.64
Sample and Select	73.61	50.98	38.56	56.26	64.32	51.64	34.94	54.53	45.10	55.36	19.40	54.94

Table 1 : Accuracy and CalAUC (%) by method, across datasets and models, comparing single-generation and multi-generation (

K=4

sampled) scoring strategies for OpenAI GPT-4.1 and Ministral-3-3B. Bold with green highlight marks the best value in each dataset/model/metric column.

Panel of Figure 1 — Figure 1 : Best-performing method per model and dataset, with models ordered by decreasing parameter count (left to right). Each point represents the best scoring strategy for that model/dataset pair. Figure (a) Accuracy declines predictably with decreasing model size and increasing domain specialization. Figure (b) Calibration-AUC remains stable across both dimensions, showing no systematic dependence on scale or domain difficulty.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

		OpenAI GPT-4.1		GPT-OSS-120B		Qwen3.5-9B		Gemma-4-E4B-it		Ministral-3-3B
Dataset	Method	Acc.	CalAUC	Acc.	CalAUC	Acc.	CalAUC	Acc.	CalAUC	Acc.	CalAUC
TQA	P(True) (single gen)	64.95	70.84	59.98	50.87	60.71	73.09	48.47	71.15	36.47	57.18
	Seq likelihood	71.89	47.92	60.00	62.71	58.87	56.76	50.31	55.06	37.58	57.00
	Seq len-norm likelihood	71.64	52.71	60.12	69.19	59.85	63.54	50.06	61.36	35.86	56.50
	Sample and Select	73.61	50.98	64.42	58.45	61.57	52.60	50.92	50.08	38.56	56.26
	Sample and Select w/ nota	73.37	56.59	63.19	58.31	61.08	59.77	50.80	57.75	37.21	61.78

Appendix

Table 2 : Accuracy and CalAUC (%) by method and dataset, across all models. Includes all 8 multi-generation (

K=4

sampled) methods plus single-generation P(True). Bold with green highlight marks the best value in each model/metric column per dataset.

Model	Parameters	Checkpoint / Endpoint	Access
GPT-4.1	undisclosed	openai/gpt-4.1	OpenAI API
GPT-OSS-120B	120B	openai/gpt-oss-120b	vLLM (HF)
Qwen3.5-9B	9B	Qwen/Qwen3.5-9B	vLLM (HF)
Gemma-4-E4B-it	4B (eff.)	google/gemma-4-e4b-it	vLLM (HF)
Ministral-3-3B	3B	mistralai/Ministral-3B-Instruct-2512	vLLM (HF)

Appendix

Table 3: Models evaluated. HF = Hugging Face Hub checkpoint.

Explore similar work

Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs
May 8, 2026Sree Bhattacharyya, Samarth Khanna, Leona Chen +3Self-Evaluation Large Language Model Reliability
Department of Informatics and Intelligent Systems, College of Information Sciences, The Pennsylvania State University. · Department of Computer Science and Engineering, College of Engineering, The Pennsylvania State University.
An Analysis of Training-Free Self-Reported Confidence in Language Models
Sep 17, 2026Lukas Meyer, Sofia Rossi, Wei Chen +2Confidence Calibration Large Language Model Reliability
DreamAI
Latent Confidence Alignment for LLM Self-Assessment
Jun 20, 2026Ting-Yu Chen, Tingting Yu, Pei-Cing Huang +3Self-Evaluation Confidence Calibration
Department of Information Management National Sun Yat-Sen University Kaohsiung, Taiwan · Kaohsiung Medical University Hospital Kaohsiung Medical University Kaohsiung, Taiwan

May 8, 2026, cs.CLJ/K move · Enter open · S save

Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs

Sree Bhattacharyya, Samarth Khanna, Leona Chen, Lucas Craig+2

Department of Informatics and Intelligent Systems, College of Information Sciences, The Pennsylvania State University. · Department of Computer Science and Engineering, College of Engineering, The Pennsylvania State University.

Self-Evaluation Large Language Model Reliability Confidence Calibration Competence

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly used in settings where reliable self-assessment is critical. Assessing model reliability has evolved from using probabilistic correctness estimates to, more recently, eliciting verbalized confidence. Confidence, however, has been shown to be an inconsistent and overoptimistic predictor of model correctness. Drawing on cognitive appraisal theory, a framework from human psychology that decomposes self-evaluation into multiple components, we propose a multidimensional perspective on model self-assessment. We elicit six appraisal-based dimensions of self-assessment, alongside confidence, and evaluate their utility for predicting model failure across 12 LLMs and 38 tasks spanning eight domains. We find that competence-related appraisal dimensions, particularly effort and ability, consistently match or outperform confidence across most settings. Effort additionally yields less overoptimistic estimates that remain stable across model sizes. In contrast, affective dimensions provide marginally predictive signals. Furthermore, the most informative dimension varies systematically with task characteristics: effort is most predictive for reasoning-intensive tasks, while ability and confidence dominate on retrieval-oriented tasks. Broadly, our findings indicate that structured multidimensional self-assessment is a promising approach to improving the reliability and safety of language model deployment across diverse real-world settings.

Abstract

Explore similar work

Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs

An Analysis of Training-Free Self-Reported Confidence in Language Models

Latent Confidence Alignment for LLM Self-Assessment