As large language models (LLMs) are increasingly used to simulate human behavior, their operational validity is typically assessed through accuracy alone, implicitly treating human behavior as deterministic. We argue that on noisy digital trace data, a model's performance depends not only on predicting the correct outcome but on its calibration. We present a case study testing whether the token probabilities of open-weight LLMs can serve as confidence estimates in a Next Like Prediction task: predicting how a user of the Luxembourgish news platform RTL.lu rates a comment, conditioned on their prior interactions. Evaluating Ministral 3 (3B, 8B), we find that, with in-context learning only, the tested models are overconfident and are outperformed on accuracy and calibration by a simple statistical baseline. Accuracy and calibration do not move together: scaling from 3B to 8B improves accuracy while degrading calibration, and longer behavioral histories raise accuracy while confidence declines. Task-specific supervised fine-tuning acts as an effective calibration intervention, reducing Expected Calibration Error from 0.2 to 0.006 while improving accuracy from 0.536 to 0.758. We conclude that confidence estimation should be evaluated alongside accuracy to improve validity when using LLM-based agents.
Figures & tables
Model
Accuracy ( ↑ )
ROC-AUC ( ↑ )
Brier Score ( ↓ )
ECE ( ↓ )
Conf Cor. ( ↑ )
Conf Incor. ( ↓ )
Statistical Baseline
0.5900
0.5488
0.2476
0.0642
0.6581
0.6346
Ministral 3 3B
0.4906
0.6128
0.2813
0.1650
0.6632
0.6482
Ministral 3 3B (SFT)
0.7089
0.7600
0.1880
0.0087
0.7303
0.6434
Ministral 3 8B
0.5355
0.5872
0.2937
0.2004
0.7494
0.7202
Ministral 3 8B (SFT)
0.7578
0.8253
0.1618
0.0056
0.7828
0.6616
Gemini 3.5 Flash Lite
0.6868
-
-
-
-
-
Table 1: Comparative Model Performance and Calibration. Results detail discrete accuracy and probabilistic calibration across all evaluated configurations. Best values per metric are bolded .
Figure 1: Reliability diagram comparing model calibration on the Next Like Prediction task.
Model
History → Acc
History → Conf
Δ100
r
Δ100
r
Statistical Baseline
0.195
0.49∗∗∗
-0.017
−0.20
Ministral 3 3B
0.100
0.23
-0.068
−0.71∗∗∗
Ministral 3 3B (SFT)
0.095
0.34∗∗
0.050
0.49∗∗∗
Ministral 3 8B
0.247
0.50∗∗∗
-0.087
−0.64∗∗∗
Ministral 3 8B (SFT)
0.086
0.33∗∗
0.045
0.43∗∗∗
Table 2: Contextual Scaling. Δ100 represents the linear change in accuracy and confidence per 100 interactions, rescaled from a slope fitted over the observed range (at most 72 interactions) and r denotes the Pearson correlation coefficient. Significance levels: ∗p<0.05 , ∗∗p<0.01 , ∗∗∗p<0.001 .
Accurate confidence estimation is critical for reliable use of large language models (LLMs). Prior work on LLM calibration largely focuses on response-level confidence, which estimates the correctness of a single generated output. However, this formulation is misaligned with many practical settings where the central question is how likely a model is to solve a query overall. We show that this mismatch results from the stochastic nature of modern LLM decoding, under which single-response correctness fails to reflect underlying model capability. To address this issue, we introduce capability calibration, a new evaluation framework for measuring how well query-level confidence aligns with a model's expected accuracy on individual queries. We formally distinguish capability calibration (CC) from response calibration (RC) and show that the two differ both theoretically and empirically. We further show that CC is better suited than RC to applications like pass@k prediction and inference budget allocation. Finally, we evaluate common confidence estimation methods to understand the practical feasibility of CC.
Large language models (LLM) can state their confidence in an answer (verbalized confidence), or the confidence can be read from the probability of the answer's tokens (token confidence). Comparisons of the two ask which is better calibrated, but the token probability of an answer depends on the text that precedes it. Across five open LLMs and four question-answering datasets, we fix each answer, its verbalized confidence, and its correctness label, and vary how each score is measured. The scores themselves change with the protocol, and so does the comparison between them. Changing the context in which the token probability is read shifts the gap between the two scores by about the size of the gap itself (.287 vs. .304 in expected calibration error, ECE; .189 vs. .145 in AUROC). The 95% interval of the shift excludes zero in 19 and 17 of the 20 settings, and under AUROC the shift has the same sign in all of them, so the context changes how answers are ranked. The better score flips in 8 settings under each metric, counting only flips that the intervals support, and the shift also appears in four larger models of up to 72B parameters. Most of the shift appears with changes to the instruction, not to the position of the answer. Other measurement choices also matter, and stated confidence depends on how a supplied answer is written. We turn these results into a reporting checklist for LLM confidence and its comparisons.
We investigate the calibration of large language models' (LLMs') confidence across diverse tasks. The results of our preregistered study show that the current crop of LLMs are, like people, too sure they are right: confidence exceeds accuracy, on average. Importantly, however, this tendency is moderated by a powerful hard-easy effect, wherein overconfidence is greatest on difficult tests; by contrast, easy tests actually show substantial underconfidence. We develop LifeEval, a test for evaluating model calibration across levels of difficulty.