Standard calibration evaluation and recalibration methods assume access to predicted probabilities. We study one-bit threshold access, a setting in which probabilities are hidden and the evaluator observes only whether the predicted probability exceeds a single chosen threshold per sample. Under this restriction, we show that classifiers can nevertheless be evaluated and recalibrated without reconstructing their hidden scores. We derive a consistent estimator of True Calibration Error, unbiased estimators of strictly proper scoring rules, and a one-bit objective for Temperature Scaling whose expectation matches the standard log-loss objective. Across these settings, we quantify the statistical cost incurred by replacing standard probability-access procedures by our one-bit estimators. Finally, we instantiate the framework for black-box LLMs, where the logit_bias API parameter allows us to realize the required threshold queries in zero-shot binary question answering.
Figures & tables
Figure 1: One-bit calibration auditing. (a) Empirical MSE-optimal resolution M⋆ versus N for A=4 , compared with the predicted N1/5 scaling. (b) MSE convergence for A=2 under one-bit and probability access, with reference rates N−2/5 and N−2/3 . (c) Query cost versus calibration-error estimation accuracy across four LLMs. (d) Signed one-bit calibration contributions across score intervals for Mistral-7B on MMLU, compared with the white-box reference.
Figure 2: One-bit proper-score estimation and Temperature Scaling. (a) Oracle variance-minimizing β⋆ within the Beta(β,β) family as confident errors become more common. (b) Variance inflation of one-bit Log-loss relative to probability access; uniform sampling degrades quickly, while Beta(1/2,1/2) remains stable. (c) Excess population Log-loss after TS on the synthetic family; centered one-bit TS with Beta(1/2,1/2) closely tracks probability-access TS. (d) Average LLM recalibration efficiency versus API queries per calibration example. One-bit TS recovers over 98% of the probability-access TS improvement with one query, while the Carlini et al. baseline requires K≥5 queries for comparable performance.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Empirical MSE-minimizing number of bins. For each (A,N) , M⋆(A,N) minimizes the Monte Carlo MSE over M∈{2,…,40} . The solid line shows the fitted power law and the dashed line the theoretical scaling M∝N1/5 . The fitted exponents are 0.24 , 0.18 , 0.22 , 0.22 , 0.21 , and 0.20 for A=2,4,6,8,10,12 , respectively. Error bars show 95% bootstrap intervals for M⋆ .
Figure 4: MSE as a function of the number of bins. Mean squared error of the one-bit calibration estimator over 500 independent trials for A∈{2,4,6,8,10,12} and four representative sample sizes. Each curve shows the MSE over M∈{2,…,40} , and stars mark the empirical minimizers M⋆(A,N) . As N increases, the minimizing number of bins shifts toward larger values.
Figure 5: MSE convergence of the one-bit calibration estimator. MSE is computed over 500 independent trials using the empirically selected M⋆(A,N) at each sample size. The dashed line shows the theoretical upper-bound rate N−2/5 . The empirical log–log slopes are approximately −0.41 , −0.88 , −0.75 , −0.61 , −0.65 , and −0.69 for A=2,4,6,8,10,12 , respectively.
Figure 6: Additional synthetic proper-score experiments. (a) Variance inflation of the one-bit Log-loss estimator under T∼Beta(β,β) ; stars mark the variance-minimizing β⋆ on the tested grid, while the vertical lines indicate Beta(1/2,1/2) and Uniform sampling. (b) Log-loss variance inflation versus the fraction of confident errors for Uniform, fixed Beta(1/2,1/2) , and oracle threshold proposals. (c) Empirical JBrier,1 under Uniform thresholds, compared with the theoretical bound JBrier,1≤1/3 . (d) Brier-score RMSE between one-bit and probability-access risk estimates as the sample size increases. (e) Log-loss RMSE for Uniform and Beta(1/2,1/2) thresholds. Panels (d)–(e) use A∈{2,6,12} . All Monte Carlo results use R=500 independent trials.
Figure 7: Finite-sample Temperature Scaling on the synthetic family. Top row: excess population Log-loss R(α)−R(α⋆) as the calibration-set size grows. Bottom row: mean absolute error in the fitted inverse temperature, ∣α−α⋆∣ . We compare probability-access TS with centered one-bit TS using Uniform and Beta(1/2,1/2) threshold proposals. Endpoint-focused sampling becomes increasingly helpful as the scores become more extreme.
Figure 8: Effect of centering the one-bit TS objective. Empirical objectives for probability-access TS, the direct one-bit construction, and the centered one-bit construction at A=6 and Ncal=2048 . The direct objective approaches zero near α=0 , whereas the centered objective has the correct boundary behavior and retains a useful minimum near α⋆ .
Model
Mean Abs. Probability Shift
Mistral v0.3-7B
23.2%
Llama 3.1-8B
19.7%
Qwen 2.5-7B
5.6%
Gemma 2-9B
4.0%
Appendix
Table 1: Sensitivity to the A/B token mapping. Mean absolute change in the probability assigned to the same semantic answer when the A/B labels are swapped.
Model
ID A
Decoded A
ID B
Decoded B
Qwen 2.5-7B
32
A
33
B
Llama 3.1-8B
32
A
33
B
Mistral v0.3-7B
29509
A
29528
B
Gemma 2-9B
235280
A
235305
B
Appendix
Table 2: Contextual token IDs for the designated A/B answer labels.
Figure 9: Magnitude of threshold-dependent logit-bias interventions. Empirical CDF of ∣σ−1(t)∣ for thresholds drawn from Uniform (0,1) and Beta (1/2,1/2) . The Beta proposal samples thresholds closer to 0 and 1 , leading to larger interventions more frequently. Dashed lines indicate the 99th percentiles.
Estimator
Model Calls
Generated Tokens / Call
Logit Interventions
One-bit (ours)
1
1
1
Verbalized Confidence
1
10
0
Sampling Frequency ( K )
K
1
0
Carlini Binary Search ( K )
K
1
K
Appendix
Table 3: Query cost per sample for the evaluated restricted-access methods.
Figure 10: Calibration-estimation error across model–dataset pairs. Mean squared error relative to the probability-access ECE reference as a function of the number of queries per example. Results are shown for the four evaluated LLMs on BoolQ (top) and MMLU (bottom). The one-bit estimator uses one threshold query, while Sampling Frequency and Carlini-style logit extraction use repeated queries. The vertical range is restricted for readability; numerical results are reported in Table 4 .
Single query
Sampling
Carlini
Model
Dataset
ECE ref.
One-bit
Verb.
K=1
K=4
K=8
K=1
K=4
K=8
Gemma-2-9B
BoolQ
0.218
0.025
0.115
0.110
0.024
0.010
0.101
0.003
0.002
Llama-3.1-8B
BoolQ
0.114
0.018
0.208
0.305
0.082
0.044
0.271
0.035
0.001
Mistral-7B-v0.3
BoolQ
0.265
0.011
0.089
0.201
0.047
0.011
0.192
0.009
0.001
Qwen2.5-7B
BoolQ
0.346
0.057
0.143
0.030
0.004
0.002
0.040
<0.001
<0.001
Gemma-2-9B
MMLU
0.131
0.001
0.110
0.164
0.043
0.021
0.133
0.004
<0.001
Appendix
Table 4: Calibration-auditing results across all model–dataset pairs. The third column reports the probability-access ECE reference. All remaining entries report root mean squared error (RMSE) relative to this reference. One-bit and Verbalized Confidence use one query per example; Sampling and Carlini-style extraction are reported for K∈{1,4,8} queries. Lower RMSE is better.
Figure 11: One-bit calibration profiles across all model–dataset pairs. Signed calibration-error contributions for the four evaluated LLMs on BoolQ (top) and MMLU (bottom). Bars show the one-bit estimates and curves show the corresponding probability-access contributions. Positive values indicate underconfidence and negative values indicate overconfidence. The one-bit profiles generally recover the main structure and sign of the probability-access profiles.
Figure 12: One-bit proper-score estimation on LLMs. RMSE relative to the probability-access risk for all eight model–dataset pairs, averaged over R=500 threshold draws. (a) Brier score with Uniform thresholds. (b) Log-loss with Uniform and Beta(1/2,1/2) thresholds; the vertical axis is logarithmic. Each one-bit estimate uses one threshold query per example.
Figure 13: Temperature Scaling on individual LLM–dataset pairs. Percentage of the probability-access TS improvement recovered on the held-out set. One-bit Uniform and Beta(1/2,1/2) use one threshold query per calibration example; Carlini-style extraction uses K repeated queries. The dashed line denotes probability-access TS. Values below 50% fall outside the displayed range.
Accurate confidence estimation is critical for reliable use of large language models (LLMs). Prior work on LLM calibration largely focuses on response-level confidence, which estimates the correctness of a single generated output. However, this formulation is misaligned with many practical settings where the central question is how likely a model is to solve a query overall. We show that this mismatch results from the stochastic nature of modern LLM decoding, under which single-response correctness fails to reflect underlying model capability. To address this issue, we introduce capability calibration, a new evaluation framework for measuring how well query-level confidence aligns with a model's expected accuracy on individual queries. We formally distinguish capability calibration (CC) from response calibration (RC) and show that the two differ both theoretically and empirically. We further show that CC is better suited than RC to applications like pass@k prediction and inference budget allocation. Finally, we evaluate common confidence estimation methods to understand the practical feasibility of CC.
Large Language Models (LLMs) exhibit systematic biases across demographic groups. Auditing is proposed as an accountability tool for black-box LLM applications, but suffers from resource-intensive query access. We conceptualise auditing as uncertainty estimation over a target fairness metric and introduce BAFA, the Bounded Active Fairness Auditor for query-efficient auditing of black-box LLMs. BAFA maintains a version space of surrogate models consistent with queried scores and computes uncertainty intervals for fairness metrics (e.g., Δ AUC) via constrained empirical risk minimisation. Active query selection narrows these intervals to reduce estimation error. We evaluate BAFA on two standard fairness dataset case studies: CivilComments and Bias-in-Bios, comparing against stratified sampling, power sampling, and ablations. BAFA achieves target error thresholds with up to 40× fewer queries than stratified sampling (e.g., 144 vs 5,956 queries at ε=0.02 for CivilComments) for tight thresholds, demonstrates substantially better performance over time, and shows lower variance across runs. These results suggest that active sampling can reduce resources needed for independent fairness auditing with LLMs, supporting continuous model evaluations.
David Hartmann, Lena Pohlmann, Lelia Hanslik +3
Weizenbaum Institut Berlin · Technische Universität Berlin · FIZ Karlsruhe +2
Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's {\em behavioral robustness} to irrelevant or misleading information. In this paper, we argue that a model's true confidence should reflect its stability under cognitive pressure. We introduce CaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction. CaliDist quantifies how an LLM's predictions and uncertainty change when its input prompt is perturbed with semantic distractors. This stability (or lack thereof) signal is then used to adaptively scale the model's initial confidence score. Our extensive experiments on seven Natural Language Understanding classification benchmarks using six distinct LLMs show that CaliDist consistently achieves lower Expected Calibration Error (ECE) and Brier Score compared with strong baselines. Remarkably, our method reduces the ECE from 23% to 7% on average--a relative improvement of 70%--demonstrating that behavioral stability is a powerful signal for calibration. We make our code and datasets available at github.com/m-anas-j/CaliDist.
Mohammad Anas Jawad, Cornelia Caragea
Computer Science, University of Illinois Chicago, USA.