Evaluating and Recalibrating Probabilistic Classifiers from One-Bit Threshold Queries

Roman Plaud; Antoine Saillenfest; Matthieu Labeau; Thomas Bonald; Willem Waegeman

cs.LGSep 4, 2026

Evaluating and Recalibrating Probabilistic Classifiers from One-Bit Threshold Queries

Authors: Roman Plaud, Antoine Saillenfest, Matthieu Labeau, Thomas Bonald, Willem Waegeman

Organizations: Institut Polytechnique de Paris · Onepoint, France · Ghent University

Abstract

Standard calibration evaluation and recalibration methods assume access to predicted probabilities. We study one-bit threshold access, a setting in which probabilities are hidden and the evaluator observes only whether the predicted probability exceeds a single chosen threshold per sample. Under this restriction, we show that classifiers can nevertheless be evaluated and recalibrated without reconstructing their hidden scores. We derive a consistent estimator of True Calibration Error, unbiased estimators of strictly proper scoring rules, and a one-bit objective for Temperature Scaling whose expectation matches the standard log-loss objective. Across these settings, we quantify the statistical cost incurred by replacing standard probability-access procedures by our one-bit estimators. Finally, we instantiate the framework for black-box LLMs, where the logit_bias API parameter allows us to realize the required threshold queries in zero-shot binary question answering.

Figures & tables

Figure 1: One-bit calibration auditing. (a) Empirical MSE-optimal resolution

M^{\star}

versus

N

for

A=4

, compared with the predicted

N^{1/5}

scaling. (b) MSE convergence for

A=2

under one-bit and probability access, with reference rates

N^{-2/5}

and

N^{-2/3}

. (c) Query cost versus calibration-error estimation accuracy across four LLMs. (d) Signed one-bit calibration contributions across score intervals for Mistral-7B on MMLU, compared with the white-box reference.

Figure 2: One-bit proper-score estimation and Temperature Scaling. (a) Oracle variance-minimizing

\beta^{\star}

within the

\mathrm{Beta}(\beta,\beta)

family as confident errors become more common. (b) Variance inflation of one-bit Log-loss relative to probability access; uniform sampling degrades quickly, while

\mathrm{Beta}(1/2,1/2)

remains stable. (c) Excess population Log-loss after TS on the synthetic family; centered one-bit TS with

\mathrm{Beta}(1/2,1/2)

closely tracks probability-access TS. (d) Average LLM recalibration efficiency versus API queries per calibration example. One-bit TS recovers over

98\%

of the probability-access TS improvement with one query, while the Carlini et al. baseline requires

K\geq 5

queries for comparable performance.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Empirical MSE-minimizing number of bins. For each

(A,N)

M^{\star}(A,N)

minimizes the Monte Carlo MSE over

M\in\{2,\ldots,40\}

. The solid line shows the fitted power law and the dashed line the theoretical scaling

M\propto N^{1/5}

. The fitted exponents are

0.24

0.18

0.22

0.22

0.21

, and

0.20

for

A=2,4,6,8,10,12

, respectively. Error bars show

95\%

bootstrap intervals for

M^{\star}

Figure 4: MSE as a function of the number of bins. Mean squared error of the one-bit calibration estimator over

500

independent trials for

A\in\{2,4,6,8,10,12\}

and four representative sample sizes. Each curve shows the MSE over

M\in\{2,\ldots,40\}

, and stars mark the empirical minimizers

M^{\star}(A,N)

. As

N

increases, the minimizing number of bins shifts toward larger values.

Figure 5: MSE convergence of the one-bit calibration estimator. MSE is computed over

500

independent trials using the empirically selected

M^{\star}(A,N)

at each sample size. The dashed line shows the theoretical upper-bound rate

N^{-2/5}

. The empirical log–log slopes are approximately

-0.41

-0.88

-0.75

-0.61

-0.65

, and

-0.69

for

A=2,4,6,8,10,12

, respectively.

Figure 6: Additional synthetic proper-score experiments. (a) Variance inflation of the one-bit Log-loss estimator under

T\sim\mathrm{Beta}(\beta,\beta)

; stars mark the variance-minimizing

\beta^{\star}

on the tested grid, while the vertical lines indicate

\mathrm{Beta}(1/2,1/2)

and Uniform sampling. (b) Log-loss variance inflation versus the fraction of confident errors for Uniform, fixed

\mathrm{Beta}(1/2,1/2)

, and oracle threshold proposals. (c) Empirical

J_{\mathrm{Brier},1}

under Uniform thresholds, compared with the theoretical bound

J_{\mathrm{Brier},1}\leq 1/3

. (d) Brier-score RMSE between one-bit and probability-access risk estimates as the sample size increases. (e) Log-loss RMSE for Uniform and

\mathrm{Beta}(1/2,1/2)

thresholds. Panels (d)–(e) use

A\in\{2,6,12\}

. All Monte Carlo results use

R=500

independent trials.

Figure 7: Finite-sample Temperature Scaling on the synthetic family. Top row: excess population Log-loss

R(\widehat{\alpha})-R(\alpha^{\star})

as the calibration-set size grows. Bottom row: mean absolute error in the fitted inverse temperature,

|\widehat{\alpha}-\alpha^{\star}|

. We compare probability-access TS with centered one-bit TS using Uniform and

\mathrm{Beta}(1/2,1/2)

threshold proposals. Endpoint-focused sampling becomes increasingly helpful as the scores become more extreme.

Figure 8: Effect of centering the one-bit TS objective. Empirical objectives for probability-access TS, the direct one-bit construction, and the centered one-bit construction at

A=6

and

N_{\mathrm{cal}}=2048

. The direct objective approaches zero near

\alpha=0

, whereas the centered objective has the correct boundary behavior and retains a useful minimum near

\alpha^{\star}

Model	Mean Abs. Probability Shift
Mistral v0.3-7B	23.2%
Llama 3.1-8B	19.7%
Qwen 2.5-7B	5.6%
Gemma 2-9B	4.0%

Appendix

Table 1: Sensitivity to the A/B token mapping. Mean absolute change in the probability assigned to the same semantic answer when the A/B labels are swapped.

Model	ID A	Decoded A	ID B	Decoded B
Qwen 2.5-7B	32	A	33	B
Llama 3.1-8B	32	A	33	B
Mistral v0.3-7B	29509	A	29528	B
Gemma 2-9B	235280	A	235305	B

Appendix

Table 2: Contextual token IDs for the designated A/B answer labels.

Figure 9: Magnitude of threshold-dependent logit-bias interventions. Empirical CDF of

|\sigma^{-1}(t)|

for thresholds drawn from Uniform

(0,1)

and Beta

(1/2,1/2)

. The Beta proposal samples thresholds closer to

0

and

1

, leading to larger interventions more frequently. Dashed lines indicate the 99th percentiles.

Estimator	Model Calls	Generated Tokens / Call	Logit Interventions
One-bit (ours)	1	1	1
Verbalized Confidence	1	10	0
Sampling Frequency ( $K$ )	$K$	1	0
Carlini Binary Search ( $K$ )	$K$	1	$K$

Appendix

Table 3: Query cost per sample for the evaluated restricted-access methods.

Figure 10: Calibration-estimation error across model–dataset pairs. Mean squared error relative to the probability-access ECE reference as a function of the number of queries per example. Results are shown for the four evaluated LLMs on BoolQ (top) and MMLU (bottom). The one-bit estimator uses one threshold query, while Sampling Frequency and Carlini-style logit extraction use repeated queries. The vertical range is restricted for readability; numerical results are reported in Table 4 .

Model	Dataset	ECE ref.	One-bit	Verb.	$K=1$	$K=4$	$K=8$	$K=1$	$K=4$	$K=8$
			Single query		Sampling			Carlini
Gemma-2-9B	BoolQ	0.218	0.025	0.115	0.110	0.024	0.010	0.101	0.003	0.002
Llama-3.1-8B	BoolQ	0.114	0.018	0.208	0.305	0.082	0.044	0.271	0.035	0.001
Mistral-7B-v0.3	BoolQ	0.265	0.011	0.089	0.201	0.047	0.011	0.192	0.009	0.001
Qwen2.5-7B	BoolQ	0.346	0.057	0.143	0.030	0.004	0.002	0.040	$<0.001$	$<0.001$
Gemma-2-9B	MMLU	0.131	0.001	0.110	0.164	0.043	0.021	0.133	0.004	$<0.001$

Appendix

Table 4: Calibration-auditing results across all model–dataset pairs. The third column reports the probability-access ECE reference. All remaining entries report root mean squared error (RMSE) relative to this reference. One-bit and Verbalized Confidence use one query per example; Sampling and Carlini-style extraction are reported for

K\in\{1,4,8\}

queries. Lower RMSE is better.

Figure 11: One-bit calibration profiles across all model–dataset pairs. Signed calibration-error contributions for the four evaluated LLMs on BoolQ (top) and MMLU (bottom). Bars show the one-bit estimates and curves show the corresponding probability-access contributions. Positive values indicate underconfidence and negative values indicate overconfidence. The one-bit profiles generally recover the main structure and sign of the probability-access profiles.

Figure 12: One-bit proper-score estimation on LLMs. RMSE relative to the probability-access risk for all eight model–dataset pairs, averaged over

R=500

threshold draws. (a) Brier score with Uniform thresholds. (b) Log-loss with Uniform and

\mathrm{Beta}(1/2,1/2)

thresholds; the vertical axis is logarithmic. Each one-bit estimate uses one threshold query per example.

Figure 13: Temperature Scaling on individual LLM–dataset pairs. Percentage of the probability-access TS improvement recovered on the held-out set. One-bit Uniform and

\mathrm{Beta}(1/2,1/2)

use one threshold query per calibration example; Carlini-style extraction uses

K

repeated queries. The dashed line denotes probability-access TS. Values below

50\%

fall outside the displayed range.

Language Model Calibration

Language model calibration studies whether models’ confidence estimates match their actual accuracy across tasks, prompts, and deployment settings.

See all 381 papers

On Calibration of Large Language Models: From Response To Capability
Feb 14, 2026Sin-Han Yang, Cheng-Kuang Wu, Chieh-Yen Lin +3LLM Evaluation Confidence Estimation in Language Models
Appier AI Research · National Taiwan University
Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs
Jan 6, 2026David Hartmann, Lena Pohlmann, Lelia Hanslik +3LLM Auditing Algorithmic Fairness
Weizenbaum Institut Berlin · Technische Universität Berlin · FIZ Karlsruhe +2
CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction
Jun 4, 2026Mohammad Anas Jawad, Cornelia CarageaConfidence Estimation in Language Models Language Model Calibration
Computer Science, University of Illinois Chicago, USA.

Feb 14, 2026J/K move · Enter open · S save

On Calibration of Large Language Models: From Response To Capability

Sin-Han Yang, Cheng-Kuang Wu, Chieh-Yen Lin, Yun-Nung Chen+2

Appier AI Research · National Taiwan University

LLM Evaluation Confidence Estimation in Language Models Language Model Calibration LLM Reliability LLM Uncertainty Estimation Confidence Estimation

Open paper AskPDF arXiv

Accurate confidence estimation is critical for reliable use of large language models (LLMs). Prior work on LLM calibration largely focuses on response-level confidence, which estimates the correctness of a single generated output. However, this formulation is misaligned with many practical settings where the central question is how likely a model is to solve a query overall. We show that this mismatch results from the stochastic nature of modern LLM decoding, under which single-response correctness fails to reflect underlying model capability. To address this issue, we introduce capability calibration, a new evaluation framework for measuring how well query-level confidence aligns with a model's expected accuracy on individual queries. We formally distinguish capability calibration (CC) from response calibration (RC) and show that the two differ both theoretically and empirically. We further show that CC is better suited than RC to applications like pass@k prediction and inference budget allocation. Finally, we evaluate common confidence estimation methods to understand the practical feasibility of CC.

Abstract

Language Model Calibration

Explore similar work

On Calibration of Large Language Models: From Response To Capability

Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs

CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction