Zero-shot large language models (LLMs) are increasingly used as text classifiers, but their labels are only useful when we know to what extent we can trust them. We compare black-box methods (verbalised confidence and sampling) with white-box methods that read token probabilities (P(True), and multiple logit-scoring methods using constrained generation). We evaluate eleven open-weight LLMs (1B-32B parameters) on five classification datasets with 3 to 151 labels, measuring classification performance (accuracy, F1-macro), discrimination (AUROC), and calibration (ECE and Brier score). The choice of confidence method rarely impacts accuracy significantly, but logit-based scoring methods lead to much better discrimination (AUROC 0.74-0.77) than black-box methods (AUROC 0.57-0.66). Replacing labels with single-token identifiers preserves discrimination but costs 3-6 accuracy points on average. We also introduce decision-only scoring, which only scores positions where labels diverge in the prefix tree (trie). It reduces decode steps by 88-96% on the three largest label sets and performs comparably to full-label scoring. The best open-weight model-method configurations (Gemma 4 31B and Qwen 3.5 9B) are competitive with the commercial Jev 1.13.0 classifier.
Figures & tables
Method
Accuracy
F1-macro
AUROC
ECE
Brier
Unconstrained (no conf)
0.724 ∗
0.715 ∗
0.500
0.276
0.276
Unconstrained (no conf ABC)
0.671
0.663
0.500
0.329
0.329
Unconstrained (10 samples)
0.724 ∗
0.715 ∗
0.569
0.254
0.255
Unconstrained + P(True)
0.724 ∗
0.715 ∗
0.741 ∗
0.227
0.235
Unconstrained JSON (no conf)
0.723 ∗
0.718 ∗
0.500
0.277
0.277
Unconstrained JSON (no conf ABC)
0.686
0.680
0.500
0.314
0.314
Table 1: Mean score over all model–dataset pairs (excluding Llama 3.2 1B, Llama 3.2 3B). Bold indicates the best mean. ∗ indicates a method is not significantly different from the method with the best mean rank (two-sided Wilcoxon signed-rank test paired over model-dataset pairs, Holm-adjusted p>0.05 ).
Table 2: Best model–method combinations for each metric, best constrained method (accuracy), best ABC-logit method (accuracy), compared against Jev 1.13.0. Scores are means across all datasets.
Figure 2: Rate of errors made by each model (all unconstrained inference methods: plaintext, JSON with no confidence, verbal pre and post, Likert and percentage) plotted against number of model parameters (in billions) and grouped by model family.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Banking77
CLINC150
FEVER
SST-5
20NG
Llama 3.2 1B
+1.66% / -11.59%
+3.00% / -6.07%
+1.13% / -0.19%
+0.00% / -0.00%
+0.10% / -0.13%
Llama 3.2 3B
+0.55% / -0.39%
+2.56% / -1.82%
+0.59% / -0.15%
+0.09% / -0.18%
+0.21% / -0.13%
Gemma 4 E2B
+0.16% / -0.03%
+0.31% / -2.78%
+0.00% / -0.10%
+0.00% / -0.00%
+0.11% / -0.83%
Gemma 4 E4B
+0.03% / -0.03%
+0.04% / -0.02%
+0.00% / -0.00%
+0.00% / -0.00%
+0.00% / -0.00%
Gemma 4 31B
+0.00% / -0.00%
+0.04% / -0.00%
+0.01% / -0.01%
+0.00% / -0.05%
+0.03% / -0.00%
Qwen2.5 7B
+0.26% / -0.36%
+0.05% / -0.05%
+0.15% / -0.06%
+0.05% / -0.00%
+0.07% / -0.00%
Appendix
Table 3: The percentage of data points on which constrained plaintext generation helps ( + ) (constrained output is correct, unconstrained output is either incorrect or unparsable) and hinders ( − ) (unconstrained output is correct, constrained output is incorrect) classification performance by model and dataset. Both constrained and unconstrained generation use greedy decoding.
Banking77
CLINC150
FEVER
SST-5
20NG
Llama 3.2 1B
+4.22% / -20.13%
+18.20% / -8.62%
+32.25% / -35.92%
+11.22% / -24.75%
+3.93% / -3.13%
Llama 3.2 3B
+2.63% / -1.40%
+3.11% / -3.56%
+4.45% / -1.48%
+2.44% / -3.57%
+2.09% / -5.25%
Gemma 4 E2B
+0.75% / -0.91%
+0.58% / -0.91%
+1.01% / -7.62%
+3.35% / -5.29%
+1.24% / -2.27%
Gemma 4 E4B
+0.29% / -0.49%
+0.62% / -3.98%
+0.33% / -1.00%
+1.49% / -1.86%
+0.59% / -1.28%
Gemma 4 31B
+0.29% / -0.19%
+0.09% / -0.13%
+0.11% / -0.06%
+1.49% / -0.81%
+0.10% / -0.15%
Qwen2.5 7B
+1.92% / -2.79%
+3.56% / -3.80%
+0.93% / -0.86%
+1.76% / -2.81%
+1.80% / -1.57%
Appendix
Table 4: The percentage of data points on which constrained JSON generation helps ( + ) (constrained output is correct, unconstrained output is either incorrect or unparsable) and hinders ( − ) (unconstrained output is correct, constrained output is incorrect) classification performance by model and dataset. Both constrained and unconstrained generation use greedy decoding.
Method
Llama 3.2 1B
Llama 3.2 3B
Gemma 4 E2B
Gemma 4 E4B
Gemma 4 31B
Qwen2.5 7B
Qwen3 8B
Qwen3 14B
Qwen3 32B
Qwen3.5 4B
Qwen3.5 9B
Unconstrained (no conf)
0.245
0.487
0.645
0.728
0.801
0.681
0.706
0.731
0.749
0.720
0.751
Unconstrained (no conf ABC)
0.162
0.341
0.552
0.677
0.795
0.607
0.619
0.702
0.716
0.658
0.710
Unconstrained (10 samples)
0.245
0.490
0.646
0.729
0.801
0.680
0.707
0.731
0.749
0.720
0.751
Unconstrained + P(True)
0.245
0.487
0.645
0.728
0.801
0.681
0.706
0.731
0.749
0.720
0.751
Unconstrained JSON (no conf)
0.273
0.515
0.656
0.725
0.800
0.669
0.709
0.733
0.741
0.718
0.752
Unconstrained JSON (no conf ABC)
0.160
0.421
0.604
0.694
0.798
0.604
0.658
0.715
0.720
0.669
0.713
Appendix
Table 5: Accuracy (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.
Method
Llama 3.2 1B
Llama 3.2 3B
Gemma 4 E2B
Gemma 4 E4B
Gemma 4 31B
Qwen2.5 7B
Qwen3 8B
Qwen3 14B
Qwen3 32B
Qwen3.5 4B
Qwen3.5 9B
Unconstrained (no conf)
0.175
0.470
0.637
0.723
0.795
0.665
0.693
0.732
0.746
0.701
0.741
Unconstrained (no conf ABC)
0.134
0.315
0.546
0.681
0.789
0.588
0.603
0.695
0.710
0.648
0.703
Unconstrained (10 samples)
0.189
0.478
0.637
0.723
0.795
0.665
0.694
0.732
0.746
0.700
0.741
Unconstrained + P(True)
0.175
0.470
0.637
0.723
0.795
0.665
0.693
0.732
0.746
0.701
0.741
Unconstrained JSON (no conf)
0.235
0.503
0.661
0.721
0.793
0.663
0.701
0.738
0.739
0.697
0.748
Unconstrained JSON (no conf ABC)
0.098
0.389
0.609
0.698
0.790
0.590
0.650
0.710
0.715
0.657
0.703
Appendix
Table 6: F1-macro (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.
Method
Llama 3.2 1B
Llama 3.2 3B
Gemma 4 E2B
Gemma 4 E4B
Gemma 4 31B
Qwen2.5 7B
Qwen3 8B
Qwen3 14B
Qwen3 32B
Qwen3.5 4B
Qwen3.5 9B
Unconstrained (no conf)
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
Unconstrained (no conf ABC)
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
Unconstrained (10 samples)
0.581
0.654
0.525
0.559
0.515
0.539
0.532
0.535
0.581
0.673
0.659
Unconstrained + P(True)
0.473
0.754
0.685
0.719
0.767
0.724
0.726
0.751
0.786
0.749
0.759
Unconstrained JSON (no conf)
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
Unconstrained JSON (no conf ABC)
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
0.500
Appendix
Table 7: AUROC (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.
Method
Llama 3.2 1B
Llama 3.2 3B
Gemma 4 E2B
Gemma 4 E4B
Gemma 4 31B
Qwen2.5 7B
Qwen3 8B
Qwen3 14B
Qwen3 32B
Qwen3.5 4B
Qwen3.5 9B
Unconstrained (no conf)
0.755
0.513
0.355
0.272
0.199
0.319
0.294
0.269
0.251
0.280
0.249
Unconstrained (no conf ABC)
0.838
0.659
0.448
0.323
0.205
0.393
0.381
0.298
0.284
0.342
0.290
Unconstrained (10 samples)
0.495
0.386
0.344
0.253
0.196
0.302
0.281
0.257
0.231
0.221
0.201
Unconstrained + P(True)
0.331
0.323
0.333
0.225
0.187
0.273
0.266
0.229
0.190
0.179
0.158
Unconstrained JSON (no conf)
0.727
0.485
0.344
0.275
0.200
0.331
0.291
0.267
0.259
0.282
0.248
Unconstrained JSON (no conf ABC)
0.840
0.579
0.396
0.306
0.202
0.396
0.342
0.285
0.280
0.331
0.287
Appendix
Table 8: ECE (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.
Method
Llama 3.2 1B
Llama 3.2 3B
Gemma 4 E2B
Gemma 4 E4B
Gemma 4 31B
Qwen2.5 7B
Qwen3 8B
Qwen3 14B
Qwen3 32B
Qwen3.5 4B
Qwen3.5 9B
Unconstrained (no conf)
0.755
0.513
0.355
0.272
0.199
0.319
0.294
0.269
0.251
0.280
0.249
Unconstrained (no conf ABC)
0.838
0.659
0.448
0.323
0.205
0.393
0.381
0.298
0.284
0.342
0.290
Unconstrained (10 samples)
0.456
0.372
0.344
0.254
0.196
0.302
0.282
0.258
0.231
0.226
0.206
Unconstrained + P(True)
0.308
0.305
0.335
0.232
0.187
0.277
0.267
0.231
0.198
0.206
0.182
Unconstrained JSON (no conf)
0.727
0.485
0.344
0.275
0.200
0.331
0.291
0.267
0.259
0.282
0.248
Unconstrained JSON (no conf ABC)
0.840
0.579
0.396
0.306
0.202
0.396
0.342
0.285
0.280
0.331
0.287
Appendix
Table 9: Top-label Brier (mean score) across datasets, by model. Bold score represents the best performing for that model. Significance not tested as there are too few cells for the test to reach significance.
Zero-shot text classification assigns texts to categories without task-specific labeled training data, using language models, embeddings, or label descriptions.
Large language models (LLMs) are increasingly deployed as black-box classifiers in pipelines that automate confident decisions and route uncertain ones to human review. Such selective prediction needs a confidence score that an operator can threshold at a chosen risk level. Prior work asks whether LLM confidence is well calibrated or well ranked; we ask a complementary, deployment-oriented question that has been largely overlooked: at what resolution can the score be thresholded? We call the answer the score granularity gap. Through a controlled comparison of seven ways to build a confidence score, from a single verbalized number, to token probabilities, to querying the model many times and combining the answers, across 25 model-dataset pairs (9 LLMs, 3 benchmarks), we find that single-shot verbalized confidence, once correctly converted to a class probability, ranks cases surprisingly well, yet takes only a handful of distinct values. It therefore offers an operator only a few coarse thresholds, no matter how well it ranks. We show which constructions widen this gap, at what inference cost, and with what effect on ranking, notably that multi-query aggregation helps weak models but can degrade already-strong ones. We translate these trade-offs into concrete deployment guidance.
Uncertainty quantification (UQ) is essential for the safe deployment of large language models (LLMs). Existing methods, from verbalized confidence to ones requiring multiple generations, are often zero-shot and produce scores quantifying uncertainty without the need for labelled data. Nonetheless, in practice one must always evaluate their performance on a dataset of interest before deployment. In this work we show that, by leveraging this dataset, we consistently outperform these existing scores. Specifically, we build simple classifiers that predict LLM response correctness by using these scores and the correctness of similar queries as features. Our method produces minimal computational overhead, making it a cheap and straightforward enhancement for UQ in LLMs for real-world applications.
Sokhna Diarra Mbacke, Mouloud Belbahri, Gabriel Loaiza-Ganem
How reliably can a small language model estimate its own correctness? The answer determines whether local-to-cloud routing-escalating queries a cheap local model cannot handle-can work without supervised training data. As inference costs dominate large language model (LLM) deployment budgets, routing most queries to a cheap local model while reserving expensive cloud calls for hard cases is an increasingly common cost-control strategy. We compare zero-shot confidence signals against RouteLLM-style supervised baselines across three 7-8B model families and two datasets (1,000 and 500 queries per model, respectively). Average token log-probability, which requires no training data, matches or exceeds supervised baselines in-distribution (Area Under the Receiver Operating Characteristic curve (AUROC) 0.650-0.714 vs. 0.644-0.676) and substantially outperforms them out-of-distribution (0.717-0.833 vs. 0.512-0.564), because it measures a property of the model's generation rather than the query distribution. This paper further proposes retrieval-conditional self-assessment, a pre-generation signal that selectively injects retrieved knowledge when similarity is high, improving over bare self-assessment by up to +0.069 AUROC at 3-10x lower latency than log-probability. A supervised baseline trained on 1,000 labeled examples never exceeds the zero-shot signal. We release all code, data, and experiment logs.