LLMs Are Already Confidence-Aware Zero-Shot Classifiers
Organizations: University of Sheffield Department of Computer Science
Abstract
Zero-shot large language models (LLMs) are increasingly used as text classifiers, but their labels are only useful when we know to what extent we can trust them. We compare black-box methods (verbalised confidence and sampling) with white-box methods that read token probabilities (P(True), and multiple logit-scoring methods using constrained generation). We evaluate eleven open-weight LLMs (1B-32B parameters) on five classification datasets with 3 to 151 labels, measuring classification performance (accuracy, F1-macro), discrimination (AUROC), and calibration (ECE and Brier score). The choice of confidence method rarely impacts accuracy significantly, but logit-based scoring methods lead to much better discrimination (AUROC 0.74-0.77) than black-box methods (AUROC 0.57-0.66). Replacing labels with single-token identifiers preserves discrimination but costs 3-6 accuracy points on average. We also introduce decision-only scoring, which only scores positions where labels diverge in the prefix tree (trie). It reduces decode steps by 88-96% on the three largest label sets and performs comparably to full-label scoring. The best open-weight model-method configurations (Gemma 4 31B and Qwen 3.5 9B) are competitive with the commercial Jev 1.13.0 classifier.
Figures & tables
| Method | Accuracy | F1-macro | AUROC | ECE | Brier |
| Unconstrained (no conf) | 0.724 ∗ | 0.715 ∗ | 0.500 | 0.276 | 0.276 |
| Unconstrained (no conf ABC) | 0.671 | 0.663 | 0.500 | 0.329 | 0.329 |
| Unconstrained (10 samples) | 0.724 ∗ | 0.715 ∗ | 0.569 | 0.254 | 0.255 |
| Unconstrained + P(True) | 0.724 ∗ | 0.715 ∗ | 0.741 ∗ | 0.227 | 0.235 |
| Unconstrained JSON (no conf) | 0.723 ∗ | 0.718 ∗ | 0.500 | 0.277 | 0.277 |
| Unconstrained JSON (no conf ABC) | 0.686 | 0.680 | 0.500 | 0.314 | 0.314 |
| Model | Method | Accuracy | F1-macro | AUROC | ECE | Brier |
| Gemma 4 31B | Unconstrained JSON verbal (pre, Likert) | 0.803 | 0.796 | 0.642 | 0.120 | 0.145 |
| Gemma 4 31B | Unconstrained JSON verbal (post, Likert) | 0.802 | 0.796 | 0.673 | 0.107 | 0.140 |
| Qwen 3.5 9B | Constrained (Google length norm) | 0.750 | 0.741 | 0.802 | 0.107 | 0.153 |
| Qwen 3.5 9B | Constrained (length norm, decision-only) | 0.751 | 0.744 | 0.793 | 0.053 | 0.142 |
| Gemma 4 31B | Constrained JSON (Google length norm, decision-only) | 0.802 | 0.795 | 0.775 | 0.190 | 0.190 |
| Gemma 4 31B | Constrained JSON (ABC) | 0.798 | 0.790 | 0.772 | 0.196 | 0.196 |
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| Banking77 | CLINC150 | FEVER | SST-5 | 20NG | |
| Llama 3.2 1B | +1.66% / -11.59% | +3.00% / -6.07% | +1.13% / -0.19% | +0.00% / -0.00% | +0.10% / -0.13% |
| Llama 3.2 3B | +0.55% / -0.39% | +2.56% / -1.82% | +0.59% / -0.15% | +0.09% / -0.18% | +0.21% / -0.13% |
| Gemma 4 E2B | +0.16% / -0.03% | +0.31% / -2.78% | +0.00% / -0.10% | +0.00% / -0.00% | +0.11% / -0.83% |
| Gemma 4 E4B | +0.03% / -0.03% | +0.04% / -0.02% | +0.00% / -0.00% | +0.00% / -0.00% | +0.00% / -0.00% |
| Gemma 4 31B | +0.00% / -0.00% | +0.04% / -0.00% | +0.01% / -0.01% | +0.00% / -0.05% | +0.03% / -0.00% |
| Qwen2.5 7B | +0.26% / -0.36% | +0.05% / -0.05% | +0.15% / -0.06% | +0.05% / -0.00% | +0.07% / -0.00% |
| Banking77 | CLINC150 | FEVER | SST-5 | 20NG | |
| Llama 3.2 1B | +4.22% / -20.13% | +18.20% / -8.62% | +32.25% / -35.92% | +11.22% / -24.75% | +3.93% / -3.13% |
| Llama 3.2 3B | +2.63% / -1.40% | +3.11% / -3.56% | +4.45% / -1.48% | +2.44% / -3.57% | +2.09% / -5.25% |
| Gemma 4 E2B | +0.75% / -0.91% | +0.58% / -0.91% | +1.01% / -7.62% | +3.35% / -5.29% | +1.24% / -2.27% |
| Gemma 4 E4B | +0.29% / -0.49% | +0.62% / -3.98% | +0.33% / -1.00% | +1.49% / -1.86% | +0.59% / -1.28% |
| Gemma 4 31B | +0.29% / -0.19% | +0.09% / -0.13% | +0.11% / -0.06% | +1.49% / -0.81% | +0.10% / -0.15% |
| Qwen2.5 7B | +1.92% / -2.79% | +3.56% / -3.80% | +0.93% / -0.86% | +1.76% / -2.81% | +1.80% / -1.57% |
| Method | Llama 3.2 1B | Llama 3.2 3B | Gemma 4 E2B | Gemma 4 E4B | Gemma 4 31B | Qwen2.5 7B | Qwen3 8B | Qwen3 14B | Qwen3 32B | Qwen3.5 4B | Qwen3.5 9B |
| Unconstrained (no conf) | 0.245 | 0.487 | 0.645 | 0.728 | 0.801 | 0.681 | 0.706 | 0.731 | 0.749 | 0.720 | 0.751 |
| Unconstrained (no conf ABC) | 0.162 | 0.341 | 0.552 | 0.677 | 0.795 | 0.607 | 0.619 | 0.702 | 0.716 | 0.658 | 0.710 |
| Unconstrained (10 samples) | 0.245 | 0.490 | 0.646 | 0.729 | 0.801 | 0.680 | 0.707 | 0.731 | 0.749 | 0.720 | 0.751 |
| Unconstrained + P(True) | 0.245 | 0.487 | 0.645 | 0.728 | 0.801 | 0.681 | 0.706 | 0.731 | 0.749 | 0.720 | 0.751 |
| Unconstrained JSON (no conf) | 0.273 | 0.515 | 0.656 | 0.725 | 0.800 | 0.669 | 0.709 | 0.733 | 0.741 | 0.718 | 0.752 |
| Unconstrained JSON (no conf ABC) | 0.160 | 0.421 | 0.604 | 0.694 | 0.798 | 0.604 | 0.658 | 0.715 | 0.720 | 0.669 | 0.713 |
| Method | Llama 3.2 1B | Llama 3.2 3B | Gemma 4 E2B | Gemma 4 E4B | Gemma 4 31B | Qwen2.5 7B | Qwen3 8B | Qwen3 14B | Qwen3 32B | Qwen3.5 4B | Qwen3.5 9B |
| Unconstrained (no conf) | 0.175 | 0.470 | 0.637 | 0.723 | 0.795 | 0.665 | 0.693 | 0.732 | 0.746 | 0.701 | 0.741 |
| Unconstrained (no conf ABC) | 0.134 | 0.315 | 0.546 | 0.681 | 0.789 | 0.588 | 0.603 | 0.695 | 0.710 | 0.648 | 0.703 |
| Unconstrained (10 samples) | 0.189 | 0.478 | 0.637 | 0.723 | 0.795 | 0.665 | 0.694 | 0.732 | 0.746 | 0.700 | 0.741 |
| Unconstrained + P(True) | 0.175 | 0.470 | 0.637 | 0.723 | 0.795 | 0.665 | 0.693 | 0.732 | 0.746 | 0.701 | 0.741 |
| Unconstrained JSON (no conf) | 0.235 | 0.503 | 0.661 | 0.721 | 0.793 | 0.663 | 0.701 | 0.738 | 0.739 | 0.697 | 0.748 |
| Unconstrained JSON (no conf ABC) | 0.098 | 0.389 | 0.609 | 0.698 | 0.790 | 0.590 | 0.650 | 0.710 | 0.715 | 0.657 | 0.703 |
| Method | Llama 3.2 1B | Llama 3.2 3B | Gemma 4 E2B | Gemma 4 E4B | Gemma 4 31B | Qwen2.5 7B | Qwen3 8B | Qwen3 14B | Qwen3 32B | Qwen3.5 4B | Qwen3.5 9B |
| Unconstrained (no conf) | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 |
| Unconstrained (no conf ABC) | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 |
| Unconstrained (10 samples) | 0.581 | 0.654 | 0.525 | 0.559 | 0.515 | 0.539 | 0.532 | 0.535 | 0.581 | 0.673 | 0.659 |
| Unconstrained + P(True) | 0.473 | 0.754 | 0.685 | 0.719 | 0.767 | 0.724 | 0.726 | 0.751 | 0.786 | 0.749 | 0.759 |
| Unconstrained JSON (no conf) | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 |
| Unconstrained JSON (no conf ABC) | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 | 0.500 |
| Method | Llama 3.2 1B | Llama 3.2 3B | Gemma 4 E2B | Gemma 4 E4B | Gemma 4 31B | Qwen2.5 7B | Qwen3 8B | Qwen3 14B | Qwen3 32B | Qwen3.5 4B | Qwen3.5 9B |
| Unconstrained (no conf) | 0.755 | 0.513 | 0.355 | 0.272 | 0.199 | 0.319 | 0.294 | 0.269 | 0.251 | 0.280 | 0.249 |
| Unconstrained (no conf ABC) | 0.838 | 0.659 | 0.448 | 0.323 | 0.205 | 0.393 | 0.381 | 0.298 | 0.284 | 0.342 | 0.290 |
| Unconstrained (10 samples) | 0.495 | 0.386 | 0.344 | 0.253 | 0.196 | 0.302 | 0.281 | 0.257 | 0.231 | 0.221 | 0.201 |
| Unconstrained + P(True) | 0.331 | 0.323 | 0.333 | 0.225 | 0.187 | 0.273 | 0.266 | 0.229 | 0.190 | 0.179 | 0.158 |
| Unconstrained JSON (no conf) | 0.727 | 0.485 | 0.344 | 0.275 | 0.200 | 0.331 | 0.291 | 0.267 | 0.259 | 0.282 | 0.248 |
| Unconstrained JSON (no conf ABC) | 0.840 | 0.579 | 0.396 | 0.306 | 0.202 | 0.396 | 0.342 | 0.285 | 0.280 | 0.331 | 0.287 |
| Method | Llama 3.2 1B | Llama 3.2 3B | Gemma 4 E2B | Gemma 4 E4B | Gemma 4 31B | Qwen2.5 7B | Qwen3 8B | Qwen3 14B | Qwen3 32B | Qwen3.5 4B | Qwen3.5 9B |
| Unconstrained (no conf) | 0.755 | 0.513 | 0.355 | 0.272 | 0.199 | 0.319 | 0.294 | 0.269 | 0.251 | 0.280 | 0.249 |
| Unconstrained (no conf ABC) | 0.838 | 0.659 | 0.448 | 0.323 | 0.205 | 0.393 | 0.381 | 0.298 | 0.284 | 0.342 | 0.290 |
| Unconstrained (10 samples) | 0.456 | 0.372 | 0.344 | 0.254 | 0.196 | 0.302 | 0.282 | 0.258 | 0.231 | 0.226 | 0.206 |
| Unconstrained + P(True) | 0.308 | 0.305 | 0.335 | 0.232 | 0.187 | 0.277 | 0.267 | 0.231 | 0.198 | 0.206 | 0.182 |
| Unconstrained JSON (no conf) | 0.727 | 0.485 | 0.344 | 0.275 | 0.200 | 0.331 | 0.291 | 0.267 | 0.259 | 0.282 | 0.248 |
| Unconstrained JSON (no conf ABC) | 0.840 | 0.579 | 0.396 | 0.306 | 0.202 | 0.396 | 0.342 | 0.285 | 0.280 | 0.331 | 0.287 |
Zero-Shot Text Classification
Zero-shot text classification assigns texts to categories without task-specific labeled training data, using language models, embeddings, or label descriptions.
See all 38 papers