Confidence Calibration under Ambiguous Ground Truth

Linwei Tao; Haoyang Luo; Minjing Dong; Chang Xu

cs.LGMar 24, 2026

Confidence Calibration under Ambiguous Ground Truth

Authors: Linwei Tao, Haoyang Luo, Minjing Dong, Chang Xu

Organizations: L. Tao and C. Xu are with the School of Computer Science, University of Sydney, Sydney, NSW 2006, Australia. · H. Luo and M. Dong are with the Department of Computer Science, City University of Hong Kong, Hong Kong SAR, China.

Abstract

Confidence calibration assumes a unique ground-truth label per input, yet this assumption fails wherever annotators genuinely disagree. Post-hoc calibrators fitted on majority-voted labels, the standard single-label targets used in practice, can appear well-calibrated under conventional evaluation yet remain substantially miscalibrated against the underlying annotator distribution. We show that this failure is structural: under simplifying assumptions, Temperature Scaling is biased toward temperatures that underestimate annotator uncertainty, with true-label miscalibration increasing monotonically with annotation entropy. To address this, we develop a family of ambiguity-aware post-hoc calibrators that optimise proper scoring rules against the full label distribution and require no model retraining. Our methods span progressively weaker annotation requirements: Dirichlet-Soft leverages the full annotator distribution and achieves the best overall calibration quality across settings; Monte Carlo Temperature Scaling with a single annotation per example (MCTS S=1) matches full-distribution calibration across all benchmarks, demonstrating that pre-aggregated label distributions are unnecessary; and Label-Smooth Temperature Scaling (LS-TS) operates with voted labels alone by constructing data-driven pseudo-soft targets from the model's own confidence. Experiments on four benchmarks with real multi-annotator distributions (CIFAR-10H, ChaosNLI) and clinically-informed synthetic annotations (ISIC~2019, DermaMNIST) show that Dirichlet-Soft reduces true-label ECE by 55-87% relative to Temperature Scaling, while LS-TS reduces ECE by 9-77% without any annotator data.

Figures & tables

Fig. 1 : Left: a CIFAR-10H image with dispersed human votes (cat/dog/bird), illustrating perceptual ambiguity in low-resolution vision. Middle: a ChaosNLI premise–hypothesis pair with split entailment/neutral judgments, illustrating semantic ambiguity. Right: an ISIC 2019 melanoma image paired with the clinician-informed label distribution used in our medical experiments, showing clinically plausible MEL/NV confusion. CIFAR-10H and ChaosNLI distributions are empirical human label distributions; the ISIC panel uses the dermatologist confusion model described in Appendix I .

Fig. 2 : Motivating example: all standard calibration methods fail. (a) Toy dataset generation: only the middle Gaussian cluster is ambiguous, with

\pi=[0,0.70,0.30]

; orange points are drawn as label 1 (70%) and red points as label 2 (30%), while all receive the same voted label 1. (b) Summary of the toy results: all three voted-label calibrators (TS, Platt, Histogram Binning) lower

\mathrm{ECE}_{\text{voted}}

but increase

\mathrm{ECE}_{\text{true}}

, so voted-label evaluation masks the failure. (c) Stratified

\mathrm{ECE}_{\text{true}}

for TS, Platt, and Histogram Binning: ambiguous examples are those from the middle Gaussian cluster (where annotators disagree); clear examples are those from the two unambiguous clusters (class 0 and class 2). The residual error is concentrated in ambiguous examples for all three methods.

Fig. 3 : Empirical validation of Proposition 2 . Test examples are grouped into equal-frequency bins by normalised annotation entropy

H(x)/\log K

. Each point shows the mean pointwise true-label calibration error

|\hat{p}_{\hat{c}}(x)-\pi_{\hat{c}}(x)|

for Temperature Scaling (error bars:

\pm 1

s.e.). Across all three dataset–architecture combinations, TS error increases monotonically with annotation entropy, as predicted by Proposition 2 . ChaosNLI operates in a higher-entropy regime and exhibits uniformly higher error (

19

–

25\%

Method	ECE (%)	$T^{*}$
TS (voted labels)	4.29	$2.030$
MCTS $S=1$	$1.53\pm 0.01$	$3.174\pm 0.041$
MCTS $S=5$	$1.53\pm 0.02$	$3.166\pm 0.014$
MCTS $S=20$	$1.52\pm 0.00$	$3.187\pm 0.013$
MCTS $S=50$	$1.52\pm 0.01$	$3.185\pm 0.007$
MCTS $S=200$	$1.52\pm 0.00$	$3.178\pm 0.003$

TABLE I : MCTS convergence on CIFAR-10H (ResNet-50). Mean

\pm

std over 5 seeds;

S

= MC annotation samples per calibration example.

CIFAR-10H ( $m{\approx}51$ annotations per image, $K{=}10$ )
	ResNet-50						ViT-B/16
Method	$T$	ECE $\downarrow$	aECE $\downarrow$	cwECE $\downarrow$	Br $\downarrow$	NLL $\downarrow$	$T$	ECE $\downarrow$	aECE $\downarrow$	cwECE $\downarrow$	Br $\downarrow$	NLL $\downarrow$
Voted-label baselines
Uncalibrated	—	4.97	5.71	1.09	0.120	0.692	—	4.99	5.36	1.06	0.111	0.678
TS	2.03	4.29	4.25	0.91	0.112	0.363	2.04	4.48	4.40	0.93	0.106	0.346
ATS	—	4.40	4.37	0.94	0.113	0.350	—	4.54	4.47	0.94	0.106	0.345

TABLE II : CIFAR-10H and ChaosNLI results. ECE/aECE/cwECE averaged over 100 sampled labels

\tilde{y}\sim\hat{\pi}

; Brier/NLL computed against soft target

\hat{\pi}

(Eq. 5 ). Best per architecture in bold (Oracle TS excluded). Oracle TS fits TS on the test set with soft labels (cheating upper bound on soft NLL; not a strict ECE upper bound).

Fig. 4 : Reliability diagrams for four representative methods: CIFAR-10H ResNet-50 (ECE

{}_{\text{true}}

). Red shading indicates overconfidence; green indicates underconfidence. Uncal and TS remain overconfident; soft-label temperature methods (MCTS/SLTS) substantially correct this; Dirichlet-Soft reduces the residual gap further. Reliability diagrams for all remaining methods are provided in Appendix H .

Fig. 5 : Reliability diagrams for four representative methods: ChaosNLI RoBERTa-Large (ECE

{}_{\text{true}}

). Same method selection as Figure 4 . Red shading indicates overconfidence; green indicates underconfidence. NLI models are severely overconfident before calibration; ambiguity-aware methods substantially correct this.

ISIC 2019 ( $m{=}9$ synthetic annotators, $K{=}8$ , mean agreement $75\%$ )
	EfficientNet-B4						ViT-S/16
Method	$T$	ECE $\downarrow$	aECE $\downarrow$	cwECE $\downarrow$	Br $\downarrow$	NLL $\downarrow$	$T$	ECE $\downarrow$	aECE $\downarrow$	cwECE $\downarrow$	Br $\downarrow$	NLL $\downarrow$
Voted-label baselines
Uncalibrated	—	25.87	25.97	6.64	0.600	2.686	—	21.28	21.17	6.21	0.650	1.803
TS	1.77	18.72	18.70	5.02	0.559	1.661	1.23	17.04	16.95	5.49	0.630	1.578
ATS	1.70	18.83	18.80	5.03	0.560	1.733	1.12	17.53	17.48	5.46	0.631	1.730

TABLE III : ISIC 2019 and DermaMNIST results. ECE/aECE/cwECE averaged over 100 sampled labels

\tilde{y}\sim\hat{\pi}

; Brier/NLL computed against soft target

\hat{\pi}

(Eq. 5 ). Best per architecture in bold (Oracle TS excluded). Oracle TS fits TS on the test set with soft labels (cheating upper bound on soft NLL; not a strict ECE upper bound).

Fig. 6 : Inter-reader confusion matrix for ISIC 2019. Entry $C_{ij}$ is the probability that a dermatologist labels an image as class $j$ given consensus label $i$ . Orange dashed box: MEL/NV most-confused pair ( $73$ – $76\%$ diagonal). Purple dotted: AK/BKL/SCC high-confusion cluster. Mean diagonal agreement $\bar{C}_{ii}=75\%$ .

Fig. 7 : Reliability diagrams: ISIC 2019 EfficientNet-B4 (ECE

{}_{\text{true}}

). Red shading = overconfidence; green = underconfidence. Hard-label baselines remain severely overconfident at 18–26% ECE; ambiguity-aware methods substantially correct this, with IR-Soft achieving the lowest ECE.

Fig. 8 : Reliability diagrams: DermaMNIST ResNet-18 (ECE

{}_{\text{true}}

). Same layout as Figure 4 . TS leaves 22% ECE; Dirichlet-Soft reduces it to 3.9% while achieving the best Brier/NLL.

	CIFAR R50			CIFAR ViT			Chaos RoBERTa			Chaos DeBERTa
Method	ECE	Br	NLL	ECE	Br	NLL	ECE	Br	NLL	ECE	Br	NLL
TS (voted)	3.90	.1126	.346	4.01	.1043	.320	10.55	.537	.886	11.63	.549	.900
Fixed-LS ( $\varepsilon{=}0.1$ )	7.44	.1207	.332	6.00	.1058	.305	6.46	.525	.866	7.53	.536	.882
Ent-LS	3.16	.1116	.321	3.69	.1037	.308	3.17	.520	.862	8.58	.539	.885
CC-LS	1.48	.1111	.296	1.69	.1016	.279	4.15	.522	.873	2.54	.529	.880
LS-TS (ours)	1.49	.1111	.296	1.62	.1016	.279	4.16	.522	.873	2.65	.529	.881

TABLE IV : LS-TS ablation: annotation-free smoothing strategies. ECE

{}_{\text{true}}

(%), Brier score, and NLL on CIFAR-10H and ChaosNLI. All methods use the same single-temperature family; only the pseudo-target construction differs. Fixed-LS uses

\varepsilon{=}0.1

regardless of the data; Ent-LS sets

\varepsilon_{i}{=}H(\hat{p}(x_{i}))/\log K

per instance; CC-LS computes a per-class smoothing weight

\varepsilon_{k}{=}\mathrm{mean}_{i:\,y^{*}_{i}=k}(1-\hat{p}_{y^{*}}(x_{i}))

. Best annotation-free method per column in bold .

Dataset	Arch	TS	SLTS	Dir.-Soft	IR-Soft
CIFAR-10H	R50	$4.25\pm 0.25$	$1.61\pm 0.12$	$1.37\pm 0.09$	$0.89\pm 0.12$
CIFAR-10H	ViT-B/16	$4.39\pm 0.20$	$0.83\pm 0.14$	$0.74\pm 0.07$	$0.88\pm 0.16$
ChaosNLI	RoBERTa-L	$11.54\pm 0.54$	$3.85\pm 0.34$	$2.90\pm 0.14$	$2.40\pm 0.18$
ChaosNLI	DeBERTa-v3	$12.73\pm 0.60$	$4.41\pm 0.54$	$3.80\pm 0.35$	$2.61\pm 0.41$
ISIC 2019	ENet-B4	$18.84\pm 0.17$	$9.80\pm 0.22$	$8.45\pm 0.20$	$1.79\pm 0.16$
ISIC 2019	ViT-S/16	$16.93\pm 0.16$	$7.30\pm 0.13$	$6.20\pm 0.19$	$2.05\pm 0.13$

TABLE V : Multi-seed stability. Mean

\pm

std of ECE

{}_{\text{true}}

(%) over 5 seeds. For CIFAR-10H/ChaosNLI the seed varies the cal/test split; for ISIC/DermaMNIST it varies the annotation sample.

Fig. 9 : Robustness of ambiguity-aware calibration. (a) Multi-seed stability (CIFAR-10H R50, 5 seeds). (b)–(c) ECE

{}_{\text{true}}

vs. calibration set size on CIFAR-10H R50 and ChaosNLI RoBERTa-L. (d) ECE

{}_{\text{true}}

vs. annotation count (CIFAR-10H R50). Rankings are stable across seeds, cal-set sizes, and annotation counts. Full results for all benchmarks are in Appendices G , E , and F .

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. 10 : Calibration-set-size ablation: CIFAR-10H. Left: ResNet-50. Right: ViT-B/16. TS’s ECE is flat across all calibration set sizes; ambiguity-aware methods converge within 5–10%.

Fig. 11 : Calibration-set-size ablation: ChaosNLI. Left: RoBERTa-Large. Right: DeBERTa-v3-base. NLI models show the same pattern: TS is flat near 10–13% while soft methods stabilise near 3–4%.

Fig. 12 : Calibration-set-size ablation: ISIC 2019. Left: EfficientNet-B4. Right: ViT-S/16. IR-Soft achieves the lowest ECE across all calibration set sizes on both architectures.

Fig. 13 : Calibration-set-size ablation: DermaMNIST. Left: ResNet-18. Right: ViT-S/16. Ambiguity-aware methods converge quickly; TS remains flat regardless of calibration set size.

Fig. 14 : Annotation-count ablation: CIFAR-10H (both architectures). ECE

{}_{\text{true}}

vs. annotations per image

m

for ResNet-50 (left) and ViT-B/16 (right). Ambiguity-aware methods are stable for

m\geq 2

; TS rises with

m

due to target mismatch; LS-TS converges by

m\approx 4

Fig. 15 : Multi-seed bar charts: CIFAR-10H. Left: ResNet-50. Right: ViT-B/16.

Fig. 16 : Multi-seed bar charts: ChaosNLI. Left: RoBERTa-Large. Right: DeBERTa-v3-base.

Fig. 17 : Multi-seed bar charts: ISIC 2019. Left: EfficientNet-B4. Right: ViT-S/16. Seed varies the annotation generation; val/test split is fixed.

Fig. 18 : Multi-seed bar charts: DermaMNIST. Left: ResNet-18. Right: ViT-S/16. Seed varies the annotation generation; val/test split is fixed.

Fig. 19 : Reliability diagrams (summary): CIFAR-10H ViT-B/16.

Fig. 20 : Reliability diagrams (all methods): CIFAR-10H ResNet-50 (ECE

{}_{\text{true}}

Fig. 21 : Reliability diagrams (all methods): CIFAR-10H ViT-B/16 (ECE

{}_{\text{true}}

Fig. 22 : Reliability diagrams (summary): ChaosNLI DeBERTa-v3-base.

Fig. 23 : Reliability diagrams (all methods): ChaosNLI RoBERTa-Large (ECE

{}_{\text{true}}

). NLI models are severely overconfident before calibration; ambiguity-aware methods substantially correct this.

Fig. 24 : Reliability diagrams (all methods): ChaosNLI DeBERTa-v3-base (ECE

{}_{\text{true}}

Fig. 25 : Reliability diagrams (summary): ISIC 2019 EfficientNet-B4.

Fig. 26 : Reliability diagrams (summary): ISIC 2019 ViT-S/16.

Fig. 27 : Reliability diagrams (all methods): ISIC 2019 EfficientNet-B4 (ECE

{}_{\text{true}}

Fig. 28 : Reliability diagrams (all methods): ISIC 2019 ViT-S/16 (ECE

{}_{\text{true}}

Fig. 29 : Reliability diagrams (summary): DermaMNIST ResNet-18.

Fig. 30 : Reliability diagrams (summary): DermaMNIST ViT-S/16.

Fig. 31 : Reliability diagrams (all methods): DermaMNIST ResNet-18 (ECE

{}_{\text{true}}

Fig. 32 : Reliability diagrams (all methods): DermaMNIST ViT-S/16 (ECE

{}_{\text{true}}

Fig. 33 : Inter-reader confusion matrix for ISIC 2019. Entry $C_{ij}$ is the probability that a dermatologist labels an image as class $j$ given the consensus label is $i$ . Diagonal entries (bold) are the per-condition agreement rates, calibrated to match Liu et al. [ 40 ] Table 2. Orange dashed box: MEL/NV most-confused pair. Purple dotted highlights: AK/BKL/SCC high-confusion cluster. Mean diagonal agreement $\bar{C}_{ii}=75\%$ .

	MEL	NV	BCC	AK	BKL	DF	VL	SCC
MEL	0.73	0.14	0.02	0.03	0.08	0.00	0.00	0.00
NV	0.15	0.76	0.01	0.01	0.06	0.01	0.00	0.00
BCC	0.02	0.01	0.81	0.05	0.07	0.01	0.01	0.02
AK	0.03	0.01	0.04	0.65	0.11	0.00	0.00	0.16
BKL	0.12	0.05	0.03	0.10	0.62	0.00	0.00	0.08
DF	0.01	0.02	0.02	0.01	0.02	0.87	0.03	0.02

Appendix

TABLE VI : Numerical values of the inter-reader confusion matrix for ISIC 2019. Row = consensus (majority-vote) label; column = annotator’s label. Each row sums to 1. Diagonal entries (bold) are per-condition agreement rates, calibrated to Liu et al. [ 40 ] Table 2; mean diagonal agreement

\bar{C}_{ii}=75\%

	C10H R50		C10H ViT		NLI RoBERTa		NLI DeBERTa		ISIC ENet		ISIC ViT		Derm R18		Derm ViT
Method	ECE	Br	ECE	Br	ECE	Br	ECE	Br	ECE	Br	ECE	Br	ECE	Br	ECE	Br
TS	4.29	.112	4.48	.106	10.55	.537	11.63	.549	18.72	.559	17.04	.630	22.25	.687	24.65	.683
ATS	4.40	.113	4.54	.106	11.36	.538	13.09	.552	18.83	.560	17.53	.631	22.82	.687	25.35	.686
LS-TS	1.57	.109	2.37	.103	4.16	.522	2.65	.529	9.34	.528	15.49	.619	5.05	.630	7.36	.615

Appendix

TABLE VII: ATS vs. TS and LS-TS across all 8 settings (annotation-free methods only). ECE

{}_{\text{true}}

(%) and Brier. ATS predicts a per-instance temperature from logit features but is trained with the same voted-label NLL as TS. Bold = best annotation-free method per column.

Post-Hoc Calibration

Post-hoc calibration adjusts a trained model’s probabilities or confidence scores to better match observed accuracy, often without retraining.

See all 81 papers

Temperature Scaling Is Not Enough: Calibration Gaps Under Human Label Distributions
Jul 15, 2026Wisdom DogahPost-Hoc Calibration Language Model Calibration
Faculty of Computing and Mathematical Sciences, University of Mines and Technology (UMaT), Tarkwa, Ghana. · BlackMatrix AI Research, Accra, Ghana.
Noisy-Label Calibration with Multiple Unreliable Labels
Oct 14, 2026Keisuke Hanada, Futoshi FutamiModel Calibration Probability Calibration
Department of Biostatistics, Wakayama Medical University, Wakayama, Japan · RIKEN AIP, Tokyo, Japan · Graduate School of Engineering Science, The University of Osaka, Osaka, Japan +1
Quantile Adaptive Temperature Scaling for Confidence Calibration
Jun 19, 2026Omprakash Chakraborty, Leo Fillioux, Ismail Ben Ayed +1Post-Hoc Calibration Probability Calibration
1 ÉTS Montréal, Canada · Université Paris-Saclay, CentraleSupélec, Gustave Roussy, INSERM, CDSU, IHU PRISM, Gif-sur-Yvette

Jul 15, 2026J/K move · Enter open · S save

Temperature Scaling Is Not Enough: Calibration Gaps Under Human Label Distributions

Wisdom Dogah

Faculty of Computing and Mathematical Sciences, University of Mines and Technology (UMaT), Tarkwa, Ghana. · BlackMatrix AI Research, Accra, Ghana.

Post-Hoc Calibration Language Model Calibration Model Calibration

Open paper AskPDF arXiv

Temperature scaling is the dominant post-hoc calibration method in modern deep learning. Its theoretical justification rests on an assumption that is rarely stated explicitly: that ground-truth labels are one-hot and deterministic. In practice, labels are frequently soft, crowd-sourced, or genuinely distributional, reflecting real disagreement among human annotators rather than annotation noise. We study whether temperature scaling retains its calibration properties when this assumption is violated, and whether any resulting degradation depends on model scale. Using CIFAR-10H and ChaosNLI, two publicly available datasets with human-annotated soft label distributions, we evaluate three model scales per modality under both hard one-hot and soft distributional label targets. Across all nine configurations we find a positive soft-label calibration gap: temperature scaling calibrated on hard labels consistently underperforms an oracle calibrated directly on soft labels, with Brier Score gaps ranging from 0.002 to 0.134. The gap grows monotonically with model scale in the vision domain and on the SNLI-derived split of ChaosNLI, and is substantially larger in the language domain (mean gap 0.079) than in vision (mean gap 0.003). A scale-ordering reversal on the MNLI-derived split remains after matched-domain training; we treat it as inconclusive for the scale hypothesis and attribute it primarily to near-chance accuracy on that split. As a second post-hoc baseline, multiclass isotonic regression yields the same qualitative conclusion: positive soft-label gaps in all nine configurations, and larger gaps in language than in vision. These findings suggest that calibration protocols built on majority-vote labels systematically misstate model reliability wherever label ambiguity is structural, with direct consequences for deployment in safety-critical settings.

Abstract

Post-Hoc Calibration

Explore similar work

Temperature Scaling Is Not Enough: Calibration Gaps Under Human Label Distributions

Noisy-Label Calibration with Multiple Unreliable Labels

Quantile Adaptive Temperature Scaling for Confidence Calibration