SCALE: Synthetic Calibration via Agreement Labeling in Embedding Space

Wenjun Liu; Saeed Hassanpour

cs.CVSep 30, 2026

SCALE: Synthetic Calibration via Agreement Labeling in Embedding Space

Authors: Wenjun Liu, Saeed Hassanpour

Organizations: Department of Computer Science Dartmouth College Hanover, NH, USA · Departments of Biomedical Data Science, Computer Science, and Epidemiology Dartmouth College, Hanover, NH, USA

Abstract

Foundation models for computational pathology are usually evaluated using AUC and accuracy, while calibration is often left untested. This matters because a model can be accurate on average but still assign overly confident probabilities to cases that are difficult even for pathologists. We study calibration across eight pathology foundation models. Using pathologist agreement as a measure of diagnostic difficulty, we find that calibration error is consistently higher on low-agreement cases than on high-agreement cases. This pattern is not apparent from aggregate expected calibration error (ECE) alone. We then propose synthetic agreement calibration, a method for improving calibration without collecting multi-annotator labels. Given a trained linear probe, we select high-confidence embeddings as class anchors and interpolate between anchors from opposite classes. The interpolation weights encode a continuous notion of diagnostic ambiguity, which we use as a synthetic agreement signal to retrain the probe with agreement-aware label smoothing. On MHIST, which includes annotations from seven pathologists, synthetic agreement calibration recovers most of the calibration improvement obtained by label smoothing based on real pathologist agreement, while substantially reducing low-agreement ECE relative to the uncalibrated baseline. Discrimination metrics are preserved. On PatchCamelyon and BreakHis, public histopathology datasets without multi-annotator labels, the method improves calibration across the evaluated foundation models, whereas annotator-dependent approaches cannot be used without additional expert annotation.

Figures & tables

Figure 1: Overview of SCALE. (a) The

K

highest-confidence embeddings per class are selected as anchors. (b) Linear interpolation between opposing-class anchors generates embeddings at varying synthetic difficulty levels, parameterized by

\alpha

. Points near

\alpha{=}0.5

lie close to the decision boundary and represent synthetically hard cases. (c) The interpolation weight is mapped to a soft label through a smoothing function (piecewise shown). (d) The linear probe is retrained on original embeddings with hard labels plus interpolated embeddings with soft labels.

Method	AUC (%)	ECE (%)
Baseline	84.5 $\pm$ 0.9 (84.7 $\pm$ 0.8)	9.1 $\pm$ 1.5 (8.9 $\pm$ 1.4)
Vanilla LS	85.4 $\pm$ 0.7 (85.6 $\pm$ 0.6)	3.4 $\pm$ 0.7 (3.2 $\pm$ 0.6)
Agree-Linear	85.9 $\pm$ 0.8 (86.1 $\pm$ 0.9)	5.5 $\pm$ 0.8 (5.3 $\pm$ 0.7)
Agree-Piecewise	86.2 $\pm$ 0.5 (86.4 $\pm$ 0.4)	3.1 $\pm$ 0.5 (2.9 $\pm$ 0.4)
Agree-Nonlinear	86.1 $\pm$ 0.8 (86.3 $\pm$ 0.7)	3.0 $\pm$ 0.7 (2.8 $\pm$ 0.6)

Table 1: Replication of Wei et al. (2022) on MHIST. End-to-end ResNet, 10 seeds. Published values in parentheses.

Model	Overall	High-Agree	Med-Agree	Low-Agree
GigaPath	0.045	0.035	0.065	0.111
UNI	0.022	0.018	0.045	0.150
CONCH	0.045	0.032	0.058	0.170
CHIEF	0.038	0.028	0.055	0.198
CTransPath	0.023	0.019	0.048	0.180
Phikon	0.039	0.030	0.060	0.210

Table 2: Baseline ECE stratified by agreement level (no calibration applied).

Figure 2: Baseline calibration error stratified by diagnostic difficulty. Low-agreement cases show markedly higher ECE across all eight models.

Method	Overall ECE	Low-Agree ECE	Low-Agree $\Delta$ %	AUC	$p$
No calibration	0.035	0.191	—	90.4	—
Temp scaling	0.031	0.175	$-$ 8.1%	90.4
Vanilla LS	0.030	0.151	$-$ 20.8%	90.4
Real-agree PW †	0.028	0.123	$-$ 35.2%	90.3
SCALE (ours)	0.030	0.135	$-$ 29.2%	90.4	$<$ 0.001

Table 3: Calibration and discrimination on MHIST, averaged across 8 foundation models (mean over 10 seeds). † Requires multi-annotator labels.

p

: paired

t

-test on model-averaged low-agreement ECE across 10 matched seeds. Per-model results with standard deviations are reported in Appendix B .

Figure 3: Low-agreement ECE across eight models. SCALE (blue) substantially improves over the uncalibrated baseline (red), temperature scaling (orange), and vanilla label smoothing (dark orange), tracking close to real-agreement smoothing (green). The gap between vanilla LS and SCALE demonstrates the value of difficulty-aware synthetic agreement beyond uniform smoothing.

Smoothing	Real-Agree ECE	SCALE ECE	$\Delta$ ECE	% of Real Gain
Linear	0.143	0.154	+0.011	77%
Piecewise	0.123	0.135	+0.011	83%
Nonlinear	0.127	0.141	+0.014	78%

Table 4: Real vs. synthetic agreement, same smoothing function (low-agreement ECE averaged across 8 models). SCALE recovers the majority of the real-agreement benefit across all three variants.

Figure 4: Sensitivity to

K

(averaged across 8 models). All variants plateau around

K{=}10

. Dashed lines: baseline (red) and real-agreement piecewise (green).

	PatchCamelyon			BreakHis
Model	Base	Van. LS	SCALE	Base	Van. LS	SCALE
GigaPath	0.052	0.046	0.041	0.048	0.043	0.039
UNI	0.038	0.034	0.030	0.042	0.038	0.035
CONCH	0.048	0.043	0.038	0.050	0.045	0.041
CHIEF	0.045	0.040	0.036	0.055	0.049	0.045
CTransPath	0.035	0.031	0.028	0.046	0.041	0.038

Table 5: Generalization to datasets without multi-annotator labels (overall ECE, mean over 10 seeds). SCALE outperforms both baseline and vanilla LS across all models. AUC and standard deviations are in Appendix B .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: The three agreement-aware label smoothing functions from Wei et al. [2022] . The

x

-axis is annotator agreement (

n_{k}/N_{a}

); the

y

-axis is the soft label.

Model	Method	Overall ECE	Low-Agree ECE	AUC (%)	Acc (%)
GigaPath	No calibration	0.045 $\pm$ 0.004	0.111 $\pm$ 0.013	93.5 $\pm$ 0.5	89.9 $\pm$ 0.5
	Temp scaling	0.040 $\pm$ 0.004	0.095 $\pm$ 0.011	93.5 $\pm$ 0.5	89.9 $\pm$ 0.5
	Vanilla LS	0.039 $\pm$ 0.004	0.085 $\pm$ 0.010	93.4 $\pm$ 0.4	89.8 $\pm$ 0.4
	Real-agree PW †	0.037 $\pm$ 0.003	0.070 $\pm$ 0.008	93.6 $\pm$ 0.5	90.0 $\pm$ 0.6
	SCALE (ours)	0.039 $\pm$ 0.003	0.078 $\pm$ 0.008	93.5 $\pm$ 0.5	89.9 $\pm$ 0.6
UNI	No calibration	0.022 $\pm$ 0.002	0.150 $\pm$ 0.022	91.2 $\pm$ 0.6	87.2 $\pm$ 0.6

Appendix

Table 6: Full MHIST results with standard deviations (10 seeds). † Requires multi-annotator labels.

	PatchCamelyon				BreakHis
Model	Base ECE	Van. LS ECE	SCALE ECE	AUC	Base ECE	Van. LS ECE	SCALE ECE	AUC
GigaPath	0.052 $\pm$ 0.005	0.046 $\pm$ 0.005	0.041 $\pm$ 0.004	96.2 $\pm$ 0.3	0.048 $\pm$ 0.005	0.043 $\pm$ 0.004	0.039 $\pm$ 0.004	94.5 $\pm$ 0.4
UNI	0.038 $\pm$ 0.004	0.034 $\pm$ 0.003	0.030 $\pm$ 0.003	95.8 $\pm$ 0.3	0.042 $\pm$ 0.004	0.038 $\pm$ 0.004	0.035 $\pm$ 0.004	93.8 $\pm$ 0.4
CONCH	0.048 $\pm$ 0.005	0.043 $\pm$ 0.004	0.038 $\pm$ 0.004	95.5 $\pm$ 0.3	0.050 $\pm$ 0.005	0.045 $\pm$ 0.005	0.041 $\pm$ 0.004	93.2 $\pm$ 0.4
CHIEF	0.045 $\pm$ 0.004	0.040 $\pm$ 0.004	0.036 $\pm$ 0.004	94.8 $\pm$ 0.3	0.055 $\pm$ 0.006	0.049 $\pm$ 0.005	0.045 $\pm$ 0.005	92.5 $\pm$ 0.4
CTransP.	0.035 $\pm$ 0.004	0.031 $\pm$ 0.003	0.028 $\pm$ 0.003	93.5 $\pm$ 0.3	0.046 $\pm$ 0.005	0.041 $\pm$ 0.004	0.038 $\pm$ 0.004	91.8 $\pm$ 0.5

Appendix

Table 7: Full generalization results with standard deviations (10 seeds). Vanilla LS included for comparison. AUC is reported for SCALE; baseline and Vanilla LS AUC remain within seed-level variation.

Model	Vanilla LS	Random Anchors	SCALE (Top- $K$ )
GigaPath	0.085	0.081	0.078
UNI	0.116	0.106	0.100
CONCH	0.133	0.122	0.115
CHIEF	0.159	0.149	0.142
CTransPath	0.145	0.136	0.130
Phikon	0.169	0.159	0.152

Appendix

Table 8: Anchor selection ablation (low-agreement ECE on MHIST, piecewise smoothing,

K{=}10

Explore similar work

Jul 10, 2026cs.CV

ALICE: Learning a General-Purpose Pathology Foundation Model from Vision, Vision-Language, and Slide-Level Experts

Foundation models are reshaping computational pathology, yet their capabilities remain shaped by pretraining objectives, data sources, and spatial scales, fragmenting complementary expertise across separate backbones. Here we present ALICE, a unified foundation model trained through multi-stage agglomerative distillation that sequentially distills eight vision-only, vision-language, and slide-level teacher models into dedicated modules of a single backbone. ALICE is pretrained on 24,985,184 tile-level pathology images and 155,604 high-resolution images, and evaluated across 21 task scenarios, 96 downstream tasks, and 48 data sources, spanning region-of-interest tissue analysis, vision-language multimodal evaluation, and whole-slide clinical assessment. In all three evaluation settings, ALICE achieved the best average rank among task-matched pathology foundation models. These results demonstrate that agglomerative distillation can consolidate complementary capabilities from specialized models into a unified backbone for broad computational pathology applications. The model is available at https://github.com/WonderLandxD/ALICE.

Jiawen Li, Tian Guan, Huijuan Shi +5

Institute of Biopharmaceutical and Health Engineering, Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Department of Pathology, The First Affiliated Hospital of Sun Yat-sen University, Guangzhou, China · Department of Engineering Science, University of Oxford, Oxford, UK +1

Pathology Foundation Models Probe-Logit Distillation

Ask PDF

May 4, 2026cs.CV

Multi-Rater Calibrated Segmentation Models

Objective: Accurate probability estimates are essential for the safe deployment of medical image segmentation models in clinical decision-making. However, modern deep segmentation networks are often poorly calibrated, a problem exacerbated when multiple expert annotations exhibit substantial disagreement. While inter-rater variability is typically treated as noise, it provides valuable information about intrinsic annotation ambiguity that must be reflected in model confidence. Methods: We improve the probabilistic calibration of medical image segmentation models by reformulating multi-rater supervision as an ordinal learning problem. Voxel-wise annotator agreement is treated as an ordered target, linking predictive confidence to the empirical variability in training data. This formulation allows the use of ordinal-aware scoring rules, such as the Ranked Probability Score ordinal loss, combined with a standard binary objective to preserve discriminative performance. Results: We evaluated the proposed approach across four public segmentation benchmarks spanning ophthalmology, histopathology, and thoracic imaging. Calibration was assessed using a multi-rater extension of expected calibration error. Results consistently show that ordinal-aware training yields substantially improved calibration with respect to inter-rater agreement without degrading segmentation accuracy. Conclusions: Treating multi-rater annotations as ordered information provides a principled and architecture-agnostic route to more reliable probabilistic segmentation models.

Meritxell Riera-Marín, Javier García López, Júlia Rodríguez-Comas +2

aSycai Technologies SL, Scientific and Technical Department, Barcelona, Spain · bBCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain · cInstitució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, Spain +1

Semi-Supervised Medical Image Segmentation Inter-Annotator Agreement

Ask PDF

Sep 26, 2026cs.CV

Multimodal LLMs Outperform Pathology Foundation Models in Cross-Domain Histological Similarity

State-of-the-art pathology foundation models, trained on millions of histology tiles, can fail to preserve tissue similarity when comparisons cross slide or institution boundaries. We show that general-purpose multimodal LLMs, without being trained as pathology foundation models, consistently outperform these specialized models in cross-domain histological similarity judgments. Using a relative similarity framework that we release as the MOSAIC (Model Similarity Assessment across Institutions and Cohorts) benchmark, we evaluate 17 models across 6 datasets and find that pathology encoders often rank same-institution, different-disease tiles as more similar than same-disease, different-institution tiles, a clinically dangerous failure mode invisible to standard within-domain evaluations. LLMs appear less susceptible to this failure, likely because they perform semantic visual comparison of morphology and tissue architecture rather than relying on shortcut features tied to acquisition context. Scaling training data does not resolve the problem for pathology encoders, implicating the learning objective rather than data coverage. Our results expose a fundamental robustness gap in current pathology foundation models and establish multimodal LLMs as a viable alternative for cross-institutional retrieval, dataset harmonization, and multi-site quality control. Code and data will be released upon acceptance.

Yishu Zhang, Yun Li, Daiwei Zhang

University of North Carolina at Chapel Hill

Pathology Foundation Models Multimodal Clinical Data Multimodal Foundation Model Cross-Modality Wireless Foundation Models Multimodal Large Language Models

Ask PDF