Align Before You Combine: Reference Space Calibration for Supervision Without Ground Truth

Jackson Eshbaugh; Jorge Silveyra

cs.LGOct 7, 2026

Align Before You Combine: Reference Space Calibration for Supervision Without Ground Truth

Authors: Jackson Eshbaugh, Jorge Silveyra

Organizations: Department of Computer Science Lafayette College Easton, PA, USA

Abstract

We introduce a calibration-first framework that produces supervision scores without access to ground-truth labels or a shared annotation space. Our framework aligns subset-specific scorers using a synthetic ordinal reference space before fusion. This reference space is constructed from ordered calibration features that represent the latent concept, providing a common scale on which otherwise incomparable scorer outputs can be aligned. Because our calibration procedure uses the reference space rather than training samples, it is independent of the training set's empirical distribution. Across three benchmark datasets, our framework consistently outperforms uncalibrated averaging and achieves higher primary-metric point estimates on the evaluation metrics than the best individual scorer. Performance relative to sample-dependent baselines varies by domain, with absolute differences below 0.02 on Ames Housing and below 0.01 on Breast Cancer Wisconsin and Wine Quality. After Bonferroni correction, differences remain significant for all three comparisons on Ames Housing and one on Breast Cancer Wisconsin. Additionally, we show that using fewer calibration levels per feature can closely approximate higher-resolution results at substantially lower computational cost. Together, these results support our framework as a viable approach to construct supervision scores when neither ground-truth labels nor a shared annotation space is available.

Figures & tables

Panel of Figure 1 — Figure 1: Calibration. Subset-specific scorers are evaluated over the synthetic ordinal reference space to estimate normalization statistics $(\mu_{j},\sigma_{j})$ .

\toprule Dataset	Method	Primary [95% CI]	$\Delta=\kappa-\mathrm{method}$ [95% paired CI]	Bonf. [98.33% CI]
\midrule Ames Housing	Calibrated Fusion ( $\kappa$ )	$0.890\,[0.876,\,0.903]$	—	—
	Empirical $z$ -Score Avg.	$0.908\,[0.895,\,0.919]$	$-0.0172\,[-0.0225,\,-0.0122]$	$[-0.0238,\,-0.0111]$
	Borda Rank Averaging	$0.905\,[0.893,\,0.916]$	$-0.0148\,[-0.0214,\,-0.0085]$	$[-0.0230,\,-0.0072]$
	Unsupervised PCA (PC1)	$\mathbf{0.909\,[0.896,\,0.919]}$	$-0.0181\,[-0.0233,\,-0.0131]$	$[-0.0245,\,-0.0120]$
	Uncalibrated Average	$0.859\,[0.840,\,0.877]$	$+0.0311\,[0.0143,\,0.0485]$	$[0.0109,\,0.0524]$
	Best Single Scorer	$0.8150$	—	—

Table 1: Cross-domain validation on the primary metric. Point estimates are reported with percentile-bootstrap 95% confidence intervals

(B=50{,}000)

. Pairwise intervals report

\Delta=\kappa-\mathrm{method}

, with positive values favoring calibrated fusion. Bonferroni-adjusted 98.33% confidence intervals control the family-wise error rate across the three datasets separately for each baseline. The primary metric is Spearman’s

\rho

for Ames Housing and Wine Quality and AUROC for Breast Cancer Wisconsin. Additional metrics are available in Appendix C .

\toprule Varied Scorer	Level 1	Level 2	Level 3	Level 4	Level 5
\midrule Text / HVAC	1.10	0.43	0.26	-0.57	-0.57
Structured / HVAC	1.26	0.76	0.26	-0.24	-0.74
Text / Insulation	0.94	0.94	-0.09	-0.24	-0.53
Structured / Insulation	0.91	0.41	-0.09	-0.59	-1.09
\bottomrule

Table 2: Concept-specific supervision scores under scorer-level sensitivity analysis. Larger values indicate greater retrofit need.

\toprule Text Evidence	Structured Evidence	HVAC	Insulation
\midrule Level 1	Level 1	2.10	1.94
Level 1	Level 5	0.10	-0.06
Level 5	Level 1	0.43	0.47
Level 5	Level 5	-1.57	-1.53
\bottomrule

Table 3: Concept-specific supervision scores. Level 1 is least efficient and Level 5 is most efficient.

Figure 4: Spearman $\rho$ against $k=15$ vs EnergyPlus compute time for all tested values of $k$ .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

\toprule Resolution	Selected levels
\midrule $k=2$	1, 15
$k=3$	1, 8, 15
$k=5$	1, 5, 8, 11, 15
$k=7$	1, 3, 6, 8, 10, 13, 15
$k=9$	1, 3, 5, 6, 8, 10, 11, 13, 15
$k=11$	1, 2, 4, 5, 7, 8, 9, 11, 12, 14, 15

Appendix

Table 4: Selected ordinal levels from the

k=15

master calibration grid for each calibration resolution.

\toprule Level	Wall $R$	Roof $R$	Heating COP	Cooling COP
\midrule 1	4	10	0.7	1
2	5	13.333	0.7333	1.3333
3	6	16.667	0.7667	1.6667
4	7	20	0.8	2
5	8.5	22.5	0.825	2.25
6	10	25	0.85	2.5

Appendix

Table 5: Physical values associated with the

k=15

calibration levels.

Figure 5: Growth in the number of calibration points with calibration resolution $k$ . The calibration space grows as $k^{4}$ because four calibration features are varied independently.

\toprule Resolution	Points	Runtime	Spearman $\rho$	Top-10% Overlap
\midrule $k=5$	625	0.24h	0.9871	88.56 %
$k=7$	2401	0.94h	0.9939	92.42 %
$k=9$	6561	2.56h	0.9987	96.66 %
$k=15$	50625	19.76h	1.0000	100.00 %
\bottomrule

Appendix

Table 6: Sensitivity results from selected

k

-values.

Figure 6: Spearman correlation between supervision scores generated at each calibration resolution and the $k=15$ reference space.

\toprule Calibration dimension	Feature	Level 1–Level 5
\midrule Physical scale	GrLivArea	600–3500
	TotalBsmtSF	300–1800
\midrule Quality and condition	OverallQual	2–10
	OverallCond	3–9
\midrule Age and modernization	YearBuilt	1920–2010
	YearRemodAdd	1950–2010

Appendix

Table 7: Calibration dimensions and endpoint values used for Ames Housing.

\toprule Calibration dimension	Feature	Level 1–Level 5
\midrule Size	Mean radius	7–28
	Mean perimeter	45–190
	Mean area	150–2500
\midrule Shape irregularity	Mean compactness	0.02–0.35
	Mean concavity	0–0.45
	Mean concave points	0–0.20

Appendix

Table 8: Calibration dimensions and endpoint values used for Breast Cancer Wisconsin.

\toprule Calibration dimension	Feature	Level 1–Level 5
\midrule Acidity	Volatile acidity	1.58–0.12
	Citric acid	0.0–1.0
\midrule Additives	Chlorides	0.611–0.012
	Sulphates	0.33–2.0
\midrule Preservatives	Free sulfur dioxide	72.0–1.0
	Total sulfur dioxide	289.0–6.0

Appendix

Table 9: Calibration dimensions and endpoint values used for Wine Quality.

\toprule Dataset	Method	Secondary	Additional
\midrule Ames Housing	Calibrated Fusion ( $\kappa$ )	0.7197	73.97%
	Empirical $z$ -Score Avg.	0.7444	78.08%
	Borda Rank Averaging	0.7359	73.29%
	Unsupervised PCA (PC1)	0.7456	78.08%
	Uncalibrated Average	0.6890	73.29%
	Best Single Scorer	0.6365	68.49%

Appendix

Table 10: Secondary and additional cross-domain validation metrics. For Ames Housing and Wine Quality, the secondary metric is Kendall’s

\tau

and the additional metric is top-10% overlap. For Breast Cancer Wisconsin, the secondary metric is average precision and the additional metric is Spearman correlation with malignancy status. Bold values indicate the highest observed value within each dataset and metric.

\toprule Dataset	Subset-Specific Scorer	Metric	Value
\midrule Ames Housing	Physical scale	Spearman $\rho$	0.8150
	Age and modernization		0.6814
	Garage capacity		0.6496
	Quality and condition		0.6197
\midrule Breast Cancer	Worst-case features	AUROC	0.9705
Wisconsin	Size		0.9391

Appendix

Table 11: Primary validation performance of each subset-specific scorer across all three evaluation datasets.

\toprule Calibration Feature	Increasing Efficiency $\longrightarrow$
\midrule HVAC heating COP	0.7	0.8	0.9	0.95	1.0
HVAC cooling COP	1.0	2.0	3.0	3.5	4.0
Roof R-value	10	20	30	40	50
Wall R-value	4	7	13	20	30
\bottomrule

Appendix

Table 12: Calibration levels used to construct the Synthetic Homes ordinal reference space.

\toprule Level	HVAC Component	Insulation Component
\midrule 1	The home has an aging central HVAC system showing substantial exterior corrosion and poor operating efficiency.	Attic insulation is minimal, with exposed joists visible throughout, causing significant heat loss.
2	The HVAC system appears to be in working condition but is an older standard-efficiency model.	No signs of added insulation were observed in the basement ceiling, suggesting potential energy inefficiency.
3	The home has a functioning central HVAC system of moderate age and efficiency, with no major defects observed but no recent high-efficiency upgrades.	Walls appear to be adequately insulated based on construction year, though no upgrades were observed.
4	The HVAC system was recently replaced with a standard high-efficiency model and is expected to operate efficiently.	Blown-in insulation is present in the attic to a depth of approximately 10 inches, providing good thermal resistance.
5	A state-of-the-art HVAC system with smart thermostats and variable-speed compressors was recently installed, maximizing energy efficiency.	High-performance spray foam insulation was installed throughout the walls, attic, and basement, providing maximum energy efficiency.
\bottomrule

Appendix

Table 13: Text components used to construct home inspection notes at each ordinal efficiency level.

\toprule Dataset	Method	Spearman $\rho$ with Item-Level PC1
\midrule BIG5	Calibrated Fusion ( $\kappa$ )	0.615
HEXACO	Calibrated Fusion ( $\kappa$ )	0.252
MGKT	Calibrated Fusion ( $\kappa$ )	0.987
\bottomrule

Appendix

Table 14: Agreement between calibrated fusion and the dominant latent factor extracted from item-level responses.

Explore similar work

SCALE: Synthetic Calibration via Agreement Labeling in Embedding Space
Sep 30, 2026Wenjun Liu, Saeed HassanpourPathology Foundation Models Computational Pathology
Department of Computer Science Dartmouth College Hanover, NH, USA · Departments of Biomedical Data Science, Computer Science, and Epidemiology Dartmouth College, Hanover, NH, USA
CalArena: A Large-Scale Post-Hoc Calibration Benchmark
May 28, 2026Eugène Berta, David Holzmüller, Francis Bach +1Multiclass Classification
Inria - Ecole Normale Supérieure PSL Research University · Inria
Unified Approach for Weakly Supervised Multicalibration
May 11, 2026Futoshi Futami, Takashi IshidaCalibrated Uncertainty
The University of Osaka / RIKEN AIP / The University of Tokyo · The University of Tokyo / RIKEN AIP

Sep 30, 2026, cs.CVJ/K move · Enter open · S save

SCALE: Synthetic Calibration via Agreement Labeling in Embedding Space

Wenjun Liu, Saeed Hassanpour

Department of Computer Science Dartmouth College Hanover, NH, USA · Departments of Biomedical Data Science, Computer Science, and Epidemiology Dartmouth College, Hanover, NH, USA

Pathology Foundation Models Computational Pathology

Open paper AskPDF arXiv

Foundation models for computational pathology are usually evaluated using AUC and accuracy, while calibration is often left untested. This matters because a model can be accurate on average but still assign overly confident probabilities to cases that are difficult even for pathologists. We study calibration across eight pathology foundation models. Using pathologist agreement as a measure of diagnostic difficulty, we find that calibration error is consistently higher on low-agreement cases than on high-agreement cases. This pattern is not apparent from aggregate expected calibration error (ECE) alone. We then propose synthetic agreement calibration, a method for improving calibration without collecting multi-annotator labels. Given a trained linear probe, we select high-confidence embeddings as class anchors and interpolate between anchors from opposite classes. The interpolation weights encode a continuous notion of diagnostic ambiguity, which we use as a synthetic agreement signal to retrain the probe with agreement-aware label smoothing. On MHIST, which includes annotations from seven pathologists, synthetic agreement calibration recovers most of the calibration improvement obtained by label smoothing based on real pathologist agreement, while substantially reducing low-agreement ECE relative to the uncalibrated baseline. Discrimination metrics are preserved. On PatchCamelyon and BreakHis, public histopathology datasets without multi-annotator labels, the method improves calibration across the evaluated foundation models, whereas annotator-dependent approaches cannot be used without additional expert annotation.

Abstract

Explore similar work

SCALE: Synthetic Calibration via Agreement Labeling in Embedding Space

CalArena: A Large-Scale Post-Hoc Calibration Benchmark

Unified Approach for Weakly Supervised Multicalibration