STAMP: Predicting Out-of-Distribution Generalization without Target Data

Md Kawsher Mahbub; Milon Biswas

cs.LGSep 26, 2026

STAMP: Predicting Out-of-Distribution Generalization without Target Data

Authors: Md Kawsher Mahbub, Milon Biswas

Organizations: Department of Computer Science and Engineering NPI University of Bangladesh Manikganj, Bangladesh · Department of Computer & Information Sciences Towson University Towson, MD 21252

Abstract

Predicting whether a trained model will generalize under distribution shift remains difficult, especially when target-domain data are unavailable. We introduce STAMP (Semantic Temporal Augmented Model Prediction), a source-only, target-label-free criterion that estimates out-of-distribution (OOD) performance from paired source-domain images. STAMP computes the output-space correlation ratio $η^2=S_B/S_T$ by contrasting semantically stable pairs with random pairs: higher $η^2$ indicates that model outputs vary with semantic identity rather than nuisance variation. On 44 chest X-ray models spanning CNNs, ViTs, MetaFormers, foundation models, and SSL/VLM probes, temporal STAMP attains Spearman correlations of $0.844$ -- $0.855$ with macro AUROC on VinDr-CXR, CheXpert, and MIMIC-CXR; a class-matched variant improves single-class RSNA from $0.311$ to $0.663$ . STAMP attains the best average source-only medical ranking and outperforms the target-domain ATC and AoTL estimators without any target data. On 27 ImageNet models, temperature-scaled STAMPTS attains $ρ{=}0.984$ on ObjectNet and $ρ\geq0.905$ on four additional distribution shifts, with partial correlations of $0.662$ -- $0.949$ after controlling for ImageNet accuracy. Requiring approximately 12s per model on one GPU, STAMP is a practical pre-deployment model-selection and auditing tool.

Figures & tables

Dataset	$\rho_{\mathrm{s}}$	95 % CI	$p$	$n$
VinDr-CXR	0.855	[0.72, 0.92]	$<10^{-10}$	44
CheXpert	0.851	[0.71, 0.92]	$<10^{-10}$	44
MIMIC-CXR	0.844	[0.70, 0.91]	$<10^{-10}$	44
RSNA (macro)	0.311	[0.02, 0.56]	0.044	44
RSNA (CM, Pneu.)	0.663	[0.44, 0.80]	$<10^{-4}$	44

Table 1: Spearman

\rho_{\mathrm{s}}

between STAMP and macro-AUROC (

n{=}44

). RSNA (CM) uses class-matched Pneumonia-only STAMP .

Figure 1: STAMP vs. OOD macro-AUROC across 44 medical architectures , colored by pretraining objective. VinDr, CheXpert, and MIMIC show strong, consistent positive trends; RSNA (class-matched Pneumonia-only STAMP shown) is weaker and noisier, consistent with the class-overlap mismatch discussed in Section 8 (Proposition 2 ).

Method	Dom.	VinDr	Chex.	RSNA	MIMIC	Avg.	$n$
NIH AUROC (ref.)	lbl	.861	.869	.291	.887	.727	44
Entropy	src	$-.242$	$-.126$	$+.030$	$-.186$	$-.131$	40
Mean Confidence	src	$-.444$	$-.346$	$+.044$	$-.440$	$-.296$	40
ECE	src	$+.398$	$+.212$	$+.111$	$+.227$	$+.237$	44
MC Dropout	src	$+.359$	$+.287$	$+.021$	$+.283$	$+.238$	31
TTA Multi	src	$+.297$	$+.344$	$+.442$	$+.294$	$+.344$	40

Table 2: Spearman

\rho_{\mathrm{s}}

baseline comparison.

n

varies by metric depending on which caches were available (see column

n

); source-only best in bold .

Figure 2: STAMP-TS predicts OOD top-1 accuracy across 27 ImageNet-pretrained models on five distribution-shift benchmarks. Partial $\rho$ controls for ImageNet-1K top-1 accuracy and decreases monotonically with shift severity (Proposition 2 ), from ObjectNet (0.949) to ImageNet-C (0.662). ViT-Tiny, excluded by the $T>3.0$ calibration criterion, is shown for reference only.

Benchmark	Shift	$\rho_{\mathrm{s}}$	Part. $\rho_{\mathrm{s}}$	LOO range	95 % CI	Boot. CI
ObjectNet	Hard covariate	0.984	0.949	[0.982, 0.988]	[0.964, 0.993]	[0.935, 0.996]
Dollar Street	Geographic	0.940	0.804	[0.932, 0.966]	[0.870, 0.972]	[0.799, 0.991]
ImageNet-A	Natural adversarial	0.928	0.755	[0.919, 0.942]	[0.846, 0.967]	[0.808, 0.971]
ImageNet-R	Style/texture	0.911	0.700	[0.900, 0.936]	[0.812, 0.959]	[0.736, 0.982]
ImageNet-C	Synthetic corruption	0.905	0.662	[0.893, 0.926]	[0.800, 0.956]	[0.745, 0.976]

Table 3: STAMP-TS Spearman

\rho_{\mathrm{s}}

across OOD benchmarks (

n{=}27

). All

p\leq 0.0001

(two-sided permutation). Partial

\rho_{\mathrm{s}}

controls for ImageNet-1K top-1 accuracy.

Objective	$n$	$\mu$	$\sigma$	Range
CE-Supervised	31	0.786	0.011	[0.754, 0.805]
SSL	5	0.717	0.029	[0.666, 0.754]
VL-Contrastive	3	0.674	0.034	[0.626, 0.701]
TXV	4	0.641	0.101	[0.493, 0.739]

Table 4: STAMP stratified by pretraining objective for the 43 models with a standard objective grouping. The hybrid RAD-DINO checkpoint, which combines a self-supervised radiology backbone with an NIH-supervised linear head, is analyzed separately. Kruskal–Wallis

H{=}25.5

p{=}1.2{\times}10^{-5}

. MWU CE-Sup

>

SSL:

p{=}2.7{\times}10^{-6}

, Cliff’s

\delta{=}+1.0

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Label-free	Target-free	Arch.-agnostic	Closed-form	$O(n)$ fwd.
LogME ( You et al., 2021 )	✗	✗	✓	✗	✓
LEEP ( Nguyen et al., 2020 )	✗	✗	✓	✗	✓
H-score ( Bao et al., 2019 )	✗	✗	✓	✗	✓
$\mathcal{H}\Delta\mathcal{H}$ ( Ben-David et al., 2010 )	✓	✗	✓	✓	✗
FID ( Heusel et al., 2017 )	✓	✗	✓	✗	✓
MMD ( Gretton et al., 2012 )	✓	✗	✓	✗	✓

Appendix

Table 5: Theoretical properties of source-only OOD proxy metrics. “Target-free” denotes no target-domain data; “closed-form” denotes an analytically available generalization bound in terms of the metric.

A5(ii): centroid gap vs. STAMP rank			A5(iii): $\hat{L}_{\theta}$ vs. STAMP
Family	$\rho$	$p$	Family	$\rho$	$p$
All ( $n{=}40$ )	$+0.73$	$<0.001$	CNN family	$+0.236$	$0.397$
CNN ( $n{=}15$ )	$+0.61$	$0.016$	ViT/Modern	$+0.182$	$0.499$
ViT/Modern ( $n{=}16$ )	$+0.68$	$0.004$	All (pooled)	$+0.545$	$<0.001$
			labeled STAMP (pooled)	$+0.594$	$<0.001$

Appendix

Table 6: Verification of identifiability conditions A5(ii) (centroid-gap ordering, left) and A5(iii) (Lipschitz proxy vs. temporal STAMP , right). Within-family Lipschitz correlations are non-significant, while the pooled correlation reflects architecture-family differences.

Variant	$\rho_{\mathrm{s}}$	$p$
$-$ SV only	$-0.19$	$0.24$
AV only	$+0.64$	$<0.001$
SV/AV (raw ratio)	$-0.89$	$<10^{-10}$
STAMP $=1{-}$ SV/AV	$+0.89$	$<10^{-10}$

Appendix

Table 7: Component ablation on VinDr-CXR (

n{=}40

, custom models with full ablation caches).

Figure 3: (a) Component ablation: the full ratio formulation $\textsc{STAMP}=1-\mathrm{SV}/\mathrm{AV}$ is necessary; neither component alone is predictive. (b) Temporal (unlabeled same-patient) pairs uniformly outperform label-matched pairs, consistent with NIH label-noise contamination of the labeled pair pool.

Figure 4: Controlled architecture experiment: five ViT-B/16-family checkpoints differing only in pretraining objective. STAMP rank reproduces the OOD rank exactly on VinDr and near-exactly on CheXpert/MIMIC, showing the pretraining-objective effect is not an architecture confound.

Figure 5: STAMP by pretraining regime. NIH-ViT models edge out NIH-CNN and NIH-Meta in both STAMP and VinDr AUROC; all three NIH-trained regimes separate completely from Foundation-model probes. Panel (C) shows STAMP also predicts the residual from the population accuracy-on-the-line, not just the raw AUROC.

Figure 6: Layer-wise STAMP ( $n{=}35$ models with feature caches). (a) Absolute STAMP at each layer, sorted by VinDr AUROC. (b) STAMP gain per layer transition; the mid-to-penultimate transition is the key predictive signal.

Figure 7: The Universal Generalization Motif: output-layer STAMP and the mid-to-penultimate STAMP gain are the two dominant layer-wise predictors of VinDr OOD AUROC ( $n{=}35$ ); early- and mid-layer STAMP alone are uninformative.

Figure 8: Nuisance ratio (SV aug /SV random ) is negatively correlated with both STAMP and VinDr OOD AUROC ( $n{=}35$ ), corroborating that models which suppress non-semantic augmentation noise in output space also generalize better (Section 7 ).

Figure 9: Pre-deployment screening: Alarm F1 for STAMP vs. the always-alarm baseline as a function of the failure threshold $\delta$ . STAMP achieves Precision $=1.0$ (zero false alarms) in the clinically strict regime $\delta\leq 0.821$ .

Figure 10: Probit-scale accuracy-on-the-line: STAMP (color) matches labeled NIH-AUROC as a predictor of OOD AUROC across all four datasets (Steiger $p\geq 0.541$ ; $n{=}44$ model with usable NIH labels).

Figure 11: STAMP-TS vs. Dollar Street top-1 accuracy (left) and vs. the ImageNet-1K baseline predictor (right), colored by architecture family. STAMP-TS improves over the ID-accuracy baseline ( $\rho{=}0.940$ vs. $\rho{=}0.902$ ).

Figure 12: STAMP and VinDr OOD AUROC stratified by pretraining objective, showing complete distributional separation (Cliff’s $\delta{=}+1.0$ ) between CE-Supervised and SSL/VL-Contrastive/TXV groups.

CE-Supervised fine-tuned (31 models, two-stage protocol)
Architecture	STAMP	95 % CI	SV	AV
ResNet-18	0.7764	[0.763, 0.790]	0.00500	0.02238
ResNet-34	0.7818	[0.769, 0.795]	0.00465	0.02130
ResNet-50	0.7802	[0.768, 0.793]	0.00480	0.02182
ResNet-101	0.7760	[0.762, 0.790]	0.00493	0.02200
ResNet-152	0.7728	[0.759, 0.786]	0.00512	0.02254

Appendix

Table 8: Temporal STAMP scores for all 44 medical models. Bootstrap 95 % CIs use 10,000 resamples. SV and AV are the mean same-patient and cross-patient squared output distances, respectively. RAD-DINO is reported as a hybrid foundation-model checkpoint and is not assigned to the standard objective grouping in Table 4 .

Architecture	Temporal STAMP	Synthetic STAMP	$\boldsymbol{\Delta}$
ResNet-18	.7764	.9528	+.177
ResNet-34	.7818	.9472	+.165
ResNet-50	.7802	.9601	+.180
ResNet-101	.7760	.9336	+.158
ResNet-152	.7728	.9433	+.171
DenseNet-121	.7832	.9548	+.172

Appendix

Table 9: Temporal STAMP vs. labeled (same-label) STAMP for all 31 custom models.

\Delta{=}

Temporal

-

Labeled.

\Delta>0

for every model (mean

+0.161

, range

[+0.110,+0.186]

), confirming that label-noise inflation of SV in the labeled variant suppresses its predictive range. Models are sorted by temporal STAMP score.

Method	Images	Time (s)	Target?
STAMP (ours)	4,000	$\approx$ 12	No
ATC ( Garg et al., 2022 )	32,010	$\approx$ 96	Yes
NIH AUROC (labeled)	25,596	$\approx$ 77	Yes $+$ labels
AoTL ( Baek et al., 2022 ) ( $M{=}44$ )	1,126,224	$\approx$ 3,379	Yes $+$ multi-model

Appendix

Table 10: Wall-clock cost per model (single T4 GPU, batch size 32, FP16 inference). “Images” counts unique images across all passes; STAMP ’s 4,000-image pair pool is cached across all models. AoTL cost assumes

M{=}44

models evaluated jointly on the same target images. STAMP is the only method requiring zero target-domain data; all others require at least unlabeled target samples.

Corruption type	$\rho_{\mathrm{s}}$
Gaussian noise	0.891
Shot noise	0.892
Impulse noise	0.892
Defocus blur	0.868
Glass blur	0.813
Motion blur	0.874

Appendix

Table 11: STAMP-TS Spearman

\rho_{\mathrm{s}}

per corruption type on ImageNet-C (

n{=}27

retained models, all

p{<}0.001

, two-sided permutation test with 5,000 iterations). Each row reports

\rho_{\mathrm{s}}

between STAMP-TS and mean top-1 accuracy across severity levels 1–5 for that corruption type. Mean across corruptions:

0.876

. Min: contrast (

0.756

). Max: brightness (

0.929

$n$	$\rho_{\mathrm{s}}{=}0.5$	$\rho_{\mathrm{s}}{=}0.7$	$\rho_{\mathrm{s}}{=}0.8$	$\rho_{\mathrm{s}}{=}0.9$
27	0.682	0.966	0.998	${>}0.999$
30	0.721	0.977	0.999	${>}0.999$
44	0.865	0.997	${>}0.999$	${>}0.999$

Appendix

Table 12: Statistical power of the two-sided Spearman rank correlation test (

\alpha{=}0.05

) as a function of sample size

n

and true effect size

\rho_{\mathrm{s}}

. Power was estimated via Monte Carlo simulation (10,000 draws per cell) drawing rank-correlated pairs at each

\rho_{\mathrm{s}}

value. At

n{=}27

(natural image study) and

n{=}44

(medical study), all reported correlations (

\rho_{\mathrm{s}}\geq 0.905

and

\rho_{\mathrm{s}}\geq 0.844

respectively) correspond to power

>0.998

Explore similar work

Can Circuit Alignment Predict OOD Generalization?
Sep 25, 2026Ayan Banerjee, Abhra Chaudhuri, Josep Llados +2Distribution Shift Robustness OOD Generalization
Computer Vision Center, Universitat Autònoma de Barcelona · Fujitsu Research of Europe · Indian Statistical Institute, Kolkata +1
Label-Free Foundational Model Selection for Medical Image Classification under Distribution Shift via Pseudo Label Discrepancy
Aug 26, 2026Juan Iñaki Larrea, Lucas Mansilla, Enzo FerranteModel Selection Medical Imaging Foundation Models
Applied Artificial Intelligence Lab (LIAA), Computer Sciences Institute (ICC), Universidad de Buenos Aires - CONICET, Argentina · Research Institute for Signals, Systems and Computational Intelligence, sinc(i), Universidad Nacional del Litoral - CONICET, Argentina
MaRS: Robust Out-of-Distribution Detection via Mahalanobis Residual Scoring
Jun 21, 2026Francesco Di Salvo, Sebastian Doerrich, Christian LedigMedical Imaging Foundation Models Medical Image Anomaly Detection
XAI Lab Bamberg, University of Bamberg, Germany

Sep 25, 2026, cs.LGJ/K move · Enter open · S save

Can Circuit Alignment Predict OOD Generalization?

Ayan Banerjee, Abhra Chaudhuri, Josep Llados, Umapada Pal+1

Computer Vision Center, Universitat Autònoma de Barcelona · Fujitsu Research of Europe · Indian Statistical Institute, Kolkata+1

Distribution Shift Robustness OOD Generalization Neural Network Generalization Representation Alignment

Open paper AskPDF arXiv

Can out-of-distribution (OOD) generalization be predicted from a trained model's weights alone, without any target-domain data? Existing representational similarity metrics (CKA, SVCCA, RSA) compare activations rather than forecast generalization. We show they are provably insensitive to structural rerouting in the computational graph, the very change distribution shift induces. We close this gap with the Circuit Alignment Score (CAS), which compares class-specific circuits across domains via graph kernels, decomposed into same-class coherence and cross-class confusion. Casting CAS as a Lebesgue integral over the domain distribution, we prove its Monte Carlo estimate recovers the ground-truth ranking of learners by OOD accuracy, with pairwise inversion error vanishing at rate

O(1/M)

, where

M

is the number of sampled domains. Across

48

learners on PACS, CAS attains

0.88

rank correlation with OOD accuracy, versus

0.58

(CKA),

0.23

(SVCCA), and

0.14

(RSA), with similar trends on other benchmarks and even against data-dependent methods, making it the first provably consistent predictor of distributional robustness requiring neither target-domain data nor labels. The code is available at: https://github.com/ayanban011/ACE

Abstract

Explore similar work

Can Circuit Alignment Predict OOD Generalization?

Label-Free Foundational Model Selection for Medical Image Classification under Distribution Shift via Pseudo Label Discrepancy

MaRS: Robust Out-of-Distribution Detection via Mahalanobis Residual Scoring