STRADAViT: Self-Supervised Domain Adaptation of Vision Transformer Backbones for Radio Astronomy

Andrea DeMarco; Ian Fenech Conti; Hayley Camilleri; Ardiana Bushi; Simone Riggi

astro-ph.IMMar 31, 2026

STRADAViT: Self-Supervised Domain Adaptation of Vision Transformer Backbones for Radio Astronomy

Authors: Andrea DeMarco, Ian Fenech Conti, Hayley Camilleri, Ardiana Bushi, Simone Riggi

Organizations: Institute of Space Sciences and Astronomy, University of Malta, Msida, MSD2080, Malta · School of Physics and Astronomy, Royal Observatory, University of Edinburgh, Edinburgh, United Kingdom · Istituto Nazionale di Astrofisica (INAF), Osservatorio Astrofisico di Catania, Catania, Italy

Abstract

Next-generation radio astronomy surveys are delivering millions of resolved sources, yet scalable morphology analysis remains difficult across heterogeneous telescopes and imaging pipelines. We present STRADAViT, a self-supervised continued-pretraining framework for learning transferable radio-astronomy encoders from Vision Transformer (ViT) backbones. It combines mixed-survey data curation, radio astronomy-aware training-view generation, and a ViT-MAE-initialized encoder family with optional register tokens. It supports reconstruction-only, contrastive-only, and two-stage branches. Our pretraining dataset comprises 512x512 radio astronomy cutouts drawn from four complementary sources (MeerKAT, ASKAP, LOFAR/LoTSS, and SKA SDC1 simulated data). We evaluate transfer with linear probing (LP) and fine-tuning (FT) on three morphology benchmarks spanning binary and multi-class settings (MiraBest, LoTSS DR2, and Radio Galaxy Zoo). An exploratory three-fold ablation grid guides selection of a register-based two-stage checkpoint using a fixed cross-dataset criterion. Across subsequent 15-seed paired downstream evaluations on fixed partitions, this checkpoint improves linear-probe Macro-F1 over its ViT-MAE initialization on all three benchmarks and improves fine-tuning on MiraBest and RGZ DR1, while LoTSS DR2 fine-tuning declines; all six differences remain statistically supported after Holm correction. A parallel DINOv2 experiment yields mixed adaptation effects: the procedure transfers, but the benefit is not uniform. STRADAViT thus improves frozen ViT representations while retaining clear dataset-dependent limitations and remaining below task-specialized methods on standard MiraBest classification.

Figures & tables

Panel of Figure 1 — Figure 1: High-level overview of STRADAViT. Mixed-survey radio cutouts are standardized and converted into radio astronomy-aware training views, then used in reconstruction-only, contrastive-only, or two-stage continued-pretraining branches. The adapted encoders are evaluated under the same downstream linear-probing and fine-tuning protocol across the three morphology benchmarks.

Source	Cutouts	Ratio
MGCLS (MeerKAT), DR1	139500	0.24
ASKAP (CASDA)	261390	0.44
LoTSS (LOFAR)	177476	0.30
SKA SDC1, DR1	12288	0.02
Total	590654	1.00

Table 1: Composition of the self-supervised pretraining dataset (all images are

512\times 512

cutouts). The LoTSS count is after spatial exclusion. “Ratio” denotes the fraction of total cutouts contributed by each source.

Dataset	Samples
Radio Galaxy Zoo DR1	98,391
MiraBest	1,563
LoTSS DR2	8,805

Table 2: Evaluation datasets and retained image counts for the exploratory grid. MiraBest combines FIRST and NVSS image entries; the count is not a count of distinct sources.

Figure 2: Example cutouts from the three evaluation datasets after preprocessing with the same pipeline used for pretraining and evaluation (Section 3 ). Panels show samples of each class for (a) RGZ DR1, (b) MiraBest, and (c) the LoTSS DR2 visual-classification sample of Horton et al. (2025) .

Figure 3: Reconstruction-branch masked-reconstruction views. The first column shows the standardized parent cutout (after the per-image ZScale contrast stretch described in Section 3 ). Subsequent columns show example ROI-aligned crops produced by our single-view strategy (Section 6 ); with probability $p_{\mathrm{global}}=0.2$ the strategy instead samples a wide-field crop from the full cutout. The selected view is then transformed by progressively applied, morphology-preserving augmentations.

Panel of Figure 4 — Figure 4: Representative masked-reconstruction diagnostics. The left and right panels contain independent sets of six examples. Within each panel, rows show, from top to bottom, the original reconstruction view, the masked input presented to the model, the decoder reconstruction, and the absolute pixel residual between the original and reconstructed views. The reconstructions recover the principal source morphology and broad intensity structure, whereas residuals remain concentrated around compact peaks, sharp boundaries, and fine-scale image structure. Unlike Figure 3 , this figure assesses reconstruction behavior rather than view construction.

Operator	Mild setting	Strong setting
Dihedral transforms	$k\times 90^{\circ}$ rotation ( $k\in\{0,1,2,3\}$ ) + horizontal flip ( $p=0.5$ )	Same as mild
Asinh stretch	$p_{\mathrm{asinh}}=0.35$ , $\alpha\in[3,12]$	$p_{\mathrm{asinh}}=0.50$ , $\alpha\in[3,20]$
Affine jitter	translation $\tau=0.03$ , scale jitter $\delta=0.05$	translation $\tau=0.05$ , scale jitter $\delta=0.08$
Gaussian noise	$\sigma=0.02$	$\sigma=0.04$
Banding	reconstruction branch: $B=2$ ; contrastive wide view: disabled ( $B=0$ ); $\Delta\in[-0.3,0.3]$	$B=3$ ; $\Delta\in[-0.3,0.3]$
Flux/background perturbation	$p=0.5$ , $T=\mu+0.3\sigma_{x}$ , $f\in[0.7,0.95]$	$p=0.5$ , $T=\mu+0.3\sigma_{x}$ , $f\in[0.5,0.9]$

Table 3: Augmentation parameter settings used for view generation. “Mild” applies to the reconstruction branch and to the wide view in the contrastive objective; “Strong” applies to the additional contrastive view.

Figure 5: Contrastive-branch multi-view examples produced by our on-the-fly augmenter. The first column shows the standardized parent cutout. Each subsequent pair of columns illustrates two correlated views sampled from the same cutout and anchored to the same object-centric ROI (Section 7 ): a wider/global view with mild, morphology-preserving augmentations and an additional view with stronger corruptions. For training we sample exactly $V=2$ views per cutout per forward pass, so each step uses one such pair per cutout; the multiple pairs shown here are independent draws for visualization. In the contrastive loss, the two views of the same cutout form the positives, while all other views in the (distributed) micro-batch act as negatives.

Variant	Parameters
InfoNCE	temperature $\tau$ : $0.20\rightarrow 0.10$ (warmup 15% epochs), then $\tau=0.10$
HCL	$\tau$ as above; $\beta=1.0$ , $\tau_{+}=0$
Soft-HCL	$\tau$ as above; $\alpha=0.5$ , $\tau_{h}=0.15$

Table 4: Contrastive loss variants and hyperparameters used in the contrastive branch.

Setting	Reconstruction branch	Contrastive branch
Epochs	35	35
Patch size	16	16
Masking	mask ratio $0.75$	unmasked encoder (no patch masking)
Views per cutout	1	$V=2$
Per-device batch size	256	64
Gradient-accumulation steps	2	8

Table 5: Default ViT-MAE-branch training settings (each stage lasts 35 epochs). DINOv2 uses the batch and resolution adjustments described in the text.

View strategy	MiraBest LP	LoTSS DR2 LP	RGZ DR1 LP	MiraBest FT	LoTSS DR2 FT	RGZ DR1 FT
Standard/no-ROI views	$0.659\pm 0.011$	$0.527\pm 0.012$	$0.635\pm 0.004$	$0.715\pm 0.007$	$0.651\pm 0.014$	$0.796\pm 0.002$
ROI-aware views	$0.691\pm 0.010$	$0.560\pm 0.012$	$0.728\pm 0.003$	$0.735\pm 0.014$	$0.667\pm 0.007$	$0.822\pm 0.003$
Delta, ROI-aware $-$ standard	$+3.16$	$+3.32$	$+9.34$	$+1.98$	$+1.63$	$+2.57$

Table 6: Matched contrastive-only ablation of view-generation strategy. Values are mean Macro-F1

\pm

standard deviation over

K=3

folds; the delta is computed as ROI-aware minus standard/no-ROI views, in percentage points, before rounding the displayed means to three decimals.

Model	Protocol	MiraBest		LoTSS DR2		RGZ DR1
		Macro-F1	Weighted-F1	Macro-F1	Weighted-F1	Macro-F1	Weighted-F1
DINOv2-Base	LP	$0.717\pm 0.014$	$0.718\pm 0.014$	$0.569\pm 0.005$	$0.751\pm 0.004$	$0.661\pm 0.004$	$0.871\pm 0.001$
DINOv2-Base	FT	$0.461\pm 0.195$	$0.451\pm 0.202$	$0.708\pm 0.008$	$0.831\pm 0.002$	$0.808\pm 0.003$	$0.916\pm 0.001$
DINOv2-Small	LP	$0.713\pm 0.018$	$0.713\pm 0.018$	$0.544\pm 0.014$	$0.729\pm 0.007$	$0.648\pm 0.002$	$0.861\pm 0.001$
DINOv2-Small	FT	$0.707\pm 0.031$	$0.708\pm 0.031$	$0.704\pm 0.014$	$0.828\pm 0.004$	$0.795\pm 0.002$	$0.912\pm 0.001$
DINOv2-Base(R)	LP	$0.685\pm 0.013$	$0.686\pm 0.013$	$0.529\pm 0.009$	$0.716\pm 0.003$	$0.629\pm 0.001$	$0.862\pm 0.001$

Table 7: Baseline transfer under linear probing (LP) and full fine-tuning (FT). Entries are mean

\pm

SD over the same three folds; comparisons are descriptive. Bold marks the highest mean Macro-F1 per dataset and protocol, with Weighted-F1 breaking ties. (R) denotes register tokens.

Loss	Protocol	MiraBest		LoTSS DR2		RGZ DR1
		Macro-F1	Weighted-F1	Macro-F1	Weighted-F1	Macro-F1	Weighted-F1
No registers ( $R=0$ )
L2	LP	$0.607\pm 0.013$	$0.607\pm 0.013$	$0.433\pm 0.006$	$0.606\pm 0.005$	$0.526\pm 0.005$	$0.811\pm 0.002$
L2	FT	$0.708\pm 0.011$	$0.709\pm 0.011$	$0.661\pm 0.013$	$0.791\pm 0.003$	$0.795\pm 0.005$	$0.915\pm 0.001$
L2+L1	LP	$0.609\pm 0.006$	$0.609\pm 0.006$	$0.436\pm 0.013$	$0.599\pm 0.009$	$0.525\pm 0.001$	$0.813\pm 0.001$
L2+L1	FT	$0.713\pm 0.018$	$0.713\pm 0.018$	$0.657\pm 0.017$	$0.793\pm 0.003$	$0.795\pm 0.005$	$0.914\pm 0.001$

Table 8: Reconstruction-only transfer from ViT-MAE under linear probing (LP) and full fine-tuning (FT). Entries are mean

\pm

SD over the same three folds; comparisons are descriptive. Bold marks the highest mean Macro-F1 per dataset and protocol, within each register setting, with Weighted-F1 breaking ties.

Loss	Protocol	MiraBest		LoTSS DR2		RGZ DR1
		Macro-F1	Weighted-F1	Macro-F1	Weighted-F1	Macro-F1	Weighted-F1
No registers ( $R=0$ )
InfoNCE	LP	$0.676\pm 0.003$	$0.677\pm 0.002$	$0.538\pm 0.010$	$0.710\pm 0.007$	$0.699\pm 0.001$	$0.878\pm 0.000$
InfoNCE	FT	$0.726\pm 0.005$	$0.727\pm 0.005$	$0.672\pm 0.011$	$0.798\pm 0.004$	$0.818\pm 0.003$	$0.921\pm 0.001$
Soft-HCL	LP	$0.692\pm 0.012$	$0.693\pm 0.012$	$0.549\pm 0.009$	$0.719\pm 0.003$	$0.709\pm 0.001$	$0.883\pm 0.001$
Soft-HCL	FT	$0.719\pm 0.005$	$0.720\pm 0.015$	$0.660\pm 0.002$	$0.799\pm 0.003$	$0.824\pm 0.003$	$0.923\pm 0.001$

Table 9: Contrastive-only transfer from ViT-MAE under linear probing (LP) and full fine-tuning (FT). Entries are mean

\pm

SD over the same three folds; comparisons are descriptive. Bold marks the highest mean Macro-F1 per dataset and protocol, within each register setting, with Weighted-F1 breaking ties.

Setting	Initialization	MiraBest		LoTSS DR2		RGZ DR1
		Macro-F1	Weighted-F1	Macro-F1	Weighted-F1	Macro-F1	Weighted-F1
Linear Probe	DINOv2-Base	$0.683\pm 0.010$	$0.683\pm 0.010$	$0.551\pm 0.010$	$0.724\pm 0.006$	$0.735\pm 0.004$	$0.894\pm 0.000$
Linear Probe	DINOv2-Base(R)	$0.686\pm 0.018$	$0.687\pm 0.018$	$0.561\pm 0.005$	$0.731\pm 0.003$	$0.739\pm 0.004$	$0.895\pm 0.000$
Fine-Tuning	DINOv2-Base	$0.721\pm 0.020$	$0.722\pm 0.020$	$0.676\pm 0.009$	$0.816\pm 0.004$	$0.816\pm 0.000$	$0.919\pm 0.000$
Fine-Tuning	DINOv2-Base(R)	$0.738\pm 0.014$	$0.738\pm 0.014$	$0.678\pm 0.015$	$0.816\pm 0.006$	$0.828\pm 0.002$	$0.922\pm 0.002$

Table 10: DINOv2 initialization ablation under the contrastive-only HCL recipe (mean

\pm

std over the same

K=3

fold assignments used elsewhere). The highest mean Macro-F1 per dataset within each transfer setting is highlighted in bold; ties are broken by Weighted-F1. Differences are interpreted descriptively rather than as statistically established rankings.

Reconstruction loss	Contrastive loss	MiraBest		LoTSS DR2		RGZ DR1
		Macro-F1	Weighted-F1	Macro-F1	Weighted-F1	Macro-F1	Weighted-F1
No registers ( $R=0$ )
L2	InfoNCE	$0.670\pm 0.010$	$0.671\pm 0.010$	$0.529\pm 0.010$	$0.705\pm 0.006$	$0.693\pm 0.003$	$0.877\pm 0.001$
	Soft-HCL	$0.677\pm 0.007$	$0.678\pm 0.007$	$0.543\pm 0.010$	$0.715\pm 0.004$	$0.704\pm 0.002$	$0.881\pm 0.000$
	HCL	$0.689\pm 0.010$	$0.690\pm 0.011$	$0.545\pm 0.012$	$0.714\pm 0.006$	$0.706\pm 0.003$	$0.884\pm 0.001$
L2+L1	InfoNCE	$0.664\pm 0.013$	$0.665\pm 0.013$	$0.528\pm 0.010$	$0.701\pm 0.006$	$0.693\pm 0.004$	$0.876\pm 0.001$

Table 11: Continued pretraining (two-stage branch): linear-probe results (mean

\pm

std over

K=3

folds). The contrastive stage uses an unmasked encoder. The highest mean Macro-F1 per dataset within each register setting is highlighted in bold; ties are broken by Weighted-F1.

Reconstruction loss	Contrastive loss	MiraBest		LoTSS DR2		RGZ DR1
		Macro-F1	Weighted-F1	Macro-F1	Weighted-F1	Macro-F1	Weighted-F1
No registers ( $R=0$ )
L2	InfoNCE	$0.714\pm 0.002$	$0.715\pm 0.015$	$0.664\pm 0.013$	$0.793\pm 0.004$	$0.824\pm 0.003$	$0.922\pm 0.001$
	Soft-HCL	$0.732\pm 0.007$	$0.733\pm 0.007$	$0.661\pm 0.007$	$0.790\pm 0.004$	$0.822\pm 0.003$	$0.923\pm 0.001$
	HCL	$0.727\pm 0.003$	$0.728\pm 0.003$	$0.654\pm 0.008$	$0.793\pm 0.004$	$0.824\pm 0.004$	$0.923\pm 0.002$
L2+L1	InfoNCE	$0.715\pm 0.009$	$0.716\pm 0.009$	$0.655\pm 0.006$	$0.790\pm 0.003$	$0.821\pm 0.003$	$0.922\pm 0.001$

Table 12: Continued pretraining (two-stage branch): full fine-tuning results (mean

\pm

std over

K=3

folds). The contrastive stage uses an unmasked encoder. The highest mean Macro-F1 per dataset within each register setting is highlighted in bold; ties are broken by Weighted-F1.

Protocol	Branch	MiraBest	LoTSS DR2	RGZ DR1
LP	Contrastive-only, Soft-HCL	$0.691$	$0.560$	$0.728$
	Two-stage, L2+L1 $\rightarrow$ Soft-HCL	$0.674$	$0.564$	$0.732$
	Delta (pp)	$-1.7$	$+0.4$	$+0.4$
FT	Contrastive-only, Soft-HCL	$0.735$	$0.667$	$0.822$
	Two-stage, L2+L1 $\rightarrow$ Soft-HCL	$0.725$	$0.677$	$0.825$
	Delta (pp)	$-1.0$	$+1.0$	$+0.3$

Table 13: Architecture- and loss-matched comparison of register-based (

R=4

) Soft-HCL contrastive-only training with L2+L1 reconstruction followed by Soft-HCL. Values are mean Macro-F1 over the same three exploratory folds. Delta is

(\text{two-stage}-\text{contrastive-only})\times 100

percentage points and is interpreted descriptively.

Setting	Model / comparison	MiraBest	LoTSS DR2	RGZ DR1
Linear Probe
	Release checkpoint	$0.674$	$0.564$	$0.732$
	Delta vs ViT-MAE (pp)	$+3.0$	$+7.5$	$+14.6$
	Delta vs highest-mean stable DINOv2 (pp)	$-4.3$	$-0.5$	$+7.1$
Fine-Tuning
	Release checkpoint	$0.725$	$0.677$	$0.825$

Table 14: Mean Macro-F1 and percentage-point (pp) changes for the fixed release checkpoint (L2+L1 reconstruction, Soft-HCL contrastive refinement, and four register tokens), relative to the ViT-MAE starting point and the highest-mean stable DINOv2 comparator. The DINOv2 references are Base for all LP cells, Small for MiraBest FT, Base for LoTSS DR2 FT, and Base(R) for RGZ DR1 FT. Each delta is computed as

(\text{release-checkpoint mean Macro-F1}-\text{reference mean Macro-F1})\times 100

. Positive values indicate higher mean Macro-F1.

Dataset	Protocol	Reference	Adapted model	$\Delta$ [95% CI] (pp)	$p$	$p_{\mathrm{Holm}}$
STRADAViT versus ViT-MAE initialization
MiraBest	LP	$0.660\pm 0.007$	$0.838\pm 0.023$	$+17.78\;[16.52,19.04]$	$3.77\times 10^{-14}$	$1.51\times 10^{-13}$
MiraBest	FT	$0.825\pm 0.014$	$0.847\pm 0.010$	$+2.16\;[1.37,2.96]$	$4.17\times 10^{-5}$	$4.17\times 10^{-5}$
LoTSS DR2	LP	$0.495\pm 0.003$	$0.572\pm 0.007$	$+7.77\;[7.33,8.21]$	$1.71\times 10^{-15}$	$8.54\times 10^{-15}$
LoTSS DR2	FT	$0.700\pm 0.009$	$0.674\pm 0.006$	$-2.61\;[-3.29,-1.93]$	$9.80\times 10^{-7}$	$1.96\times 10^{-6}$
RGZ DR1	LP	$0.586\pm 0.002$	$0.733\pm 0.002$	$+14.65\;[14.52,14.78]$	$1.13\times 10^{-26}$	$6.80\times 10^{-26}$

Table 15: Repeated-seed comparisons of the fixed STRADAViT checkpoint with its ViT-MAE initialization and the HCL-adapted DINOv2-Base(R) checkpoint with its off-the-shelf initialization. Macro-F1 values are mean

\pm

standard deviation over 15 matched seeds on fixed train–test partitions. Delta is adapted model minus reference model in percentage points (pp); CI denotes its pointwise 95% confidence interval, also in pp. MiraBest uses the standard Confident

729/104

split; LoTSS DR2 and RGZ DR1 use the first exploratory fold.

p

is from a two-sided paired

t

-test, and

p_{\mathrm{Holm}}

is adjusted separately across the six comparisons within each model pair.

Figure 6: Aggregate recall-form confusion matrices (%) for MiraBest, LoTSS DR2, and RGZ DR1. Panel (a) shows the linear-probe comparison and panel (b) shows the full fine-tuning comparison. In both panels, the selected STRADAViT configuration is compared against the two reference baselines, ViT-MAE and DINOv2-Base(R), labeled “DINOv2 (Registers)” in the panels. Rows denote true classes and columns denote predicted classes. DINOv2-Base(R) provides the stable register-based reference corresponding to the initialization-portability experiment; it is not presented as the strongest baseline in every dataset and protocol.

Figure 7: UMAP projections of frozen embeddings for RGZ DR1 (panels (a)–(d)), MiraBest (panels (e)–(h)), and LoTSS DR2 (panels (i)–(l)). Columns follow the architecture-matched register-based pathway used for the selected release configuration: the unadapted ViT-MAE initialization, L2+L1 reconstruction-only training, Soft-HCL contrastive-only training, and L2+L1 followed by Soft-HCL two-stage training; the adapted checkpoints use four register tokens. For each dataset, the full labeled sample is class-balanced once with subset seed 2026 and reused across all four models; no evaluation fold is chosen based on visual separability. The resulting samples contain 8,898 RGZ DR1, 1,490 MiraBest, and 910 LoTSS DR2 images. Each panel is fitted independently to the unnormalized pooled embeddings using cosine distance, 30 neighbors, minimum distance 0.1, and UMAP seed 2026. Only within-panel class separation and mixing are interpreted.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Study and method	Reported result	Split and evaluation details
Scaife and Porter (2021) : supervised $D_{16}$ -equivariant CNN	Accuracy $96.56\pm 1.29$ % (SD)	Standard split; five training repeats; rotated test images
Bowles et al. (2021) : attention-gated CNN	Accuracy 92%	MiraBest Confident test set; published point estimate
Hossain et al. (2023) : BYOL E(2)-equivariant G-CNN	Accuracy $97.12\pm 0.40$ % (SD)	Standard split; 15 training repeats
Slijepcevic et al. (2023) : RGZ-pretrained BYOL, fine-tuned	Error $1.92\pm 0.25$ % (SE)	Standard split; 10 training repeats
Andrianomena and Tang (2023) : VDVAE + extra trees	Accuracy 82%	Standard split; frozen latent features; $64\times 64$ images
STRADAViT: linear probe	Accuracy $83.85\pm 2.33$ % (SD)	Standard split; 15 seeds; fixed release checkpoint

Appendix

Table 16: Published MiraBest Confident results and the selected STRADAViT checkpoint. The standard split contains 729 training and 104 test images. Architecture, preprocessing, training, and test-time procedures differ between studies; these are published-score comparisons, not paired model comparisons. SD denotes standard deviation and SE standard error; the uncertainty definitions are retained from each study.

Protocol	Model	Accuracy (%)	Error (%)	Macro-F1	Weighted-F1	FR I F1	FR II F1
LP	ViT-MAE	$67.76\pm 0.62$	$32.24\pm 0.62$	$0.660\pm 0.007$	$0.665\pm 0.007$	$0.584\pm 0.011$	$0.737\pm 0.004$
LP	STRADAViT	$83.85\pm 2.33$	$16.15\pm 2.33$	$0.838\pm 0.023$	$0.838\pm 0.024$	$0.835\pm 0.021$	$0.841\pm 0.027$
FT	ViT-MAE	$82.50\pm 1.37$	$17.50\pm 1.37$	$0.825\pm 0.014$	$0.825\pm 0.014$	$0.822\pm 0.016$	$0.827\pm 0.013$
FT	STRADAViT	$84.68\pm 0.99$	$15.32\pm 0.99$	$0.847\pm 0.010$	$0.847\pm 0.010$	$0.840\pm 0.010$	$0.853\pm 0.010$

Appendix

Table 17: Repeated linear-probe (LP) and fine-tuning (FT) comparison on the fixed MiraBest Confident

729/104

split. Each model/protocol combination is trained 15 times with matched random seeds. Values are mean

\pm

standard deviation; accuracy and error are percentages, and F1 scores are on the

[0,1]

scale. Paired Macro-F1 differences, confidence intervals, and significance tests are reported in Table 15 .

Explore similar work

Cross-simulator transfer with foundation model summaries: Towards robust SKA-era reionization inference
Aug 26, 2026Yannic Pietschke, Caroline Heneka, Ayodele Ore +1Simulation-Based Inference Transfer Learning
Institut für Theoretische Physik, Universität Heidelberg, Philosophenweg 16, 69120 Heidelberg, Germany · Max Planck Institute for Radioastronomy, Auf dem Hügel 69, Bonn, 53121, Germany
Clustering Guided Domain-Specific Pretrained Foundation Model for Very High-Resolution Arctic Remote Sensing
May 28, 2026Amal S. Perera, Chandi Witharana, Elias Manos +2Domain-Adaptive Pretraining Vision Foundation Model Adaptation
Department of Natural Resources and the Environment, University of Connecticut, Storrs, CT · Woodwell Climate Research Center, Falmouth, MA.
What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers
Aug 19, 2026Jesse PonnockVision Transformer Transformer Attention
Johns Hopkins University

Aug 26, 2026J/K move · Enter open · S save

Cross-simulator transfer with foundation model summaries: Towards robust SKA-era reionization inference

Yannic Pietschke, Caroline Heneka, Ayodele Ore, Romain Meriot

Institut für Theoretische Physik, Universität Heidelberg, Philosophenweg 16, 69120 Heidelberg, Germany · Max Planck Institute for Radioastronomy, Auf dem Hügel 69, Bonn, 53121, Germany

Simulation-Based Inference Transfer Learning Self-Supervised Pre-Training Vision Transformer

Open paper AskPDF arXiv

Simulation-based inference (SBI) for parameter estimation is vulnerable to model misspecification: neural summaries and density estimators trained on a specific forward model typically fail when applied to data drawn from another model, or from real observations, and no training simulator can capture the full observational pipeline of a real measurement exactly. We show that a self-supervised Vision Transformer (ViT), pretrained label-free on a fast approximate simulator, produces transferable data summaries that generalize across simulators. Without retraining, it can be reused as a frozen encoder to infer astrophysical parameters from a completely different simulator that resolves the radiative transfer explicitly, on which it has never seen either data or parameters. As a concrete use case in 21cm cosmology, SKATR, a ViT pretrained with a Joint Embedding Predictive Architecture (JEPA), serves as a foundation model for reionization inference from upcoming SKA measurements: SKATR is pretrained once on 67k low-cost, noiseless semi-numerical 21cmFAST lightcones, then frozen and applied to hydrodynamical Loreli II lightcones, where a lightweight conditional flow matching head infers five astrophysical parameters; the encoder is never shown Loreli data, its parameters, or any noise. In our comparison, SKATR yields the most precise and best-calibrated posteriors across all five parameters, matching the accuracy of the fully-supervised in-domain baseline while requiring 2.6x fewer radiative-transfer simulations. Under realistic SKA AA* noise, only SKATR remains simultaneously accurate, informative, and calibrated, outperforming even a supervised baseline retrained from scratch on noisy data. Self-supervised pretraining on computationally efficient semi-numerical simulations is therefore a viable route to calibrated, simulator- and noise-agnostic reionization inference for the SKA-era.

Abstract

Explore similar work

Cross-simulator transfer with foundation model summaries: Towards robust SKA-era reionization inference

Clustering Guided Domain-Specific Pretrained Foundation Model for Very High-Resolution Arctic Remote Sensing

What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers