RINO: Renormalization Group Distillation with No Labels

Zichun Hao; Raghav Kansal; Abhijith Gandrakota; Chang Sun; Haoyang Li; Zihan Zhao; Ngadiuba Jennifer; Javier Duarte

hep-exOct 11, 2026

RINO: Renormalization Group Distillation with No Labels

Authors: Zichun Hao, Raghav Kansal, Abhijith Gandrakota, Chang Sun, Haoyang Li, Zihan Zhao, Ngadiuba Jennifer, Javier Duarte, +1 more

Organizations: Division of Physics, Mathematics and Astronomy, California Institute of Technology, Pasadena, CA 91125 · Physics Division, Fermi National Accelerator Laboratory, Batavia, IL 60510

Abstract

Supervised machine learning in high-energy physics (HEP) requires training on labeled simulations of particle collisions, which are known to be unfaithful to the underlying physics. This leads to significant inaccuracies and biases in the transfer of trained models to real data, limiting the power of deep learning methods. Self-supervised learning (SSL) is a promising alternative for pretraining directly on real, unlabeled data. However, existing techniques largely operate with inductive biases inspired by domains such as images and natural language. We propose RINO (Renormalization Group DIstillation with NO Labels), which, for the first time, exploits the fundamentally multi-scale renormalization group structure of high-energy jet formation as a physically grounded source of SSL views. We define teacher-student representations across energy scales using different levels of hierarchical clustering, aligned via DINO-style self-distillation. To mirror the background-dominated nature of real collider data, RINO pretrains exclusively on jets from quantum chromodynamics (QCD) processes and is evaluated on out-of-distribution (OOD) transfer to rare signal classes. In our controlled data-efficiency study, RINO matches fully supervised training on JetNet after finetuning on only 1% of the labeled data. With the full labeled dataset, RINO improves OOD classification accuracy on top-quark jets by 5.9% and, without retraining, improves AUC on the unseen Higgs boson decay channel $H \to 4q$ by 4.1% over the strongest SSL baseline. Scale-structure diagnostics reveal that DINO-style self-distillation permits our backbone to retain scale-dependent structure. Ablation studies show that this accounts for an 8.6% OOD accuracy gap with respect to alternative SSL paradigms such as SimCLR. Code is available at https://github.com/zichunhao/RINO.

Figures & tables

Panel of Figure 1 — Figure 1: (a) A QCD parton shower (here, a $q\bar{q}$ pair radiating softer gluons and quarks) is analogous to a renormalization group (RG) flow down to the hadronization scale. The $k_{\mathrm{T}}$ clustering approximately inverts this process, so stopping at different numbers of subjets $N$ generates the multi-scale views used for RINO self-distillation. (b) RINO training pipeline (DINO term of Eq. 2 ). The same jet is clustered at two $k_{\mathrm{T}}$ stopping points (here teacher $N{=}6$ , student $N{=}3$ ). The teacher produces soft labels for the student to match via cross-entropy. Teacher parameters are updated from the student via an exponential moving average (EMA). This figure shows one representative student-teacher pair; in practice the loss aligns each student view to every teacher view at a different scale (Sec. 3.2 ).

SSL baselines
Method	$k$ -NN	Linear probe	Linear head	MLP head
Supervised	$0.722\pm 0.022$	$0.714\pm 0.030$	$0.784\pm 0.033$	$0.786\pm 0.023$
JetCLR ( Dillon et al., 2022 )	$0.780\pm 0.002$	$0.793\pm 0.001$	$0.791\pm 0.048$	$0.700\pm 0.031$
OmniJet- $\alpha$ ( Birk et al., 2024 )	$0.859\pm 0.001$	$0.869\pm 0.002$	$0.749\pm 0.080$	$0.751\pm 0.050$
MPMv1 ( Golling et al., 2024 )	$0.882\pm 0.002$	$0.907\pm 0.002$	$0.809\pm 0.008$	$0.800\pm 0.019$
MPMv2 ( Leigh et al., 2025 )	$0.896\pm 0.001$	$0.912\pm 0.001$	$0.817\pm 0.027$	$0.760\pm 0.016$

Table 1: Top-tagging accuracy on JetClass (top vs. QCD). The linear and MLP heads are finetuned on JetNet only (OOD);

k

-NN and the linear probe are fit on JetClass labels and measure frozen-embedding quality. The bottom block groups all methods using

k_{\mathrm{T}}

-clustering views (ours), differing only in the training loss. Each entry is

\mu\pm\sigma

; bold marks the best per column; underline marks entries within

1\sigma

of the best.

Figure 2: OOD top-tagging accuracy versus JetNet data fraction. RINO with

1\%

of labels matches supervised training on

100\%

within uncertainty. Bands show

\pm 1\sigma

over

10

finetuning seeds (MLP head); the dotted line marks supervised training on

100\%

of labels.

SSL baselines
Method	$H\to 4q$		$H\to b\bar{b}$
	ACC	AUC	ACC	AUC
Supervised	$0.707\pm 0.026$	$0.800\pm 0.029$	$0.623\pm 0.020$	$0.726\pm 0.030$
JetCLR ( Dillon et al., 2022 )	$0.727\pm 0.067$	$0.803\pm 0.042$	$\mathbf{0.706\pm 0.041}$	$\underline{0.782\pm 0.018}$
MPMv1 ( Golling et al., 2024 )	$0.741\pm 0.027$	$0.819\pm 0.018$	$0.612\pm 0.026$	$0.705\pm 0.017$
MPMv2 ( Leigh et al., 2025 )	$0.757\pm 0.033$	$0.859\pm 0.018$	$0.648\pm 0.023$	$\underline{0.778\pm 0.010}$

Table 2: Cross-topology transfer ( JetNet

\to

JetClass , linear head). Models are finetuned on top tagging only and evaluated on unseen JetClass signal topologies. Bold marks the best per column; underline marks entries within one standard deviation of the best.

Configuration	OOD top ACC
iBOT only	$0.736\pm 0.028$
+ Cross-scale self-distillation	$0.839\pm 0.005$ $\uparrow 14.0\%$
+ Teacher HP tuning	$0.847\pm 0.003$ $\uparrow 1.0\%$

Table 3: Ablation: contribution of DINO self-distillation. All variants share the same polar-binned PE, backbone, and finetuning protocol. Arrows give the relative gain over the previous row.

Panel of Figure 3 — Figure 3: Factorization of RINO’s OOD accuracy gain. The top horizontal arrow replaces JetCLR augmentations with $k_{\mathrm{T}}$ scale views; the bottom horizontal arrow adds DINO-based multi-scale distillation across $k_{\mathrm{T}}$ views. Vertical arrows compare contrastive and self-distillation objectives. RINO is shown before teacher tuning ( $0.839$ , Table 3 ); the final model reaches $0.847$ (Table 1 ).

Method	$W_{1}$ (QCD) $\downarrow$	$W_{1}$ (top) $\downarrow$	MMD 2 (QCD) $\downarrow$	MMD 2 (top) $\downarrow$	OOD ACC $\uparrow$
Supervised	$0.746$	$0.728$	$0.515$	$0.564$	$0.786\pm 0.023$
JetCLR	$0.132$	$0.127$	$0.196$	$0.191$	$0.700\pm 0.031$
MPMv1	$0.241$	$0.262$	$0.133$	$0.162$	$0.800\pm 0.019$
MPMv2	$0.375$	$0.348$	$0.338$	$0.305$	$0.760\pm 0.016$
OmniJet- $\alpha$	$0.815$	$0.752$	$0.571$	$0.552$	$0.751\pm 0.050$
JetCLR-scale	$0.133$	$0.145$	$0.045$	$0.048$	$0.780\pm 0.023$

Table 4: Domain shift in finetuned representations ( JetNet vs. JetClass ). Lower

W_{1}

and MMD 2 indicate more domain-invariant representations.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Group	Hyperparameter	Value
Optimization	Optimizer	AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.999$ )
	Peak learning rate	$3\times 10^{-3}$
	Weight decay	$0.05$
	Gradient clipping (max norm)	$3.0$
Schedule	Total epochs	$100$
	LR warmup	epochs $0$ – $19$ , $10^{-6}\to 1.0$ (cosine)

Appendix

Table 5: RINO pretraining hyperparameters (production configuration).

Feature	Definition	$\mu$	$\sigma$
$\log p_{\mathrm{T}}$	$\log p_{\mathrm{T},i}$ , constituent transverse momentum	$1.7$	$1.8$
$\log E$	$\log E_{i}$ , constituent energy	$2.0$	$1.8$
$\log(p_{\mathrm{T}}/p_{\mathrm{T}}^{\mathrm{jet}})$	$\log(p_{\mathrm{T},i}/p_{\mathrm{T}}^{\mathrm{jet}})$ , relative $p_{\mathrm{T}}$	$-4.7$	$1.8$
$\log(E/E^{\mathrm{jet}})$	$\log(E_{i}/E^{\mathrm{jet}})$ , relative energy	$-4.7$	$1.8$
$\Delta\eta$	$\eta_{i}-\eta_{\mathrm{jet}}$ , rapidity offset	$0.0$	$0.14$
$\Delta\phi$	$\phi_{i}-\phi_{\mathrm{jet}}$ (wrapped to $[-\pi,\pi)$ ), azimuthal offset	$0.0$	$0.14$

Appendix

Table 6: Per-particle input features and the fixed standardization

(\mu,\sigma)

applied by the input pipeline. The model receives

(x-\mu)/\sigma

for each feature.

Per-particle collision rate (%)
Scheme	$20\!\times\!20$	$50\!\times\!50$	$80\!\times\!80$	$\mathbf{100\!\times\!100}$	$120\!\times\!120$
(patches)	400	2,500	6,400	10,000	14,400
Cartesian	64.76	35.98	22.19	16.82	13.13
Log-Cartesian	42.70	13.61	6.27	4.19	3.00
Polar	32.96	9.87	4.52	3.03	2.16
Per-jet collision rate (%)

Appendix

Table 7: Bin-collision rates (%) for positional-encoding schemes across grid sizes, measured on

5\!\times\!10^{5}

JetClass QCD jets. Lower collision rates reduce ambiguity among masked iBOT tokens. The production choice is Polar

100\!\times\!100

(bold).

Figure 4: Cumulative distribution of constituent multiplicity for QCD jets in JetClass (

12.5

M jets across train, validation, and test splits). The shaded region marks the

k_{\mathrm{T}}

view range

N\in\{2,\ldots,16\}

; only

11\%

of jets have

\leq 16

constituents, confirming that the coarse views are genuine coarse-grainings.

$N$	$q_{N}$ [GeV]	$p_{\mathrm{T}}^{\mathrm{soft}}/p_{\mathrm{T}}^{\mathrm{jet}}$
$2$	$9.91\ [6.07,17.09]$	$0.23\ [0.12,0.36]$
$3$	$5.70\ [3.69,9.01]$	$0.088\ [0.042,0.156]$
$4$	$3.89\ [2.57,5.99]$	$0.044\ [0.021,0.084]$
$6$	$2.25\ [1.48,3.39]$	$0.018\ [0.009,0.034]$
$8$	$1.49\ [0.95,2.26]$	$0.010\ [0.006,0.019]$
$16$	$0.46\ [0.24,0.76]$	$0.0033\ [0.0021,0.0054]$

Appendix

Table 8: Physical scales of the exclusive-

k_{\mathrm{T}}

views: merge scale

q_{N}

and softest-subjet momentum fraction, median [IQR] over JetClass QCD jets with more than

16

constituents.

Hyperparameter	Value
Optimizer	AdamW ( $\beta_{1}=0.9$ , $\beta_{2}=0.999$ )
Head learning rate	$1\times 10^{-4}$
Backbone learning-rate factor (after unfreeze)	$1.0$ ( i.e. , same as head)
Weight decay	$0.05$
LP-FT: backbone freeze epochs	$5$
LR warmup epochs	$10$

Appendix

Table 9: Finetuning hyperparameters (locked across all backbones and datasets).

Method	Default	Worst	Best (target-selected)
Supervised	$0.786\pm 0.023$	$0.771\pm 0.033$	$0.786\pm 0.023$
MPMv1 ( Golling et al., 2024 )	$0.800\pm 0.019$	$0.790\pm 0.018$	$0.800\pm 0.019$
MPMv2 ( Leigh et al., 2025 )	$0.760\pm 0.016$	$0.760\pm 0.016$	$0.810\pm 0.017$
RINO	$\mathbf{0.847\pm 0.003}$	$\mathbf{0.843\pm 0.005}$	$\mathbf{0.852\pm 0.003}$

Appendix

Table 10: Sensitivity of OOD top-tagging accuracy (

\mu\pm\sigma

over

10

finetuning seeds; JetNet

\to

JetClass , MLP head,

100\%

of JetNet labels) to the finetuning weight decay and dropout. Default is the Table 1 value (weight decay

0.05

, dropout

0.1

); Worst and Best are the lowest and highest mean over the five settings of App. A.7 . Best is selected on JetClass accuracy and is a retrospective upper bound. Bold marks the best per column.

Configuration	$k$ -NN	LP	OOD ACC
DINO only, mixed views, no PE	$0.859$	$0.923$	$0.677\pm 0.016$
+ iBOT	$0.897$	$0.918$	$0.776\pm 0.018$ $\uparrow 14.6\%$
+ $k_{\mathrm{T}}$ multi-scale views, $p_{\mathrm{T}}$ -rank PE	$0.905$	$0.916$	$0.828\pm 0.004$ $\uparrow 6.7\%$
+ Polar-binned PE	$0.919$	$0.919$	$0.839\pm 0.005$ $\uparrow 1.3\%$
+ Teacher HP tuning (production)	$0.908$	$0.916$	$0.847\pm 0.003$ $\uparrow 1.0\%$

Appendix

Table 11: Progressive construction of the RINO recipe. Each row builds on the previous row.

Axis	Configuration	$k$ -NN	LP	OOD ACC
PE choice $\mid$ mixed views, DINO+iBOT
	No PE	$0.897$	$0.918$	$0.776\pm 0.018$
	$p_{\mathrm{T}}$ -rank	$0.902$	$0.919$	$0.777\pm 0.028$ $\uparrow 0.1$
	Polar-binned	$0.909$	$0.922$	$0.730\pm 0.009$ $\downarrow 5.9$
PE choice $\mid$ $k_{\mathrm{T}}$ multi-scale views, DINO+iBOT
	$p_{\mathrm{T}}$ -rank	$0.905$	$0.916$	$0.828\pm 0.004$

Appendix

Table 12: Independent comparisons of design choices. Each block varies one axis while holding others fixed. Arrows give the relative change (%) with respect to the first row of each block.

Figure 5: Per-jet mean cosine similarity across $k_{\mathrm{T}}$ clustering scales. Left to right, top to bottom: MPMv2 shows monotonic decay (input-similarity pattern); JetCLR-scale converges to near-invariance; RINO backbone retains scale-dependent geometry organized along the student / teacher role assignment; RINO backbone + projection head realizes cross-scale invariance in the $2048$ -dimensional head output, including the unclustered view.

Figure 6: [CLS] cosine similarity across $k_{\mathrm{T}}$ scales for the standard (left) and swapped (right) teacher-student assignments. Red = teacher scales, blue = student scales. The grouping of $N\!=\!4$ changes with its role: as a student-only scale (left) it is closest to $N\!=\!3$ (cosine $0.85$ , vs. $0.65$ to $N\!=\!6$ ); as a teacher scale (right) it is closest to $N\!=\!6$ (cosine $0.87$ , vs. $0.51$ to $N\!=\!3$ ).

SSL baselines
Method	$t\to bqq$	$H\to 4q$	$H\to b\bar{b}$
Supervised	$0.786\pm 0.023$	$0.699\pm 0.038$	$0.623\pm 0.028$
JetCLR	$0.700\pm 0.031$	$0.696\pm 0.028$	$0.686\pm 0.015$
MPMv1	$0.800\pm 0.019$	$0.769\pm 0.006$	$0.674\pm 0.017$
MPMv2	$0.760\pm 0.016$	$0.780\pm 0.025$	$0.692\pm 0.029$
OmniJet- $\alpha$	$0.751\pm 0.050$	$0.684\pm 0.028$	$0.600\pm 0.017$

Appendix

Table 13: Cross-topology transfer ACC ( JetNet

\to

JetClass , MLP head).

SSL baselines
Method	MLP head	Linear head
Supervised	$0.929\pm 0.001$	$0.929\pm 0.001$
JetCLR ( Dillon et al., 2022 )	$0.921\pm 0.001$	$0.921\pm 0.001$
MPMv1 ( Golling et al., 2024 )	$0.929\pm 0.002$	$0.927\pm 0.001$
MPMv2 ( Leigh et al., 2025 )	$0.932\pm 0.001$	$\underline{0.933\pm 0.001}$
OmniJet- $\alpha$ ( Birk et al., 2024 )	$0.760\pm 0.009$	$0.756\pm 0.008$

Appendix

Table 14: Top Quark Tagging

\to

JetClass transfer: top-tagging accuracy (

\mu\pm\sigma

over

6

seeds) on the JetClass test set of models finetuned on Top Quark Tagging . These are transfer results, not comparable to in-distribution results on the standard Top Quark Tagging test set. The leading methods saturate near

0.93

. Bold marks the best per column; underline marks entries within

1\sigma

of the best.

Method	$t\to bqq$	$H\to 4q$
MPMv2	$\mathbf{0.982\pm 0.001}$	$\mathbf{0.900\pm 0.005}$
RINO	$0.981\pm 0.000$	$0.896\pm 0.002$
MPMv1	$0.980\pm 0.000$	$0.889\pm 0.001$
JetCLR	$0.976\pm 0.001$	$0.877\pm 0.005$

Appendix

Table 15: Top Quark Tagging

\to

JetClass cross-topology AUC (linear head, 6 seeds). With the smaller domain gap, the four methods reach similar AUC.

SSL baselines
Method	$[0,0.3)$	$[0.3,0.5)$	$[0.5,0.7)$	$[0.7,0.9)$	$[0.9,0.95)$	$[0.95,1]$	$\langle\|\mathrm{SF}-1\|\rangle\downarrow$
Supervised	$1.120\pm 0.053$	$0.947\pm 0.022$	$0.933\pm 0.023$	$0.960\pm 0.041$	$0.958\pm 0.073$	$\mathbf{0.972\pm 0.128}$	$0.059$
JetCLR	$1.134\pm 0.032$	$0.942\pm 0.012$	$0.903\pm 0.020$	$0.898\pm 0.023$	$0.964\pm 0.029$	$1.256\pm 0.084$	$0.114$
MPMv1	$1.162\pm 0.047$	$\mathbf{0.966\pm 0.013}$	$0.892\pm 0.014$	$0.884\pm 0.030$	$0.947\pm 0.043$	$1.117\pm 0.083$	$0.098$
MPMv2	$0.928\pm 0.024$	$0.867\pm 0.021$	$0.898\pm 0.008$	$1.019\pm 0.022$	$1.278\pm 0.048$	$1.999\pm 0.112$	$0.267$
OmniJet- $\alpha$	$1.073\pm 0.029$	$0.938\pm 0.020$	$0.949\pm 0.031$	$0.975\pm 0.026$	$\mathbf{1.001\pm 0.046}$	$1.110\pm 0.159$	$0.054$

Appendix

Table 16: Top Quark Tagging

\to

JetClass scale factors (Eq. 8 ) per interval of the source signal-score CDF (the last containing the highest-scoring

5\%

of jets) and their mean absolute departure from unity (last column). Entries are mean

\pm

standard deviation over finetuning runs; bold marks the value closest to unity per bin.

Panel of Figure 7 — Figure 7: Intended deployment of RINO. Labels enter only in stage 2, and stage 3 is the standard calibration used for supervised taggers. The bottom row lists the simulated stand-ins used in this work, where JetClass top jets and all JetClass labels enter only in stage 3, for evaluation.

Figure 8: Full data-efficiency curves (all seven methods) with

\pm 1\sigma

bands over

10

seeds.

Method	MLP	Linear	$t$
RINO	$\mathbf{0.847\pm 0.003}$	$\mathbf{0.838\pm 0.009}$	$\mathbf{+3.0}$
Supervised	$0.786\pm 0.023$	$0.784\pm 0.033$	$+0.2$
MPMv1	$0.800\pm 0.019$	$0.809\pm 0.008$	$-1.4$
MPMv2	$0.760\pm 0.016$	$0.817\pm 0.027$	$-5.7$
JetCLR	$0.700\pm 0.031$	$0.791\pm 0.048$	$-5.0$
JetCLR-scale	$0.780\pm 0.023$	$0.803\pm 0.017$	$-2.5$

Appendix

Table 17: MLP vs. linear head on OOD top tagging ( JetNet

\to

JetClass ).

Method	$W_{1}$ (QCD) $\downarrow$	$W_{1}$ (top) $\downarrow$	MMD 2 (QCD) $\downarrow$	MMD 2 (top) $\downarrow$
JetCLR	$0.128$ $+.004$	$0.125$ $+.002$	$0.223$ $-.027$	$0.212$ $-.021$
OmniJet- $\alpha$	$0.181$ $+.634$	$0.246$ $+.506$	$0.095$ $+.476$	$0.193$ $+.359$
MPMv1	$0.232$ $+.009$	$0.274$ $-.012$	$0.120$ $+.013$	$0.183$ $-.021$
MPMv2	$0.409$ $-.034$	$0.374$ $-.026$	$0.427$ $-.089$	$0.409$ $-.104$
JetCLR-scale	$0.092$ $+.041$	$0.071$ $+.074$	$0.017$ $+.028$	$\mathbf{0.009}$ $+.039$
RINO	$\mathbf{0.055}$ $-.012$	$\mathbf{0.052}$ $-.008$	$\mathbf{0.012}$ $-.005$	$0.011$ $-.004$

Appendix

Table 18: Domain shift in pretrained representations ( JetNet vs. JetClass ), before finetuning. Colored values give the change after finetuning (Table 4 ):

-

(green) = finetuning reduces the gap,

+

(red) = finetuning increases it.

Method	Backbone	Attention	Input	Loss
RINO	JTE enc.	Bidir.	Continuous	DINO+iBOT
Supervised	JTE enc.	Bidir.	Continuous	Cross-ent.
MPMv1	JTE enc.	Bidir.	Continuous	Masked VQ pred.
MPMv2	JTE enc.	Bidir.	Continuous	Masked K-means + L1 + flow
JetCLR	JTE enc.	Bidir.	Continuous	NT-Xent
JetCLR-scale	JTE enc.	Bidir.	Continuous	NT-Xent+MAE

Appendix

Table 19: Baseline implementation summary. All methods share the same 7 kinematic input features (Table 6 ), the same QCD pretraining data, and the same finetuning recipe (App. A.6 ).

Method	Time/epoch (min)	Total (h)
JetCLR	${\sim}10$	${\sim}20$
JetCLR-scale	${\sim}25$	${\sim}45$
MPMv1 †	${\sim}30$	${\sim}50$
MPMv2	${\sim}40$	${\sim}65$
OmniJet- $\alpha$ †	${\sim}20$	${\sim}30$
iBOT-only	${\sim}50$	${\sim}80$

Appendix

Table 20: Per-method pretraining wall-clock cost (QCD-only JetClass subset, estimated for

100

epochs).

\dagger

Requires a shared VQ-VAE tokenizer (

{\sim}8

h additional, trained once).

Explore similar work

Jun 12, 2026

JetParticle-JEPA: An Efficient Self-Supervised Representation Learning method for Jet Tagging in High-Energy Physics

Jet tagging at the Large Hadron Collider increasingly relies on deep learning models trained on massive simulated datasets, leading to high computational costs and limited robustness to detector mismodeling. We introduce JetParticle-JEPA (JP-JEPA), a self-supervised Joint-Embedding Predictive Architecture that learns physically meaningful jet representations directly from continuous particle clouds without tokenization or reconstruction of raw inputs. Built on a Particle Transformer backbone, JP-JEPA predicts latent representations of masked particles while preserving fine-grained kinematic correlations. On the JetClass benchmark, JP-JEPA achieves performance comparable to fully supervised state-of-the-art methods on the full dataset, surpasses supervised baselines in low-label regimes, and significantly outperforms existing SSL approaches. On Top Quark and Quark-Gluon Tagging benchmarks, it remains on par with supervised methods. The learned representations also exhibit strong robustness to missing detector information and improved uncertainty behavior, highlighting JP-JEPA as a promising foundation-model framework for robust and data-efficient jet physics at the LHC.

Guillaume Letellier, Antonin Vacheret, Frédéric Jurie

LPCC · GREYC, Normandy University, Unicaen, ENSICAEN, UMR CNRS 6072, F-14000 Caen, France · LPC, Normandy University, Unicaen, ENSICAEN, IN2P3, UMR CNRS 6534, F-14000 Caen, France

Self-Supervised Learning Joint-Embedding Predictive Architecture Jet Tagging High-Energy Physics Data-Efficient Learning

Ask PDF

Nov 25, 2025

Pre-train to Gain: Robust Learning Without Clean Labels

Training deep networks with noisy labels leads to poor generalization and degraded accuracy due to overfitting to label noise. Existing approaches for learning with noisy labels often rely on the availability of a clean subset of data. By pre-training a feature extractor on the target dataset without labels using in-domain self-supervised learning (SSL), followed by standard supervised training on the same noisy dataset, we can train a more noise robust model without requiring a subset with clean labels. We evaluate both contrastive and non-contrastive SSL pre-training methods across datasets with synthetic and real-world label noise, demonstrating the broad applicability of our approach across large-scale datasets, diverse downstream tasks, and model architectures. Across all noise rates, in-domain self-supervised pre-training consistently improves classification accuracy and downstream label-error detection (F1 and Balanced Accuracy) compared with supervised training from scratch. The performance gap widens as the noise rate increases, demonstrating improved robustness. Notably, our approach achieves comparable results to ImageNet and DinoV2 pre-trained models at low noise levels, while substantially outperforming them under high noise conditions.

David Szczecina, Nicholas Pellegrino, Paul Fieguth

Systems Design Engineering University of Waterloo Waterloo, Ontario, Canada

Self-Supervised Pre-Training Learning with Noisy Labels

Ask PDF

Jun 19, 2026

Radial Basis Function Networks as Projection Heads in Self-Supervised Learning

Self-supervised learning (SSL) typically relies on a backbone encoder followed by a small multilayer perceptron (MLP) projection head, which is conventionally discarded after training, while backbone quality is assessed via costly linear probing on labeled data. We argue that this approach including discarding the projector is rather computationally wasteful. Instead, we propose replacing the MLP head with a radial basis function network (RBFN), whose interpretable center and shape parameters can be exploited to judge representation quality without labels or a separate classifier. To this end, we introduce Scale-Normalized Separation (SNS), a novel label-free quality metric derived solely from the kernel centers and shapes learned during training. Across five canonical SSL architectures (MoCo, SimCLR, BYOL, SwAV and SimSiam) and four image classification datasets, we show that RBFN projection heads are competitive drop-in replacements for standard MLP projectors. We recommend constructing them with three RBF layers activated by the Gaussian radial basis function. Moreover, SNS exhibits strong to very strong positive correlation with established logistic regression metrics, demonstrating that a trained RBFN projector can act as a reliable proxy for backbone representation quality. We additionally publish a novel PyTorch compatible image classification dataset based on Google's Open Images V7 to facilitate reproducible research into representation learning.

Andreas Schliebitz, Heiko Tapken, Martin Atzmueller

Osnabrück University of Applied Sciences, Albrechtstr. 30, 49076 Osnabrück, Germany · Osnabrück University, Semantic Information Systems Group,2026 Wachsbleiche 27, 49090 Osnabrück Germany · German Research Center for Artificial Intelligence (DFKI), Hamburger Str. 24, 49084 Osnabrück, Germany

Representation Learning Self-Supervised Visual Representation Learning Radial Basis Function Networks

Ask PDF