SuperJEPA: Pretraining with a Supervision-Augmented Joint-Embedding Predictive Architecture for High Energy Physics

Dylan Sheldon Rankin; Ho Fung Tsoi

hep-exOct 26, 2026

SuperJEPA: Pretraining with a Supervision-Augmented Joint-Embedding Predictive Architecture for High Energy Physics

Authors: Dylan Sheldon Rankin, Ho Fung Tsoi

Organizations: Department of Physics and Astronomy, University of Pennsylvania, Philadelphia, PA, USA

Abstract

Foundation models often obtain powerful results through very large unlabeled training datasets that dwarf available labeled datasets in size, but high energy physics already provides large samples of high-fidelity labeled simulation. We therefore ask whether self-supervised objectives improve over supervised pretraining alone when large labeled samples already exist. We introduce SuperJEPA, a joint-embedding predictive architec- ture augmented with a supervised objective, and find improved downstream jet-tagging performance across multiple tasks, model sizes, and pretraining dataset sizes. The gains persist for both frozen and fine-tuned encoders, motivating predictive self-supervision in future large-scale particle-physics foundation models.

Figures & tables

Figure 1: Schematic of the training method and terms that contribute to the loss function.

Configuration	$(*)$ $H\to b\bar{b}$ Rej 50%	$H\to c\bar{c}$ Rej 50%	$(*)$ $H\to gg$ Rej 50%	$H\to l\nu qq^{\prime}$ Rej 90%	$H\to 4q$ Rej 50%	$(*)$ $t\to bqq^{\prime}$ Rej 50%	$t\to bl\nu$ Rej 90%	$W\to qq^{\prime}$ Rej 50%	$Z\to q\bar{q}$ Rej 50%	$(*)$ $Z\to\nu\nu$ Rej 50%
No attention pooling
Model M (4M)
Supervised	966(3)	135(1)	106	33(1)	74	3634(44)	227(6)	62	41	377(1)
SuperJEPA ( $\alpha=0.1$ )	393(2)	86(1)	98	83(2)	93	1876(3)	285(7)	62	38	357(1)
( $\alpha=1$ )	1556(5)	231(1)	117	72(1)	94	4285(26)	384(8)	65	43	388
Model M (400k)

Table 1: Results on the JetClass dataset for supervised and SuperJEPA models pretrained only on the 4 classes denoted with

(*)

. Numbers in parentheses next to models represent the size of the JetClass pretraining dataset. Numbers in parentheses next to results represent uncertainty from three classifier training runs. Downstream models are shown both with and without attention pooling.

	JetNet					Top Tagging
	No attention pooling							Attention pooling
Configuration	$q$ Rej 50%	$g$ Rej 50%	$w$ Rej 50%	$z$ Rej 50%	$t$ Rej 50%	Rej 50%	Rej 30%	Rej 50%	Rej 30%
Model M (4M)
Supervised	38	35	117(1)	261(3)	168(1)	344(3)	1256(27)	391(5)	1416(49)
SuperJEPA ( $\alpha=0.1$ )	38	37	120(2)	291(8)	201(2)	326(6)	1239(50)	353(7)	1342(68)
( $\alpha=1$ )	38	39	95(1)	247(2)	197(6)	365(8)	1540(25)	403(9)	1883(10)

Table 2: Results on the JetNet and Top tagging datasets for supervised and SuperJEPA models pretrained only on 4 classes from JetClass. Numbers in parentheses next to models represent the size of the JetClass pretraining dataset. Numbers in parentheses next to results represent uncertainty from three classifier training runs. Downstream models are shown both with and without attention pooling.

Configuration	$H\to b\bar{b}$ Rej 50%	$H\to c\bar{c}$ Rej 50%	$H\to gg$ Rej 50%	$H\to l\nu qq^{\prime}$ Rej 90%	$H\to 4q$ Rej 50%	$t\to bqq^{\prime}$ Rej 50%	$t\to bl\nu$ Rej 90%	$W\to qq^{\prime}$ Rej 50%	$Z\to q\bar{q}$ Rej 50%	$Z\to\nu\nu$ Rej 50%
No attention pooling
Supervised	2360(3)	424(1)	150	1444(3)	235	7772(31)	4749(23)	93	65	449(1)
SuperJEPA ( $\alpha=0.1$ )	1776(2)	311	135	995(2)	195	4398(7)	2709(18)	86	58	423(1)
( $\alpha=1$ )	2530(3)	458(2)	153	1508(3)	242	8461(77)	5079(4)	95	66	451
Attention pooling
Supervised	2349(22)	423(1)	150	1431(11)	233(1)	7643(60)	4697(17)	93	64	448(1)

Table 3: Results on the JetClass dataset for supervised and SuperJEPA models pretrained on 20M total samples from all 10 classes. Numbers in parentheses next to models represent the size of the JetClass pretraining dataset. Numbers in parentheses next to results represent uncertainty from three classifier training runs. Downstream models are shown both with and without attention pooling.

Configuration	AUC	Rej 50%	Rej 30%
Model M (20M, 10 classes)
Supervised	0.9861(1)	474.4(4.3)	1915(29)
SuperJEPA ( $\alpha=0.1$ )	0.9858	408.8(3.5)	1798(16)
( $\alpha=1$ )	0.9862(1)	470.9(5.4)	1980(46)
Model M (4M, 4 classes)
Supervised	0.9857	416.8(5.6)	1524(52)

Table 4: Results after fine-tuning on the top tagging dataset for supervised and SuperJEPA models. Numbers in parentheses next to models represent the size of the JetClass pretraining dataset. Numbers in parentheses next to results represent uncertainty from three finetuning runs.

Panel of Figure 2 — Figure 2: Corner plots of four per-particle embedding features as produced by an encoder trained using only standard supervision (a) and SuperJEPA with $\alpha=1$ (b). The distribution shown in blue represents classes seen during pretraining, while the other colors represent new classes not used in pretraining.

Figure 3: t-SNE plots of the CLS token for different models. Supervised models are shown in the top row, SuperJEPA $(\alpha=0.1)$ models are shown in the middle row, and SuperJEPA $(\alpha=1)$ models are shown in the bottom row. Model and pretraining dataset size increase from left to right. All JetClass classes are shown.

Self-Supervised Pre-Training

Self-supervised pre-training learns transferable representations from unlabeled data to reduce the labels needed for downstream tasks.

See all 250 papers

JetParticle-JEPA: An Efficient Self-Supervised Representation Learning method for Jet Tagging in High-Energy Physics
Jun 12, 2026Guillaume Letellier, Antonin Vacheret, Frédéric JurieSelf-Supervised Learning Joint-Embedding Predictive Architecture
LPCC · GREYC, Normandy University, Unicaen, ENSICAEN, UMR CNRS 6072, F-14000 Caen, France · LPC, Normandy University, Unicaen, ENSICAEN, IN2P3, UMR CNRS 6534, F-14000 Caen, France
Pre-Training for Simulation-Based Science: A Study on Jet Foundation Model Training Objectives
Jun 12, 2026Ibrahim Elsharkawy, Joschka Birk, Vinicius Mikuni +3Self-Supervised Pre-Training High-Energy Physics
Department of Physics, University of Toronto and Vector Institute, Toronto, ON, Canada · NERSC, Lawrence Berkeley National Laboratory, Berkeley, California, USA · Institut für Experimentalphysik, Universität Hamburg, 22761 Hamburg, Germany +3
Similarity Pairing with Energy Mover's Distance for Self-Supervised Pre-Training at the LHC
Sep 15, 2026Ho Fung Tsoi, Dylan RankinRepresentation Learning Self-Supervised Pre-Training
Department of Physics and Astronomy, University of Pennsylvania, USA

Jun 12, 2026J/K move · Enter open · S save

JetParticle-JEPA: An Efficient Self-Supervised Representation Learning method for Jet Tagging in High-Energy Physics

Guillaume Letellier, Antonin Vacheret, Frédéric Jurie

LPCC · GREYC, Normandy University, Unicaen, ENSICAEN, UMR CNRS 6072, F-14000 Caen, France · LPC, Normandy University, Unicaen, ENSICAEN, IN2P3, UMR CNRS 6534, F-14000 Caen, France

Self-Supervised Learning Joint-Embedding Predictive Architecture Jet Tagging High-Energy Physics Data-Efficient Learning

Open paper AskPDF arXiv

Jet tagging at the Large Hadron Collider increasingly relies on deep learning models trained on massive simulated datasets, leading to high computational costs and limited robustness to detector mismodeling. We introduce JetParticle-JEPA (JP-JEPA), a self-supervised Joint-Embedding Predictive Architecture that learns physically meaningful jet representations directly from continuous particle clouds without tokenization or reconstruction of raw inputs. Built on a Particle Transformer backbone, JP-JEPA predicts latent representations of masked particles while preserving fine-grained kinematic correlations. On the JetClass benchmark, JP-JEPA achieves performance comparable to fully supervised state-of-the-art methods on the full dataset, surpasses supervised baselines in low-label regimes, and significantly outperforms existing SSL approaches. On Top Quark and Quark-Gluon Tagging benchmarks, it remains on par with supervised methods. The learned representations also exhibit strong robustness to missing detector information and improved uncertainty behavior, highlighting JP-JEPA as a promising foundation-model framework for robust and data-efficient jet physics at the LHC.

Abstract

Self-Supervised Pre-Training

Explore similar work

JetParticle-JEPA: An Efficient Self-Supervised Representation Learning method for Jet Tagging in High-Energy Physics

Pre-Training for Simulation-Based Science: A Study on Jet Foundation Model Training Objectives

Similarity Pairing with Energy Mover's Distance for Self-Supervised Pre-Training at the LHC