KerJEPA: Kernel Discrepancies for Euclidean Self-Supervised Learning

Eric Zimmermann; Harley Wiltzer; Justin Szeto; David Alvarez-Melis; Lester Mackey

cs.LGDec 22, 2025

KerJEPA: Kernel Discrepancies for Euclidean Self-Supervised Learning

Authors: Eric Zimmermann, Harley Wiltzer, Justin Szeto, David Alvarez-Melis, Lester Mackey

Organizations: Microsoft Research, Cambridge, MA United States · Mila–Québec AI Institute, Montreal, Canada · McGill University, Montreal, Canada

Abstract

Breakthroughs in self-supervised Joint-Embedding Predictive Architectures (JEPAs) have established that regularizing Euclidean representations toward isotropic Gaussian priors yields provable gains in training stability and downstream generalization. We introduce a new, flexible family of Euclidean joint embedding self-supervised learning algorithms with kernel-based regularizers. One instance of this family corresponds to the SIGReg Epps-Pulley regularizer which approximates a sliced Maximum Mean Discrepancy (MMD) with a Gaussian prior and Gaussian kernel. By expanding the class of viable kernels and priors and computing the closed-form high-dimensional limit of sliced discrepancies, we develop alternative methods with improved training stability and design flexibility and leverage those techniques to train strong vision learners at scale.

Figures & tables

Figure 1: Impacts of slicing in various dimensions for the MMD and KSD regularizer on ImageNette, measuring test accuracy over various output dimensions. Insufficient slices slow convergence rates as a function of dimension and introduces instability in the early half of training.

Discrepancy	Type	Base Kernel	Prior	Slices	Linear
MMD	Unsliced	Gaussian	Gaussian	–	91.29
	Sliced (Analytic)	Gaussian	Gaussian	$\infty$	91.13
	Sliced (Finite)	Gaussian	Gaussian	1024	91.13
	Sliced (Finite)	Gaussian	Laplace	1024	90.25
KSD	Unsliced	Gaussian	Gaussian	–	91.31
	Unsliced	Gaussian	Laplace	–	91.18

Table 1: ImageNette regularization ablations. Top-1 accuracy (%) across different settings on models trained for 800 epochs with dimension of 128. Unsliced heavy-tail kernels perform best.

Backbone	Method	k-NN	Linear
ViT-S/16	DINO	74.45	76.17
	iBOT	75.15	77.08
	KerJEPA $\mathtt{CWMMD}$	76.09	77.85
	KerJEPA $\mathtt{CWKSD}$	75.40	77.65
ViT-B/16	DINO	76.10	77.99
	iBOT	77.02	79.37

Table 2: ImageNet-1k classification. Top-1 accuracy (%) on frozen

{\mathtt{[CLS]}}

features. KerJEPA out-competes baselines and shows favorable scaling across model sizes.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Distribution	Notation	$p(x)$	$\nabla_{x}\log p(x)$	$\phi(\omega)$
Gaussian	$\mathcal{N}(0,\sigma^{2}I_{d})$	$\displaystyle\frac{1}{(2\pi\sigma^{2})^{d/2}}\exp\left(-\frac{1}{2\sigma^{2}}\\|x\\|^{2}_{2}\right)$	$\displaystyle-\frac{1}{\sigma^{2}}x$	$\displaystyle\exp\left(-\frac{1}{2}\sigma^{2}\\|\omega\\|^{2}_{2}\right)$
Laplace	$\textrm{Lap}(0,\sigma I_{d})$	$\displaystyle\frac{\Gamma(d/2)}{2\pi^{d/2}\sigma^{d}\Gamma(d)}\exp\left(-\frac{1}{\sigma}\\|x\\|_{2}\right)$	$\displaystyle-\frac{1}{\sigma}\frac{x}{\\|x\\|_{2}}$	$\displaystyle\left(1+\sigma^{2}\\|\omega\\|^{2}_{2}\right)^{-\frac{d+1}{2}}$
Student-t	$\mathcal{T}_{\nu}(0,\sigma^{2}I_{d})$	$\displaystyle\frac{\Gamma(\frac{\nu+d}{2})}{\Gamma(\frac{\nu}{2})(\nu\pi\sigma^{2})^{d/2}}\left(1+\frac{1}{\nu\sigma^{2}}\\|x\\|^{2}_{2}\right)^{-\frac{\nu+d}{2}}$	$\displaystyle-\frac{\nu+d}{\nu\sigma^{2}+\\|x\\|^{2}_{2}}x$	$\displaystyle\frac{K_{\nu/2}(\sqrt{\nu}\sigma\\|\omega\\|_{2})(\sqrt{\nu}\sigma\\|\omega\\|_{2})^{\nu/2}}{\Gamma(\nu/2)2^{\nu/2-1}}$

Appendix

Table 3: Isotropic prior distributions: probability density

p(x)

, scores

\nabla_{x}\log p(x)

, and their characteristic function

\phi(\omega)

used to construct different discrepancy objectives.

$k(x,y)$	$\nabla_{x}k(x,y)$	$\operatorname{tr}(\nabla_{x}\nabla_{y}k(x,y))$
Gaussian	$\displaystyle-2\gamma(x-y)e^{-\gamma\\|x-y\\|_{2}^{2}}$	$\displaystyle(2\gamma d-4\gamma^{2}\\|x-y\\|_{2}^{2})e^{-\gamma\\|x-y\\|_{2}^{2}}$
IMQ	$\displaystyle-2\alpha\beta(x-y)(1+\alpha\\|x-y\\|_{2}^{2})^{-(\beta+1)}$	$\displaystyle 4\alpha^{2}\beta(\beta+1)\\|x-y\\|_{2}^{2}(1+\alpha\\|x-y\\|_{2}^{2})^{-(\beta+2)}$
Bochner	$\displaystyle\int_{\mathbb{R}^{d}}\mathrm{i}\omega\,e^{\mathrm{i}\omega^{\top}(x-y)}\rho(\omega)\,d\omega$	$\displaystyle\int_{\mathbb{R}^{d}}\\|\omega\\|_{2}^{2}e^{\mathrm{i}\omega^{\top}(x-y)}\rho(\omega)\,d\omega$

Appendix

Table 4: Gradients and Hessians for the Gaussian, IMQ, and general shift-invariant kernels..

Figure 2: Pseudo-code for the finite-sliced discrepancy regularization losses for both Gaussian and Laplace priors using the Gaussian kernel. Integration is done as Gauss-Hermite quadrature approximation. Other choice of kernel may require additional quadrature estimation

Backbone	Method	mIoU	mAcc
ViT-S/16	iBOT	33.01	42.34
	DINO	30.29	39.05
	KerJEPA $\mathtt{CWMMD}$	30.67	39.05
	KerJEPA $\mathtt{CWKSD}$	30.54	38.76
ViT-B/16	iBOT	38.08	47.41
	DINO	34.77	43.93

Appendix

Table 5: Semantic segmentation on ADE20K. mIoU/mAcc on frozen patch features. iBOT shows strongest performance but is trained with masked objectives.

Panel of Figure 3 — Figure 3: DAVIS layer ablation across model sizes. Despite KerJEPA having weaker spatial features, they show favorable scale, with top performance even surpassing iBOT at intermediate layers without direct patch supervision.

Joint-Embedding Predictive Architecture

Joint-Embedding Predictive Architecture (JEPA) learns representations by predicting latent features of hidden or future data from observed context.

See all 240 papers

UR-JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures
May 31, 2026Triet M. LeJoint-Embedding Predictive Architecture Geometric Representation Learning
LocaLeJEPA: Maximum Entropy Representations under Geometric Priors
Oct 20, 2026Kartik Tandon, Alessio Borgi, Pavel Snopov +8Representation Learning Joint-Embedding Predictive Architecture
University of Pennsylvania · Sapienza University of Rome · University of Cambridge +6
$λ$ -JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learning
Sep 28, 2026Berker Demirel, Clémentine Dominé, Valentino Maiorca +3Spectral Regularization Joint-Embedding Predictive Architecture
Institute of Science and Technology Austria (ISTA) Am Campus 1, 3400 Klosterneuburg, Austria

May 31, 2026J/K move · Enter open · S save

UR-JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures

Triet M. Le

Joint-Embedding Predictive Architecture Geometric Representation Learning Self-Supervised Visual Representation Learning Latent-Space Regularization

Open paper AskPDF arXiv

A central difficulty in training Joint-Embedding Predictive Architectures (JEPAs) is preventing representation collapse. LeJEPA addresses this by enforcing an isotropic Gaussian target on the embeddings via Sketched Isotropic Gaussian Regularization (SIGReg). This target is in tension with the manifold hypothesis, which expects embeddings to concentrate on a low-dimensional subset of the ambient space. We propose UR-JEPA, which targets a uniformly

n

-rectifiable measure of local tangent dimension

n

at small scales, realized through a Gaussian-kernel smoothed Carleson-type square function

\mathcal{L}^{\text{CGLT}}

, with a complementary Jones

β

-number formulation. On Inet10, UR-JEPA(

\mathcal{L}^{\text{CGLT}}

) attains

0.9141 \pm 0.0014

for a

+0.83

,pp gain over LeJEPA(

\mathcal{L}^{\text{SIGReg}}

) with

\sim 30\%

lower seed standard deviation; on matched-recipe Galaxy10~SDSS, a single-seed ImageNet-

100

run, and a

3

-seed EuroSAT remote-sensing run, the two methods lie in the same peak-accuracy band at convergence, with UR-JEPA retaining its lower-seed-variance signature. On EuroSAT the in-domain pair is competitive at

96.0

96.1\%

with large remote-sensing foundation-model transfer at a

25\times

smaller backbone. The distinction is geometric: direct visualization of the projector output distribution shows that on all four datasets UR--JEPA(

\mathcal{L}^{\text{CGLT}}

) produces a global PCA spectrum with a

4

5

order-of-magnitude drop at index

\sim 20

25

out of

D = 32

, while LeJEPA's spectrum is near-flat (top-to-bottom ratio at most

3.6

). Per-dimension marginals are simultaneously near-Gaussian for both methods (mean Shapiro-Wilk

W \in [0.992, 0.996]

) as a Diaconis-Freedman consequence. At matched accuracy the two regularizers therefore yield structurally distinct projected representations.

Abstract

Joint-Embedding Predictive Architecture

Explore similar work

UR-JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures

LocaLeJEPA: Maximum Entropy Representations under Geometric Priors

λλλ-JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learning

$λ$ -JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learning