Shared Gaussianization: What Gaussian Regularizers Certify About Contrastive Learning, and What They Miss

Ruoyu Zhao; Yuting Chen; Jinheng Zhang; Zhehao Zou; Tong Che

cs.LGOct 7, 2026

Shared Gaussianization: What Gaussian Regularizers Certify About Contrastive Learning, and What They Miss

Authors: Ruoyu Zhao, Yuting Chen, Jinheng Zhang, Zhehao Zou, Tong Che

Organizations: City University of Hong Kong · Georgia Institute of Technology · University of Pennsylvania · The Chinese University of Hong Kong · NVIDIA Research

Abstract

What can a distribution-matching regularizer such as SIGReg in LeJEPA certify about contrastive learning? We study shared Gaussianization (SG), a characteristic-function Gaussianity test on the average of two normalized views, scaled by an independent $χ_d$ radius. Because disagreeing views shorten the average, one test detects both misalignment and non-uniformity. SG vanishes exactly at the aligned, uniform minimizers of population InfoNCE, and under equal marginals it bounds the InfoNCE excess by $4\cdot 3^{3/4}β$ times the square root of the SG loss, plus a term linear in the loss. The square-root rate and this dimension-free constant are sharp, and no squared mean-embedding distance on view pairs achieves a faster rate. With an explicit alignment term, a rotation-invariant uniformity test gives a linear bound if and only if its spectrum dominates that of InfoNCE's kernel $e^{βu^\top v}$ ; SG's own test does, Gaussian kernels $e^{-γ\|u-v\|^2}$ qualify exactly when $γ\ge β/2$ , and moment matching never does. Away from the optimum, the objectives differ. Along an isotropic nuisance channel, pure SG lowers its loss by adding per-view nuisance whenever the shared code is non-uniform. An alignment weight above the channel's gain makes the nuisance-free solution a strict local minimizer; for LeJEPA, the same rule gives a critical SIGReg weight that decreases with the batch size. At finite batch size, an off-diagonal U-statistic removes a plug-in bias toward misalignment. In controlled latent-variable models, pure SG retains per-view style, an alignment weight above the measured gain removes it, and for LeJEPA at three batch sizes the measured gain separates the encoders that retain style from those that do not. InfoNCE training also reaches a lower SG $_{0.2}$ loss than SG $_{0.2}$ training from scratch, which points to an optimization gap.

Figures & tables

Figure 1: Theory on controlled families ( Appendix B ). (a) InfoNCE excess relative to the leading term

2\beta k_{0}\sqrt{\mathcal{J}}

of Theorem 1 (

d=16

\beta=2

): misaligned families attain the sharp constant, the mixture path of Proposition 3 stays at

0.77

, and the aligned von Mises–Fisher (vMF) path has excess

\Theta(\mathcal{J})

. (b) The plug-in estimator is minimized away from alignment (dotted), whereas the U-statistic follows

\mathcal{J}

(extremal family,

d=16

; markers: Monte Carlo). (c) Along the isotropic channel, pure SG’s optimal misalignment for residual non-uniformity of degree

\ell

follows

\rho_{\ell}\mathcal{J}_{\rm unif}

( Theorem 3 ,

d=8

;

\rho_{\ell}

as after ( 11 )); InfoNCE’s is zero. (d) With

\mathcal{J}_{\alpha}

, the rescaled misalignment collapses onto

[1-\alpha/\alpha_{c}]_{+}

, where

\alpha_{c}=g

is the channel’s gain.

Figure 2: Condition (U) (

d=8

\beta=5

). (a) Eigenvalues

\mu_{\ell}

of zonal squared MMDs relative to those of

e^{\beta u^{\top}v}

, normalized at

\ell=1

: the ratio is bounded below for SG’s lift and Gaussian kernels with

\gamma\geq\beta/2

, and not for

\gamma=2

or moment matching (

\mu_{\ell}=0

for

\ell\geq 3

). (b) Exact

\mathcal{U}_{\beta}

over the squared MMD of

e^{-\gamma\|u-v\|^{2}}

for a law perturbed in degree

\ell

; bounded in

\ell

if and only if

\gamma\geq\beta/2

( Corollary 1 ).

Figure 3: Gain against price in trained encoders (content–style model,

d=8

; filled: style detected). (a) SG α : style is detected only when the own-channel gain exceeds

\alpha

. (b) LeJEPA, three batch sizes: style is detected exactly when

Bg_{S}

exceeds the price

(1-\lambda)/\lambda

. (c) Warm starts stay close to InfoNCE; the style added by pure SG vanishes once

\alpha

exceeds their gains (shaded; dashed: detection threshold).

from scratch
	vMF	content–style
Method	CKA	CKA	style	$\mathcal{U}_{\beta}$	$\mathcal{J}$	$\mathcal{J}_{0.2}$
InfoNCE	$0.997$	$0.982$	$-0.02$	$0.02$	$0.17$	$1.7$
SG	$0.938$	$0.646$	$0.11$	$0.05$	$0.85$	$20.6$
SG 0.2	$0.993$	$0.828$	$-0.02$	$0.16$	$0.98$	$3.6$
LeJEPA, $\lambda=0.01$	$0.981$	$0.696$	$-0.02$	$0.26$	$1.57$	$6.0$

Table 1: Controlled study (

d=8

\beta=5

; mean over 3 seeds). CKA to InfoNCE encoders of other seeds; style: held-out probe

R^{2}

(detected if

>0.02

);

\mathcal{J}

and

\mathcal{J}_{0.2}=\mathcal{J}+0.2\Delta

, in units of

10^{-3}

, are evaluated on held-out pairs for every row, whatever the training loss. Warm starts switch InfoNCE to SG or SG 0.2 after 4k of 8k steps. Standard deviations: Tables 3 and 3 .

Figure 4: Condition (U) in trained encoders (alignment term

+

uniformity term; content–style model,

d=8

\beta=5

). (a)

\mathcal{U}_{\beta}

of encoders trained with the Gaussian uniformity loss at scale

\gamma

(one point per seed; blue:

\gamma\geq\beta/2

; amber:

\gamma<\beta/2

; gray line: mean over seeds) and mean over 3 seeds for moment matching (dashed) and InfoNCE (dotted). (b) Residual non-uniformity

\|\hat{q}_{\ell}\|^{2}

per harmonic degree, relative to InfoNCE’s (mean over seeds; dotted: equal to InfoNCE).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Method	$\cos$	$\mathcal{J}$	$\mathcal{U}_{\beta}$	excess	$R^{2}_{c}$	CKA	style
InfoNCE	$0.985\pm 0.001$	$0.17\pm 0.01$	$0.02\pm{<}0.005$	$0.10\pm{<}0.005$	$0.77\pm 0.01$	$0.98\pm{<}0.005$	$-0.02\pm 0.01$
SG, $\hat{\mathcal{J}}_{U}$	$0.802\pm 0.006$	$0.85\pm 0.02$	$0.05\pm 0.01$	$1.03\pm 0.02$	$0.57\pm 0.04$	$0.65\pm 0.03$	$0.11\pm 0.01$
InfoNCE $\to$ SG	$0.961\pm 0.002$	$0.12\pm 0.01$	$0.02\pm{<}0.005$	$0.21\pm 0.01$	$0.75\pm 0.01$	$0.96\pm{<}0.005$	$0.14\pm 0.01$
InfoNCE $\to$ SG 0.2	$0.981\pm 0.001$	$0.27\pm 0.01$	$0.04\pm{<}0.005$	$0.13\pm{<}0.005$	$0.76\pm 0.01$	$0.98\pm{<}0.005$	$-0.02\pm 0.01$
SG 0.2	$0.973\pm 0.001$	$0.98\pm 0.04$	$0.16\pm{<}0.005$	$0.29\pm 0.01$	$0.67\pm{<}0.005$	$0.83\pm 0.01$	$-0.02\pm{<}0.005$
SG 0.05	$0.934\pm 0.002$	$0.79\pm 0.04$	$0.12\pm{<}0.005$	$0.45\pm 0.01$	$0.63\pm{<}0.005$	$0.77\pm 0.01$	$-0.02\pm{<}0.005$

Appendix

Table 2: Content–style model, all metrics (mean

\pm

s.d. over 3 seeds; Table 1 reports a subset, together with the vMF CKA of Table 3 ). Here

\cos

is the mean pair cosine

\mathbb{E}\,U^{\top}V

;

\mathcal{J}

in units of

10^{-3}

; excess is

\mathcal{L}_{\beta}-\mathcal{L}^{\star}_{\beta}=2\beta\Delta+\mathcal{U}_{\beta}

\beta=5

;

R^{2}_{c}

is the held-out linear decodability of the content; CKA and style as in Table 1 .

\hat{\mathcal{J}}_{U}

is the unbiased estimator; InfoNCE

\to

SG and InfoNCE

\to

SG 0.2 switch from InfoNCE halfway through training;

\lambda

is LeJEPA’s SIGReg weight.

Explore similar work

Jul 10, 2026cs.LG

Similarity search generalisation in contrastive learning with InfoNCE loss

Similarity search is a primary application of embedding models trained by contrastive learning. For one of the most popular contrastive learning loss functions, InfoNCE, we show that the population risk with

k

negative samples is

O(1/k)

close to an expected cross-entropy which quantifies deviation between i) a softmax similarity search over unseen data using the learned embedding function, and ii) an idealised softmax search over the same data but using similarity implicitly represented in the positive sample generator. This complements existing interpretations of InfoNCE in the

k\to\infty

limit which are phrased in terms of mutual information, and alignment versus uniformity in embeddings. To quantify generalisation performance, we introduce a new continuity bound for the InfoNCE loss, obtained via Gâteaux differentiation. The bound preserves the structure of averaging over negative samples present in the loss function and features an ``inverse temperature'' parameter which can be tuned to account for the algorithmic temperature. For embedding functions which are Lipschitz in a parameter, this yields a simple demonstration that the averaging effect of

k

negative samples in the InfoNCE loss carries over to stabilisation of the generalisation error as

k

grows.

Nick Whiteley

School of Mathematics, University of Bristol, U.K.

Contrastive Learning Text Embeddings Cross-Entropy Losses Loss Function Similarity and Metrics

Ask PDF

May 31, 2026cs.LG

UR-JEPA: Uniform Rectifiability as a Regularizer for Joint-Embedding Predictive Architectures

A central difficulty in training Joint-Embedding Predictive Architectures (JEPAs) is preventing representation collapse. LeJEPA addresses this by enforcing an isotropic Gaussian target on the embeddings via Sketched Isotropic Gaussian Regularization (SIGReg). This target is in tension with the manifold hypothesis, which expects embeddings to concentrate on a low-dimensional subset of the ambient space. We propose \emph{UR-JEPA}, which targets a uniformly

n

-rectifiable measure of local tangent dimension

n

at small scales, realized through a Gaussian-kernel smoothed Carleson-type square function

\mathcal{L}^{\text{CGLT}}

, with a complementary Jones

β

-number formulation. On Inet10, UR-JEPA(

\mathcal{L}^{\text{CGLT}}

) attains

0.9141 \pm 0.0014

for a

+0.83

,pp gain over LeJEPA(

\mathcal{L}^{\text{SIGReg}}

) with

\sim 30\%

lower seed standard deviation; on matched-recipe Galaxy10~SDSS, a single-seed ImageNet-

100

run, and a

3

-seed EuroSAT remote-sensing run, the two methods lie in the same peak-accuracy band at convergence, with UR-JEPA retaining its lower-seed-variance signature. On EuroSAT the in-domain pair is competitive at

96.0

96.1\%

with large remote-sensing foundation-model transfer at a

25\times

smaller backbone. The distinction is geometric: direct visualization of the projector output distribution shows that on all four datasets UR--JEPA(

\mathcal{L}^{\text{CGLT}}

) produces a global PCA spectrum with a

4

5

order-of-magnitude drop at index

\sim 20

25

out of

D = 32

, while LeJEPA's spectrum is near-flat (top-to-bottom ratio at most

3.6

). Per-dimension marginals are simultaneously near-Gaussian for both methods (mean Shapiro-Wilk

W \in [0.992, 0.996]

) as a Diaconis-Freedman consequence. At matched accuracy the two regularizers therefore yield structurally distinct projected representations.

Triet M. Le

Joint-Embedding Predictive Architectures S-Jepa Regularization Regularity Gaussian Mixture Models Principal Component Analysis

Ask PDF

Jan 27, 2026cs.LG

The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence

While InfoNCE underlies modern contrastive learning, its geometric mechanisms remain under-characterized beyond the canonical alignment--uniformity decomposition. We develop a measure-theoretic framework in which representation measures evolve on a fixed embedding manifold. In the large-batch limit, we prove value and gradient consistency, linking the stochastic objective to explicit deterministic energy landscapes and revealing a geometric bifurcation between unimodal and symmetric multimodal regimes. In the unimodal case, the intrinsic energy is strictly convex and admits a unique Gibbs equilibrium, showing that entropy acts as a tie-breaker within the aligned basin. In the multimodal case, the intrinsic geometry becomes cross-coupled and contains a persistent negative symmetric divergence term: each modality's marginal reshapes the effective landscape of the other, allowing strong pairwise alignment to coexist with a persistent modality gap. Controlled synthetic experiments and analyses of pretrained CLIP representations support these predictions. Overall, our results shift the analytical lens from pointwise discrimination to population geometry, showing that pairwise alignment alone is insufficient to control cross-modal marginal structure.

Yichao Cai, Zhen Zhang, Yuhang Liu +1

Australian Institute for Machine Learning, Adelaide University · Responsible AI Research Centre

Multimodal Contrastive Learning Contrastive Learning Cross-Modal Free-Energy Landscape Divergence Populations

Ask PDF