InfoNCE Induces Gaussian Distribution

Roy Betser; Eyal Gofer; Meir Yossef Levi; Guy Gilboa

cs.LGFeb 27, 2026

InfoNCE Induces Gaussian Distribution

Authors: Roy Betser, Eyal Gofer, Meir Yossef Levi, Guy Gilboa

Organizations: Technion - Israel Institute of Technology

Abstract

Contrastive learning has become a cornerstone of modern representation learning, allowing training with massive unlabeled data for both task-specific and general (foundation) models. A prototypical loss in contrastive training is InfoNCE and its variants. In this work, we show that the InfoNCE objective induces Gaussian structure in representations that emerge from contrastive training. We establish this result in two complementary regimes. First, we show that under certain alignment and concentration assumptions, projections of the high-dimensional representation asymptotically approach a multivariate Gaussian distribution. Next, under less strict assumptions, we show that adding a small asymptotically vanishing regularization term that promotes low feature norm and high feature entropy leads to similar asymptotic results. We support our analysis with experiments on synthetic and CIFAR-10 datasets across multiple encoder architectures and sizes, demonstrating consistent Gaussian behavior. This perspective provides a principled explanation for commonly observed Gaussianity in contrastive representations. The resulting Gaussian model enables principled analytical treatment of learned representations and is expected to support a wide range of applications in contrastive learning.

Figures & tables

Figure 1: Illustration. Contrastive learning yields (approximately) Gaussian representations.

Figure 2: Uniformity vs. alignment across settings. A simple linear encoder trained on synthetic Laplace data exhibits (i) near-optimal alignment across all configurations and (ii) steadily improving uniformity as batch size or dimensionality grow.

Figure 3: Synthetic data experiments. Left: representation norm statistics vs. batch size (curves denote dimension), showing thin-shell concentration with increasing $d$ and $N$ . Top middle/right: norm histograms illustrating radius tightening. Bottom: normality diagnostics (AD, DP), with averages in the Gaussian acceptance range.

Figure 4: CIFAR-10 training dynamics. A two-layer MLP trained with InfoNCE on CIFAR-10 exhibits increasing Gaussianity over training. Left: representation norms concentrate as indicated by declining CV (Eq. 20 ). Middle: the AD statistic decreases from non-Gaussian levels into the normal range. Right: the fraction of coordinates passing the DP normality test rises steadily.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: vMF exponential tilt distribution for different concentration scales kappa ( $\kappa$ ).

Figure 6: Thin-shell concentration across pretrained models. Radius distributions of representations from supervised models (DenseNet, ResNet) and contrastive models (CLIP, DINO). All models exhibit thin-shell concentration, with contrastive methods showing tighter clustering (lower CV, Eq. ( 20 )).

Figure 7: Alignment and uniformity vs. batch size. Histogram view of cosine similarities for positive pairs (alignment) and negatives (uniformity), corresponding to Fig. 2 . As batch size increases, alignment remains high while uniformity improves, with negative-pair similarities concentrating near zero. The middle panel is a zoom of the left; the right panel shows that at very low dimensionality, increasing batch size yields little uniformity gain.

Figure 8: Alignment and uniformity vs. dimensionality. Histogram view of cosine similarities for positive pairs (alignment) and negatives (uniformity), corresponding to Fig. 2 . As dimensionality increases, alignment stays high while uniformity improves, pushing negative-pair similarities toward zero. The middle panel is a zoom of the left; the right panel highlights that with very small batch sizes, increasing dimensionality offers limited uniformity improvement.

Figure 9: Whitening and uniformity: unnormalized representations. Cosine similarity histograms of negatives for CLIP (image, text) and DINO, before (raw) and after whitening. Unnormalized representations benefit from whitening, with distributions pushed closer to zero, reflecting enhanced uniformity. The y-axis (“Density”) represents the relative count in each bin (normalized by the total number of samples) rather than the probability density function.

Figure 10: Whitening and uniformity: normalized representations. Cosine similarity histograms of negatives for CLIP (image, text) and DINO, before (normalized) and after whitening. Normalized representations are already close to uniform; whitening provides a modest but consistent improvement.

Figure 11: Encoder “pushforward”. On synthetic data, the encoder maps Laplace-distributed inputs to approximately Gaussian representations. Because both source and target families admit tractable likelihoods, we can score entire sets and observe consistently high correlation across different augmentation strengths.

Contrastive Learning

Contrastive learning trains representations by bringing related examples together and separating unrelated ones for recognition, retrieval, and transfer.

See all 322 papers

Shared Gaussianization: What Gaussian Regularizers Certify About Contrastive Learning, and What They Miss
Oct 7, 2026Ruoyu Zhao, Yuting Chen, Jinheng Zhang +2Contrastive Learning Multi-View Contrastive Learning
City University of Hong Kong · Georgia Institute of Technology · University of Pennsylvania +2
The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning
Jun 2, 2026Justinas Zaliaduonis, Patrick Putzky, Till Richter +1Contrastive Learning Self-Supervised Learning
Technical University of Munich, Munich, Germany · Stanford University, Stanford, USA · Merantix Momentum GmbH, Berlin, Germany +2
The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence
Jan 27, 2026Yichao Cai, Zhen Zhang, Yuhang Liu +1Cross-Modal Learning Contrastive Learning
Australian Institute for Machine Learning, Adelaide University · Responsible AI Research Centre

Oct 7, 2026J/K move · Enter open · S save

Shared Gaussianization: What Gaussian Regularizers Certify About Contrastive Learning, and What They Miss

Ruoyu Zhao, Yuting Chen, Jinheng Zhang, Zhehao Zou+1

City University of Hong Kong · Georgia Institute of Technology · University of Pennsylvania+2

Contrastive Learning Multi-View Contrastive Learning Self-Supervised Visual Representation Learning Latent Space Alignment

Open paper AskPDF arXiv

What can a distribution-matching regularizer such as SIGReg in LeJEPA certify about contrastive learning? We study shared Gaussianization (SG), a characteristic-function Gaussianity test on the average of two normalized views, scaled by an independent

χ_d

radius. Because disagreeing views shorten the average, one test detects both misalignment and non-uniformity. SG vanishes exactly at the aligned, uniform minimizers of population InfoNCE, and under equal marginals it bounds the InfoNCE excess by

4\cdot 3^{3/4}β

times the square root of the SG loss, plus a term linear in the loss. The square-root rate and this dimension-free constant are sharp, and no squared mean-embedding distance on view pairs achieves a faster rate. With an explicit alignment term, a rotation-invariant uniformity test gives a linear bound if and only if its spectrum dominates that of InfoNCE's kernel

e^{βu^\top v}

; SG's own test does, Gaussian kernels

e^{-γ\|u-v\|^2}

qualify exactly when

γ\ge β/2

, and moment matching never does. Away from the optimum, the objectives differ. Along an isotropic nuisance channel, pure SG lowers its loss by adding per-view nuisance whenever the shared code is non-uniform. An alignment weight above the channel's gain makes the nuisance-free solution a strict local minimizer; for LeJEPA, the same rule gives a critical SIGReg weight that decreases with the batch size. At finite batch size, an off-diagonal U-statistic removes a plug-in bias toward misalignment. In controlled latent-variable models, pure SG retains per-view style, an alignment weight above the measured gain removes it, and for LeJEPA at three batch sizes the measured gain separates the encoders that retain style from those that do not. InfoNCE training also reaches a lower SG

_{0.2}

loss than SG

_{0.2}

training from scratch, which points to an optimization gap.

Abstract

Contrastive Learning

Explore similar work

Shared Gaussianization: What Gaussian Regularizers Certify About Contrastive Learning, and What They Miss

The Loss Is Not Enough: Sampling Conditions and Inductive Bias in Contrastive Representation Learning

The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-modal Divergence