Internal Discrepancy for Contrastive Learning: Generalization Bounds and Downstream Behaviors

Jialiang Wang; Qing Ma; Xiong Zhou; Gangfeng Hu; Haoliang Li; Xianming Liu

cs.LGOct 23, 2026

Internal Discrepancy for Contrastive Learning: Generalization Bounds and Downstream Behaviors

Authors: Jialiang Wang, Qing Ma, Xiong Zhou, Gangfeng Hu, Haoliang Li, Xianming Liu

Organizations: Harbin Institute of Technology, Harbin, China · City University of Hong Kong, Hong Kong, China

Abstract

Contrastive learning has emerged as a popular paradigm for training representation models. While prior studies pursue more practical and less restrictive assumptions to enhance theoretical understanding of contrastive learning, they essentially describe the interconnections among representations within the data or induced by the function class. To encapsulate these assumptions, we propose the measure of internal discrepancy that quantifies the difference between the mean representation of positive samples and latent cluster centers. We then prove an internal discrepancy-aware generalization bound for downstream supervised tasks following contrastive learning, without requiring additional assumptions. This generalization bound not only serves as a unifying framework connecting various assumptions but also motivates us to further reduce the internal discrepancy, thereby introducing the relaxed k-means loss as an approximation. Moreover, we investigate the downstream behaviors of contrastive learning from the perspective of contrastive learning-incurred supervised losses. In the presence of class imbalance, we shed light on why contrastive learning can facilitate imbalanced learning. Finally, we investigate the optimality of these incurred supervised losses, proving several intriguing properties.

Figures & tables

Figure 1: Schematic comparison of large and small internal discrepancy. Colors denote latent classes, while dashed regions highlight local positive pairs. The filled and open circles indicate

f(x)

and

f(x^{+})

. The diamond and star mark

\mathbb{E}_{x^{+}\sim\mathcal{D}_{\mathrm{pos}}(\cdot\mid x)}f(x^{+})

and

\mathbb{E}_{x\sim{\mathcal{D}}^{c}}f(x)

, respectively. The expected distance between the conditional positive mean and the class mean equals

\mathrm{Disc}(f)

. The right panel depicts more compact class and local regions, accompanied by a shorter segment in the illustrated configuration.

Panel of Figure 2 — Figure 2: Illustration of validation accuracies (%) on CIFAR-10, CIFAR-100, and ImageNet-100 using cross entropy, InfoNCE-incurred loss as per Theorem 4 , and multi-class binary cross entropy defined in Equation 31 . The depicted results demonstrate that these three loss functions deliver comparable performance.

Dataset	Method	Sampling repetitions $t$
Dataset	Method	$10$	$20$	$40$	$100$
CIFAR-10	CE	94.79
	$k=1$	94.79	94.93	94.83	94.91
	$k=10$	95.20	94.87	95.05	94.71
	$k=20$	94.81	94.82	94.85	94.65
	$k=40$	94.76	94.73	94.79	94.73

Table 1: Validation accuracies (%) on the CIFAR-10 and CIFAR-100 datasets. These results are obtained through supervised learning using cross entropy (CE) or InfoNCE-incurred loss (with the number of negatives

k

and the number of sampling repetitions

t

). As is evident from the results, the InfoNCE-incurred supervised loss, when optimized with suitable values of

k

and

t

, can yield performance that is on par with or even surpasses that of the traditional CE approach.

Panel of Figure 3 — Figure 3: Linear probing accuracies (%) on CIFAR-10, CIFAR-100, and ImageNet-100 using unsupervised contrastive learning with the InfoNCE loss and the relaxed k-means loss, i.e., $\widehat{L}_{\mathrm{con}}+\lambda\cdot\widehat{L}_{\mathrm{k\text{-}means}}(f)$ , where $\tau=0.5$ .

Panel of Figure 4 — Figure 4: Validation accuracies (%) on long-tailed CIFAR-10 and long-tailed CIFAR-100 datasets using supervised losses incurred by InfoNCE with different numbers of negatives and imbalance ratios $\{10,20,50,100\}$ , where the imbalance ratio denotes the sample size ratio between the largest and smallest classes.

Dataset	Method	Imbalance Ratio
Dataset	Method	10	20	50	100
CIFAR-10-LT	CE	87.85	83.26	75.12	66.19
	BS	89.33	86.22	81.99	76.12
	$k=10$	89.35	85.55	79.90	74.34
	$k=20$	89.44	86.42	79.90	73.72
	$k=200$	88.43	86.38	81.13	74.64

Table 2: Validation accuracies (%) on long-tailed CIFAR-10 and long-tailed CIFAR-100 datasets. These results are obtained through supervised learning using cross entropy (CE), balanced softmax (BS), the InfoNCE-incurred loss (with the number of negatives

k

and the number of sampling repetitions

t=20

), and the SpeCL-incurred ( SpeCL-i ) loss, where SpeCL-i † denotes the results with normalized features and classifier weights. It is evident that the supervised loss incurred by InfoNCE outperforms CE, and SpeCL-i † demonstrates superior performance compared to SpeCL-i . Furthermore, the performance of the InfoNCE-incurred loss generally improves as the number of negatives increases, particularly in scenarios with larger imbalance ratios.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

$\lambda$	0	0.01	0.02	0.05
CIFAR-10	91.73 $\pm$ 0.06	91.68 $\pm$ 0.05	91.88 $\pm$ 0.07	91.85 $\pm$ 0.08
CIFAR-100	77.32 $\pm$ 0.08	77.37 $\pm$ 0.05	77.35 $\pm$ 0.10	77.54 $\pm$ 0.11
ImageNet-100	81.09 $\pm$ 0.20	81.22 $\pm$ 0.18	81.40 $\pm$ 0.23	81.28 $\pm$ 0.48
$\lambda$	0.1	0.2	0.5	1
CIFAR-10	91.80 $\pm$ 0.04	89.84 $\pm$ 0.08	86.57 $\pm$ 0.16	81.51 $\pm$ 0.20
CIFAR-100	77.37 $\pm$ 0.06	75.04 $\pm$ 0.07	69.03 $\pm$ 0.25	65.87 $\pm$ 1.51

Appendix

Table 3: Linear probing accuracies (%) on CIFAR-10, CIFAR-100, and ImageNet-100 using unsupervised contrastive learning with the InfoNCE loss and the relaxed k-means loss, i.e.,

\widehat{L}_{\mathrm{con}}+\lambda\cdot\widehat{L}_{\mathrm{k\text{-}means}}(f)

, where

\lambda\in\{0,0.01,0.02,0.05,0.1,0.2,0.5,1\}

and

\tau=0.5

Class Index	1	2	3	4	5	6	7	8	9	10
$\frac{1}{\rho_{\mathcal{T}}(c)}$	4.09	5.28	6.82	8.81	11.38	14.69	18.97	24.50	31.68	40.86
$\mu_{c}^{\top}w_{c}$ (train)	4.08	5.21	6.50	8.12	10.54	12.36	17.17	21.79	27.12	34.11
$\mu_{c}^{\top}w_{c}$ (test)	4.08	5.21	6.51	8.06	10.46	12.37	17.17	21.79	27.12	34.11

Appendix

Table 4: Illustration of

\frac{1}{\rho_{\mathcal{T}}(c)}

and

\mu_{c}^{\top}w_{c}

on long-tailed CIFAR-10 with an imbalance ratio of 10.

Contrastive Learning

Contrastive learning trains representations by bringing related examples together and separating unrelated ones for recognition, retrieval, and transfer.

See all 322 papers

Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets
May 11, 2026Thuan Nguyen, Shuchin Aeron, D. Richard Brown +1Class-Imbalanced Learning Contrastive Learning
Department of Engineering, Engineering Technology East Tennessee State University · Department of Electrical and Computer Engineering Tufts University · Department of Electrical and Computer Engineering Worcester Polytechnic Institute +1
Statistical Consistency and Generalization of Contrastive Representation Learning
May 4, 2026Yuanfan Li, Xiyuan Wei, Tianbao Yang +1Contrastive Learning Supervised Contrastive Learning
University of Sydney · 2Texas A&M University
Rethinking Representation Geometry in Supervised Contrastive Learning under Long-Tailed Data
Oct 23, 2026Joonhyeok Jeong, Jy-yong Sohn, Kibok LeeContrastive Learning Supervised Contrastive Learning
Department of Statistics and Data Science, Yonsei University

May 11, 2026J/K move · Enter open · S save

Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets

Thuan Nguyen, Shuchin Aeron, D. Richard Brown, Prakash Ishwar

Department of Engineering, Engineering Technology East Tennessee State University · Department of Electrical and Computer Engineering Tufts University · Department of Electrical and Computer Engineering Worcester Polytechnic Institute+1

Class-Imbalanced Learning Contrastive Learning Representation Collapse Neural Representation Geometry Neural Collapse

Open paper AskPDF arXiv

In this paper, we provide a computable characterization of the geometry of optimal representations in Contrastive Learning (CL) when the classes are imbalanced. When classes are balanced and the representation dimension is greater than the number of classes, it is well-known that the optimal representations exhibit Neural Collapse (NC), i.e., representations from the same class collapse to their class means and the class means form an Equiangular Tight Frame (ETF). For imbalanced classes and a large, generalized family of CL losses, we prove that the optimal representations of all samples from the same class collapse to their class means and their geometry exhibits an angular symmetry structure that is determined by the relative class proportions. In general, we show that the geometry can be determined by solving a convex optimization problem. Exploiting this symmetry structure, we analytically investigate a special case where class imbalance is extreme and prove that CL exhibits a phenomenon called Minority Collapse (MC) where all samples from the minority classes (classes with small probabilities) collapse into a single vector, whenever the class imbalance exceeds a threshold, which in turn depends on the regularity properties of the CL loss used and on the number of negative samples. Numerical results are provided to illustrate these phenomena and corroborate the theoretical results. We conclude by identifying a number of open problems.

Abstract

Contrastive Learning

Explore similar work

Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets

Statistical Consistency and Generalization of Contrastive Representation Learning

Rethinking Representation Geometry in Supervised Contrastive Learning under Long-Tailed Data