Contrastive learning has emerged as a popular paradigm for training representation models. While prior studies pursue more practical and less restrictive assumptions to enhance theoretical understanding of contrastive learning, they essentially describe the interconnections among representations within the data or induced by the function class. To encapsulate these assumptions, we propose the measure of internal discrepancy that quantifies the difference between the mean representation of positive samples and latent cluster centers. We then prove an internal discrepancy-aware generalization bound for downstream supervised tasks following contrastive learning, without requiring additional assumptions. This generalization bound not only serves as a unifying framework connecting various assumptions but also motivates us to further reduce the internal discrepancy, thereby introducing the relaxed k-means loss as an approximation. Moreover, we investigate the downstream behaviors of contrastive learning from the perspective of contrastive learning-incurred supervised losses. In the presence of class imbalance, we shed light on why contrastive learning can facilitate imbalanced learning. Finally, we investigate the optimality of these incurred supervised losses, proving several intriguing properties.
Figures & tables
Figure 1: Schematic comparison of large and small internal discrepancy. Colors denote latent classes, while dashed regions highlight local positive pairs. The filled and open circles indicate f(x) and f(x+) . The diamond and star mark Ex+∼Dpos(⋅∣x)f(x+) and Ex∼Dcf(x) , respectively. The expected distance between the conditional positive mean and the class mean equals Disc(f) . The right panel depicts more compact class and local regions, accompanied by a shorter segment in the illustrated configuration.
Figure 2: Illustration of validation accuracies (%) on CIFAR-10, CIFAR-100, and ImageNet-100 using cross entropy, InfoNCE-incurred loss as per Theorem 4 , and multi-class binary cross entropy defined in Equation 31 . The depicted results demonstrate that these three loss functions deliver comparable performance.
Dataset
Method
Sampling repetitions t
10
20
40
100
CIFAR-10
CE
94.79
k=1
94.79
94.93
94.83
94.91
k=10
95.20
94.87
95.05
94.71
k=20
94.81
94.82
94.85
94.65
k=40
94.76
94.73
94.79
94.73
Table 1: Validation accuracies (%) on the CIFAR-10 and CIFAR-100 datasets. These results are obtained through supervised learning using cross entropy (CE) or InfoNCE-incurred loss (with the number of negatives k and the number of sampling repetitions t ). As is evident from the results, the InfoNCE-incurred supervised loss, when optimized with suitable values of k and t , can yield performance that is on par with or even surpasses that of the traditional CE approach.
Figure 3: Linear probing accuracies (%) on CIFAR-10, CIFAR-100, and ImageNet-100 using unsupervised contrastive learning with the InfoNCE loss and the relaxed k-means loss, i.e., Lcon+λ⋅Lk-means(f) , where τ=0.5 .
Figure 4: Validation accuracies (%) on long-tailed CIFAR-10 and long-tailed CIFAR-100 datasets using supervised losses incurred by InfoNCE with different numbers of negatives and imbalance ratios {10,20,50,100} , where the imbalance ratio denotes the sample size ratio between the largest and smallest classes.
Dataset
Method
Imbalance Ratio
10
20
50
100
CIFAR-10-LT
CE
87.85
83.26
75.12
66.19
BS
89.33
86.22
81.99
76.12
k=10
89.35
85.55
79.90
74.34
k=20
89.44
86.42
79.90
73.72
k=200
88.43
86.38
81.13
74.64
Table 2: Validation accuracies (%) on long-tailed CIFAR-10 and long-tailed CIFAR-100 datasets. These results are obtained through supervised learning using cross entropy (CE), balanced softmax (BS), the InfoNCE-incurred loss (with the number of negatives k and the number of sampling repetitions t=20 ), and the SpeCL-incurred ( SpeCL-i ) loss, where SpeCL-i † denotes the results with normalized features and classifier weights. It is evident that the supervised loss incurred by InfoNCE outperforms CE, and SpeCL-i † demonstrates superior performance compared to SpeCL-i . Furthermore, the performance of the InfoNCE-incurred loss generally improves as the number of negatives increases, particularly in scenarios with larger imbalance ratios.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
λ
0
0.01
0.02
0.05
CIFAR-10
91.73 ± 0.06
91.68 ± 0.05
91.88 ± 0.07
91.85 ± 0.08
CIFAR-100
77.32 ± 0.08
77.37 ± 0.05
77.35 ± 0.10
77.54 ± 0.11
ImageNet-100
81.09 ± 0.20
81.22 ± 0.18
81.40 ± 0.23
81.28 ± 0.48
λ
0.1
0.2
0.5
1
CIFAR-10
91.80 ± 0.04
89.84 ± 0.08
86.57 ± 0.16
81.51 ± 0.20
CIFAR-100
77.37 ± 0.06
75.04 ± 0.07
69.03 ± 0.25
65.87 ± 1.51
Appendix
Table 3: Linear probing accuracies (%) on CIFAR-10, CIFAR-100, and ImageNet-100 using unsupervised contrastive learning with the InfoNCE loss and the relaxed k-means loss, i.e., Lcon+λ⋅Lk-means(f) , where λ∈{0,0.01,0.02,0.05,0.1,0.2,0.5,1} and τ=0.5 .
Class Index
1
2
3
4
5
6
7
8
9
10
ρT(c)1
4.09
5.28
6.82
8.81
11.38
14.69
18.97
24.50
31.68
40.86
μc⊤wc (train)
4.08
5.21
6.50
8.12
10.54
12.36
17.17
21.79
27.12
34.11
μc⊤wc (test)
4.08
5.21
6.51
8.06
10.46
12.37
17.17
21.79
27.12
34.11
Appendix
Table 4: Illustration of ρT(c)1 and μc⊤wc on long-tailed CIFAR-10 with an imbalance ratio of 10.
Contrastive learning trains representations by bringing related examples together and separating unrelated ones for recognition, retrieval, and transfer.
In this paper, we provide a computable characterization of the geometry of optimal representations in Contrastive Learning (CL) when the classes are imbalanced. When classes are balanced and the representation dimension is greater than the number of classes, it is well-known that the optimal representations exhibit Neural Collapse (NC), i.e., representations from the same class collapse to their class means and the class means form an Equiangular Tight Frame (ETF). For imbalanced classes and a large, generalized family of CL losses, we prove that the optimal representations of all samples from the same class collapse to their class means and their geometry exhibits an angular symmetry structure that is determined by the relative class proportions. In general, we show that the geometry can be determined by solving a convex optimization problem. Exploiting this symmetry structure, we analytically investigate a special case where class imbalance is extreme and prove that CL exhibits a phenomenon called Minority Collapse (MC) where all samples from the minority classes (classes with small probabilities) collapse into a single vector, whenever the class imbalance exceeds a threshold, which in turn depends on the regularity properties of the CL loss used and on the number of negative samples. Numerical results are provided to illustrate these phenomena and corroborate the theoretical results. We conclude by identifying a number of open problems.
Thuan Nguyen, Shuchin Aeron, D. Richard Brown +1
Department of Engineering, Engineering Technology East Tennessee State University · Department of Electrical and Computer Engineering Tufts University · Department of Electrical and Computer Engineering Worcester Polytechnic Institute +1
Contrastive representation learning (CRL) underpins many modern foundation models. Despite recent theoretical progress, existing analyses suffer from several key limitations: (i) the statistical consistency of CRL remains poorly understood; (ii) available generalization bounds deteriorate as the number of negative samples increases, contradicting the empirical benefits of large negative sets; and (iii) the retrieval performance of CRL has received limited theoretical attention. In this paper, we develop a unified statistical learning theory for CRL. For downstream tasks, we evaluate retrieval quality using an AUC-type population criterion and show that the contrastive loss is statistically consistent with optimal ranking. We further establish a calibration-style inequality that quantitatively relates excess contrastive risk to excess retrieval suboptimality. For upstream training, we study both supervised and self-supervised contrastive objectives and derive generalization bounds of order O(1/m+1/n) and O(1/m+1/n), respectively, where m denotes the number of negative samples and n the number of anchor points. These bounds not only explain the empirical advantages of large negative sets but also reveal an explicit trade-off between m and n. Extensive experiments on large-scale vision--language models corroborate our theoretical predictions.
Supervised contrastive learning and its balanced variants have demonstrated their effectiveness for representation learning under long-tailed distributions. However, although prior analyses suggest that their optimal representations may exhibit a regular simplex geometry, whether such a geometry is indeed exhibited remains insufficiently understood. In this work, we present a theoretical framework for these objectives that characterizes whether a regular simplex is formed at the optimum. By applying this to representative objectives, we show that these objectives fail to form a regular simplex at the optimum because they remain dependent on the number of samples per class. Motivated by this analysis, we introduce structural balancing that can be applied to contrastive objectives, and propose a fully balanced contrastive objective that provably forms a regular simplex geometry at its optimum. Both theoretical and empirical results demonstrate that the proposed method yields a more balanced representation space and improves performance on long-tailed recognition and transfer learning tasks.
Joonhyeok Jeong, Jy-yong Sohn, Kibok Lee
Department of Statistics and Data Science, Yonsei University