Rethinking Representation Geometry in Supervised Contrastive Learning under Long-Tailed Data

Joonhyeok Jeong; Jy-yong Sohn; Kibok Lee

cs.LGOct 23, 2026

Rethinking Representation Geometry in Supervised Contrastive Learning under Long-Tailed Data

Authors: Joonhyeok Jeong, Jy-yong Sohn, Kibok Lee

Organizations: Department of Statistics and Data Science, Yonsei University

Abstract

Supervised contrastive learning and its balanced variants have demonstrated their effectiveness for representation learning under long-tailed distributions. However, although prior analyses suggest that their optimal representations may exhibit a regular simplex geometry, whether such a geometry is indeed exhibited remains insufficiently understood. In this work, we present a theoretical framework for these objectives that characterizes whether a regular simplex is formed at the optimum. By applying this to representative objectives, we show that these objectives fail to form a regular simplex at the optimum because they remain dependent on the number of samples per class. Motivated by this analysis, we introduce structural balancing that can be applied to contrastive objectives, and propose a fully balanced contrastive objective that provably forms a regular simplex geometry at its optimum. Both theoretical and empirical results demonstrate that the proposed method yields a more balanced representation space and improves performance on long-tailed recognition and transfer learning tasks.

Figures & tables

Panel of Figure 1 — Figure 1: Visualizations of embedding spaces learned by three contrastive objectives (SCL, BCL, and FBCL) on a subset of ImageNet: three random classes sampled under (a) a balanced setting and (b) an imbalanced setting with class cardinality ratios of 1:1:1 and 1000:10:1, respectively. Angles are measured between class centers. Under (a), all three methods form a regular simplex; under (b), only FBCL forms it, whereas SCL and BCL fail.

Method	ImageNet-LT				iNaturalist
Method	Many	Medium	Few	All	Many	Medium	Few	All
OLTR ( Liu et al., 2019 )	43.2	35.1	18.5	35.6	59.0	64.1	64.9	63.9
$\tau$ -norm ( Kang et al., 2020 )	56.6	44.2	27.4	46.7	65.6	65.3	65.9	65.6
cRT ( Kang et al., 2020 )	58.8	44.0	26.1	47.3	69.0	71.1	70.2	70.6
PaCo ∗ ( Cui et al., 2021 )	55.3	50.2	26.7	48.9	60.2	67.7	68.5	67.2
GPaCo ∗ ( Cui et al., 2023 )	59.4	53.4	30.5	52.6	65.6	72.2	72.0	71.4

Table 2: Comparison of long-tailed recognition methods and the effect of structural balancing on ImageNet-LT and iNaturalist. ∗ denotes results reproduced with 90 epochs on ImageNet-LT and 100 epochs on iNaturalist for a fair comparison. Bold indicates the best results. Standard deviations are reported in Appendix C.5 .

Method	ImageNet-LT
Method	Many	Medium	Few	All
SimCLR ( Chen et al., 2020 )	65.0	52.6	36.2	55.2 $\pm$ 0.2
SCL ( Khosla et al., 2020 )	65.8	53.3	36.1	55.8 $\pm$ 0.4
BCL ( Zhu et al., 2022 )	65.8	53.7	36.2	56.0 $\pm$ 0.1
FBCL (Ours)	66.4	54.1	37.5	56.6 $\pm$ 0.1

Table 1: Comparison of classification accuracy across contrastive objectives on ImageNet-LT.

Table 4: Comparison of intra-class variance and Top-1 accuracy. (a) Results on synthetic data. (b) Ablation study of positive balancing (PB), negative balancing (NB), and structural balancing (SB) on ImageNet-LT. Details are provided in Appendix C.5 .

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Schematic overview of our framework.

Dataset	# of classes	Training	Validation	Test	Metric
CIFAR-10	10	45000	5000	10000	Top-1 accuracy
CIFAR-100	100	45000	5000	10000	Top-1 accuracy
MIT67	67	4690	670	1340	Top-1 accuracy
DTD (split1)	47	1880	1880	1880	Top-1 accuracy
CUB200	200	4994	1000	5794	Top-1 accuracy
Food	101	68175	7575	25250	Top-1 accuracy

Appendix

Table 5: Dataset information

Method	ImageNet-LT				iNaturalist
Method	Many	Medium	Few	All	Many	Medium	Few	All
SimCLR ( Chen et al., 2020 )	65.0 $\pm$ 0.1	52.6 $\pm$ 0.2	36.2 $\pm$ 0.1	55.2 $\pm$ 0.2	68.1 $\pm$ 0.3	70.7 $\pm$ 0.2	69.9 $\pm$ 0.3	70.1 $\pm$ 0.2
SCL ( Khosla et al., 2020 )	65.8 $\pm$ 0.4	53.3 $\pm$ 0.4	36.1 $\pm$ 0.5	55.8 $\pm$ 0.4	68.4 $\pm$ 0.3	71.1 $\pm$ 0.4	70.2 $\pm$ 0.5	70.4 $\pm$ 0.3
BCL ( Zhu et al., 2022 )	65.8 $\pm$ 0.2	53.7 $\pm$ 0.2	36.2 $\pm$ 0.1	56.0 $\pm$ 0.1	70.0 $\pm$ 0.7	72.0 $\pm$ 0.4	71.1 $\pm$ 0.2	71.3 $\pm$ 0.6
FBCL (Ours)	66.4 $\pm$ 0.2	54.1 $\pm$ 0.2	37.5 $\pm$ 0.4	56.6 $\pm$ 0.1	70.4 $\pm$ 0.3	72.8 $\pm$ 0.2	71.9 $\pm$ 0.3	72.2 $\pm$ 0.1

Appendix

Table 6: Statistics over three runs, on contrastive objectives on ImageNet-LT and iNaturalist.

Method	Many	Medium	Few	Acc
CE	$0.459$ $\pm$ 0.02	$0.533$ $\pm$ 0.03	$0.721$ $\pm$ 0.03	$73.8$ $\pm$ 0.16
SCL	$0.272$ $\pm$ 0.01	$0.346$ $\pm$ 0.02	$0.575$ $\pm$ 0.05	$74.1$ $\pm$ 2.74
BCL	$0.250$ $\pm$ 0.02	$0.385$ $\pm$ 0.03	$0.749$ $\pm$ 0.07	$74.1$ $\pm$ 2.36
FBCL (Ours)	$0.409$ $\pm$ 0.02	$0.453$ $\pm$ 0.02	$0.438$ $\pm$ 0.03	$75.5$ $\pm$ 3.10

Appendix

Table 7: Statistics over five runs on intra-class variance and Top-1 accuracy on synthetic data.

Method	PB	NB	SB	Intra-class variance			Acc.
Method	PB	NB	SB	Many	Medium	Few	Acc.
PB (SCL)	✓	✗	✗	0.211 $\pm$ 0.001	0.342 $\pm$ 0.003	0.420 $\pm$ 0.001	51.5 $\pm$ 0.18
PB+NB (BCL)	✓	✓	✗	0.210 $\pm$ 0.005	0.382 $\pm$ 0.004	0.465 $\pm$ 0.011	51.7 $\pm$ 0.45
PB+SB	✓	✗	✓	0.340 $\pm$ 0.004	0.387 $\pm$ 0.002	0.467 $\pm$ 0.003	51.5 $\pm$ 0.25
NB+SB	✗	✓	✓	0.660 $\pm$ 0.009	0.620 $\pm$ 0.006	0.601 $\pm$ 0.009	51.4 $\pm$ 0.16
PB+NB+SB (Ours)	✓	✓	✓	0.267 $\pm$ 0.003	0.320 $\pm$ 0.006	0.379 $\pm$ 0.003	52.1 $\pm$ 0.15

Appendix

Table 8: Statistics over five runs of the ablation study over balancing components on CIFAR-100-LT (IR=100).

Method	PB	NB	SB	Intra-class variance			Acc.
Method	PB	NB	SB	Many	Medium	Few	Many	Medium	Few	All
PB (SCL)	✓	✗	✗	0.189 $\pm$ 0.001	0.298 $\pm$ 0.003	0.357 $\pm$ 0.001	65.76 $\pm$ 0.38	53.33 $\pm$ 0.40	36.13 $\pm$ 0.46	55.78 $\pm$ 0.41
PB+NB (BCL)	✓	✓	✗	0.167 $\pm$ 0.021	0.293 $\pm$ 0.003	0.362 $\pm$ 0.006	65.83 $\pm$ 0.15	53.74 $\pm$ 0.17	36.23 $\pm$ 0.06	56.03 $\pm$ 0.07
PB+SB	✓	✗	✓	0.368 $\pm$ 0.004	0.502 $\pm$ 0.003	0.509 $\pm$ 0.003	66.07 $\pm$ 0.12	53.40 $\pm$ 0.30	36.22 $\pm$ 0.36	55.92 $\pm$ 0.19
NB+SB	✗	✓	✓	0.597 $\pm$ 0.002	0.623 $\pm$ 0.002	0.602 $\pm$ 0.001	65.20 $\pm$ 0.20	53.03 $\pm$ 0.29	35.47 $\pm$ 0.32	55.35 $\pm$ 0.17
PB+NB+SB (Ours)	✓	✓	✓	0.196 $\pm$ 0.008	0.281 $\pm$ 0.010	0.339 $\pm$ 0.008	66.44 $\pm$ 0.15	54.12 $\pm$ 0.21	37.50 $\pm$ 0.40	56.64 $\pm$ 0.12

Appendix

Table 9: Ablation study of positive balancing (PB), negative balancing (NB), and structural balancing (SB) on ImageNet-LT. Statistics over three runs are given.

Method	ImageNet-LT				iNaturalist
Method	Many	Medium	Few	All	Many	Medium	Few	All
DBM+BCL	64.4 $\pm$ 0.1	54.3 $\pm$ 0.2	41.4 $\pm$ 0.3	56.5 $\pm$ 0.2	66.3 $\pm$ 0.3	71.8 $\pm$ 0.5	73.3 $\pm$ 0.3	71.8 $\pm$ 0.1
DBM+FBCL (Ours)	64.9 $\pm$ 0.2	54.5 $\pm$ 0.2	42.3 $\pm$ 0.3	56.9 $\pm$ 0.2	66.5 $\pm$ 0.2	72.4 $\pm$ 0.4	73.8 $\pm$ 0.2	72.3 $\pm$ 0.2

Appendix

Table 10: Statistics over three runs of FBCL and BCL combined with DBM.

Method	ImageNet-LT				iNaturalist
Method	Many	Medium	Few	All	Many	Medium	Few	All
ProCo	66.5 $\pm$ 0.1	54.8 $\pm$ 0.5	38.1 $\pm$ 0.4	57.1 $\pm$ 0.2	70.4 $\pm$ 0.4	73.1 $\pm$ 0.1	71.9 $\pm$ 0.6	72.3 $\pm$ 0.2
ProCo + SB (Ours)	66.8 $\pm$ 0.2	55.2 $\pm$ 0.1	38.8 $\pm$ 0.2	57.4 $\pm$ 0.1	70.7 $\pm$ 0.6	73.4 $\pm$ 0.4	72.3 $\pm$ 0.4	72.7 $\pm$ 0.5

Appendix

Table 11: Statistics over three runs of structural balancing combined with ProCo on ImageNet-LT and iNaturalist.

Method	CIFAR-100-LT
Method	IR=100	IR=50	IR=10
BCL	$51.7\pm 0.5$	$55.7\pm 0.4$	$63.7\pm 0.1$
FBCL	$52.1\pm 0.2$ (+0.4)	$56.0\pm 0.2$ (+0.3)	$64.0\pm 0.1$ (+0.3)
DBM + BCL	$50.4\pm 1.0$	$55.6\pm 0.1$	$65.7\pm 0.2$
DBM + FBCL	$51.2\pm 0.5$ (+0.8)	$55.9\pm 0.3$ (+0.3)	$65.8\pm 0.1$ (+0.1)
LLM-AutoDA + BCL	$50.6\pm 0.6$	$55.5\pm 0.2$	$64.8\pm 0.1$

Appendix

Table 12: Statistics over five runs of FBCL or BCL with other methods on CIFAR-100-LT.

	CIFAR-100-LT
Imbalance Factor	100	50	10
Top-1 accuracy
ProCo	$51.9\pm 0.2$	$56.2\pm 0.2$	$64.6\pm 0.4$
ProCo + SB	$52.1\pm 0.3$	$56.3\pm 0.21$	$64.7\pm 0.1$
Few-class accuracy
ProCo	32.1 $\pm$ 0.6	43.2 $\pm$ 0.7	-

Appendix

Table 13: Statistics over five runs of structural balancing applied to ProCo ( Du et al., 2024 ) on CIFAR-100-LT dataset. Adding structural balancing improves few class accuracy.

Table 14: Cosine similarity across class centers on CIFAR-100-LT.

Method	ImageNet-LT
Method	Many ( $\downarrow$ )	Medium ( $\downarrow$ )	Few ( $\downarrow$ )	All ( $\downarrow$ )
BCL	0.154 $\pm$ 0.003	0.106 $\pm$ 0.004	0.104 $\pm$ 0.005	0.124 $\pm$ 0.004
FBCL	0.137 $\pm$ 0.001	0.088 $\pm$ 0.002	0.079 $\pm$ 0.005	0.103 $\pm$ 0.002

Appendix

Table 15: Cosine similarity across class centers on ImageNet-LT.

Method	CIFAR-100-LT (IR=100)
Method	Many	Medium	Few	Overall
BCL	66.04 $\pm$ 0.55	52.57 $\pm$ 0.49	33.78 $\pm$ 1.04	51.65 $\pm$ 0.45
BCL + CB Sampling	51.32 $\pm$ 0.25	56.63 $\pm$ 0.38	30.17 $\pm$ 1.60	46.84 $\pm$ 0.39
BCL + Re-weighting	65.51 $\pm$ 1.01	52.69 $\pm$ 0.54	33.56 $\pm$ 0.98	51.45 $\pm$ 0.45
SCL + CB-weighting	64.84 $\pm$ 0.80	51.44 $\pm$ 0.52	33.14 $\pm$ 0.64	50.65 $\pm$ 0.27
BCL + CB-weighting	64.77 $\pm$ 0.59	51.46 $\pm$ 0.85	33.02 $\pm$ 0.03	50.59 $\pm$ 0.49

Appendix

Table 16: Comparison with class-balanced sampling and other re-weighting methods. Statistics over five runs are provided.

Contrastive Learning

Contrastive learning trains representations by bringing related examples together and separating unrelated ones for recognition, retrieval, and transfer.

See all 318 papers

Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets
May 11, 2026Thuan Nguyen, Shuchin Aeron, D. Richard Brown +1Class-Imbalanced Learning Contrastive Learning
Department of Engineering, Engineering Technology East Tennessee State University · Department of Electrical and Computer Engineering Tufts University · Department of Electrical and Computer Engineering Worcester Polytechnic Institute +1
A Unified Geometric Framework for Weighted Contrastive Learning
May 13, 2026Raphael Vock, Edouard Duchesnay, Benoit DufumierContrastive Learning Neural Representation Geometry
GAIA Lab, NeuroSpin, CEA, CNRS · Université Paris-Saclay, Gif-sur-Yvette, France
From Perturbation Correction to Geometry-Aware Sampling: Sharpness-Guided Equilibrium Sampling for Balanced Flat Minima in Long-Tailed Learning
Jul 24, 2026Jiaxin Deng, Junbiao PangSharpness-Aware Minimization Long-Tail Learning
School of Information Science and Technology, Beijing University of Technology, Beijing, China

May 11, 2026J/K move · Enter open · S save

Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets

Thuan Nguyen, Shuchin Aeron, D. Richard Brown, Prakash Ishwar

Department of Engineering, Engineering Technology East Tennessee State University · Department of Electrical and Computer Engineering Tufts University · Department of Electrical and Computer Engineering Worcester Polytechnic Institute+1

Class-Imbalanced Learning Contrastive Learning Representation Collapse Neural Representation Geometry Neural Collapse

Open paper AskPDF arXiv

In this paper, we provide a computable characterization of the geometry of optimal representations in Contrastive Learning (CL) when the classes are imbalanced. When classes are balanced and the representation dimension is greater than the number of classes, it is well-known that the optimal representations exhibit Neural Collapse (NC), i.e., representations from the same class collapse to their class means and the class means form an Equiangular Tight Frame (ETF). For imbalanced classes and a large, generalized family of CL losses, we prove that the optimal representations of all samples from the same class collapse to their class means and their geometry exhibits an angular symmetry structure that is determined by the relative class proportions. In general, we show that the geometry can be determined by solving a convex optimization problem. Exploiting this symmetry structure, we analytically investigate a special case where class imbalance is extreme and prove that CL exhibits a phenomenon called Minority Collapse (MC) where all samples from the minority classes (classes with small probabilities) collapse into a single vector, whenever the class imbalance exceeds a threshold, which in turn depends on the regularity properties of the CL loss used and on the number of negative samples. Numerical results are provided to illustrate these phenomena and corroborate the theoretical results. We conclude by identifying a number of open problems.

Abstract

Contrastive Learning

Explore similar work

Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets

A Unified Geometric Framework for Weighted Contrastive Learning

From Perturbation Correction to Geometry-Aware Sampling: Sharpness-Guided Equilibrium Sampling for Balanced Flat Minima in Long-Tailed Learning