Multi-Marginal Inverse Optimal Transport for Contrastive Learning Via Explicit Anchor-Positive-Negative Coupling

Ngoc-Hai Nguyen; Thuan Nguyen; Prakash Ishwar; Shuchin Aeron

stat.MLSep 27, 2026

Multi-Marginal Inverse Optimal Transport for Contrastive Learning Via Explicit Anchor-Positive-Negative Coupling

Authors: Ngoc-Hai Nguyen, Thuan Nguyen, Prakash Ishwar, Shuchin Aeron

Organizations: Department of Electrical and Computer Engineering Tufts University · Department of Engineering, Engineering Technology East Tennessee State University · Department of Electrical and Computer Engineering Boston University

Abstract

Inverse Optimal Transport (OT) based methods for representation learning learn representations such that the global OT coupling between a pair of data marginals in the representation space, concentrates on the positive pairs. This is in contrast to previous methods that primarily focused on pairwise matching. However, these methods $\textit{DO NOT}$ utilize negative pairs and hence are not truly contrastive in their approach. We show that this leads to issues of dimensional collapse and hence degraded downstream performance. To alleviate this, we develop a novel multi-marginal (MM) inverse OT (IOT) contrastive learning (CL) approach called Neg-MMIOT-CL, which learns representations such that the global multi-marginal OT (MMOT) coupling between a triple of data marginals, with respect to a carefully designed ground-cost between triplets of data points in the representation space, concentrates on the anchor-positive-negative $\textit{triplets}$ . For a latent class model, we empirically show that Neg-MMIOT-CL alleviates dimensional collapse. Furthermore, for a specific choice of ground cost for all triplets in representation space, we prove that the optimal representation configuration for Neg-MMIOT-CL exhibits equiangular property for within-class and across-class representations, which translates to Neural-Collapse when the representation dimension is larger than the number of classes minus one -- a result that is $\textit{previously established only}$ for pairwise contrastive learning methods. Finally, we propose Neg-IOT-CL-PushPull, that is a computationally efficient alternative to Neg-MMIOT-CL, alleviating the high cost of computing MMOT plans needed during implementation. We apply these methods on both synthetic and real-world datasets and show significant improvements over existing OT-based contrastive learning methods.

Figures & tables

\tilde{P}_{ijk}=\frac{\mathbf{1}\!\left((i,j,k)\in\mathcal{A}\right)}{|\mathcal{A}|}.

Algorithm 1 Neg-MMIOT-CL and Neg-IOT-CL-PushPull batch training

Figure 1: Testing Neural Collapse: NC1 measures within-class variability with smaller values indicating stronger class collapse. NC2 measures how close the class means are to the simplex-ETF geometry. Both NC2 quantities should approach zero when the class means form a simplex ETF. The numbers in the legends are the NC metrics in the last iteration.

Panel of Figure 2 — Figure 2: Testing Dimensional Collapse. We plot the sorted (max)-normalized eigenvalues of the centered class-mean covariance matrix at the final epoch. A rapidly decaying spectrum indicates that the class means occupy only a few effective directions, i.e., dimensional collapse.

Setting	Method	MNIST		SVHN		CIFAR-10		CIFAR-100		Tiny-ImageNet
		ResNet50	ViT-B/16	ResNet50	ViT-B/16	ResNet50	ViT-B/16	ResNet50	ViT-B/16	ResNet50	ViT-B/16
SCL	InfoNCE	99.36	99.49	91.96	94.07	91.39	90.40	74.01	73.59	62.60	46.32
	InvaSpread	98.98	99.61	81.79	85.35	78.46	72.62	63.26	67.08	55.06	44.19
	Standard OT	99.21	99.60	92.08	94.24	90.78	84.72	72.71	73.49	61.20	45.27
	Neg-IOT-CL-PushPull (Ours)	99.30	99.64	91.89	94.38	89.59	88.69	74.80	74.67	62.04	46.54
	Neg-MMIOT-CL (Ours)	99.41	99.66	92.74	94.87	90.64	91.97	75.17	74.62	63.54	48.02

Table 1: Average linear-probe accuracy (%) for SCL and UCL across datasets using ResNet50 and ViT-B/16 backbones. Bold values indicate the best result within each setting, dataset, and backbone.

CLIP-Loss	Image $\rightarrow$ Text		Text $\rightarrow$ Image		CIFAR-10		CIFAR-100
	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
infoNCE	67.22	92.51	65.91	94.61	23.37	74.82	6.36	27.59
Standard OT	43.92	81.86	30.90	70.54	26.62	78.56	4.77	22.95
Standard OT - Uniform	64.25	92.40	63.22	92.06	25.97	78.67	6.51	27.68
DBOT	15.89	46.68	15.52	45.92	24.98	70.36	5.72	27.19
Fused-Gromov	14.32	43.92	13.94	42.87	24.99	76.06	3.72	19.40

Table 2: MS-COCO retrieval and zero-shot transfer results. Bold values indicate the best result

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

\tilde{P}_{ijk}=\begin{cases}1,&(i,j,k)\in\mathcal{A}\\ 0,&\text{otherwise}\end{cases}

Appendix

Algorithm 2 Supervised CL with Neg-MMIOT-CL

(\tilde{P}_{+})_{ij}=\begin{cases}1,&(i,j)\in\mathcal{A}^{+}\\ 0,&\text{otherwise}\end{cases}\qquad(\tilde{P}_{-})_{ik}\!=\!=\begin{cases}1,&(i,k)\in\mathcal{A}^{-}\\ 0,&\text{otherwise}\end{cases}

Appendix

Algorithm 3 Supervised CL with Neg-IOT-CL-PushPull

\mathcal{B}^{(t)}=\{(x_{i},y_{i}):i\in\bigcup_{c=1}^{K}\mathcal{S}_{\mathfrak{c}}^{(t)}\}.

Appendix

Algorithm 4 Class-Uniform Batch Sampler (only for Synthetic Experiments under the Theoretical Setting)

Figure 3: Visualization on $\mathbb{R}^{2}$ . We use tSNE method for projection.

Figure 4: Visualization on $\mathbb{R}^{2}$ . We use tSNE method for projection.

Figure 5: Neural Collapse Observation when $\psi$ is negative log sigmoid

Panel of Figure 6 — Figure 6: Dimensional Collapse in the settings of different numbers of classes when $\psi$ is negative log sigmoid

SCL	MNIST
	ResNet18		ResNet34		ResNet50		ViT-B/16
	Linear	k-NN	Linear	k-NN	Linear	k-NN	Linear	k-NN
infoNCE	$99.25\pm 0.10$	$99.24\pm 0.13$	$99.27\pm 0.04$	$99.27\pm 0.04$	$99.36\pm 0.07$	$99.30\pm 0.12$	$99.49\pm 0.13$	$99.46\pm 0.10$
InvaSpread	$99.27\pm 0.10$	$99.03\pm 0.13$	$98.91\pm 0.08$	$98.76\pm 0.13$	$98.98\pm 0.09$	$99.17\pm 0.07$	$99.61\pm 0.08$	$99.59\pm 0.03$
Standard OT	$99.22\pm 0.04$	$99.20\pm 0.05$	$99.18\pm 0.10$	$99.18\pm 0.13$	$99.21\pm 0.04$	$99.17\pm 0.04$	$99.60\pm 0.05$	$99.58\pm 0.09$
Neg-IOT-CL-PushPull (Ours)	$\mathbf{99.33\pm 0.06}$	$\mathbf{99.35\pm 0.10}$	$\mathbf{99.33\pm 0.11}$	$\mathbf{99.31\pm 0.10}$	$99.30\pm 0.13$	$99.29\pm 0.07$	$99.64\pm 0.10$	$99.64\pm 0.05$

Appendix

Table 3: SCL results on MNIST with different backbones.

SCL	SVHN
	ResNet18		ResNet34		ResNet50		ViT-B/16
	Linear	k-NN	Linear	k-NN	Linear	k-NN	Linear	k-NN
infoNCE	$91.76\pm 0.33$	$91.71\pm 0.21$	$\mathbf{91.73\pm 0.27}$	$\mathbf{91.77\pm 0.43}$	$91.96\pm 0.42$	$91.94\pm 0.29$	$94.07\pm 0.38$	$94.04\pm 0.37$
InvaSpread	$86.55\pm 0.28$	$87.21\pm 0.36$	$82.65\pm 0.34$	$85.94\pm 0.23$	$81.79\pm 0.30$	$80.80\pm 0.30$	$85.35\pm 0.45$	$84.78\pm 0.48$
Standard OT	$91.24\pm 0.32$	$91.16\pm 0.18$	$91.58\pm 0.13$	$91.61\pm 0.15$	$92.08\pm 0.25$	$91.99\pm 0.31$	$94.24\pm 0.47$	$94.12\pm 0.36$
Neg-IOT-CL-PushPull (Ours)	$91.03\pm 0.14$	$90.99\pm 0.27$	$91.55\pm 0.27$	$91.56\pm 0.13$	$91.89\pm 0.14$	$91.83\pm 0.41$	$94.38\pm 0.33$	$94.38\pm 0.35$

Appendix

Table 4: SCL results on SVHN with different backbones.

SCL	CIFAR-10
	ResNet18		ResNet34		ResNet50		ViT-B/16
	Linear	k-NN	Linear	k-NN	Linear	k-NN	Linear	k-NN
infoNCE	$85.01\pm 0.25$	$84.82\pm 0.47$	$86.21\pm 0.27$	$86.06\pm 0.29$	$\mathbf{91.39\pm 0.34}$	$\mathbf{90.98\pm 0.43}$	$90.40\pm 0.38$	$90.32\pm 0.46$
InvaSpread	$73.01\pm 0.42$	$76.01\pm 0.49$	$75.24\pm 0.33$	$75.59\pm 0.31$	$78.46\pm 0.40$	$77.53\pm 0.47$	$72.62\pm 0.24$	$72.41\pm 0.56$
Standard OT	$83.76\pm 0.30$	$83.47\pm 0.34$	$83.29\pm 0.24$	$84.40\pm 0.41$	$90.78\pm 0.55$	$90.80\pm 0.48$	$84.72\pm 0.29$	$84.55\pm 0.57$
Neg-IOT-CL-PushPull (Ours)	$82.15\pm 0.20$	$82.66\pm 0.54$	$83.23\pm 0.20$	$82.21\pm 0.36$	$89.59\pm 0.55$	$89.62\pm 0.55$	$88.69\pm 0.53$	$88.54\pm 0.16$

Appendix

Table 5: SCL results on CIFAR-10 with different backbones.

SCL	CIFAR-100
	ResNet18		ResNet34		ResNet50		ViT-B/16
	Linear	k-NN	Linear	k-NN	Linear	k-NN	Linear	k-NN
infoNCE	$69.72\pm 0.58$	$69.46\pm 0.39$	$70.61\pm 0.33$	$70.33\pm 0.50$	$74.01\pm 0.35$	$73.56\pm 0.40$	$73.59\pm 0.33$	$73.58\pm 0.34$
InvaSpread	$55.46\pm 0.53$	$54.32\pm 0.81$	$64.84\pm 0.62$	$63.10\pm 0.46$	$63.26\pm 0.81$	$62.24\pm 0.34$	$67.08\pm 0.49$	$66.68\pm 0.30$
Standard OT	$69.24\pm 0.56$	$69.20\pm 0.58$	$70.53\pm 0.82$	$70.41\pm 0.72$	$72.71\pm 0.30$	$72.88\pm 0.34$	$73.49\pm 0.56$	$73.59\pm 0.70$
Neg-IOT-CL-PushPull (Ours)	$69.26\pm 0.29$	$69.46\pm 0.52$	$70.76\pm 0.76$	$70.56\pm 0.74$	$74.80\pm 0.75$	$74.19\pm 0.30$	$\mathbf{74.67\pm 0.69}$	$\mathbf{74.83\pm 0.67}$

Appendix

Table 6: SCL results on CIFAR-100 with different backbones.

SCL	Tiny-ImageNet
	ResNet18		ResNet34		ResNet50		ViT-B/16
	Linear	k-NN	Linear	k-NN	Linear	k-NN	Linear	k-NN
infoNCE	$54.56\pm 0.65$	$54.34\pm 0.55$	$55.02\pm 0.60$	$54.52\pm 0.54$	$62.60\pm 0.62$	$\mathbf{63.30\pm 1.00}$	$46.32\pm 0.86$	$46.37\pm 0.68$
InvaSpread	$51.84\pm 0.84$	$51.26\pm 0.42$	$51.20\pm 0.44$	$49.98\pm 0.38$	$55.06\pm 0.67$	$53.72\pm 0.56$	$44.19\pm 0.91$	$43.33\pm 0.82$
Standard OT	$54.64\pm 0.86$	$53.92\pm 0.61$	$55.44\pm 0.36$	$55.08\pm 0.92$	$61.20\pm 0.83$	$60.84\pm 0.36$	$45.27\pm 0.80$	$45.19\pm 0.41$
Neg-IOT-CL-PushPull (Ours)	$\mathbf{57.28\pm 0.74}$	$\mathbf{56.88\pm 0.47}$	$\mathbf{56.74\pm 0.80}$	$\mathbf{56.38\pm 0.79}$	$62.04\pm 0.62$	$61.42\pm 1.02$	$46.54\pm 0.54$	$46.58\pm 0.80$

Appendix

Table 7: SCL results on Tiny-ImageNet with different backbones.

UCL	MNIST
	ResNet18		ResNet34		ResNet50		ViT-B/16
	Linear	k-NN	Linear	k-NN	Linear	k-NN	Linear	k-NN
infoNCE	$97.63\pm 0.08$	$98.56\pm 0.05$	$97.43\pm 0.09$	$98.22\pm 0.05$	$97.73\pm 0.08$	$98.20\pm 0.05$	$98.40\pm 0.16$	$98.82\pm 0.12$
InvaSpread	$98.00\pm 0.18$	$97.37\pm 0.18$	$97.42\pm 0.05$	$96.93\pm 0.10$	$97.65\pm 0.15$	$96.89\pm 0.06$	$97.95\pm 0.14$	$97.70\pm 0.13$
Standard OT	$97.70\pm 0.09$	$98.07\pm 0.06$	$\mathbf{97.60\pm 0.13}$	$\mathbf{98.39\pm 0.18}$	$97.20\pm 0.06$	$98.09\pm 0.11$	$98.35\pm 0.05$	$98.74\pm 0.13$
Neg-IOT-CL-PushPull (Ours)	$\mathbf{97.94\pm 0.11}$	$\mathbf{98.62\pm 0.08}$	$97.49\pm 0.13$	$98.31\pm 0.05$	$96.56\pm 0.14$	$98.07\pm 0.13$	$98.61\pm 0.14$	$98.91\pm 0.12$

Appendix

Table 8: UCL results on MNIST with different backbones.

UCL	SVHN
	ResNet18		ResNet34		ResNet50		ViT-B/16
	Linear	k-NN	Linear	k-NN	Linear	k-NN	Linear	k-NN
infoNCE	$77.44\pm 0.60$	$81.21\pm 0.39$	$77.70\pm 0.46$	$81.86\pm 0.63$	$78.13\pm 0.36$	$83.00\pm 0.43$	$84.67\pm 0.57$	$86.24\pm 0.21$
InvaSpread	$74.36\pm 0.53$	$71.49\pm 0.37$	$72.78\pm 0.46$	$69.47\pm 0.47$	$73.48\pm 0.63$	$70.53\pm 0.38$	$75.13\pm 0.38$	$73.88\pm 0.33$
Standard OT	$74.32\pm 0.29$	$79.42\pm 0.59$	$75.65\pm 0.52$	$80.12\pm 0.61$	$75.96\pm 0.37$	$79.69\pm 0.38$	$83.94\pm 0.33$	$85.76\pm 0.23$
Neg-IOT-CL-PushPull (Ours)	$79.28\pm 0.25$	$81.79\pm 0.44$	$79.42\pm 0.43$	$81.38\pm 0.34$	$78.63\pm 0.55$	$80.85\pm 0.38$	$87.45\pm 0.30$	$87.02\pm 0.26$

Appendix

Table 9: UCL results on SVHN with different backbones.

UCL	CIFAR-10
	ResNet18		ResNet34		ResNet50		ViT-B/16
	Linear	k-NN	Linear	k-NN	Linear	k-NN	Linear	k-NN
infoNCE	$75.30\pm 0.65$	$76.00\pm 0.57$	$75.45\pm 0.30$	$76.47\pm 0.51$	$76.81\pm 0.52$	$76.69\pm 0.62$	$84.57\pm 0.53$	$85.84\pm 0.61$
InvaSpread	$67.27\pm 0.45$	$65.83\pm 0.74$	$65.92\pm 0.38$	$64.54\pm 0.65$	$64.88\pm 0.52$	$63.69\pm 0.30$	$64.13\pm 0.49$	$63.28\pm 0.64$
Standard OT	$75.33\pm 0.62$	$75.27\pm 0.34$	$75.94\pm 0.72$	$76.77\pm 0.62$	$78.12\pm 0.57$	$77.67\pm 0.35$	$83.22\pm 0.43$	$86.28\pm 0.35$
Neg-IOT-CL-PushPull (Ours)	$\mathbf{77.43\pm 0.25}$	$\mathbf{76.05\pm 0.52}$	$\mathbf{77.02\pm 0.65}$	$\mathbf{76.34\pm 0.77}$	$\mathbf{78.18\pm 0.63}$	$\mathbf{77.89\pm 0.29}$	$\mathbf{87.56\pm 0.23}$	$\mathbf{86.96\pm 0.36}$

Appendix

Table 10: UCL results on CIFAR-10 with different backbones.

UCL	CIFAR-100
	ResNet18		ResNet34		ResNet50		ViT-B/16
	Linear	k-NN	Linear	k-NN	Linear	k-NN	Linear	k-NN
infoNCE	$48.34\pm 0.32$	$49.16\pm 0.31$	$48.09\pm 0.44$	$48.60\pm 0.48$	$49.05\pm 0.75$	$49.51\pm 0.79$	$56.99\pm 0.89$	$55.19\pm 0.37$
InvaSpread	$40.37\pm 0.77$	$40.63\pm 0.82$	$44.23\pm 0.57$	$45.45\pm 0.55$	$44.51\pm 0.44$	$45.83\pm 0.44$	$44.17\pm 0.49$	$39.59\pm 0.36$
Standard OT	$46.15\pm 0.38$	$46.04\pm 0.31$	$46.35\pm 0.88$	$45.54\pm 0.41$	$45.54\pm 0.91$	$45.60\pm 0.84$	$55.46\pm 0.62$	$54.21\pm 0.35$
Neg-IOT-CL-PushPull (Ours)	$\mathbf{51.81\pm 0.79}$	$\mathbf{48.41\pm 0.73}$	$\mathbf{51.04\pm 0.40}$	$\mathbf{48.13\pm 0.44}$	$\mathbf{52.20\pm 0.71}$	$\mathbf{49.13\pm 0.52}$	$55.45\pm 0.88$	$54.52\pm 0.91$

Appendix

Table 11: UCL results on CIFAR-100 with different backbones.

UCL	Tiny-ImageNet
	ResNet18		ResNet34		ResNet50		ViT-B/16
	Linear	k-NN	Linear	k-NN	Linear	k-NN	Linear	k-NN
infoNCE	$53.68\pm 0.69$	$50.35\pm 0.44$	$50.90\pm 0.96$	$48.34\pm 0.77$	$55.23\pm 1.18$	$53.56\pm 0.92$	$51.45\pm 0.46$	$49.88\pm 0.48$
InvaSpread	$48.20\pm 0.92$	$46.61\pm 0.86$	$46.14\pm 0.76$	$42.51\pm 1.15$	$46.85\pm 1.08$	$43.51\pm 0.62$	$43.18\pm 0.52$	$39.66\pm 1.15$
Standard OT	$50.45\pm 0.97$	$46.68\pm 0.88$	$51.78\pm 0.47$	$48.75\pm 1.12$	$55.29\pm 0.70$	$52.92\pm 1.15$	$53.12\pm 0.43$	$51.44\pm 0.79$
Neg-IOT-CL-PushPull (Ours)	$50.88\pm 0.47$	$47.77\pm 0.66$	$51.70\pm 1.18$	$48.56\pm 0.96$	$55.37\pm 0.86$	$52.17\pm 0.65$	$54.35\pm 1.02$	$52.90\pm 1.06$

Appendix

Table 12: UCL results on Tiny-ImageNet with different backbones.

Learning Rate	Linear	k-NN
$5\times 10^{-4}$	79.52	79.40
$1\times 10^{-3}$	82.00	81.92
$5\times 10^{-3}$	82.15	82.66
0.01	80.00	79.85
0.03	77.70	77.81
0.1	75.95	75.04

Appendix

Table 13: Performance on CIFAR-10 of Neg-IOT-CL-PushPull under different hyperparameter settings using ResNet-18 as the backbone in SCL.

Panel of Figure 7 — Figure 7: Efficiency and scaling comparison of OT-based objectives. Left : linear-probe accuracy versus wall-clock seconds per training step for different batch sizes. Points closer to the upper-left indicate a better efficiency–performance tradeoff. Right : peak GPU memory as a function of batch size. Neg-IOT-CL-PushPull stays close to Standard OT in runtime while avoiding the steep runtime and memory growth of Neg-MMIOT-CL

	CIFAR-100				Tiny-ImageNet
SCL	ResNet18	ResNet34	ResNet50	ViT-B/16	ResNet18	ResNet34	ResNet50	ViT-B/16
InfoNCE	3.22	4.40	5.66	13.23	6.44	9.37	11.08	27.68
Standard OT	3.40	4.86	6.46	13.59	6.92	9.35	11.56	28.39
InvaSpread	3.16	4.48	5.85	13.02	6.87	9.49	11.81	28.20
OT-PushPull	3.69	5.62	7.04	13.87	7.69	10.19	12.73	29.64
MMIOT	6.01	7.04	9.64	16.71	10.25	13.39	16.05	35.19

Appendix

Table 14: Average training time (seconds per epoch) for SCL methods on CIFAR-100 and Tiny-ImageNet.

	CIFAR-100				Tiny-ImageNet
UCL	ResNet18	ResNet34	ResNet50	ViT-B/16	ResNet18	ResNet34	ResNet50	ViT-B/16
InfoNCE	71.76	74.05	75.55	86.44	244.79	250.85	277.16	288.26
Standard OT	72.38	76.26	76.37	86.80	231.37	282.39	351.46	286.67
InvaSpread	75.21	80.41	83.78	87.61	257.83	324.82	360.12	287.85
OT-PushPull	73.54	76.28	77.34	87.89	294.22	283.76	351.88	288.30
MMIOT	79.28	82.51	82.30	100.57	273.87	294.03	376.97	300.93

Appendix

Table 15: Average training time (seconds per epoch) for UCL methods on CIFAR-100 and Tiny-ImageNet.

Explore similar work

Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets
May 11, 2026Thuan Nguyen, Shuchin Aeron, D. Richard Brown +1Contrastive Learning Imbalanced Classification
Department of Engineering, Engineering Technology East Tennessee State University · Department of Electrical and Computer Engineering Tufts University · Department of Electrical and Computer Engineering Worcester Polytechnic Institute +1
Positive Pair Geometry Matters: Optimal Transport for Contrastive Learning of Visual Representations
Sep 21, 2026Akshit Nanda, Shahzad Ahmad, Ram Prasad PadhyContrastive Learning Data Augmentation
Department of Computer Science and Engineering, Indian Institute of Technology Bhubaneswar · Østfold University of Applied Sciences, NTNU
Optimal VC Dimension of Contrastive Learning with Margin
Sep 30, 2026Dionysis Arvanitakis, Vaggos Chatziafratis, Yiyuan Luo +1Contrastive Learning Voice Conversion
Northwestern University · UC Santa Cruz

May 11, 2026, cs.LGJ/K move · Enter open · S save

Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets

Thuan Nguyen, Shuchin Aeron, D. Richard Brown, Prakash Ishwar

Department of Engineering, Engineering Technology East Tennessee State University · Department of Electrical and Computer Engineering Tufts University · Department of Electrical and Computer Engineering Worcester Polytechnic Institute+1

Contrastive Learning Imbalanced Classification Neural Collapse Symmetry

Open paper AskPDF arXiv

In this paper, we provide a computable characterization of the geometry of optimal representations in Contrastive Learning (CL) when the classes are imbalanced. When classes are balanced and the representation dimension is greater than the number of classes, it is well-known that the optimal representations exhibit Neural Collapse (NC), i.e., representations from the same class collapse to their class means and the class means form an Equiangular Tight Frame (ETF). For imbalanced classes and a large, generalized family of CL losses, we prove that the optimal representations of all samples from the same class collapse to their class means and their geometry exhibits an angular symmetry structure that is determined by the relative class proportions. In general, we show that the geometry can be determined by solving a convex optimization problem. Exploiting this symmetry structure, we analytically investigate a special case where class imbalance is extreme and prove that CL exhibits a phenomenon called Minority Collapse (MC) where all samples from the minority classes (classes with small probabilities) collapse into a single vector, whenever the class imbalance exceeds a threshold, which in turn depends on the regularity properties of the CL loss used and on the number of negative samples. Numerical results are provided to illustrate these phenomena and corroborate the theoretical results. We conclude by identifying a number of open problems.

Abstract

Explore similar work

Optimal Representations for Generalized Contrastive Learning with Imbalanced Datasets

Positive Pair Geometry Matters: Optimal Transport for Contrastive Learning of Visual Representations

Optimal VC Dimension of Contrastive Learning with Margin