Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity

Naoki Yoshida; Satoshi Hayakawa; Yuhta Takida; Toshimitsu Uesaka; Hiromi Wakaki; Yuki Mitsufuji

cs.LGOct 17, 2025

Theoretical Refinement of CLIP by Utilizing Linear Structure of Optimal Similarity

Authors: Naoki Yoshida, Satoshi Hayakawa, Yuhta Takida, Toshimitsu Uesaka, Hiromi Wakaki, Yuki Mitsufuji

Organizations: The University of Tokyo · Sony Group Corporation · Sony AI

Abstract

In this study, we propose an enhancement to the similarity computation mechanism in multimodal contrastive pretraining frameworks such as CLIP. Prior theoretical research has demonstrated that the optimal similarity metrics between paired modalities should correspond to the pointwise mutual information (PMI) between the two modalities. However, the current implementations of CLIP and its variants fail to fully utilize the underlying linear structure of PMI. We therefore propose KME-CLIP, which leverages this structure through the inner product in a reproducing kernel Hilbert space (RKHS). We theoretically prove that, under our assumptions, the KME-CLIP similarity can bring the contrastive loss arbitrarily close to its optimal value, which is attained by PMI, as the size of the point set grows, and we empirically evaluate KME-CLIP against CLIP and its kernel-based variants across several retrieval and classification tasks.

Figures & tables

		CC3M			MSCOCO			Flickr30K
Text retrieval	Model	Top1	Top5	Top10	Top1	Top5	Top10	Top1	Top5	Top10
CC3M	CLIP	21.95	41.97	51.01	13.64	32.34	43.57	26.46	52.18	62.68
	WPSE	21.39	41.36	50.51	14.78	33.75	44.86	27.22	52.80	63.92
	KME-CLIP	24.22	44.85	53.51	14.14	34.25	45.65	28.18	55.72	65.92
CC12M	CLIP	22.82	42.89	52.22	24.56	48.25	60.15	44.76	73.30	81.94
	WPSE	22.59	42.70	51.94	24.29	48.85	61.07	45.80	73.74	82.40

Table 1 : Text retrieval (top) and image retrieval (bottom) results. We report the top-1, 5, 10 accuracy (%).

	Model	Average	ImageNet	CIFAR-10	CIFAR-100	STL-10	Food-101	Caltech-101	Cars	Aircraft	Flowers	EuroSAT	DTD	Pets	SUN397
CC3M	CLIP	26.33	20.95	56.77	24.38	82.80	14.06	50.07	1.46	1.20	11.64	13.84	13.88	14.10	37.20
	WPSE	27.38	22.17	59.40	30.78	81.64	14.80	50.70	1.27	1.13	14.01	16.20	13.83	12.78	37.27
	KME-CLIP	29.31	22.66	64.31	31.32	85.23	16.17	53.67	1.29	1.04	16.54	17.68	12.39	16.26	42.49
CC12M	CLIP	43.48	36.99	75.41	42.77	92.36	45.97	71.60	17.44	2.48	25.58	31.72	19.47	55.59	47.88
	WPSE	43.37	36.92	74.54	42.22	92.24	43.30	71.53	18.62	3.80	22.32	32.98	19.47	54.52	51.36
	KME-CLIP	45.67	39.07	78.06	46.63	92.74	49.14	76.62	17.52	3.45	25.00	31.28	21.60	60.67	51.94

Table 2 : Zero-shot classification results. Mean per-class accuracy (%) is reported for Caltech-101, Aircraft, Flowers, and Pets datasets. Top-1 accuracy (%) is used for all other datasets.

Image point-set size $m_{\mathcal{X}}$	(CLIP)	2	10	50	100	197
Top-1 accuracy	(23.66)	23.71	23.93	24.35	24.79	25.57

Table 3 : The average of top1 accuracy (%) for text retrieval and image retrieval tasks.

Dataset	Metric	CLIP	KME-CLIP (Bary)	KME-CLIP (LogSim)
MSCOCO	Gap ( $\downarrow$ )	0.508	0.402	–
MSCOCO	Align ( $\uparrow$ )	0.380	0.418	0.494
ImageNet	Gap ( $\downarrow$ )	0.426	0.357	–
ImageNet	Align ( $\uparrow$ )	0.298	0.318	0.408
CIFAR-10	Gap ( $\downarrow$ )	0.826	0.754	–
CIFAR-10	Align ( $\uparrow$ )	0.300	0.278	0.432

Table 4 : Modality-gap and true-pair alignment diagnostics. Gap is computed from vector embeddings, and lower is better. Align is computed by cosine similarity for CLIP and barycentric KME-CLIP, and by the constant-corrected LogSim diagnostic for KME-CLIP; higher is better.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Config	Value
Total epochs	50
Warmup epochs	2
Warmup start learning rate	$10^{-6}$
Warmup end learning rate	$5.0\times 10^{-4}$
End learning rate	$10^{-5}$
AdamW $\beta$	$\beta_{1},\beta_{2}=(0.9,0.98)$

Appendix

Table 5 : Hyper-parameter for pre-training the encoders

	Model	Average	ImageNet	CIFAR-10	CIFAR-100	STL-10	Food-101	Caltech-101	Cars	Aircraft	Flowers	EuroSAT	DTD	Pets	SUN397
CC3M	CLIP	72.29	59.59	88.75	70.33	92.94	67.02	82.34	37.45	44.20	92.02	95.88	66.01	72.88	70.32
	WPSE	74.29	61.34	89.51	70.89	93.63	69.80	85.22	44.35	48.51	92.57	95.10	68.03	75.22	71.64
	KME-CLIP	74.44	62.02	89.73	70.94	93.10	69.76	84.62	45.69	49.28	93.16	95.80	67.82	74.64	71.13
CC12M	CLIP	79.45	67.79	91.23	73.96	95.56	78.79	88.50	66.55	48.77	93.46	95.74	74.41	81.58	76.50
	WPSE	81.17	69.71	91.98	74.78	95.69	79.81	89.93	71.45	57.60	94.93	96.02	72.66	83.62	77.05
	KME-CLIP	78.31	67.76	91.92	75.24	95.41	78.28	88.13	60.22	46.54	93.52	96.16	73.72	74.89	76.20

Appendix

Table 6 : Linear classification results. Mean per-class accuracy (%) is reported for Caltech-101, Aircraft, Flowers, and Pets datasets. Top-1 accuracy (%) is used for all other datasets.

Image point-set size $m_{\mathcal{X}}$	CLIP	2	10	50	100	197
Accuracy (%)	23.66	23.71	23.93	24.35	24.79	25.57
Inference time with two GPUs (sec)	126	138	140	159	186	257

Appendix

Table 7 : Top1 accuracy (%) and inference time required, averaged over text and image retrieval on validation split of CC3M.

Image point-set size $m_{\mathcal{X}}$	CLIP	2	10	50	100	197
Training time (sec/epoch)	1604	1413	1439	925	722	945
Number of GPUs for training	2	2	2	4	8	8

Appendix

Table 8 : Number of H100 GPUs and time required for training by KME-CLIP and CLIP on CC3M.

	CC3M			MSCOCO			Flickr30K
Model	Top1	Top5	Top10	Top1	Top5	Top10	Top1	Top5	Top10
Text retrieval
CLIP	$19.17\pm 0.75$	$38.81\pm 0.51$	$48.28\pm 0.37$	$13.16\pm 0.46$	$\underline{32.02\pm 0.71}$	$\underline{43.10\pm 0.86}$	$25.35\pm 0.51$	$51.62\pm 0.36$	$63.31\pm 0.10$
WPSE	$18.60\pm 0.87$	$38.31\pm 1.15$	$47.63\pm 1.38$	$\underline{13.26\pm 1.03}$	$31.37\pm 1.71$	$42.22\pm 1.86$	$24.48\pm 2.29$	$50.64\pm 2.14$	$61.75\pm 1.93$
KME-CLIP	$\underline{20.60\pm 0.83}$	$\underline{40.90\pm 1.02}$	$\underline{50.55\pm 1.33}$	$12.60\pm 0.16$	$31.44\pm 0.21$	$42.84\pm 0.34$	$\underline{26.16\pm 0.68}$	$\underline{53.09\pm 0.68}$	$\underline{64.02\pm 0.09}$
KME-no-log	$\mathbf{20.86\pm 0.22}$	$\mathbf{41.23\pm 0.92}$	$\mathbf{51.02\pm 0.83}$	$\mathbf{16.02\pm 0.65}$	$\mathbf{36.03\pm 0.74}$	$\mathbf{47.17\pm 0.92}$	$\mathbf{30.32\pm 0.57}$	$\mathbf{56.49\pm 0.76}$	$\mathbf{67.67\pm 0.63}$

Appendix

Table 9 : Matched retrieval results after CC3M pretraining with global batch size

512

. Values are percentages reported as mean

\pm

sample standard deviation over three seeds. Bold indicates the highest mean and underlining indicates the second-highest mean in each column. Text retrieval denotes image-to-text retrieval, and image retrieval denotes text-to-image retrieval.

Dataset	CLIP	WPSE	KME-CLIP	KME-no-log
Average	$26.89\pm 0.86$	$27.24\pm 1.73$	$\underline{28.68\pm 0.96}$	$\mathbf{29.18\pm 0.17}$
ImageNet	$19.61\pm 0.27$	$19.50\pm 1.23$	$\underline{21.16\pm 0.32}$	$\mathbf{22.72\pm 0.32}$
CIFAR-10	$60.75\pm 6.26$	$60.20\pm 7.53$	$\underline{61.79\pm 5.89}$	$\mathbf{64.48\pm 2.89}$
CIFAR-100	$27.88\pm 0.67$	$26.98\pm 2.20$	$\underline{29.89\pm 2.28}$	$\mathbf{32.66\pm 1.26}$
STL-10	$84.60\pm 0.49$	$\underline{84.73\pm 0.28}$	$84.30\pm 1.99$	$\mathbf{84.92\pm 1.34}$
Food-101	$13.15\pm 0.99$	$13.16\pm 0.78$	$\mathbf{15.15\pm 1.05}$	$\underline{14.72\pm 1.55}$

Appendix

Table 10 : Matched zero-shot classification results after CC3M pretraining with global batch size

512

. Values are percentages reported as mean

\pm

sample standard deviation over three seeds. Mean per-class accuracy is used for Caltech-101, Aircraft, Flowers, and Pets; top-1 accuracy is used otherwise. Average is computed over the 13 datasets within each seed before aggregation across seeds. Bold indicates the highest mean and underlining indicates the second-highest mean in each row.

Conditional KL
	Final embedding dimension $d$
$\eta$	$2$	$4$	$8$	$16$
$0.1$	$-0.397\pm 0.003$	$-0.030\pm 0.005$	$+0.095\pm 0.004$	$+0.109\pm 0.005$
$0.03$	$-0.373\pm 0.008$	$+0.054\pm 0.006$	$+0.168\pm 0.004$	$+0.181\pm 0.005$
$0.01$	$-0.355\pm 0.007$	$+0.080\pm 0.006$	$+0.193\pm 0.004$	$+0.207\pm 0.005$
$0.003$	$-0.348\pm 0.003$	$+0.088\pm 0.007$	$+0.203\pm 0.005$	$+0.217\pm 0.005$

Appendix

Table 11 : Contamination sweep at

N=32

: within-seed paired difference (KME-no-log minus KME-CLIP), reported as mean

\pm

sample standard deviation over three seeds. Positive values favor KME-CLIP. The paired difference is computed seed by seed before aggregation.

$d$	$\eta$	CLIP	WPSE	KME-no-log	KME-CLIP	Bayes
$4$	$0.1$	$12.40\pm 0.26$	$16.16\pm 0.34$	$17.04\pm 1.47$	$18.73\pm 0.33$	$18.82\pm 0.10$
	$0.03$	$13.25\pm 0.40$	$18.21\pm 1.03$	$18.91\pm 1.24$	$20.17\pm 0.18$	$20.44\pm 0.49$
	$0.01$	$14.13\pm 0.54$	$18.83\pm 0.82$	$18.98\pm 1.98$	$20.72\pm 0.12$	$20.87\pm 0.60$
	$0.003$	$14.21\pm 0.22$	$18.46\pm 0.82$	$18.72\pm 1.03$	$20.74\pm 0.12$	$20.95\pm 0.56$
$8$	$0.1$	$16.73\pm 0.25$	$16.76\pm 0.58$	$17.69\pm 0.34$	$18.57\pm 0.69$	$18.82\pm 0.10$
	$0.03$	$18.36\pm 0.63$	$17.95\pm 0.33$	$18.88\pm 0.79$	$20.39\pm 0.38$	$20.44\pm 0.49$

Appendix

Table 12 : Hard retrieval at

N=32

with one held-out positive and

1{,}023

marginal negatives. Symmetric top-10 accuracy (%), reported as mean

\pm

sample standard deviation over three seeds. “Bayes” is the analytic density-ratio ranker and gives the attainable ceiling under this protocol.

Explore similar work

UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels
Apr 17, 2026Hangke Sui, Yuqing Wang, Minh N DoContrastive Alignment Large Multimodal Models
Department of Electrical & Computer Engineering, The Grainger College of Engineering, UIUC · Coordinated Science Laboratory, University of Illinois Urbana-Champaign (UIUC) · Siebel School of Computing and Data Science, The Grainger College of Engineering, UIUC +1
KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models
Jun 2, 2026Youqi Wu, Mohammad Jalali, Farzan FarniaContrastive Learning Vision-Language Foundation Models
Department of Computer Science and Engineering, The Chinese University of Hong Kong.
CLIP-RD: Relational Distillation for Efficient CLIP Knowledge Distillation
Date pendingJeannie Chung, Hanna Jang, Ingyeong Yang +2Contrastive Language-Image Pre-Training Model Similarity and Metrics
Ewha Womans University, Seoul 03760, South Korea

Apr 17, 2026, cs.LGJ/K move · Enter open · S save

UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels

Hangke Sui, Yuqing Wang, Minh N Do

Department of Electrical & Computer Engineering, The Grainger College of Engineering, UIUC · Coordinated Science Laboratory, University of Illinois Urbana-Champaign (UIUC) · Siebel School of Computing and Data Science, The Grainger College of Engineering, UIUC+1

Contrastive Alignment Large Multimodal Models Unified Framework Similarity and Metrics Kernel Hilbert Spaces

Open paper AskPDF arXiv

Contrastive objectives power state-of-the-art multimodal models, but their training remains slow, relying on long stochastic optimization. We propose a Unified Framework for Efficient Contrastive Alignment via Kernels (UniCon), which spans linear and nonlinear encoders as well as one-to-one and many-to-many alignments. At its core, UniCon introduces the contrastive similarity weight matrix

S(γ)

, which enables closed-form global solutions that provably replace minibatch back-propagation with exact updates. Through the lens of reproducing kernel Hilbert spaces (RKHS), UniCon provides a kernelized perspective that unifies contrastive alignment and reveals its connection to spectral methods. To validate the theory, we conduct experiments on synthetic, unimodal, multimodal, and zero-shot tasks, demonstrating that UniCon achieves substantial efficiency gains while preserving generality and strong empirical performance.

Abstract

Explore similar work

UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels

KODA: Contrastive Representation Comparison and Alignment for Vision-Language Foundation Models

CLIP-RD: Relational Distillation for Efficient CLIP Knowledge Distillation