Contrastive learning has become a fundamental approach in both uni-modal and multi-modal frameworks. This learning paradigm pulls positive pairs of samples closer while pushing negatives apart. In the uni-modal setting (e.g., image-based learning), previous research has shown that the strength of these forces can be controlled through the temperature parameter. In this work, we propose Multi-Modal Temperature and Margin Schedules (MM-TS), extending the concept of uni-modal temperature scheduling to multi-modal contrastive learning. Our method dynamically adjusts the temperature in the contrastive loss during training, modulating the attraction and repulsion forces in the multi-modal setting. Additionally, recognizing that standard multi-modal datasets often follow imbalanced, long-tail distributions, we adapt the temperature based on the local distribution of each training sample. Specifically, samples from dense clusters are assigned a higher temperature to better preserve their semantic structure. Furthermore, we demonstrate that temperature scheduling can be effectively integrated within a max-margin framework, thereby unifying the two predominant approaches in multi-modal contrastive learning: InfoNCE loss and max-margin objective. We evaluate our approach on four widely used image- and video-language datasets, Flickr30K, MSCOCO, EPIC-KITCHENS-100, and YouCook2, and show that our dynamic temperature and margin schedules improve performance and lead to new state-of-the-art results in the field.
Figures & tables
Figure 1: Individual impact of negatives to the loss depends on the temperature parameter. With the small temperature, only hard negatives impact the loss, whereas impact of easy negatives is negligible. With the large temperature impact of easy negatives to the loss increases.
Figure 2: Visualization of our MM-TS approach for computing the InfoNCE loss. First, we cluster text annotations to estimate the distribution of the training data. For each cluster, we assign the respective cluster-based shift ( sh(c) ), where to the larger cluster we assign a larger shift. The base temperature τbase follows the cosine schedule. Next, we adjust the temperature for each individual sample based on the estimate cluster-based shifts resulting in τi . The updated individual temperature τi is used in a standard InfoNCE loss.
Figure 3: Detailed visualization of temperature calculation for every cluster c on every training iteration t based on the cluster distribution. Given the cosine schedule for the base temperature τbase amplitude α , oscillation period T and cluster-based shifts sh(c) , we calculate temperature for each cluster.
Figure 4
Figure 6: Qualitative samples from different estimated clusters for CC3M. The clusters are estimated using K-Means clustering. The corresponding cluster sizes from left to right are: 69669, 13708, 2904. With yellow we highlight manually identified topic for the corresponding cluster. For more text cluster examples see supmat.
Method
Back-
Flickr30K
MSCOCO
bone
IR@1
TR@1
Avg.
IR@1
TR@1
Avg.
CLIP [ 42 ]
RN50
40.9
50.9
45.9
21.3
26.9
24.1
CLIP + Ours
RN50
41.5
54.3
47.9
21.2
28.4
24.8
Table 1: Performance comparison of pretraining on CC3M and zero-shot evaluation on MSCOCO and Flickr30K. We combine CLIP with our method.
Method
Back-
mAP
nDCG
bone
V → T
T → V
Avg.
V → T
T → V
Avg.
MME [ 50 ]
TBN
43.0
34.0
38.5
50.1
46.9
48.5
JPoSE [ 50 ]
TBN
49.9
38.1
44.1
55.5
51.6
53.5
EgoVLP [ 26 ]
TSF-B
49.9
40.5
45.0
60.9
57.9
59.4
LaViLa [ 59 ]
TSF-B
55.2
45.7
50.5
66.5
63.4
65.0
AVION [ 58 ]
ViT-B
55.7
48.2
52.0
67.8
65.3
66.5
Table 2: Performance comparison of MIR on EPIC-KITCHENS-100 dataset. We combine AVION with our method.
Method
Text-to-Video
R@1
R@5
R@10
UniVL [ 32 ]
28.9
57.6
70.0
MELTR [ 22 ]
33.7
63.1
74.8
VLM [ 51 ]
27.1
56.9
69.4
VAST [ 1 ]
50.4
74.3
80.8
VAST + Ours
53.0
77.1
84.5
Table 3: Performance comparison on the YouCook2 dataset. We combine VAST with our method. The evaluation is done with Text-to-Video Recall@1,5,10. All benchmarks are multi-modal, containing both audio and subtitles. Methods that also incorporate audio and subtitle modalities (in addition to vision for video representation) are highlighted with a gray background.
Method
mAP
nDCG
V → T
T → V
Avg.
V → T
T → V
Avg.
MI-MM
55.3
47.6
51.4
67.6
64.8
66.2
w/ TS
58.4
48.6
53.6
69.4
64.8
66.9
w/ ICS
56.4
48.4
52.4
68.2
66.0
67.1
w/ TS&ICS
58.8
48.9
53.9
68.9
65.8
67.3
CLIP
47.0
39.8
43.1
64.4
61.2
62.8
Table 4: Performance comparison of AVION on EK-100 dataset (MIR) with modified max-margin- (MI-MM) and CLIP- (InfoNCE) losses. TS and ICS temperature (or margin) schedule and individual cluster shifts, respectively.
Distribution source
mAP
nDCG
V → T
T → V
Avg.
V → T
T → V
Avg.
verbs & nouns
58.2
48.7
53.4
68.3
65.5
66.9
verbs
58.5
48.7
53.6
68.0
65.3
66.7
nouns
58.8
48.9
53.9
68.9
65.3
67.3
Table 5: Performance comparison of AVION with margin distribution method based on the distribution of uniq verbs+nouns combinations, verbs and nouns
R@1
R@5
R@10
VAST (baseline)
50.4
74.3
80.8
S-BERT(text)
53.0
77.1
84.5
VAST-v(video)
52.2
76.3
84.0
CLIP-t(text)
52.5
76.4
84.4
Table 6: Comparison of retrieval performance on YouCook2 using distribution modulation based on text (S-BERT, CLIP-t) and video (VAST-v) clusters.
Method
Text-to-Video
Video-to-Text
R@1
R@5
R@10
R@1
R@5
R@10
VAST [ 1 ]
41.7
69.1
78.9
41.2
69.3
79.3
w/ TS
42.6
70.0
79.5
41.3
68.7
79.3
w/ TD
42.7
70.3
79.6
41.2
69.7
79.5
w/ TS&TD
43.2
70.3
79.7
41.8
70.3
80.0
Table 7: Performance comparison of variants of VAST on finetuning on SSv2-LT dataset. Results without refinement.
Table 8: Impact of different temperature ranges ( τmin represented with τ↓ and τmax represented with τ↑ ) and amplitude ( α ) values on the performance of VAST on SSv2-LT, evaluating R@1 Text-to-Video. The combinations marked with “-” were not done as the resulting lowest temperature for the smallest class would be ≤ 0.
Figure 14
Figure 9: Visualizations of long-tail class distributions of Epic-Kitchens100 dataset training split. Class distributions are calculated based on verbs (top-left), important nouns (top-right) and unique combinations of verbs and nouns (bottom).
Figure 10: Visualization of long-tail annotations distribution of YouCook2 dataset. Annotations distribution is calculated based on k-mean clustering (200 clusters) of the annotation embeddings. Annotation embeddings are generated using SentenceBERT model [ 43 ] .
Figure 11: Visualization of action class distribution of SSv2-LT dataset.
Method
Text-to-Video
Video-to-Text
R@1
R@5
R@10
R@1
R@5
R@10
without refinement
CLIP
41.4
65.0
74.6
42.6
66.8
75.4
w/ TS
43.2
67.8
77.2
44.2
70.5
79.1
w/ ICS
41.0
65.2
74.5
42.1
66.9
75.0
w/ TS&ICS
43.1
67.8
77.3
44.4
70.4
79.2
Table 9: Performance comparison of VAST on YouCook2 dataset (Video retrieval) with modified CLIP and MI-MM losses. TS and ICS temperature (or margin) schedule and individual cluster shifts, respectively. ”*”- marks our reproduction based on VAST codebase.
Cluster
Example Text Annotations
Topic
#1 106 samples
pour the egg and add a pinch of salt and red chili powder add cooking oil and beat an egg mix the eggs with salt and pepper add red pepper flakes to a bowl of eggs and whisk
eggs
#2 106 samples
add coconut milk fish sauce and soy sauce into the pan add rice vinegar soy sauce and sriracha to the pan add oil and green chilies to pan add garlic thyme bay leaf and tomato paste to the pan
pan
#3 106 samples
pour the mixture on hash browns spread the mixture loosely on the cooking sheet with olive oil and place in the oven mix the ingredients mix everything and let it cook
mixture
#4 101 samples
mix flour with the potato mixture add an egg and farmers cheese to the potatoes and mash add butter and milk to the pot and mash the potatoes add the mashed potatoes butter cream and kale to a pot and mix
potatoes
#5 92 samples
add chopped garlic chopped ginger and chopped onion add ginger garlic and onions sprinkle paprika and parsley on top mix red chili sugar garlic fish sauce and scallians with the radish
garlic
…
…
…
Table 10: Examples of text annotations (4 per cluster) from 5 biggest clusters (#1-#5) and 5 smallest clusters (#196-#200), obtained via kMeans clustering on SentenceBERT embeddings of training split of YouCook2 dataset.
Cluster
Example Text Annotations
Topic
#1 69669 samples
scenes of people working in the office a very long ride with the people image taken from page of people index : a list of the topics and people in the text
people
#2 54898 samples
students are looking forward to visiting public university students have fun while they learn about the environment students prepare for a perennial lesson team up to help keep students safe
students
#3 38840 samples
football player was praised after a sensational session between the sticks on tuesday football player is welcomed back after winning soccer league american football player will put his body on the line to help his team football player is the latest player to be linked with a move away from the club
football player
#4 35297 samples
actor attends the european premiere of biographical film film director attends the premiere film director attends premiere held at theater during festival tv writer at the premiere of film
film
#5 34039 samples
the truth about christmas vietnam merry christmas eve to all my wonderful followers … and to everyone else as well this would be cute to hang on the front door at christmas time things you have to do this christmas
christmas
Table 11: Examples of text annotations (4 per cluster) for 5 largest clusters, obtained via K-Means clustering on text embeddings generated using a distilled version of BERT on the training split of CC3M dataset.
Cluster
Example Text Annotations
Topic
#196 3736 samples
man with glasses looking himself in a mirror isolated on white background healthy fit young man measuring his waist with a tape measure to monitor his weight isolated on white background senior man and a kid passing a football isolated on white background portrait of a smiling young man isolated on white background
man on a white background
#197 3676 samples
silhouette of young boy playing cricket against a sunset background trees silhouetted against a sunset silhouette of palm trees during sunset at the beach silhouette of a dead tree with a sunset in the background
silhouette of a tree, sunset
#198 3105 samples
images from the girls basketball game images from the boys basketball game images vs. girls basketball game on thursday , february images from the girls basketball game in a city
basketball
#199 3105 samples
beautiful golden pink shining metal style uppercase or capital letter e in a 3d illustration with a shiny metallic soft purple red color classic font isolated on a white background with clipping path shiny purple metal lowercase or small letter w in a 3d illustration with a rough weathered metallic surface texture and classic font style isolated on a white background with clipping path shiny metal silver chrome beveled lowercase or small letter n in a 3d illustration with a glossy gray smooth metallic surface finish isolated on a white background with clipping path shiny gold metallic uppercase or capital letter v in a 3d illustration with a rich golden color and glossy smooth metal surface finish in a bold font isolated on a white background with clipping path
metallic letters
#200 2904 samples
sketches of cupcakes drawn by chalk on a blackboard teacher stands at the blackboard in classroom little boy to write with chalk on the school blackboard white chalk texture vintage stamp with map on a school blackboard schoolboy standing in front of a blackboard
blackboard
Table 12: Examples of text annotations (4 per cluster) for 5 smallest clusters, obtained via K-Means clustering on text embeddings generated using a distilled version of BERT on the training split of CC3M dataset.
Multimodal contrastive learning aligns representations from different data types so models can connect related images, text, audio, signals, and other modalities.
Contrastive learning approaches achieve strong performance by training models to bring similar samples closer while pushing dissimilar samples apart. A crucial component of contrastive learning is the temperature hyperparameter τ, which controls the penalty strength applied to negative samples. However, most existing methods either fix this hyperparameter or learn a global value during training. In this paper, we introduce TeMo, Temperature Modulation framework, a similarity-based modulation approach that adaptively adjusts the temperature for each positive-negative pair according to their similarity, enabling more fine-grained multimodal contrastive learning. Our approach seamlessly integrates temperature-modulated multimodal and unimodal losses with the standard multimodal contrastive loss by gradually transitioning between them. This design allows the model to capture both coarse- and fine-grained semantics at different training stages. Extensive experiments demonstrate that each component of TeMo consistently enhances performance across diverse zero-shot retrieval and classification tasks, establishing new state-of-the-art results.
Dhimitrios Duka, Bernt Schiele, Hilde Kuehne +1
MPI for Informatics, SIC · Tuebingen AI Center/University of Tuebingen · MIT-IBM Watson AI Lab
Memorization in machine learning models enables high performance on rare in-distribution samples by capturing their atypical patterns. However, it also causes harmful retention of noise and outliers, degrading generalization. While memorization has been extensively studied in both supervised and self-supervised learning in the vision domain, it remains unexplored in multi-modal contrastive learning. We address this gap by introducing MultiMem, the first metric designed to quantify memorization in multi-modal contrastive learning. Through our systematic analysis, we demonstrate that cross-modal semantic misalignment has the strongest influence on memorization, with text being the dominant modality driving memorization, followed by video, image, and audio. We show that targeted augmentations applied across all modalities effectively reduce memorization as measured by our MultiMem metric and improve model performance. Overall, this work establishes the first framework for measuring and mitigating memorization in multi-modal contrastive learning, preventing harmful data retention and contributing to higher-performing models.
Wenhao Wang, Franziska Boenisch, Michael Backes +1
Contrastive objectives power state-of-the-art multimodal models, but their training remains slow, relying on long stochastic optimization. We propose a Unified Framework for Efficient Contrastive Alignment via Kernels (UniCon), which spans linear and nonlinear encoders as well as one-to-one and many-to-many alignments. At its core, UniCon introduces the contrastive similarity weight matrix S(γ), which enables closed-form global solutions that provably replace minibatch back-propagation with exact updates. Through the lens of reproducing kernel Hilbert spaces (RKHS), UniCon provides a kernelized perspective that unifies contrastive alignment and reveals its connection to spectral methods. To validate the theory, we conduct experiments on synthetic, unimodal, multimodal, and zero-shot tasks, demonstrating that UniCon achieves substantial efficiency gains while preserving generality and strong empirical performance.
Hangke Sui, Yuqing Wang, Minh N Do
Department of Electrical & Computer Engineering, The Grainger College of Engineering, UIUC · Coordinated Science Laboratory, University of Illinois Urbana-Champaign (UIUC) · Siebel School of Computing and Data Science, The Grainger College of Engineering, UIUC +1