MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data

Siarhei Sheludzko; Dhimitrios Duka; Bernt Schiele; Hilde Kuehne; Anna Kukleva

cs.CVMar 9, 2026

MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data

Authors: Siarhei Sheludzko, Dhimitrios Duka, Bernt Schiele, Hilde Kuehne, Anna Kukleva

Organizations: University of Bonn · MPI for Informatics, SIC · Tuebingen AI Center/University of Tuebingen · MIT-IBM Watson AI Lab

Abstract

Contrastive learning has become a fundamental approach in both uni-modal and multi-modal frameworks. This learning paradigm pulls positive pairs of samples closer while pushing negatives apart. In the uni-modal setting (e.g., image-based learning), previous research has shown that the strength of these forces can be controlled through the temperature parameter. In this work, we propose Multi-Modal Temperature and Margin Schedules (MM-TS), extending the concept of uni-modal temperature scheduling to multi-modal contrastive learning. Our method dynamically adjusts the temperature in the contrastive loss during training, modulating the attraction and repulsion forces in the multi-modal setting. Additionally, recognizing that standard multi-modal datasets often follow imbalanced, long-tail distributions, we adapt the temperature based on the local distribution of each training sample. Specifically, samples from dense clusters are assigned a higher temperature to better preserve their semantic structure. Furthermore, we demonstrate that temperature scheduling can be effectively integrated within a max-margin framework, thereby unifying the two predominant approaches in multi-modal contrastive learning: InfoNCE loss and max-margin objective. We evaluate our approach on four widely used image- and video-language datasets, Flickr30K, MSCOCO, EPIC-KITCHENS-100, and YouCook2, and show that our dynamic temperature and margin schedules improve performance and lead to new state-of-the-art results in the field.

Figures & tables

Figure 1: Individual impact of negatives to the loss depends on the temperature parameter. With the small temperature, only hard negatives impact the loss, whereas impact of easy negatives is negligible. With the large temperature impact of easy negatives to the loss increases.

Figure 2: Visualization of our MM-TS approach for computing the InfoNCE loss. First, we cluster text annotations to estimate the distribution of the training data. For each cluster, we assign the respective cluster-based shift ( $sh(c)$ ), where to the larger cluster we assign a larger shift. The base temperature $\tau_{base}$ follows the cosine schedule. Next, we adjust the temperature for each individual sample based on the estimate cluster-based shifts resulting in $\tau_{i}$ . The updated individual temperature $\tau_{i}$ is used in a standard InfoNCE loss.

Figure 3: Detailed visualization of temperature calculation for every cluster

c

on every training iteration

t

based on the cluster distribution. Given the cosine schedule for the base temperature

\tau_{base}

amplitude

\alpha

, oscillation period

T

and cluster-based shifts

sh(c)

, we calculate temperature for each cluster.

Panel of Figure 6 — Figure 6: Qualitative samples from different estimated clusters for CC3M. The clusters are estimated using K-Means clustering. The corresponding cluster sizes from left to right are: 69669, 13708, 2904. With yellow we highlight manually identified topic for the corresponding cluster. For more text cluster examples see supmat.

Method	Back-	Flickr30K			MSCOCO
Method	bone	IR@1	TR@1	Avg.	IR@1	TR@1	Avg.
CLIP [ 42 ]	RN50	40.9	50.9	45.9	21.3	26.9	24.1
CLIP + Ours	RN50	41.5	54.3	47.9	21.2	28.4	24.8

Table 1: Performance comparison of pretraining on CC3M and zero-shot evaluation on MSCOCO and Flickr30K. We combine CLIP with our method.

Method	Back-	mAP			nDCG
Method	bone	V $\rightarrow$ T	T $\rightarrow$ V	Avg.	V $\rightarrow$ T	T $\rightarrow$ V	Avg.
MME [ 50 ]	TBN	43.0	34.0	38.5	50.1	46.9	48.5
JPoSE [ 50 ]	TBN	49.9	38.1	44.1	55.5	51.6	53.5
EgoVLP [ 26 ]	TSF-B	49.9	40.5	45.0	60.9	57.9	59.4
LaViLa [ 59 ]	TSF-B	55.2	45.7	50.5	66.5	63.4	65.0
AVION [ 58 ]	ViT-B	55.7	48.2	52.0	67.8	65.3	66.5

Table 2: Performance comparison of MIR on EPIC-KITCHENS-100 dataset. We combine AVION with our method.

Method	Text-to-Video
Method	R@1	R@5	R@10
UniVL [ 32 ]	28.9	57.6	70.0
MELTR [ 22 ]	33.7	63.1	74.8
VLM [ 51 ]	27.1	56.9	69.4
VAST [ 1 ]	50.4	74.3	80.8
VAST + Ours	53.0	77.1	84.5

Table 3: Performance comparison on the YouCook2 dataset. We combine VAST with our method. The evaluation is done with Text-to-Video Recall@1,5,10. All benchmarks are multi-modal, containing both audio and subtitles. Methods that also incorporate audio and subtitle modalities (in addition to vision for video representation) are highlighted with a gray background.

Method	mAP			nDCG
Method	V $\rightarrow$ T	T $\rightarrow$ V	Avg.	V $\rightarrow$ T	T $\rightarrow$ V	Avg.
MI-MM	55.3	47.6	51.4	67.6	64.8	66.2
w/ TS	58.4	48.6	53.6	69.4	64.8	66.9
w/ ICS	56.4	48.4	52.4	68.2	66.0	67.1
w/ TS&ICS	58.8	48.9	53.9	68.9	65.8	67.3
CLIP	47.0	39.8	43.1	64.4	61.2	62.8

Table 4: Performance comparison of AVION on EK-100 dataset (MIR) with modified max-margin- (MI-MM) and CLIP- (InfoNCE) losses. TS and ICS temperature (or margin) schedule and individual cluster shifts, respectively.

Distribution source	mAP			nDCG
Distribution source	V $\rightarrow$ T	T $\rightarrow$ V	Avg.	V $\rightarrow$ T	T $\rightarrow$ V	Avg.
verbs & nouns	58.2	48.7	53.4	68.3	65.5	66.9
verbs	58.5	48.7	53.6	68.0	65.3	66.7
nouns	58.8	48.9	53.9	68.9	65.3	67.3

Table 5: Performance comparison of AVION with margin distribution method based on the distribution of uniq verbs+nouns combinations, verbs and nouns

	R@1	R@5	R@10
VAST (baseline)	50.4	74.3	80.8
S-BERT(text)	53.0	77.1	84.5
VAST-v(video)	52.2	76.3	84.0
CLIP-t(text)	52.5	76.4	84.4

Table 6: Comparison of retrieval performance on YouCook2 using distribution modulation based on text (S-BERT, CLIP-t) and video (VAST-v) clusters.

Method	Text-to-Video			Video-to-Text
Method	R@1	R@5	R@10	R@1	R@5	R@10
VAST [ 1 ]	41.7	69.1	78.9	41.2	69.3	79.3
w/ TS	42.6	70.0	79.5	41.3	68.7	79.3
w/ TD	42.7	70.3	79.6	41.2	69.7	79.5
w/ TS&TD	43.2	70.3	79.7	41.8	70.3	80.0

Table 7: Performance comparison of variants of VAST on finetuning on SSv2-LT dataset. Results without refinement.

Panel of Table 8 — Table 8: Impact of different temperature ranges ( $\tau_{min}$ represented with $\tau_{\downarrow}$ and $\tau_{max}$ represented with $\tau_{\uparrow}$ ) and amplitude ( $\alpha$ ) values on the performance of VAST on SSv2-LT, evaluating R@1 Text-to-Video. The combinations marked with “-” were not done as the resulting lowest temperature for the smallest class would be $\leq$ 0.

Figure 9: Visualizations of long-tail class distributions of Epic-Kitchens100 dataset training split. Class distributions are calculated based on verbs (top-left), important nouns (top-right) and unique combinations of verbs and nouns (bottom).

Figure 10: Visualization of long-tail annotations distribution of YouCook2 dataset. Annotations distribution is calculated based on k-mean clustering (200 clusters) of the annotation embeddings. Annotation embeddings are generated using SentenceBERT model [ 43 ] .

Figure 11: Visualization of action class distribution of SSv2-LT dataset.

Method	Text-to-Video			Video-to-Text
Method	R@1	R@5	R@10	R@1	R@5	R@10
without refinement
CLIP	41.4	65.0	74.6	42.6	66.8	75.4
w/ TS	43.2	67.8	77.2	44.2	70.5	79.1
w/ ICS	41.0	65.2	74.5	42.1	66.9	75.0
w/ TS&ICS	43.1	67.8	77.3	44.4	70.4	79.2

Table 9: Performance comparison of VAST on YouCook2 dataset (Video retrieval) with modified CLIP and MI-MM losses. TS and ICS temperature (or margin) schedule and individual cluster shifts, respectively. ”*”- marks our reproduction based on VAST codebase.

Cluster	Example Text Annotations	Topic
#1 106 samples	pour the egg and add a pinch of salt and red chili powder add cooking oil and beat an egg mix the eggs with salt and pepper add red pepper flakes to a bowl of eggs and whisk	eggs
#2 106 samples	add coconut milk fish sauce and soy sauce into the pan add rice vinegar soy sauce and sriracha to the pan add oil and green chilies to pan add garlic thyme bay leaf and tomato paste to the pan	pan
#3 106 samples	pour the mixture on hash browns spread the mixture loosely on the cooking sheet with olive oil and place in the oven mix the ingredients mix everything and let it cook	mixture
#4 101 samples	mix flour with the potato mixture add an egg and farmers cheese to the potatoes and mash add butter and milk to the pot and mash the potatoes add the mashed potatoes butter cream and kale to a pot and mix	potatoes
#5 92 samples	add chopped garlic chopped ginger and chopped onion add ginger garlic and onions sprinkle paprika and parsley on top mix red chili sugar garlic fish sauce and scallians with the radish	garlic
…	…	…

Table 10: Examples of text annotations (4 per cluster) from 5 biggest clusters (#1-#5) and 5 smallest clusters (#196-#200), obtained via kMeans clustering on SentenceBERT embeddings of training split of YouCook2 dataset.

Cluster	Example Text Annotations	Topic
#1 69669 samples	scenes of people working in the office a very long ride with the people image taken from page of people index : a list of the topics and people in the text	people
#2 54898 samples	students are looking forward to visiting public university students have fun while they learn about the environment students prepare for a perennial lesson team up to help keep students safe	students
#3 38840 samples	football player was praised after a sensational session between the sticks on tuesday football player is welcomed back after winning soccer league american football player will put his body on the line to help his team football player is the latest player to be linked with a move away from the club	football player
#4 35297 samples	actor attends the european premiere of biographical film film director attends the premiere film director attends premiere held at theater during festival tv writer at the premiere of film	film
#5 34039 samples	the truth about christmas vietnam merry christmas eve to all my wonderful followers … and to everyone else as well this would be cute to hang on the front door at christmas time things you have to do this christmas	christmas

Table 11: Examples of text annotations (4 per cluster) for 5 largest clusters, obtained via K-Means clustering on text embeddings generated using a distilled version of BERT on the training split of CC3M dataset.

Cluster	Example Text Annotations	Topic
#196 3736 samples	man with glasses looking himself in a mirror isolated on white background healthy fit young man measuring his waist with a tape measure to monitor his weight isolated on white background senior man and a kid passing a football isolated on white background portrait of a smiling young man isolated on white background	man on a white background
#197 3676 samples	silhouette of young boy playing cricket against a sunset background trees silhouetted against a sunset silhouette of palm trees during sunset at the beach silhouette of a dead tree with a sunset in the background	silhouette of a tree, sunset
#198 3105 samples	images from the girls basketball game images from the boys basketball game images vs. girls basketball game on thursday , february images from the girls basketball game in a city	basketball
#199 3105 samples	beautiful golden pink shining metal style uppercase or capital letter e in a 3d illustration with a shiny metallic soft purple red color classic font isolated on a white background with clipping path shiny purple metal lowercase or small letter w in a 3d illustration with a rough weathered metallic surface texture and classic font style isolated on a white background with clipping path shiny metal silver chrome beveled lowercase or small letter n in a 3d illustration with a glossy gray smooth metallic surface finish isolated on a white background with clipping path shiny gold metallic uppercase or capital letter v in a 3d illustration with a rich golden color and glossy smooth metal surface finish in a bold font isolated on a white background with clipping path	metallic letters
#200 2904 samples	sketches of cupcakes drawn by chalk on a blackboard teacher stands at the blackboard in classroom little boy to write with chalk on the school blackboard white chalk texture vintage stamp with map on a school blackboard schoolboy standing in front of a blackboard	blackboard

Table 12: Examples of text annotations (4 per cluster) for 5 smallest clusters, obtained via K-Means clustering on text embeddings generated using a distilled version of BERT on the training split of CC3M dataset.

Multimodal Contrastive Learning

Multimodal contrastive learning aligns representations from different data types so models can connect related images, text, audio, signals, and other modalities.

See all 246 papers

TeMo: Temperature Modulation for Multimodal Contrastive Learning
Sep 7, 2026Dhimitrios Duka, Bernt Schiele, Hilde Kuehne +1Cross-Modal Retrieval Multimodal Classification
MPI for Informatics, SIC · Tuebingen AI Center/University of Tuebingen · MIT-IBM Watson AI Lab
MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning
Jun 20, 2026Wenhao Wang, Franziska Boenisch, Michael Backes +1Multimodal Contrastive Learning
CISPA Helmholtz Center for Information Security
UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels
Apr 17, 2026Hangke Sui, Yuqing Wang, Minh N DoContrastive Learning Cross-Modal Alignment
Department of Electrical & Computer Engineering, The Grainger College of Engineering, UIUC · Coordinated Science Laboratory, University of Illinois Urbana-Champaign (UIUC) · Siebel School of Computing and Data Science, The Grainger College of Engineering, UIUC +1

Sep 7, 2026J/K move · Enter open · S save

TeMo: Temperature Modulation for Multimodal Contrastive Learning

Dhimitrios Duka, Bernt Schiele, Hilde Kuehne, Anna Kukleva

MPI for Informatics, SIC · Tuebingen AI Center/University of Tuebingen · MIT-IBM Watson AI Lab

Cross-Modal Retrieval Multimodal Classification Multimodal Contrastive Learning

Open paper AskPDF arXiv

Contrastive learning approaches achieve strong performance by training models to bring similar samples closer while pushing dissimilar samples apart. A crucial component of contrastive learning is the temperature hyperparameter

τ

, which controls the penalty strength applied to negative samples. However, most existing methods either fix this hyperparameter or learn a global value during training. In this paper, we introduce TeMo, Temperature Modulation framework, a similarity-based modulation approach that adaptively adjusts the temperature for each positive-negative pair according to their similarity, enabling more fine-grained multimodal contrastive learning. Our approach seamlessly integrates temperature-modulated multimodal and unimodal losses with the standard multimodal contrastive loss by gradually transitioning between them. This design allows the model to capture both coarse- and fine-grained semantics at different training stages. Extensive experiments demonstrate that each component of TeMo consistently enhances performance across diverse zero-shot retrieval and classification tasks, establishing new state-of-the-art results.

Abstract

Multimodal Contrastive Learning

Explore similar work

TeMo: Temperature Modulation for Multimodal Contrastive Learning

MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning

UniCon: Unified Framework for Efficient Contrastive Alignment via Kernels