Organizations: Department of Computer Science and Engineering, Korea University · Department of English Language and Literature, Kyungpook National University
Existing multilingual embedding models often encounter challenges in cross-lingual scenarios due to imbalanced linguistic resources and less consideration of cross-lingual alignment during training. Although standardized contrastive learning approaches for cross-lingual adaptation are widely adopted, they may struggle to capture fundamental alignment between languages and degrade performance in well-aligned languages such as English. To address these challenges, we propose Cross-Lingual Enhancement in Retrieval via Reverse-training (CLEAR), a novel loss function utilizing a reverse training scheme to improve retrieval performance across diverse cross-lingual retrieval scenarios. CLEAR leverages an English passage as a bridge to strengthen alignments between the target language and English, ensuring robust performance in the cross-lingual retrieval task. Our extensive experiments demonstrate that CLEAR achieves notable improvements in cross-lingual scenarios, with gains up to 15%, particularly in low-resource languages, while minimizing performance degradation in English. Furthermore, our findings highlight that CLEAR offers promising effectiveness even in multilingual training, suggesting its potential for broad application and scalability. We release the code at https://github.com/dltmddbs100/CLEAR.
Figures & tables
Figure 1: Performance disparity of various embedding models across languages in a cross-lingual setup where English passage with other language queries in Belebele benchmark.
Figure 2: Comparison of the core idea of CLEAR with the standard InfoNCE loss. Solid arrows indicate the direct effects of training, while dashed arrows represent the indirect interactions, such as the resulting attraction or repulsion between representations.
Model
Language
English-English
English-Lang
Lang-English
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
Belebele
bge-m3
zh
94.65 (-0.90)
95.47 (-0.08)
90.35
91.69
92.64
89.81
91.63
92.02
es
95.51 (-0.04)
96.14 (+0.59)
91.99
92.81
93.39
92.05
93.50
93.82
de
95.61 (+0.06)
96.11 (+0.56)
92.81
93.98
94.58
92.90
94.30
94.52
hi
95.15 (-0.40)
95.75 (+0.20)
86.90
89.14
90.16
90.39
92.86
93.16
Table 1: Comprehensive cross-lingual evaluation results. In each cross-lingual setting, ‘Lang’ refers to the target language. The first word denotes the language of the passage, and the second one denotes the language of the query. The value in ‘()’ indicates the difference in performance from the original model.
Model
Passage-Query
Method
ar
de
zh
ru
es
hi
vi
te
bn
multilingual-e5
English-Lang
InfoNCE
82.58
89.99
86.84
89.16
89.46
83.10
86.69
76.48
81.85
CLEAR
85.12
91.87
88.85
91.68
91.98
85.94
88.85
80.22
84.51
Lang-English
InfoNCE
88.08
92.29
90.02
91.39
91.58
90.97
90.44
88.63
88.93
CLEAR
88.89
93.47
90.78
92.25
93.14
92.39
91.47
89.50
90.68
gte-multilingual
English-Lang
InfoNCE
86.38
92.12
91.92
91.80
92.44
89.13
91.26
83.53
84.42
CLEAR
87.77
92.92
92.70
92.56
93.29
90.13
91.82
85.19
85.70
Table 2: Cross-lingual evaluation results in multilingual training setup in Belebele benchmark.
Language
multilingual-e5
gte-multilingual
InfoNCE
CLEAR
InfoNCE
CLEAR
en
94.94
95.61
94.66
95.46
ar
86.82
88.42
88.37
89.10
de
91.15
93.27
92.89
93.47
zh
91.28
93.05
92.85
93.65
ru
92.15
93.09
93.07
93.83
Table 3: Monolingual performances in multilingual training setup. Each row presents the result where the passage and query are in the same language.
Figure 3: T-SNE visualization of the embeddings for multilingual-e5 with English passage and Arabic, Bengali queries after multilingual training. We randomly select 100 pairs from the Belebele and measure the distance between the embeddings of identical gold pairs.
Method
ar
de
zh
ru
es
hi
vi
te
bn
Average
CLEAR e5
83.78 / 88.23
91.94 / 92.86
88.89 / 90.75
90.59 / 92.16
91.61 / 93.13
86.10 / 92.13
88.63 / 91.95
80.97 / 88.99
85.44 / 89.92
87.55 / 91.12
[0.8pt / 1.2pt] w/o LKL
83.82 / 88.49
91.61 / 92.29
88.59 / 90.43
89.85 / 91.73
90.76 / 92.49
85.05 / 91.10
88.52 / 91.69
79.68 / 88.67
84.40 / 89.62
86.92 / 90.72
w/o Reverse
82.81 / 87.49
91.31 / 92.47
87.44 / 89.69
89.77 / 90.90
89.91 / 92.01
84.12 / 90.84
87.41 / 91.36
78.21 / 87.96
82.77 / 88.89
85.97 / 90.18
w/o LNCEen
81.95 / 87.24
90.26 / 91.55
86.98 / 89.09
89.19 / 90.98
89.88 / 91.45
83.71 / 90.21
87.67 / 90.38
79.00 / 87.21
83.58 / 88.95
85.80 / 89.67
CLEAR gte
87.91 / 91.46
93.26 / 93.95
92.67 / 92.80
92.94 / 93.33
93.28 / 93.87
89.96 / 93.05
92.23 / 93.52
86.31 / 90.92
86.64 / 92.14
90.58 / 92.78
[0.8pt/1.2pt] w/o LKL
87.13 / 91.16
92.90 / 93.70
92.60 / 92.48
92.50 / 93.28
92.92 / 93.46
89.59 / 92.89
91.80 / 93.27
85.01 / 91.18
85.79 / 91.70
90.03 / 92.57
Table 4: Ablation results (English-Lang / Lang-English) for key components in CLEAR on multilingual-e5 and gte.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Model
Details
bge-m3
BAAI/bge-m3
Chen et al. (2024)
multilingual-e5
intfloat/multilingual-e5-base
Wang et al. (2024b)
gte-multilingual
Alibaba-NLP/gte-multilingual-base
Zhang et al. (2024b)
Appendix
Table 5: Embedding models details.
Lang
bge-m3
multilingual-e5
gte-multilingual
jina-v3
InfoNCE
CLEAR
InfoNCE
CLEAR
InfoNCE
CLEAR
InfoNCE
CLEAR
Cross-lingual Scenario
ar
72.44
72.95
64.11
64.99
66.22
66.75
71.77
71.89
ru
76.93
76.77
70.84
71.83
74.36
74.76
76.99
77.59
te
68.78
69.66
55.56
59.42
68.36
69.26
67.63
67.83
bn
69.96
69.90
60.30
62.24
63.67
64.30
68.59
68.95
Appendix
Table 6: Cross-lingual evaluation results in XOR-TyDi. In the cross-lingual scenario, the results are from models fine-tuned individually for each language. In the multilingual expansion, the model is trained considering all languages equally, as in our main experiments.
Qen
When was Winehouse quoted about the album release?
Pen+
Winehouse and Ronson contributed a cover of Lesley Gore’s “It’s My Party” to the Quincy Jones tribute album Q Soul Bossa Nostra released 9 November 2010 … In July 2010, Winehouse was quoted as saying her next album would be released no later than January 2011, saying “It’s going to be very much the same as my second album, where there’s a lot of jukebox stuff and songs that are… just jukebox, really.”
Pen−
On the other hand, Arabic is divided into over 27 dialects. Almost every Arab state has at least one local dialect of its own …
Qℓ
zh: 懷恩豪斯何時提到這張專輯的發行?
de: Wann hat Winehouse die Veröffentlichung dieses Albums erwähnt?
ru: Когда Уайнхаус упомянула о выпуске этого альбома?
es: ¿Cuándo mencionó Winehouse el lanzamiento de este álbum?
Appendix
Table 7: Example of inputs in the training.
Model
Lang
English-Lang
Lang-English
InfoNCE
CLEAR
InfoNCE
CLEAR
bge-m3
ar
88.66
89.23
90.87
91.17
zh
91.86
92.40
91.63
92.10
es
92.66
93.17
93.07
93.44
de
93.77
94.30
93.99
94.66
ru
93.72
94.17
93.42
93.54
Appendix
Table 8: Cross-lingual evaluation result in Belebele trained with m2m100 translated dataset.
Figure 4: Performance variation depending on the loss component weights ( λ1 , λ2 , λ3 ) across multiple languages.
Model
Language
English-Lang
Lang-English
nDCG@5
nDCG@10
nDCG@5
nDCG@10
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
bge-m3
ar
86.62
88.22
88.29
87.77
89.06
89.02
86.42
90.66
90.93
87.63
91.61
91.91
zh
89.70
90.93
92.14
90.35
91.69
92.64
89.08
90.97
91.51
89.81
91.63
92.02
es
91.48
92.61
93.05
91.99
92.81
93.39
91.47
92.99
93.59
92.05
93.50
93.82
de
92.20
93.62
94.31
92.81
93.98
94.58
92.47
93.87
94.30
92.90
94.30
94.52
Appendix
Table 9: Results on all languages under cross-lingual scenario in Belebele.
Model
Language
English-Lang
Lang-English
nDCG@5
nDCG@10
nDCG@5
nDCG@10
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
bge-m3
ar
91.84
92.69
93.24
92.24
93.08
93.50
91.91
94.03
94.44
92.38
94.41
94.71
zh
93.85
93.66
94.34
94.04
94.03
94.68
92.74
93.80
94.50
93.18
94.21
94.82
es
95.98
95.86
95.97
96.14
95.97
96.14
95.80
96.20
96.09
95.96
96.42
96.34
de
95.29
95.64
95.91
95.52
95.75
96.04
95.89
95.53
95.91
96.17
95.67
95.93
Appendix
Table 10: Results on all languages under cross-lingual scenario in XQuAD.
Model
Language
English-Lang
Lang-English
nDCG@5
nDCG@10
nDCG@5
nDCG@10
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
bge-m3
ar
86.62
88.85
89.31
87.77
89.56
89.97
86.42
90.35
91.01
87.63
91.30
91.90
zh
89.70
92.31
92.66
90.35
92.92
93.10
89.08
91.55
92.23
89.81
92.10
92.67
es
91.48
92.03
93.22
91.99
92.66
93.44
91.47
92.60
93.29
92.05
93.26
93.81
de
92.20
93.44
94.40
92.81
93.77
94.59
92.47
93.72
93.98
92.90
94.11
94.53
Appendix
Table 11: Results on all languages under the multilingual training setup in Belebele.
Model
Language
English-Lang
Lang-English
nDCG@5
nDCG@10
nDCG@5
nDCG@10
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
Base
InfoNCE
CLEAR
bge-m3
ar
91.84
92.29
93.19
92.24
92.53
93.46
91.91
94.12
94.26
92.38
94.52
94.62
zh
93.85
94.22
94.21
94.04
94.50
94.58
92.74
93.53
93.98
93.18
93.83
94.26
es
95.98
95.12
96.04
96.14
95.37
96.29
95.80
95.68
96.03
95.96
95.95
96.26
de
95.29
95.06
95.65
95.52
95.22
95.81
95.89
95.29
95.81
96.17
95.46
95.91
Appendix
Table 12: Results on all languages under the multilingual training setup in XQuAD.
Multilingual Information Retrieval (MLIR) reflects real-world search environments in which queries and relevant documents may appear in different languages within a mixed-language corpus. However, existing embedding models are primarily optimized for Multi-Monolingual retrieval and their performance often degrades in MLIR settings. Moreover, directly applying conventional contrastive learning to MLIR can exacerbate language clustering and expose a trade-off between cross-lingual alignment and embedding uniformity. To address these limitations, we propose MIMO: Multilingual Information Retrieval via Monolingual Objectives, a two-stage framework that uses a stable English semantic space from a high-performing teacher model as an anchor. MIMO first initializes the student model's cross-lingual alignment through knowledge distillation, and then jointly optimizes distillation and cross-lingual contrastive learning to improve retrieval discrimination while preserving alignment. Extensive experiments show that MIMO consistently outperforms existing cross-lingual training baselines across various MLIR and Multi-Monolingual benchmarks. MIMO also remains competitive with off-the-shelf models of similar or larger parameter scales. Furthermore, our cross-lingual Alignment-Uniformity analysis clarifies the distinct roles of the two loss components and shows that their combination yields a favorable trade-off between alignment and uniformity.
Youngjoon Jang, Seongtae Hong, Heuiseok Lim
Department of Computer Science and Engineering, Korea University
Multilingual pretraining typically lacks explicit alignment signals, leading to suboptimal cross-lingual alignment in the representation space. In this work, we show that training standard pretrained models for cross-lingual alignment with a multi-way parallel corpus in a diverse pool of languages can substantially improve multilingual and cross-lingual representations for NLU tasks. We construct a multi-way parallel dataset using translations of English text from an off-the-shelf NMT model for a pool of six target languages and achieve strong cross-lingual alignment through contrastive learning. This leads to substantial performance gains across both seen and unseen languages for multiple tasks from the MTEB benchmark evaluated for XLM-Roberta and multilingual BERT base models. Using a multi-way parallel corpus for contrastive training yields substantial gains on bitext mining (21.3%), semantic similarity (5.3%), and classification (28.4%) compared to English-centric (En-X) bilingually parallel data, where X is sampled from a pool of multiple target languages. Furthermore, finetuning mE5 model on a small dataset with multi-way parallelism significantly improves bitext mining compared to one without, underscoring the importance of multi-way cross-lingual supervision even for models already pretrained for high-quality sentence embeddings.
Barah Fazili, Koustava Goswami
IIT Bombay, Mumbai, India · Adobe Research, Bangalore, India
Multilingual Large Language Models (LLMs) struggle with cross-lingual tasks due to data imbalances between high-resource and low-resource languages, as well as monolingual bias in pre-training. Existing methods, such as bilingual fine-tuning and contrastive alignment, can improve cross-lingual performance, but they often require extensive parallel data or suffer from instability. To address these challenges, we introduce a Cross-Lingual Mapping Task during the pre-training phase, which enhances cross-lingual alignment without compromising monolingual fluency. Our approach bi-directionally maps languages within the LLM embedding space, improving both language generation and comprehension. We further propose a Language Alignment Coefficient to robustly quantify cross-lingual consistency, even in limited-data scenarios. Experimental results on machine translation (MT), cross-lingual natural language understanding (CLNLU), and cross-lingual question answering (CLQA) show that our model achieves gains of up to 11.9 BLEU points in MT, 6.72 points in CLQA BERTScore-Precision, and more than 5% in CLNLU accuracy over strong multilingual baselines. These findings highlight the potential of incorporating cross-lingual objectives into pre-training to improve multilingual LLMs.
Weihua Zheng, Chang Liu, Zhengyuan Liu +5
Singapore University of Technology and Design, Singapore · ByteDance, Singapore · Agency for Science, Technology and Research, Singapore