Omnilingual SONAR: Cross-Lingual and Cross-Modal Sentence Embeddings Bridging Massively Multilingual Text and Speech
Authors: Omnilingual SONAR Team, João Maria Janeiro, Pere-Lluís Huguet Cabot, Ioannis Tsiamas, Yen Meng, Vivek Iyer, Guillem Ramírez, Loic Barrault, Belen Alastruey, Xiang "Tony" Cao, Yu-An Chung, Marta R. Costa-Jussa, David Dale, Kevin Heffernan, Jaehyeong Jo, Artyom Kozhevnikov, Alexandre Mourachko, Christophe Ropers, Holger Schwenk, Paul-Ambroise Duquenne
Cross-lingual sentence encoders typically cover only a few hundred languages and often trade downstream quality for stronger alignment, limiting their adoption. We introduce OmniSONAR, a new family of omnilingual, cross-lingual and cross-modal sentence embedding models that natively embed text, speech, code, and mathematical expressions in a single semantic space, while delivering state-of-the-art downstream performance at the scale of thousands of languages, from high-resource to extremely low-resource varieties. To reach this scale without representation collapse, we use progressive training. We first learn a strong foundational space for 200 languages with an LLM-initialized encoder-decoder, combining token-level decoding with a novel split-softmax contrastive loss and synthetic hard negatives. Building on this foundation, we expand to several thousands language varieties via a two-stage teacher-student encoder distillation framework. Finally, we demonstrate the cross-modal extensibility of this space by seamlessly mapping 177 spoken languages into it. OmniSONAR halves cross-lingual similarity search error on the 200-language FLORES dataset and reduces error by a factor of 15 on the 1,560-language BIBLE benchmark. It also enables strong translation, outperforming NLLB-3B on multilingual benchmarks and exceeding prior models (including much larger LLMs) by 15 chrF++ points on 1,560 languages into English BIBLE translation. OmniSONAR also performs strongly on MTEB and XLCoST. For speech, OmniSONAR achieves a 43% lower similarity-search error and reaches 97% of SeamlessM4T speech-to-text quality, despite being zero-shot for translation (trained only on ASR data). Finally, by training an encoder-decoder LM, Spectrum, exclusively on English text processing OmniSONAR embedding sequences, we unlock high-performance transfer to thousands of languages and speech for complex downstream tasks.
Figures & tables
Figure 1: The training stages. In Stage 1 , we train our LLM-initialized encoder-decoder on translation data with a decoding loss. In Stage 2 , we introduce an encoder bottleneck via pooling and train with a combination of contrastive and decoding objectives. In Stage 3 , we introduce hard negatives and continue training with a split-softmax contrastive objective and the decoding loss. In Stage 4 , we extend the space to omnilingual-level language coverage by training with teacher-student distillation on 4,200 language varieties with a combination of MSE and Contrastive objectives, while first warming-up the omnilingual tokenization with MSE-based distillation. Lastly, in Stage 5 , we extend the omnilingual space to the speech modality with teacher-student distillation using ASR data.
Example 1: Python
Source
if input_event["enable_points"] or event_info.get("bonus"):
Target
The script determines if the "enable_points" in the input_event dictionary is set to True, or if the event_info dictionary includes a key called "bonus" with any associated value.
Example 2: JavaScript
Source
const DataHandler = require(’./lib/dataHandler.js’); let windowRef; const dataHandler = new DataHandler({});
Target
A constant named DataHandler is initialized by importing a module from a file called dataHandler.js located in a folder named lib , then a variable windowRef is declared, and a constant dataHandler is created as a new instance of DataHandler , passing an empty object to its constructor.
Example 3: Math
Table 1: Examples of paired data for code and mathematical expressions.
Original Sentence
Generated hard negatives
It was carrying a special mission unit
It was not carrying a special mission unit
It is carrying a normal mission unit
She was carrying a special mission unit
It will be carrying a special mission team
It was carrying 12 special mission units
Table 2: Examples of natural language hard negatives.
Figure 2: Example of OmniSONAR natural text prompting on both encoder and decoder sides. Source sentences, which are input to the encoder, are prefixed with language identifiers. The format used is “[language name]:”. Starting from the omnilingual extension, this prefix can be randomly replaced by ”Unspecified language:”. Target sentences, provided to the decoder, include task specifications, output language information, and information about data provenance (indicating whether the translation is human-labeled, automatically mined, or back-translated), following NLLB Team (2024) . Specifically, we use the prompts such as This is a possible translation in [language name]: for translation tasks and This is a possible natural language explanation in English: for code and math explanation tasks. This input formatting is applied consistently across all training stages. We provide the full list of prompts in Appendix Table 40 .
Table 3: Loss function and training parameter configurations depending on the source language type (foundational vs new) during the omnilingual extension. † When the source language is English, we use the source embedding of the teacher.
Table 4: X-Eng cross-lingual similarity search error rates xsim/xsim++ ( ↓ ) on different multilingual test sets.
Table 5: Results for GMMLU question mining (left) for all 42 languages and those covered by the baselines (common) and XLCOST (right). xsim ( ↓ ) reported for all models.
Table 6: Comparison of xsim and xsim++ results on FLEURS test set for the 36 languages covered by SONAR. Tables 43(a) and 43(b) .
Table 7: Classification and Pair Classification results from sentence-level MTEB tasks.
Table 8: X-Eng translation quality chrF++ ( ↑ ) / xCOMET( ↑ ) on different multilingual test sets. FLORES+ reports only the 11 languages not in FLORES200.
Table 9: Speech Translation (X-Eng) BLEU score on FLEURS, comparing OmniSONAR with SONAR and other Speech Translation models.
Table 10: Results of smaller distilled models on cross-lingual similarity search (FLORES200) and downstream tasks (MTEB). xsim/xsim++ report error rates ( ↓ ) and MTEB scores are higher-is-better ( ↑ ).
Table 11: Training objectives ablations : Ablations on training objectives to learn a massively multilingual sentence embedding space on the cross-lingual similarity search task of FLORES200 dev set, as measured by xsim and xsim++.
Table 12: Contrastive Learning ablations: Effect of hyper-parameters and modeling options in Contrastive Learning on cross-lingual similarity search on FLORES200 dev set.
Table 13: Model initialization ablations : Effect of model weight initialization for sequence-to-sequence stage as well as for contrastive learning stage on respectively decoding performance (spBLEU and chrF++) and cross-lingual similariy search (xsim and xsim++) on FLORES200 dev set.
Figure 3: Loss function parameters for new languages. Results on Bible dev . chrF++ obtained with the OmniSONAR-200 decoder. Vertical dashed lines indicate the final configuration used for OmniSONAR.
BIBLE
FLORES
Model
xsim
chrF++
xsim++
chrF++
(a) Omnilingual Vocabulary Adaptation warm-up
without
4.8
37.1
5.9
55.4
with
4.8
37.7
5.6
56.0
(b) Loss function for foundational Languages
MSE
Contr. s → t
Contr. t → s
Table 14: Ablations on the OmniSONAR omnilingual extension evaluated on BIBLE and FLORES dev sets. Cross-lingual similarity error rates xsim/xsim++ ( ↓ ) and translation quality chrF++ ( ↑ ). The final OmniSONAR configuration is highlighted in gray.
Table 15: Comparison of the foundational 200-language model (OmniSONAR-200) and the final omnilingual model (OmniSONAR) in cross-lingual similarity search (xsim/xsim++ ↓ ) and translation quality (chrF++/ xCOMET ↑ ) in Bible test and FLORES devtest . For translation with each model we use their dedicated decoders.
BIBLE
FLORES
FLORES+
Vocabulary
Params
xsim
chrF++
Fert.
Thr.
xsim++
chrF++
Fert.
Thr.
xsim++
chrF++
Fert.
Thr.
200-lang Vocabularies
256K
1.5
4.7
37.5
57.5
264
5.6
55.9
43.2
523
14.3
43.3
76.4
130
Omnilingual Vocabularies
8K
1.0
4.6
37.5
73.2
263
5.6
55.8
70.3
321
14.5
43.7
78.2
335
16K
1.0
4.8
37.6
66.9
297
5.6
55.9
62.7
365
14.6
43.9
70.2
378
Table 16: Vocabulary size ablations. Results in X-Eng dev sets. The 200-language vocabulary is the one used from OmniSONAR-200, which is extended from the Llama3 128K vocabulary. Model parameters in Billions. Metrics: cross-lingual similarity error rates with xsim/xsim++( ↓ ); translation quality with chrF++ ( ↑ ) obtained with the OmniSONAR-200 decoder; Tokenizer fertility( ↓ ); and Inference Throughput( ↑ ) in sentences per second.
chrF++ (Translation Quality)
xsim (Similarity Search Error)
Feature
Importance
95% CI
Rank
Importance
95% CI
Rank
Tokenizer Fertility
22.61
[20.88, 24.75]
1
5.12
[4.43, 5.73]
2
Family Examples
14.19
[13.23, 15.13]
2
4.47
[3.99, 5.19]
3
Language Examples
12.03
[11.11, 12.99]
3
6.86
[6.07, 7.59]
1
Latitude
6.89
[6.50, 7.42]
4
3.63
[3.18, 4.23]
4
Longitude
5.53
[5.10, 6.09]
5
3.46
[3.06, 3.79]
5
Table 17: Feature importance for predicting language performance on new languages (n=1,420) using Gradient Boosting with permutation-based importance and 95% confidence intervals. Importance values indicate increase in RMSE when a feature is randomly shuffled.
Table 18: Cross-lingual transfer (CLT) on SIB200Classification: Models trained on French, evaluated zero-shot on 199 languages (all) and 80 baseline-supported languages (common), reporting average relative performance to French.
Figure 4: Cross-lingual similarity (xsim) and translation quality (chrF++) for encoders trained on progressively more language groups, evaluated on BIBLE dev X-Eng. For translation we use the OmniSONAR-200 decoder. In the Top subfigures (a) and (b), we display Resource-based grouping: Groups A-G sorted by resourcefulness. In the Bottom subfigures (c) and (d), we display Family-based grouping: Group A = Indo-European (largest family) and Groups B-F = other families. In bold is the best performance per column (excluding OmniSONAR). The ”All” column is the average across all 1,560 languages. OmniSONAR is trained on all groups plus an additional group containing 1,864 extremely low resource languages.
Table 19: Speech Translation performance (X-Eng) on unseen languages during training.
Figure 5: Architecture of the Spectrum model. An encoder processes a sequence of sentence-level OmniSONAR embeddings, while the decoder is a token-level transformer that generates answers auto-regressively from a prefixed prompt. The decoder achieves this by performing cross-attention over the outputs of the encoder. We refer to the encoder as the “SONAR tower” and the decoder as the “token tower”.
Figure 6: Training phases of the Spectrum model, divided in Pre-training (Encoder Warmup, Teacher-Forced Sentence-Level LM and Prefix-LM) and Instruction Finetuning.
Figure 7: Example of a prompt for a Classification task from SIB200 using our diverse prompt generation approach. System prompt is in red , task prompt in blue , choices block in orange and the target format prompt in brown . The question block, used exclusively for MCQ and QA tasks, is absent here. This prompt is fed to the decoder, while the input document is passed through the encoder as usual.
Figure 8: FLOPs analysis of Spectrum architecture versus token-level LLM decoder-only architecture such as Llama3 models. Greener means that Spectrum is more compute efficient than token-level decoder-only models.
AE BLEU1
AE BLEU2
AE BLEU3
AE BLEU4
AE BLEU
Spectrum auto-encode
99.9
99.8
99.7
99.6
99.71
Table 20: Auto-encoding (AE) BLEU scores obtained by splitting and encoding long documents from WikiLingua with OmniSONAR and decoding them back into English with Spectrum auto-encode .
Figure 9: Spectrum pre-training performance across XHellaswag and Spoken StoryCloze benchmarks.
Model
XBelebele
SIB200
Taxi1500
Llama 3.2 3B Instruct
53.03
65.69
19.67
Llama 3.1 8B Instruct
63.06
66.07
24.18
Spectrum 6B-3B
61.85
79.95
36.9
Table 21: Zero-shot multilingual results of Spectrum on XBelebele, SIB, and Taxi1500, compared with Llama models
T → T
S → T
Speech-XBelebele
63.64
62.74
Speech-SIB100
85.11
82.59
Table 22: Zero-shot multilingual speech understanding results of Spectrum on Speech-XBelebele and Speech-SIB. We compare performance on Text → Text (T → T) and Speech → Text (S → T) splits of each benchmark to demonstrate cross-modal transfer.
Figure 10: Average AER across language pairs for different layer numbers. We see that OmniSONAR exhibits a parabolic shape, with inner layers showing better alignment than the final layers.
Table 23: Last layer AER for different encoders.
Dataset/task
#Test languages
Example
Word alignment
14
The necessary correction will be made . Eine entsprechende Anderung wird vorgenommen werden .
Massive
52
What is today ’s forecast for Berlin DATE PLACE
PAN-X
40
REDIRECCIÓN Algarrobo ( Chile ) B-LOC I-LOC
WiC
6
\left.\begin{array}[]{@{}l@{}}\text{Bolivia holds a key \lx@text@underline{play} in this process of peace.}\\ \text{A musical \lx@text@underline{play} on the same subject was also staged.}\end{array}\right\}\ \begin{array}[]{@{}l@{}}\text{Different}\\ \text{meaning}\end{array}
Table 24: Summary of the tasks used for the evaluation of cross-lingual token representations.
Table 25: Performance for cross-lingual token representation tasks. The first column presents the mean Alignment Error Rate (AER) for the last layer. The remaining columns display the average F1 scores achieved by a classifier that was trained using English token embeddings and then evaluated on data from other languages.
Table 26: Performance for sentence-level tasks. We report xsim ( Artetxe and Schwenk, 2019 ) and xsim++ ( Chen et al., 2023c ) for cross-lingual mining and ChrF++ ( Popović, 2017a ) on the FLORES ( NLLB Team, 2024 ) dataset for translation performance.
Figure 11: Training dynamics of SONAR and OmniSONAR trained with the OmniSONAR-Token objective.
Table 27: Sentence-level performance comparison between OmniSONAR and OmniSONAR-LC on FLORES (error rate ↓ ) and MTEB (score ↑ ).
Table 28: Cross-lingual similarity search error rates on Belebele paragraphs.
Figure 12: Distribution of sentence count differences from the mean across 9 languages from FLORES paragraphs. Each subplot shows the histogram of (language sentence count - average sentence count) across all paragraphs. Red dashed lines indicate zero (perfect consistency) and the mean deviation for each language. (a) The baseline SaT-12L-sm model exhibits substantial deviation from zero, with means ranging from -0.34 to 0.64 and an overall standard deviation of 1.48 across all 201 languages, indicating inconsistent segmentation. (b) Our custom model produces distributions tightly centered around zero, with means ranging from -0.09 to 0.13 and an overall standard deviation of 0.34.
Pattern
Description
(.∗?)
Inline math (e.g., a2+b2=c2 )
(.∗?)
Display math with double dollar signs
$(.*?)$
Display math with ...
\begin{equation}(.*?)\end{equation}
Equation environment
\begin{align}(.*?)\end{align}
Align environment
\begin{align*}(.?)\end{align}
Align* environment
Appendix
Table 30: Summary of regular expressions used for extracting LaTeX math expressions and splitting sentences.
Figure 13: Three prompt templates for translation, code snippet semantic description, and mathematical notation explanation.
Figure 14: Examples of hard negatives generation across Python code (blue), JavaScript code (orange), and mathematical expressions (green).
Original sentence
Generated hard negatives
Tonight will be long.
Tonight will not be short
Yesterday may be long
Tomorrow will be short
Tonight may not be long
Last night was short
You got the archaeologist and the psychologist wrong.
You did not get the archaeologist and the psychologist right
Appendix
Table 31: Natural language hard negative examples
Figure 15: Prompt for generating xsim++ transformations with clear instructions and structure.
Code
Language
Script
Family
Subgrouping
Res.
ace_Arab
Acehnese
Arabic
Austronesian
Malayo-Polynesian
Low
ace_Latn
Acehnese
Latin
Austronesian
Malayo-Polynesian
Low
acm_Arab
Mesopotamian Arabic
Arabic
Afro-Asiatic
Semitic
Low
acq_Arab
Ta’izzi-Adeni Arabic
Arabic
Afro-Asiatic
Semitic
Low
aeb_Arab
Tunisian Arabic
Arabic
Afro-Asiatic
Semitic
Low
afr_Latn
Afrikaans
Latin
Indo-European
Germanic
High
Appendix
Table 47
Code
Language
Script
Family
Subgrouping
Res.
ita_Latn
Italian
Latin
Indo-European
Italic
High
jav_Latn
Javanese
Latin
Austronesian
Malayo-Polynesian
Low
jpn_Jpan
Japanese
Japanese
Japonic
Japanesic
High
kab_Latn
Kabyle
Latin
Afro-Asiatic
Berber
Low
kac_Latn
Jingpho
Latin
Sino-Tibetan
Brahmaputran
Low
kam_Latn
Kamba
Latin
Atlantic-Congo
Benue-Congo
Low
Appendix
Table 48
Code
Language
Script
Family
Subgrouping
Res.
smo_Latn
Samoan
Latin
Austronesian
Malayo-Polynesian
Low
sna_Latn
Shona
Latin
Atlantic-Congo
Benue-Congo
Low
snd_Arab
Sindhi
Arabic
Indo-European
Indo-Aryan
Low
som_Latn
Somali
Latin
Afro-Asiatic
Cushitic
Low
sot_Latn
Southern Sotho
Latin
Atlantic-Congo
Benue-Congo
High
spa_Latn
Spanish
Latin
Indo-European
Italic
High
Appendix
Table 32: Correspondence table from No Language Left Behind project for OmniSONAR-200 languages. This table was adapted from ( NLLB Team, 2024 ) .
Languages
ace_Arab
ace_Latn
acm_Arab
acq_Arab
aeb_Arab
afr_Latn
ajp_Arab
aka_Latn
als_Latn
amh_Ethi
apc_Arab
arb_Arab
ars_Arab
ary_Arab
arz_Arab
asm_Beng
ast_Latn
awa_Deva
ayr_Latn
azb_Arab
azj_Latn
bak_Cyrl
bam_Latn
ban_Latn
bel_Cyrl
bem_Latn
ben_Beng
bho_Deva
bjn_Arab
bjn_Latn
bod_Tibt
bos_Latn
bug_Latn
bul_Cyrl
cat_Latn
ceb_Latn
ces_Latn
cjk_Latn
ckb_Arab
crh_Latn
cym_Latn
dan_Latn
Appendix
Table 33: Complete list of languages covered by OmniSONAR-200. Languages shown in bold are supported by all models in our comparison. Underlined languages are supported by OmniSONAR-speech 3B and 7B models. Our initial training covers 202 languages total, with 81 languages supported across all compared models.
Seq2Seq
Contrastive
Dataset
pairs
dirs
source
target
pairs
dirs
source
target
BT Math
445M
56
8
8
–
–
–
–
Dictionary
153M
3.3K
182
97
–
–
–
–
Code/Math → Eng
1B
9
9
1
9.0M
9
9
1
Eng → Code
941M
8
1
8
–
–
–
–
Eng → Math
80M
1
1
1
–
–
–
–
Appendix
Table 34: Training Data Statistics for Seq2Seq and Contrastive Learning Approaches. Each dataset shows: pairs (number of translation pairs), dirs (number of translation directions, i.e., language X to Y), source (number of source languages), and target (number of target languages).
Langs
Examples
Group
Train
Dev
Total (M)
Median (K)
Note
Resource-based grouping ( Figures 4(a) and 4(b) )
A
211
140
2,764
541
High-resource
B
300
246
93
104
C
300
269
17
55
D
300
275
11
37
Appendix
Table 35: Language group statistics for resource-based (top) and family-based (bottom) grouping experiments. The resource-based grouping includes all languages with sufficient parallel data, sorted by resourcefulness. The family-based grouping assigns Indo-European to Group A and distributes other families equally across Groups B-F.
Parameter
Large
Medium
Small
Tiny
xTiny
vocab_size
256,232
128,232
64,232
32,232
16,232
hidden_size
2,048
1,792
1,536
1,152
512
intermediate_size
8,192
7,168
6,144
4,608
2,048
num_hidden_layers
16
14
12
10
8
num_attention_heads
32
28
24
18
16
num_key_value_heads
8
7
6
6
4
Appendix
Table 36: Differences across the model sizes for OmniSONAR as explained in Section 4.8
Model
xsim
xsim++
Mean
0.68
9.25
CLS
0.64
8.77
Appendix
Table 37: Ablation on different pooling strategies, evaluated on FLORES200 dev set.
model
std
mean
MEXMA
0.0312
-0.0011
mE5 large
0.0312
-0.0008
LaBSE
0.0358
0.0049
SONAR
0.0074
0.0000
OmniSONAR
0.0356
-0.0006
Appendix
Table 38: Standard deviation (std) and mean of embedding features for different models when encoding FLORES200 dev set on all common languages.
Figure 16: Singular values of embeddings from different models.
System
Source Sentence
Desired Retrieved
Actual Retrieved
OmniSONAR
O Corpo de Engenheiros dos EUA estimou que 15 cm de chuva podem romper os diques anteriormente danificados.
The U.S. Corps of Engineers estimated that 6 inches of rainfall could breach the previously damaged levees.
The U.S. Corps of Engineers estimated that 15 inches of rainfall could breach the previously damaged levees.
OmniSONAR
Os limites de velocidade anunciados sao visivelmente mais baixos do que nas secoes anteriores e subsequentes - comumente 55-65 km/h - e a estrita obediencia a eles e ainda mais importante do que o contrario.
Posted speed limits are noticeably lower than in previous and subsequent sections commonly 35-40 mph (56-64 km/h) and strict obedience to them is even more important than otherwise.
Posted speed limits are noticeably lower than in previous and subsequent sections commonly 35-90 mph (56-64 km/h) and strict obedience to them is even more important than otherwise.
MEXMA
O Corpo de Engenheiros dos EUA estimou que 15 cm de chuva podem romper os diques anteriormente danificados.
The U.S. Corps of Engineers estimated that 6 inches of rainfall could breach the previously damaged levees.
The U.S. Corps of Engineers estimated that 15 inches of rainfall could breach the previously damaged levees.
MEXMA
Reportagens televisivas divulgam a fumaca esbranquicada saindo da planta.
Television reports show white smoke coming from the plant.
Television reports show black smoke coming from the plant.
SONAR
No periodo de um ano, uma pessoa infectada pode infectar entre 10 e 15 contatos proximos.
In one year’s time, an infected person may infect 10 to 15 close contacts.
In one year’s time, an infected person will infect 10 to 15 close contacts.
SONAR
Aconteceu novamente no mesmo mes em Mashhad, outro aviao comercial entrou em uma pista e atingiu uma parede, matando dezessete pessoas.
The same month saw another airliner overrun a runway at Mashhad and strike a wall, killing seventeen.
The same month did not saw another airliner overrun a runway at Mashhad and strike a wall, killing seventeen.
Appendix
Table 39: Comparison of three systems (OmniSONAR, MEXMA, SONAR) on two examples each. For each example, the table shows the original source sentence (in Portuguese) and the desired retrieved English sentence, the actual retrieved English sentence.
Source
Prompt Template
Encoder
Source/Anchor
"<CLS><s> [LANGUAGE]:<SEP> [INPUT SENTENCE] </s>"
Decoder
Primary data
"<s> This is a possible translation in [LANGUAGE]:<SEP> [INPUT SENTENCE] </s>"
Mined data
"<s> This is a possible mined translation in [LANGUAGE]:<SEP> [INPUT SENTENCE] </s>"
Back-translations
"<s> This is a possible back-translation in [LANGUAGE]:<SEP> [INPUT SENTENCE] </s>"
Appendix
Table 40: Prompt Templates for Encoder and Decoder Components. The encoder uses classification prompts to identify language and content, while the decoder uses descriptive prompts tailored to different data sources and translation types. Placeholders [LANGUAGE] and [INPUT SENTENCE] are replaced with actual values during training.
Lang
SONAR
LaBSE
MEXMA
OmniSONAR
mE5
Lang
SONAR
LaBSE
MEXMA
OmniSONAR
mE5
Lang
SONAR
LaBSE
MEXMA
OmniSONAR
mE5
Lang
SONAR
LaBSE
MEXMA
OmniSONAR
mE5
ace_Arab
6.23
96.44
57.91
8.60
77.77
ace_Latn
0.30
32.61
8.60
0.10
8.70
acm_Arab
0.10
0.30
0.00
0.00
0.10
acq_Arab
0.00
0.10
0.00
0.00
0.10
aeb_Arab
0.40
4.94
0.20
0.20
0.99
afr_Latn
0.00
0.00
0.00
0.00
0.00
ajp_Arab
0.10
0.49
0.00
0.00
0.20
aka_Latn
0.30
53.46
44.86
0.10
7.21
als_Latn
0.00
0.00
0.00
0.00
0.00
amh_Ethi
0.00
0.00
0.00
0.00
0.40
apc_Arab
0.10
1.09
0.00
0.00
0.10
arb_Arab
0.00
0.00
0.00
0.00
0.00
ars_Arab
0.10
0.00
0.00
0.00
0.10
ary_Arab
0.99
13.14
0.99
0.89
2.77
arz_Arab
0.20
0.69
0.10
0.00
0.40
asm_Beng
0.10
1.78
0.00
0.00
0.69
ast_Latn
0.00
0.20
0.00
0.00
0.00
awa_Deva
0.99
1.09
0.89
0.89
0.99
ayr_Latn
3.85
72.13
54.25
1.68
43.97
azb_Arab
2.96
44.37
1.68
0.69
11.36
azj_Latn
0.30
0.30
0.20
0.20
0.20
bak_Cyrl
0.00
41.90
11.36
0.00
1.68
bam_Latn
4.05
65.61
52.37
2.17
14.62
ban_Latn
0.40
8.40
1.09
0.30
2.57
Appendix
Table 41: xsim results for all models in all languages, x-eng in FLORES200 devtest set.
Lang
SONAR
LaBSE
MEXMA
OmniSONAR
mE5
Lang
SONAR
LaBSE
MEXMA
OmniSONAR
mE5
Lang
SONAR
LaBSE
MEXMA
OmniSONAR
mE5
Lang
SONAR
LaBSE
MEXMA
OmniSONAR
mE5
ace_Arab
55.34
100.00
92.09
36.76
98.91
ace_Latn
16.30
82.21
49.41
8.10
48.22
acm_Arab
11.17
52.67
9.39
5.83
27.37
acq_Arab
8.30
46.54
8.20
10.87
23.02
aeb_Arab
11.76
66.60
13.64
6.72
31.72
afr_Latn
5.14
9.49
4.64
1.19
13.34
ajp_Arab
7.61
54.15
9.19
5.24
23.81
aka_Latn
19.47
92.39
85.18
10.77
42.79
als_Latn
5.14
10.97
9.39
1.78
18.48
amh_Ethi
8.60
28.85
6.23
3.36
43.48
apc_Arab
9.58
58.00
12.65
5.14
28.46
arb_Arab
6.82
35.87
6.72
2.37
19.27
ars_Arab
13.93
39.72
12.65
8.50
23.52
ary_Arab
14.03
77.08
22.23
12.45
39.03
arz_Arab
10.87
58.99
11.46
4.84
24.51
asm_Beng
14.92
62.15
11.17
6.42
41.50
ast_Latn
9.68
38.04
14.13
6.82
19.86
awa_Deva
11.07
26.98
12.06
3.75
27.67
ayr_Latn
34.78
97.43
83.10
19.27
85.77
azb_Arab
42.59
94.76
32.61
21.25
65.32
azj_Latn
14.03
17.98
10.77
6.13
29.45
bak_Cyrl
11.46
91.90
51.09
3.56
40.71
bam_Latn
30.14
95.26
85.67
17.09
61.56
ban_Latn
13.04
60.38
29.15
6.13
36.26
Appendix
Table 42: xsim++ results for all models in all languages, x-eng in FLORES200 devtest set.
Table 43: OmniSONAR-speech modality results on 36 common languages SONAR and OmniSONAR-speech models cover, x-eng on FLEURS dev set. SONAR results are for 36 different language-specific encoders, while OmniSONAR-speech are one model across all languages.
Family
Langs
Examples
xsim
chrF++
Family
Langs
Examples
xsim
chrF++
Indo-European
535
6,329,570
0.9
56.3
Huitotoan
3
107
6.1
32.3
Atlantic-Congo
652
464,728
2.8
43.3
Border
2
105
6.1
31.4
Afroasiatic
196
419,370
4.2
44.0
Eastern Daly
3
102
7.5
31.1
Turkic
82
410,656
0.6
55.4
Kunimaipan
3
96
9.3
29.9
Dravidian
39
369,957
1.9
49.7
Araucanian
3
90
4.5
38.9
Austronesian
795
315,373
3.0
41.8
Lengua-Mascoy
6
84
12.2
31.4
Appendix
Table 44: Detailed statistics and results per language family. Results in BIBLE test . Number of examples is in thousands. Families sorted by number of examples. We removed 55 families that are absent from BIBLE test .
Figure 17: UMAP visualization of the sentence “ During his time with the team, he scored 403 goals in 468 appearances. ” from FLORES devset along closest hard negatives, shown as red crosses. Lines connect the translations to their English counterpart.
Figure 18: UMAP visualization of the whole space defined by the FLORES devset for 20 languages with different scripts.
Omni-modal large language models (LLMs) have achieved remarkable progress in audio and video understanding. However, existing models struggle with long-form, multi-speaker videos, especially in understanding the entire video and recognizing who says what and when. To characterize the capability limit of current omni LLMs and guide future model development, we introduce Omnilingual-MsASR, a multilingual, multimodal benchmark for long-form Multi-speaker Automatic Speech Recognition (MsASR) covering six languages, recordings up to 30 minutes, and scenarios with up to 20 speakers. Omnilingual-MsASR provides parallel audio and video inputs to evaluate models' cross-modal consistency and their ability to exploit complementary cues across modalities. Extensive evaluations of leading omni-modal LLMs reveal substantial performance bottlenecks, while their tcpWERs frequently exceed 100%. These findings highlight fine-grained temporal localization, speaker disambiguation, and cross-modal synergy as key challenges for future omni-modal foundation models. Our work is available at https://github.com/QwenLM/Omnilingua-Bench/tree/main/Omnilingua-MSpeaker.
Yuan Ge, Pei Zhang, Xize Cheng +8
Northeastern University, China · Alibaba Token Hub, Alibaba · Kunming University of Science and Technology +1
Speakers of under-represented languages face both a language barrier, as most online knowledge is in a few dominant languages, and a modality barrier, since information is largely text-based while many languages are primarily oral. We address this for French-Wolof by training the first bilingual speech-text Matryoshka embedding model, enabling efficient retrieval of French text from Wolof speech queries without relying on a costly ASR-translation pipelines. We introduce large-scale data curation pipelines and new benchmarks, compare modeling strategies, and show that modality fusion within a frozen text Matryoshka model performs best. Although trained only for retrieval, the model generalizes well to other tasks, such as speech intent detection, indicating the learning of general semantic representations. Finally, we analyze cost-accuracy trade-offs across Matryoshka dimensions and ranks, showing that information is concentrated only in a few components, suggesting potential for efficiency improvements.
We introduce V-SONAR, a vision-language embedding space extended from the text-only embedding space SONAR (Omnilingual Embeddings Team et al., 2026), which supports 1500 text languages and 177 speech languages. To construct V-SONAR, we propose a post-hoc alignment pipeline that maps the representations of an existing vision encoder into the SONAR space. We thoroughly evaluate V-SONAR and show that its embeddings achieve competitive performance on text-to-video retrieval. Equipped with the OMNISONAR text decoder, V-SONAR further surpasses state-of-the-art vision-language models on video captioning tasks, including DREAM-1K (BLEU 23.9 vs. 19.6) and PE-VIDEO (BLEU 39.0 vs. 30.0). Leveraging V-SONAR, we first demonstrate that the Large Concept Model (LCM; LCM team et al. 2024) operating in SONAR and trained with English text only, can perform both single- and multi-visual concept understanding in a zero-shot manner. Finally, we introduce V-LCM, which extends the LCM with vision-language instruction tuning. V-LCM encodes vision and language inputs into an unified sequence of latent embeddings via V-SONAR and SONAR, and it is trained with the same latent diffusion objective for next-embedding prediction as in LCM's text-only pre-training. Experiments on a large-scale multilingual and -modal instruction-tuning data mixture highlight the potential of V-LCM: V-LCM matches state-of-the-art vision-language models on tasks covering image/video captioning and question answering, while significantly outperforming them across 61 rich- to low-resource languages out of all 62 tested languages.