Cross-lingual Matryoshka Representation Learning across Speech and Text

Yaya Sy; Dioula Doucouré; Christophe Cerisara; Irina Illina

cs.CLFeb 23, 2026

Cross-lingual Matryoshka Representation Learning across Speech and Text

Authors: Yaya Sy, Dioula Doucouré, Christophe Cerisara, Irina Illina

Organizations: LORIA, CNRS · Soynade Research

Abstract

Speakers of under-represented languages face both a language barrier, as most online knowledge is in a few dominant languages, and a modality barrier, since information is largely text-based while many languages are primarily oral. We address this for French-Wolof by training the first bilingual speech-text Matryoshka embedding model, enabling efficient retrieval of French text from Wolof speech queries without relying on a costly ASR-translation pipelines. We introduce large-scale data curation pipelines and new benchmarks, compare modeling strategies, and show that modality fusion within a frozen text Matryoshka model performs best. Although trained only for retrieval, the model generalizes well to other tasks, such as speech intent detection, indicating the learning of general semantic representations. Finally, we analyze cost-accuracy trade-offs across Matryoshka dimensions and ranks, showing that information is concentrated only in a few components, suggesting potential for efficiency improvements.

Figures & tables

Figure 1: Speech data pipeline . Raw data is filtered via Source Separation , Diarization , VAD , and Quality Filtering . The resulting speech is transcribed, bad transcriptions (in red ) are filtered, then used to generate French story, dialog and blogpost documents.

Figure 2: Late-Fusion vs. Dual architectures . In the Late-Fusion approach, speech is encoded with HuBERT, then the sequence is downsampled by a CNN (x2), projected to the LLM embedding with a matrix W , concatenated with the prompt token embeddings, and the whole is forwarded to the Matryoshka embedding LLM. In the Dual architecture, HuBERT features are pooled at the sequence level using an attention-based pooler, then projected with dimension-specific W matrices to obtain Matryoshka embeddings. In both architectures, text documents are embedded using the text-only Matryoshka embedding LLM.

Approach	dim=4096		dim=1024		dim=512		dim=256		dim=128
Approach	nDCG@5	nDCG@10	nDCG@5	nDCG@10	nDCG@5	nDCG@10	nDCG@5	nDCG@10	nDCG@5	nDCG@10
NLLB-LLM2Vec	57.98	61.53	–	–	–	–	–	–	–	–
Late-Fusion	–	–	69.85	74.49	66.86	71.04	61.58	67.05	56.13	62.30
Pipelined	–	–	57.09	62.82	54.07	59.14	45.60	51.56	41.21	47.34
Dual – Retrieval	–	–	46.96	53.70	45.27	52.48	41.97	49.78	38.47	44.40
Dual – Query Alignment	–	–	41.56	47.42	41.02	46.76	38.24	44.18	35.47	40.47

Table 1: nDCG@5 and nDCG@10 document retrieval results on Kallaama-Retrieval-Eval for the different approaches across dimensions. NLLB-LLM2Vec is not Matryoshka and uses the full model dimension (4096), while other approaches use Matryoshka dimensions. The most performant speech-based approach is colorized and the best scores are in bold.

Approach	dim=4096		dim=1024		dim=512		dim=256		dim=128
Approach	nDCG@5	nDCG@10	nDCG@5	nDCG@10	nDCG@5	nDCG@10	nDCG@5	nDCG@10	nDCG@5	nDCG@10
NLLB-LLM2Vec	55.98	59.43	–	–	–	–	–	–	–	–
Late-Fusion	–	–	57.89	61.19	55.03	58.90	50.87	54.43	41.56	46.60
Dual – Retrieval	–	–	41.28	45.54	40.18	45.06	39.84	45.17	39.24	44.34
Dual – Query Alignment	–	–	38.07	41.82	37.33	41.69	38.04	41.29	34.46	38.53

Table 2: nDCG@5 and nDCG@10 document retrieval results on Fleurs-Retrieval-Eval for the different approaches across dimensions. NLLB-LLM2Vec uses the full embedding dimension (4096), while other approaches use Matryoshka dimensions. The most performant speech-based approach is colorized and the best scores are in bold.

Table 5

Table 7

Figure 6: Costs across Matryoshka dimensions.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Split	Fleurs	Alfa	CV	Kallaama	UB
Train	8.72	16.13	34.97	33.60	4.52
Test	1.75	2.84	6.21	5.91	1.12

Appendix

Table 7: The non-synthetic ASR dataset.

Explore similar work

Low resource cross-modal alignment using HGNN to enhance speech representation
Sep 19, 2026Yannick Yomie Nzeuhang, Marie Tahon, Paulin Melatagia YontaCross-Modal Alignment Speech Processing
Department of computer sciences, University of Yaounde I, Street, Yaounde, 812, Cameroon. · IRD, UMMISCO, Street, Bondy, F-93143, France. · LIUM, Le Mans Universit´e, Avenue Olivier Messiaen, Le Mans, 72085, France.
SSEmb: Training a Generalizable Speech Retrieval Model Using Only Automatic Speech Recognition Data
Oct 13, 2026Sangkwon Park, Dongwook Lee, Youngho Cho +3Audio-Text Retrieval
ECE, Seoul National University · IPAI, Seoul National University · University of Seoul +1
Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders
Aug 5, 2026Yexing Du, Kaiyuan Liu, Youcheng Pan +4Speech Translation Mixture of Experts
Harbin Institute of Technology, Harbin, China · Pengcheng Laboratory, Shenzhen, China

Sep 19, 2026J/K move · Enter open · S save

Low resource cross-modal alignment using HGNN to enhance speech representation

Yannick Yomie Nzeuhang, Marie Tahon, Paulin Melatagia Yonta

Department of computer sciences, University of Yaounde I, Street, Yaounde, 812, Cameroon. · IRD, UMMISCO, Street, Bondy, F-93143, France. · LIUM, Le Mans Universit´e, Avenue Olivier Messiaen, Le Mans, 72085, France.

Cross-Modal Alignment Speech Processing Low-Resource Language Processing

Open paper AskPDF arXiv

Speech-text space alignment is a multimodal representation learning method consisting to map different speech and text into a shared representation space, leading to enrichment of the representation of each modality. Proposed architectures, such as SAMU-XLSR, typically follow a student/teacher framework, with the goal of fine-tuning an audio encoder to produce representations that closely match those of the text. In this way a speech representation is semantically enriched. However, such systems generally require large amounts of training data and considerable computational resource, making them difficult to apply to low resources languages under frugal constraints. The present work proposes a data-efficient space alignment method based on Heterogeneous Graph Neural Networks and link prediction. The core idea is to leverage message passing to explicitly transfer information from the text modality to the speech modality, thereby reducing the need for large training datasets and intrinsically enriching the acoustic representations, all in a more interpretable manner. Although thoroughly explored for high-resource languages, word-level tasks in speech remain relevant for certain low-resource languages. Therefore, we conducted experiments on speech-text alignment at the word level using the TIMIT (English) dataset and Yemba (a Cameroonian language). Our approach yields results comparable to those of SAMU-XLSR, a state-of-the-art method, and even surpasses it for the Yemba language in the task of word retrieval, while using far fewer resources, demonstrating its power, frugality, and efficiency.

Abstract

Explore similar work

Low resource cross-modal alignment using HGNN to enhance speech representation

SSEmb: Training a Generalizable Speech Retrieval Model Using Only Automatic Speech Recognition Data

Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders