Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion

Yexing Du; Youcheng Pan; Zekun Wang; Zheng Chu; Yichong Huang; Kaiyuan Liu; Bo Yang; Yang Xiang

cs.CLFeb 25, 2026

Scalable Multilingual Multimodal Machine Translation with Speech-Text Fusion

Authors: Yexing Du, Youcheng Pan, Zekun Wang, Zheng Chu, Yichong Huang, Kaiyuan Liu, Bo Yang, Yang Xiang,

Organizations: Harbin Institute of Technology · Pengcheng Laboratory

Abstract

Multimodal Large Language Models (MLLMs) have achieved notable success in enhancing translation performance by integrating multimodal information. However, existing research primarily focuses on image-guided methods, whose applicability is constrained by the scarcity of multilingual image-text pairs. The speech modality overcomes this limitation due to its natural alignment with text and the abundance of existing speech datasets, which enable scalable language coverage. In this paper, we propose a Speech-guided Machine Translation (SMT) framework that integrates speech and text as fused inputs into an MLLM to improve translation quality. To mitigate reliance on low-resource data, we introduce a Self-Evolution Mechanism. The core components of this framework include a text-to-speech model, responsible for generating synthetic speech, and an MLLM capable of classifying synthetic speech samples and iteratively optimizing itself using positive samples. Experimental results demonstrate that our framework surpasses all existing methods on the Multi30K multimodal machine translation benchmark, achieving new state-of-the-art results. Furthermore, on general machine translation datasets, particularly the FLORES-200, it achieves average state-of-the-art performance in 108 translation directions. Ablation studies on CoVoST-2 confirms that differences between synthetic and authentic speech have negligible impact on translation quality. The code and models are released at https://github.com/yxduir/LLM-SRT.

Figures & tables

Figure 2: Overview of Our SMT Framework. The proposed system architecture comprises two core components: (1) MLLM pretraining and (2) Self-Evolution. This framework takes text input, synthesizes speech of the text via a TTS model, and leverages the MLLM to process both text and speech features for higher-quality translation output. Self-evolution mechanism can autonomously generate training data to iteratively optimize the framework.

Modules	Param	Stage	Details
Speech Encoder	$\sim$ 635M	-	Whisper’s encoder
Speech Adapter	$\sim$ 80.5M	All	Q-Former and MLP
LLM	$\sim$ 9.2B	-	GemmaX2-28-9B
LLM adapter	$\sim$ 8.9M	III	LoRA (r=16, alpha=32)
Total	$\sim$ 10B

Table 2: MLLM Pre-training. The blue color indicates the number of trainable parameters.

Models based on Text
Models		eng $\rightarrow$ deu			eng $\rightarrow$ fra		eng $\rightarrow$ ces
Models	Test2016	Test2017	MSCOCO	Test2016	Test2017	MSCOCO	Test2016	Test2018
DeepSeek-V3.1 ( Guo et al., 2025 )	44.2 / 87.3	41.1 / 86.8	36.4 / 83.2	55.3 / 88.2	54.0 / 87.7	53.5 / 85.8	37.9 / 90.7	35.9 / 89.7
Gemma3-27B-it ( Team et al., 2025 )	43.7 / 87.1	40.3 / 86.3	36.1 / 83.2	55.4 / 87.9	54.3 / 87.9	49.6 / 85.0	36.4 / 89.9	35.9 / 89.1
NLLB-moe-54B ( Team et al., 2022 )	41.4 / 86.2	39.7 / 85.8	34.7 / 82.1	55.1 / 87.4	54.8 / 87.7	53.3 / 85.3	35.7 / 88.9	35.8 / 88.3
Qwen3-Next-80B-A3B ( Team, 2025 )	41.6 / 86.3	37.6 / 85.9	31.9 / 82.5	53.2 / 87.8	51.9 / 87.6	50.4 / 85.1	29.2 / 87.2	27.9 / 85.9

Table 3: Translation Performance on Multi30K (BLEU / COMET) MMT Benchmark. The average character length of the input English text is 59.3 .

\dagger

indicates that the scores were directly cited from other research papers.

Models		FLORES-200			WMT24++
Models	eng $\rightarrow$ 27	jpn $\rightarrow$ 27	kor $\rightarrow$ 27	cmn $\rightarrow$ 27	eng $\rightarrow$ 22	eng $\rightarrow$ 22 ( $<$ 200)
Models based on Text
DeepSeek-V3.1 ( Guo et al., 2025 )	39.3 / 88.9	26.1 / 85.7	27.7 / 85.9	27.5 / 86.2	34.1 / 83.6	31.8 / 83.4
Gemma3-27B-it ( Team et al., 2025 )	37.4 / 88.0	23.8 / 81.0	25.0 / 81.2	24.5 / 81.5	34.3 / 82.9	31.8 / 82.6
NLLB-moe-54B ( Team et al., 2022 )	35.7 / 86.3	21.8 / 81.7	23.6 / 83.7	22.8 / 82.1	25.4 / 76.9	24.4 / 77.7
Qwen3-Next-80B-A3B ( Team, 2025 )	34.5 / 86.6	22.9 / 83.8	23.9 / 83.9	24.2 / 84.3	30.5 / 81.5	29.6 / 81.6

Table 4: Translation Performance on FLORES-200 and WMT24++ (spBLEU / COMET) MT Benchmarks. The average character length of the input English text is 130.4 for FLORES-200 and 191.3 for WMT24++. The notation

<200

indicates that the input English text length is within 200 characters. Detailed results are summarized in Tables 11 , and 12 in the Appendix.

Figure 3: COMET Results by Resource Level, Categorized as Low, Medium, and High. Our model shows an improvement in translation scores, particularly for low-scoring translation directions.

Input			eng $\rightarrow$ xx					spBLEU / COMET $\uparrow$
Text	AS	SS	ara	deu	fra	ind	jpn	tur	Avg.
✓			37.7 / 86.3	45.2 / 88.0	32.1 / 86.9	47.9 / 91.5	31.5 / 90.7	36.7 / 88.8	38.5 / 88.7
	✓		32.6 / 82.2	36.6 / 82.2	27.9 / 82.6	36.8 / 85.9	26.9 / 86.5	29.3 / 83.6	31.7 / 83.8
		✓	34.1 / 83.5	39.0 / 84.0	28.9 / 83.8	36.9 / 87.4	27.1 / 87.4	30.3 / 85.0	32.7 / 85.4
✓	✓		40.1 / 86.8	46.5 / 88.3	33.6 / 87.4	48.4 / 91.6	33.6 / 90.6	37.9 / 89.1	40.0 / 89.0
✓		✓	40.1 / 86.8	46.5 / 88.2	33.6 / 87.4	48.5 / 91.6	33.5 / 90.7	37.8 / 89.1	40.0 / 89.0

Table 5: Ablation Study on the CoVoST-2 Benchmark. A comparison of configurations with different modality inputs. (AS denotes authentic speech; SS denotes synthetic speech)

Models	eng $\rightarrow$ xx					spBLEU / COMET $\uparrow$
Models	jpn	cmn	tha	khm	lao	mya	Avg.
Baseline	33.3 / 91.3	41.6 / 89.2	42.5 / 88.7	24.1 / 84.2	31.5 / 84.7	20.1 / 88.1	32.2 / 87.7
SMT-9B	35.2 / 92.7	42.6 / 91.2	44.1 / 90.3	25.6 / 83.6	34.2 / 86.3	24.3 / 88.5	34.3 / 88.8
w/o SE	34.8 / 92.1	42.3 / 89.3	42.5 / 89.7	23.0 / 81.7	31.7 / 84.3	23.4 / 86.8	33.0 / 87.3

Table 6: Ablation Study on Self-Evolution (SE) Mechanism on the FLORES-200 benchmark.

Figure 4: Self-Evolution Rounds of spBLEU / COMET (eng

\rightarrow

xx) on FLORES-200 benchmark.

Figure 5: Case Study for Under-Translation. Having undergone speech pre-training, MLLMs align text words with speech. The SMT model, which receives this speech-text fusion input, is prevented from ignoring the input text, thereby mitigating omission errors.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

ISO-3	Language	Script	Family	Subgrouping	Resource
ben	Bemba	Latin	Atlantic-Congo	Benue-Congo	Low
ces	Czech	Latin	Indo-European	Balto-Slavic	High
cmn	Chinese	Han	Sino-Tibetan	Sinitic	High
deu	German	Latin	Indo-European	Germanic	High
eng	English	Latin	Indo-European	Germanic	High
fra	French	Latin	Indo-European	Italic	High

Appendix

Table 7: 11 Languages Supported by Image-Guided MMT datasets. The resource of each language is determined according to the taxonomy classes by ( Joshi et al., 2020 ) .

ISO-3	Language	Script	Family	Subgrouping	Resource
ara	Arabic	Arabic	Afro-Asiatic	Semitic	High
ben	Bengali	Bengali	Indo-European	Indo-Aryan	Med
ces	Czech	Latin	Indo-European	Balto-Slavic	High
cmn	Chinese	Han	Sino-Tibetan	Sinitic	High
deu	German	Latin	Indo-European	Germanic	High
eng	English	Latin	Indo-European	Germanic	High

Appendix

Table 8: 28 Languages Supported by Our Model. The resource of each language is determined according to the taxonomy classes by ( Joshi et al., 2020 ) .

Model	Task	Description	Dataset	Split	Data Size	Metric
MLLM	ASR	Automatic Speech Recognition	FLEURS †	train	$\sim$ 160h	WER $\downarrow$
	ASR	Automatic Speech Recognition	Common Voice 19	train	$\sim$ 3000h	WER $\downarrow$
	SMT	Speech-Guided	FLEURS †	train	$\sim$ 160h	spBLEU / COMET $\uparrow$
	SMT	Multimodal Machine Translation	Multi30K	train	$\sim$ 40h	BLEU / COMET $\uparrow$

Appendix

Table 9: Summary of Training Datasets for SMT Models. Data size refers to the actual amount used for training, as we removed some overly long samples.

\dagger

indicates that we performed data cleaning on the dataset. Since there is an overlap between the FLEURS and FLORES datasets, we removed the overlapping portions from the FLEURS training set.

Task	Description	Dataset	Split	Metric
MT	Machine Translation	FLORES-200	devtest	spBLEU / COMET $\uparrow$
MT	Machine Translation	WMT24++	test	spBLEU / COMET $\uparrow$
MMT	Multimodal Machine Translation	Multi30K	test	BLEU / COMET $\uparrow$
S2TT	Speech-to-Text Translation	CoVoST-2	test	spBLEU / COMET $\uparrow$

Appendix

Table 10: Summary of Evaluation Benchmarks.

Direction	DeepSeek	Gemma3	NLLB-moe	Qwen3-Next	Baseline	SMT
Direction	-v3.1	-27B	-54B	-80B-A3B	Baseline	-9B
eng $\rightarrow$ ara	20.0 / 78.5	20.0 / 78.1	18.5 / 74.6	19.4 / 77.6	19.4 / 77.3	19.1 / 77.5
eng $\rightarrow$ ben	25.9 / 83.3	25.4 / 82.7	23.5 / 79.7	16.2 / 78.5	24.8 / 82.1	23.6 / 80.7
eng $\rightarrow$ ces	36.3 / 85.9	36.3 / 84.6	23.4 / 79.0	30.3 / 82.2	36.4 / 85.1	35.8 / 85.5
eng $\rightarrow$ cmn	34.3 / 84.9	36.4 / 83.5	18.0 / 69.5	37.4 / 84.9	36.9 / 83.8	36.5 / 85.3
eng $\rightarrow$ deu	37.9 / 82.6	37.9 / 81.9	28.5 / 76.3	36.2 / 81.7	37.7 / 82.3	37.3 / 82.3

Appendix

Table 11: spBLEU / COMET Scores on the WMT24++ Benchmark.

Direction	DeepSeek	Gemma3	NLLB-moe	Qwen3-Next	Baseline	SMT
Direction	-v3.1	-27B	-54B	-80B-A3B	Baseline	-9B
eng $\rightarrow$ ara	41.6 / 88.1	41.7 / 87.8	41.8 / 86.8	38.3 / 87.1	42.9 / 87.8	42.6 / 89.5
eng $\rightarrow$ ben	33.6 / 87.8	30.0 / 86.6	34.5 / 86.4	28.0 / 85.6	34.8 / 86.6	34.3 / 86.6
eng $\rightarrow$ ces	44.0 / 92.5	41.4 / 91.3	40.3 / 90.9	37.9 / 90.2	42.7 / 91.6	43.1 / 92.9
eng $\rightarrow$ cmn	35.7 / 89.2	37.2 / 88.8	22.4 / 78.0	37.0 / 89.2	41.6 / 89.2	42.6 / 91.2
eng $\rightarrow$ deu	48.5 / 89.0	46.9 / 88.7	43.8 / 87.1	46.2 / 88.5	47.1 / 88.5	47.8 / 89.7

Appendix

Table 12: spBLEU / COMET Scores on the FLORES-200 Benchmark.

Multimodal MT

See all 11 papers

Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders
Aug 5, 2026Yexing Du, Kaiyuan Liu, Youcheng Pan +4Speech Translation Mixture of Experts
Harbin Institute of Technology, Harbin, China · Pengcheng Laboratory, Shenzhen, China
Efficient Training for Cross-lingual Speech Language Models
Apr 13, 2026Yan Zhou, Qingkai Fang, Yun Hong +1Speech Language Models Cross-Modal Alignment
Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China
Bandwidth-Efficient and Privacy-Preserving Edge-Cloud Many-to-Many Speech Translation
May 27, 2026Yexing Du, Kaiyuan Liu, Youcheng Pan +4Speech Translation Speech Processing
Harbin Institute of Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China · Harbin Institute of Technology, Harbin, China

Aug 5, 2026J/K move · Enter open · S save

Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders

Yexing Du, Kaiyuan Liu, Youcheng Pan, Bo Yang+3

Harbin Institute of Technology, Harbin, China · Pengcheng Laboratory, Shenzhen, China

Speech Translation Mixture of Experts

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have achieved significant success in speech-to-text translation (S2TT). However, when processing multilingual speech inputs, a single speech encoder shared across all languages suffers from the curse of multilinguality: languages at different resource levels compete for limited representation capacity, leading to strong high-resource performance but substantial degradation on low-resource speech. To address this problem and improve multilingual consistency, we propose MSRT, a novel framework built around a resource-aware Mixture of Speech Encoders (MoSE). MoSE uses an explicit language router to assign each utterance to an appropriate expert encoder. A frozen expert preserves high-resource language capabilities, while a trainable expert adapts to and specializes in medium- and low-resource languages. We further introduce a five-stage curriculum learning strategy that substantially reduces data dependence, requiring only 10 hours of paired S2TT data per language for effective alignment. We conduct extensive experiments on 45 languages, systematically evaluating all

45 \times 44

translation directions. Our 4B-parameter model achieves state-of-the-art performance, outperforming substantially larger baselines. Empirical analyses show that MoSE improves high-, medium-, and low-resource languages simultaneously, with the largest gains on low-resource speech, thereby breaking the curse of multilinguality without compromising high-resource performance. To support future multilingual S2TT research, we release our code and models.

Abstract

Multimodal MT

Explore similar work

Breaking the Curse of Multilinguality in Many-to-Many Speech-to-Text Translation via a Resource-Aware Mixture of Speech Encoders

Efficient Training for Cross-lingual Speech Language Models

Bandwidth-Efficient and Privacy-Preserving Edge-Cloud Many-to-Many Speech Translation