Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality

Mengyu Bu; Yang Feng

cs.CLMar 18, 2026

Language on Demand, Knowledge at Core: Composing LLMs with Encoder-Decoder Translation Models for Extensible Multilinguality

Authors: Mengyu Bu, Yang Feng

Organizations: Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) · State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences, Beijing, China

Abstract

Large language models (LLMs) exhibit strong general intelligence, yet their multilingual performance remains highly imbalanced. Although LLMs encode substantial cross-lingual knowledge in a unified semantic space, they often struggle to reliably interface this knowledge with low-resource or unseen languages. Fortunately, pretrained encoder-decoder translation models already possess balanced multilingual capability, suggesting a natural complement to LLMs. In this work, we propose XBridge, a compositional encoder-LLM-decoder architecture that offloads multilingual understanding and generation to external pretrained translation models, while preserving the LLM as an English-centric core for general knowledge processing. To address the resulting representation misalignment across models, we introduce lightweight cross-model mapping layers and an optimal transport-based alignment objective, enabling fine-grained semantic consistency for multilingual generation. Experiments on four LLMs across multilingual understanding, reasoning, summarization, and generation indicate that XBridge outperforms strong baselines, especially on low-resource and previously unseen languages, without retraining the LLM.

Figures & tables

Figure 1: Overview of XBridge. Pretrained multilingual NMT models provide broad language coverage but limited general reasoning capability, while English-centric LLMs excel at general reasoning yet struggle with low-resource or unseen languages. XBridge harmonizes these strengths through model composition, offloading multilingual processing to the pretrained multilingual model while leveraging the LLM as a knowledge core.

Figure 2: Left: XBridge composes a pretrained multilingual encoder-decoder with an LLM via lightweight mapping layers for multilingual understanding and generation, keeping the LLM frozen as a knowledge core. Right: A three-stage training strategy progressively aligns heterogeneous representations and adapts the encoder and decoder.

System	Low-Resource Languages				High-Resource Languages				Average
System	Bn-En	En-Bn	Sw-En	En-Sw	Ja-En	En-Ja	De-En	En-De	X-En	En-X
NLLB-200-1.3B	37.78	32.83	42.66	36.28	29.60	19.07	46.23	39.91	37.51	31.00
MetaMath-7B
MetaMath-7B	1.46	0.67	3.33	1.75	27.62	16.76	34.36	19.42	18.62	11.92
MindMerger	30.76	-	39.43	-	22.50	-	40.05	-	31.57	-
LayAlign	30.91	-	39.02	-	22.36	-	39.43	-	31.98	-

Table 1: FLORES-101 translation results for stage 1. For clarity, we report results on two low-resource languages (Bengali, Swahili) and two high-resource languages (Japanese, German), with complete results and COMET scores in Appendix C . "X" denotes all languages except for English. We bold the best scores for each LLM group.

Panel of Figure 3 — Figure 3: Multilingual reasoning accuracy on MGSM and multilingual summarization Rouge-L on XL-Sum, with complete results in Appendix C . Models with the same base LLM share the same color scheme, where lighter shades denote baselines and darker shades denote XBridge. "XBridge-LLM" refers to English reasoning by the LLM, while "XBridge-Dec" refers to multilingual reasoning by the composed decoder. For XL-Sum, since the baselines produce English-only summaries, we translate them into target languages using NLLB-200-1.3B for evaluation.

Panel of Figure 4 — Figure 4: Ablation analysis of XBridge. We compare different ablated variants of XBridge: encoder-only augmentation "w/o Decoder", loss ablation "w/o OT", removal of stage 1 "w/o Stage 1", and joint optimization of stage 2&3 "Joint Stage 2&3". “Lrl”, “Hrl”, and “Avg” denote low-, high-resource, and average performance, respectively.

Figure 5: Cross-lingual generalization to 41 untuned languages in FLORES-101. Left: X

\rightarrow

En direction. Right: En

\rightarrow

X direction. We directly evaluate the ablation variants described in Section 5.1 . Appendix C lists the included untuned languages and provides detailed results.

Panel of Figure 6 — Figure 6: Evaluation for language-on-demand generation. Appendix C presents detailed results.

Panel of Figure 7 — Figure 7: Visualization of sentence-level representation alignment for Chinese (Zh). We compare models trained without OT (left) and with OT (right) using t-SNE.

Panel of Figure 8 — Figure 8: LLaMA3-8B composed with M2M100-1.2B. “Lrl”, “Hrl”, and “Avg” denote low-resource, high-resource, and average performance, respectively. Hollow markers placed on the bottom boundary indicate models that lack En $\rightarrow$ X translation capability. Appendix C presents detailed results.

Panel of Figure 9 — Figure 9: MetaMath-7B composed with NLLB in different sizes (600M vs. 1.3B). “Lrl”, “Hrl”, and “Avg” denote low-, high-resource, and average performance, respectively. Appendix C presents detailed results.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 10: Statistics of training datasets used in different stages.

System	Training	Inference
SFT	1.0x	1.0x
Translate-Test	-	0.55x
MindMerger	1.42x	0.85x
XBridge	0.91x	0.66x

Appendix

Table 2: Relative speed comparison.

Figure 11: Case study on multilingual reasoning. Red text indicates the language token fed to the decoder, which controls the target generation language.

Figure 12: Radar plot comparison on X-CSQA.

ISO 639-1	Language	ISO 639-1	Language	ISO 639-1	Language
Af	Akrikaans	It	Italian	Pl	Polish
Ar	Arabic	Ka	Georgian	Ps	Pashto
Az	Azerbaijani	Kk	Kazakh	Pt	Portuguese
Cs	Czech	Km	Khmer	Ro	Romanian
El	Modern Greek	Ko	Korean	Sl	Slovenian
Et	Estonian	Lt	Lithuanian	Sv	Swedish

Appendix

Table 3: The included 41 untuned languages and corresponding language codes.

MetaMath-7B
System	Bn-En	En-Bn	De-En	En-De	Es-En	En-Es	Fr-En	En-Fr	Ja-En	En-Ja
NLLB-200-1.3B	37.78	32.83	46.23	39.91	35.37	31.35	47.14	51.33	29.60	19.07
MetaMath-7B	1.46	0.67	34.36	19.42	36.33	27.42	15.54	11.64	27.62	16.76
MindMerger	30.76	-	40.05	-	28.77	-	38.65	-	22.50	-
LayerAlign	30.91	-	39.43	-	30.16	-	40.48	-	22.36	-
XBridge (Ours)	35.47	29.23	41.42	35.39	30.51	28.88	42.14	49.49	24.52	19.60

Appendix

Table 4: Detailed BLEU scores on FLORES-101 for stage 1. "X" denotes all languages except for English. We bold the best scores for each LLM group.

MetaMath-7B
System	Bn-En	En-Bn	De-En	En-De	Es-En	En-Es	Fr-En	En-Fr	Ja-En	En-Ja
NLLB-200-1.3B	88.30	85.67	88.80	86.12	86.72	85.68	88.83	87.05	87.05	86.45
MetaMath-7B	50.85	35.10	86.25	76.59	84.39	78.49	86.84	78.45	82.62	76.35
MindMerger	86.87	-	87.81	-	85.52	-	87.64	-	84.94	-
LayerAlign	87.10	-	87.77	-	85.75	-	87.77	-	84.78	-
XBridge (Ours)	88.03	84.98	88.17	85.31	86.16	85.26	88.07	86.53	86.04	85.99

Appendix

Table 5: Detailed COMET scores on FLORES-101 for stage 1. "X" denotes all languages except for English. We bold the best scores for each LLM group.

System	Bn	De	En	Es	Fr	Ja	Ru	Sw	Th	Zh	Avg
MetaMath-7B
SFT	30.00	40.40	48.00	39.60	39.20	32.40	39.60	35.20	32.80	36.00	37.32
Translate-Test	34.80	49.20	53.20	47.60	40.80	45.20	49.20	43.20	32.40	46.00	44.16
MindMerger	43.60	57.20	67.20	59.20	60.00	45.60	57.60	49.20	42.40	48.80	53.08
LayerAlign	40.40	54.80	62.80	59.20	52.40	51.60	56.40	53.20	47.20	51.20	52.92
XBridge_LLM (Ours)	48.40	54.80	67.20	60.80	57.20	47.20	57.60	46.40	45.60	52.80	53.80

Appendix

Table 6: Detailed accuracy results on MGSM. "XBridge-LLM" denotes English reasoning outputs from the LLM, while "XBridge-Dec" denotes multilingual reasoning outputs via the external decoder. We bold the best scores for each LLM group.

System	Bn	En	Es	Fr	Ja	Ru	Sw	Th	Zh	Avg
MetaMath-7B
SFT	16.79	33.27	23.68	28.42	29.20	25.42	27.01	18.68	20.16	24.74
MindMerger	17.93	27.27	21.03	26.05	28.11	22.25	26.49	19.20	18.56	22.99
LayerAlign	17.94	28.84	21.49	26.31	27.55	22.22	26.29	20.09	19.02	23.30
XBridge (Ours)	18.93	31.66	22.75	27.96	32.12	24.64	28.90	20.70	24.12	25.75
LLaMA3-8B

Appendix

Table 7: Detailed multilingual Rouge-L results on XL-Sum. For XL-Sum, since the baselines produce English summaries only, we translate them into target languages using NLLB-200-1.3B for comparison. We bold the best scores for each LLM group.

Variants	Lrl-En	En-Lrl	Hrl-En	En-Hrl	X-En	En-X
MetaMath-7B	2.87	1.05	26.50	17.36	18.62	11.92
w/o OT	35.33	22.20	32.98	25.30	33.76	24.27
w/o Decoder	36.22	-	35.01	-	35.41	-
XBridge	33.93	25.56	31.98	29.37	32.63	28.10

Appendix

Table 8: Ablation results on FLORES-101 for stage 1. "Lrl" and "Hrl" denote low- and high-resource languages, respectively. Following Shi et al. (2023) , we treat Bn, Sw, and Th as low-resource languages, and the remaining as high-resource ones. "X" denotes all languages except for English. We bold the best scores for each LLM group.

Variants	Lrl-LLM	Lrl-Dec	Hrl-LLM	Hrl-Dec	Avg-LLM	Avg-Dec
MetaMath-7B	5.60	-	50.06	-	36.72	-
SFT	32.67	-	39.31	-	37.32	-
w/o OT	48.27	47.20	56.23	54.57	53.84	52.36
w/o Decoder	48.27	-	56.91	-	54.32	-
w/o Stage 1	29.33	28.13	52.51	51.37	45.56	44.40
Joint Stage 2&3	43.07	42.53	48.63	47.26	46.96	45.84

Appendix

Table 9: Ablation accuracy results on MGSM. "Lrl" and "Hrl" denote low- and high-resource languages, respectively. "-LLM" denotes English reasoning outputs from the LLM, while "-Dec" denotes multilingual reasoning outputs via the external decoder. We bold the best scores.

Variants	Bn	En	Es	Fr	Ja	Ru	Sw	Th	Zh	Lrl	Hrl	Avg
SFT	16.79	33.27	23.68	28.42	29.20	25.42	27.01	18.68	20.16	20.83	26.69	24.74
w/o OT	18.55	31.49	22.68	27.86	32.05	24.27	28.35	20.27	23.81	22.39	27.03	25.48
XBridge	18.85	31.50	22.84	28.18	32.26	24.36	28.57	21.08	24.40	22.83	27.26	25.78

Appendix

Table 10: Ablation Rouge-L results of XL-Sum on MetaMath-7B. "Lrl" and "Hrl" denote low- and high-resource languages, respectively. We bold the best scores.

Languages	X $\rightarrow$ En				En $\rightarrow$ X
Languages	NLLB	MetaMath	w/o OT	XBridge	NLLB	MetaMath	w/o OT	XBridge
Af	58.33	17.64	53.69	54.90	42.69	12.79	37.30	41.75
Ar	42.42	24.59	40.61	38.20	33.13	10.94	16.81	26.37
Az	28.25	28.97	25.35	23.57	19.39	16.23	12.25	16.18
Cs	41.17	1.34	40.47	38.54	35.85	0.32	23.39	30.03
El	37.68	3.59	37.19	35.43	28.16	0.80	19.07	24.34

Appendix

Table 11: Detailed results on 41 untuned languages, composing MetaMath-7B with NLLB-200-1.3B. We evaluate cross-model mapping quality on FLORES-101, following the main experiments. "X" denotes all languages except for English. We bold the best scores for the LLM group.

	Bn	De	En	Es	Fr	Ja	Ru	Sw	Th	Zh	Avg
Bn	-	22.47	37.09	20.56	31.36	16.69	21.03	22.85	13.62	17.42	22.57
De	24.74	-	45.75	25.57	39.82	18.73	26.51	26.34	14.82	19.81	26.90
En	28.42	35.45	-	29.59	49.38	20.12	30.57	34.68	17.09	23.11	29.82
Es	22.21	23.42	32.00	-	33.74	16.82	21.78	22.01	12.93	17.57	22.50
Fr	24.26	28.62	46.10	26.64	-	17.75	25.90	26.99	14.47	19.96	25.63
Ja	20.76	20.05	27.63	18.59	26.94	-	18.82	19.76	12.60	17.16	20.26

Appendix

Table 12: BLEU scores for cross-lingual generation of XBridge, composing LLaMA3-8B with NLLB-200-1.3B. Rows denote the source language and columns denote the target language. The source text is encoded by the multilingual encoder, the LLM produces an English translation, and the decoder generates the target-language text conditioned on the target-language token. For X

\rightarrow

En directions, we directly evaluate the LLM outputs.

MGSM	Bn	De	En	Es	Fr	Ja	Ru	Sw	Th	Zh	Avg
Bn	46.00	44.40	45.20	45.60	40.80	42.80	44.00	46.00	44.00	44.80	44.36
De	58.40	56.40	58.40	59.20	53.20	54.80	58.40	58.00	58.40	59.20	57.44
En	63.60	62.80	63.20	63.60	59.60	61.60	62.80	63.20	61.20	64.00	62.56
Es	60.00	57.60	61.60	61.20	55.20	56.80	60.40	60.80	57.60	60.80	59.20
Fr	58.80	58.40	60.40	60.80	57.20	56.80	60.80	59.60	58.00	60.00	59.08
Ja	49.20	47.20	49.20	50.00	43.60	46.80	48.80	48.80	48.00	48.80	48.04

Appendix

Table 13: Accuracy for langauge-on-demand generation of XBridge, composing LLaMA3-8B with NLLB-200-1.3B. Rows denote the query language and columns denote the response language. The query is first processed by the encoder, the LLM produces an English response, and the decoder then generates the target language response via the target language token.

System	Lrl-En	En-Lrl	Hrl-En	En-Hrl	X-En	En-X
M2M100-1.2B	28.00	22.25	33.11	32.55	31.41	29.12
LLaMA3-8B	32.03	17.43	36.77	32.32	35.19	27.36
MindMerger	31.37	-	32.67	-	32.24	-
LayerAlign	31.28	-	32.47	-	32.08	-
XBridge (Ours)	32.24	21.66	35.86	28.67	34.65	26.33

Appendix

Table 14: Detailed FLORES-101 translation results for LLaMA3-8B composed with M2M100-1.2B. "Lrl" and "Hrl" denote low- and high-resource languages, respectively. "X" denotes all languages except for English. We bold the best scores for the LLM group.

System	Bn	De	En	Es	Fr	Ja	Ru	Sw	Th	Zh	Avg
SFT	38.80	46.40	54.00	54.80	44.40	38.80	45.20	40.80	44.00	49.20	45.64
Translate-Test	40.40	56.00	62.00	54.80	54.80	51.60	57.20	46.80	35.20	48.00	50.68
MindMerger	52.40	55.60	62.40	55.60	59.60	50.40	58.40	47.60	50.80	56.40	54.92
LayerAlign	51.20	60.00	65.20	58.00	55.60	50.00	59.20	49.20	50.80	54.00	55.32
XBridge_LLM (Ours)	49.20	57.20	66.00	60.80	52.80	54.40	54.80	47.20	52.80	60.00	55.52
XBridge_Dec (Ours)	48.40	53.20	62.80	59.20	48.40	53.20	54.00	44.40	48.40	58.80	53.08

Appendix

Table 15: Detailed MGSM results for LLaMA3-8B composed with M2M100-1.2B. "XBridge-LLM" denotes English reasoning outputs from the LLM, while "XBridge-Dec" denotes multilingual reasoning outputs via the external decoder. We bold the best scores for the LLM group.

System	Lrl-En	En-Lrl	Hrl-En	En-Hrl	X-En	En-X
NLLB-200-600M	33.59	26.43	34.83	29.52	34.42	28.49
NLLB-200-1.3B	37.09	28.86	37.72	32.08	37.51	31.00
MetaMath-7B	2.87	1.05	26.50	17.36	18.62	11.92
XBridge (600M)	31.20	23.42	29.46	26.55	30.04	25.50
XBridge (1.3B)	35.26	26.91	32.42	31.24	33.37	29.80

Appendix

Table 16: Detailed FLORES-101 translation results for MetaMath-7B composed with NLLB-200 in different sizes (600M vs. 1.3B). "Lrl" and "Hrl" denote low- and high-resource languages, respectively. "X" denotes all languages except for English. We bold the best scores for the LLM group.

System	Bn	De	En	Es	Fr	Ja	Ru	Sw	Th	Zh	Avg
SFT	38.80	46.40	54.00	54.80	44.40	38.80	45.20	40.80	44.00	49.20	45.64
XBridge_LLM (600M)	47.60	56.00	65.60	56.00	56.00	47.20	56.00	50.80	49.20	50.00	53.44
XBridge_Dec (600M)	45.20	52.80	65.60	55.60	51.60	44.40	54.40	50.80	46.00	49.20	51.56
XBridge_LLM (1.3B)	48.40	54.80	67.20	60.80	57.20	47.20	57.60	46.40	45.60	52.80	53.80
XBridge_Dec (1.3B)	47.20	52.80	66.80	60.00	51.60	44.80	56.00	45.60	44.40	52.00	52.12

Appendix

Table 17: Detailed FLORES-101 translation results for MetaMath-7B composed with NLLB-200 in different sizes (600M vs. 1.3B). "X" denotes all languages except for English. We bold the best scores for the LLM group.

Explore similar work

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs
Apr 14, 2026Daniil Gurgurov, Tom Röhr, Sebastian von Rohrscheidt +3Cross-Lingual Reasoning in Language Models Multilingual Language Models
German Research Center for Artificial Intelligence (DFKI), Saarland University · Berliner Hochschule für Technik
Bridging Linguistic Gaps: Cross-Lingual Mapping in Pre-Training and Dataset for Enhanced Multilingual LLM Performance
Apr 12, 2026Weihua Zheng, Chang Liu, Zhengyuan Liu +5Multilingual Language Models Cross-Lingual Representation Alignment
Singapore University of Technology and Design, Singapore · ByteDance, Singapore · Agency for Science, Technology and Research, Singapore
Left Behind: Cross-Lingual Transfer as a Bridge for Low-Resource Languages in Large Language Models
Mar 22, 2026Abdul-Salem BeibitkhanLow-Resource Language Processing Multilingual Language Model Evaluation
North American University

Apr 14, 2026J/K move · Enter open · S save

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs

Daniil Gurgurov, Tom Röhr, Sebastian von Rohrscheidt, Josef van Genabith+2

German Research Center for Artificial Intelligence (DFKI), Saarland University · Berliner Hochschule für Technik

Cross-Lingual Reasoning in Language Models Multilingual Language Models

Open paper AskPDF arXiv

Despite advances in multilingual capabilities, most large language models (LLMs) remain English-centric in their training and, crucially, in the reasoning traces they produce: even for non-English problems, they predominantly reason in English, leaving non-English users with intermediate reasoning they cannot follow. We address this disparity with two contributions. (a) We introduce

\texttt{ReasonXL}

, the first large-scale parallel corpus of cross-domain reasoning traces in five European languages (English, German, French, Italian, and Spanish): over 2.5M aligned samples (11B tokens) per language, each comprising prompt, reasoning trace, and final output, enabling direct supervision of language-specific reasoning. (b) Using

\texttt{ReasonXL}

, we demonstrate that LLMs can be adapted to reason entirely in a target language through a simple two-stage pipeline of supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR), matching or exceeding the base models' English-reasoning performance in two of four languages and recovering most of it in the remaining two.

Abstract

Explore similar work

ReasonXL: A Large-Scale Corpus for Language-Specific Reasoning in LLMs

Bridging Linguistic Gaps: Cross-Lingual Mapping in Pre-Training and Dataset for Enhanced Multilingual LLM Performance

Left Behind: Cross-Lingual Transfer as a Bridge for Low-Resource Languages in Large Language Models