cs.CLApr 12, 2026
SaveBridging Linguistic Gaps: Cross-Lingual Mapping in Pre-Training and Dataset for Enhanced Multilingual LLM Performance
Organizations: Singapore University of Technology and Design, Singapore · ByteDance, Singapore · Agency for Science, Technology and Research, Singapore
Abstract
Multilingual Large Language Models (LLMs) struggle with cross-lingual tasks due to data imbalances between high-resource and low-resource languages, as well as monolingual bias in pre-training. Existing methods, such as bilingual fine-tuning and contrastive alignment, can improve cross-lingual performance, but they often require extensive parallel data or suffer from instability. To address these challenges, we introduce a Cross-Lingual Mapping Task during the pre-training phase, which enhances cross-lingual alignment without compromising monolingual fluency. Our approach bi-directionally maps languages within the LLM embedding space, improving both language generation and comprehension. We further propose a Language Alignment Coefficient to robustly quantify cross-lingual consistency, even in limited-data scenarios. Experimental results on machine translation (MT), cross-lingual natural language understanding (CLNLU), and cross-lingual question answering (CLQA) show that our model achieves gains of up to 11.9 BLEU points in MT, 6.72 points in CLQA BERTScore-Precision, and more than 5% in CLNLU accuracy over strong multilingual baselines. These findings highlight the potential of incorporating cross-lingual objectives into pre-training to improve multilingual LLMs.
Figures & tables
Figure 1. Continue pre-training including NTP and CL. "<s>" is "start token", " " is the mask of the token. "x", "y", "z" are tokens from different languages.
Figure 2. Language alignment of different pre-trained models. LAC is Language Alignment Coefficient.
| Task Name | Sub-Task | Language / Language Pairs | Data Volume | Data Source |
| Monolingual Continue Pretraining | NA | English | 37,182,019 tokens | ClutureX |
| Chinese | 93,124,828 tokens | |||
| Japanese | 85,415,308 tokens | |||
| Ukrainian | 187,336,165 tokens | |||
| Czech | 185,235,692 tokens | |||
| Cross-Lingual Mapping | NA | EN-ZH, EN-CS, CS-UK, ZH-JP | 3.7M sentence pairs per language pair | WMT2024 |
Table 1. Task Details and Data Statistics
| Task | Prompts |
| MT | Translate the following <Source Language> to <Target Language>: <Source Sentence> |
| CLSum | Please read the following <Source Language> text and generate a short and precise <Target Language> summary: <Source Language Paragraph> |
| Xquad | <Source Language Paragraph> Based on the above paragraph, answer the following questions in <Target Language> or numbers. |
| Cross-Lingual Open-ended Question and Answering | Answer the following questions in <Target Language>. <Question in Source Language> |
| CLNLU | Answer the following single-choice questions and output only the option. <Source Question> <Target Options>. |
Table 2. Prompts for Different Tasks
| Lang | L-3-8B | L_NTP | L_Bi_NTP | L_CL-only | L_CL |
| EN | 30.1 | 27.4 | 26.9 | 30.7 | 25.5 |
| ZH | 22.4 | 22.1 | 21.3 | 23.3 | 20.6 |
| CS | 21.1 | 16.8 | 16.1 | 22.1 | 15.2 |
| UK | 19.1 | 17.2 | 16.9 | 19.5 | 16.0 |
| JP | 25.1 | 22.9 | 22.2 | 25.5 | 20.8 |
Table 3. Perplexity Scores of Languages for Different Models. Best scores are in bold .
| Tasks | Data Sets | Metric | Llama-3-8B | Llama_NTP | Llama_Bi_NTP | Llama_CLI | Llama_CL |
| MT | WMT-EN-ZH | BLEU | 36.2 0.5 | 37.8 0.5 (+1.6) | 44.0 0.5 (+7.8) | 49.4 0.5 (+13.1) | 48.2 0.6 (+12.0) |
| WMT-EN-CS | 19.4 0.3 | 25.1 0.4 (+5.7) | 26.3 0.4 (+6.9) | 31.4 0.6 (+12.0) | 31.2 0.5 (+11.8) | ||
| WMT-CS-UK | 18.2 0.3 | 21.0 0.4 (+2.8) | 23.4 0.4 (+5.2) | 25.1 0.4 (+6.9) | 25.8 0.5 (+7.6) | ||
| Flores-EN-ZH | 28.6 0.4 | 29.9 0.4 (+1.3) | 33.6 0.5 (+5.0) | 37.6 0.4 (+9.0) | 36.2 0.5 (+7.6) | ||
| Flores-EN-CS | 17.2 0.3 | 21.0 0.4 (+3.8) | 21.6 0.4 (+4.4) | 25.7 0.5 (+8.5) | 23.2 0.4 (+6.0) | ||
| Flores-CS-UK | 14.3 0.3 | 18.0 0.3 (+3.7) | 18.5 0.4 (+4.2) | 20.0 0.3 (+5.7) | 19.4 0.4 (+5.1) |
Table 4. Evaluation Results on Cross-Lingual Tasks for Llama-3. All results in this table are obtained by evaluating the instruction-fine-tuned models.
| Tasks | Data Sets | Metric | BLOOM-3B | BLOOM_NTP | BLOOM_Bi_NTP | BLOOM_CL |
| MT | WMT-EN-ZH | BLEU | 11.3 0.4 | 13.6 0.4 (+2.3) | 16.6 0.4 (+5.3) | 17.5 0.5 (+6.2) |
| Flores-EN-ZH | 13.0 0.4 | 16.7 0.4 (+3.7) | 18.9 0.5 (+5.9) | 22.6 0.5 (+9.6) | ||
| WMT-EN-ZH | COMET | 57.93 0.28 | 58.49 0.30 (+0.56) | 59.82 0.34 (+1.89) | 59.70 0.34 (+1.77) | |
| Flores-EN-ZH | 60.25 0.30 | 62.63 0.33 (+2.38) | 70.41 0.40 (+10.16) | 74.68 0.43 (+14.43) | ||
| CLSum | CrossSum-EN-ZH | R-1 | 6.20 | 6.70(+0.5) | 7.10(+0.9) | 7.30(+1.1) |
| R-2 | 1.90 | 2.20(+0.3) | 2.90(+1.0) | 3.40(+1.5) |
Table 5. Evaluation Results on Cross-Lingual Tasks for BLOOM. All results in this table are obtained by evaluating the instruction-fine-tuned models.
| Types of open-ended questions | Question Example |
| Creative prompts | Compose a poem about childhood. |
| Commonsense reasoning | Why do people wear sunscreen? |
| Knowledge-based queries | What are the key features of quantum computing? |
| List-style questions | List five characteristics of a good employee. |
Table 6. Examples of open-ended question types
| Rank | L-3-8B | L_NTP | L_Bi_NTP | L_CL |
| 1 | 181 | 271 | 252 | 325 |
| 2 | 332 | 182 | 281 | 190 |
| 3 | 261 | 213 | 212 | 202 |
| 4 | 163 | 271 | 192 | 220 |
| Same | 63 | |||
Table 7. Evaluation Results on Open-Ended CLQA Task. Best scores are in bold .
| Test Datasets | ||||
| WMT-en-zh | 62.04 | 67.26 | 68.87 | 71.03 |
| WMT-en-cs | 66.72 | 63.42 | 69.78 | 72.47 |
| WMT-cs-uk | 77.92 | 74.21 | 81.17 | 84.97 |
| Flores-en-zh | 75.64 | 78.31 | 81.09 | 83.05 |
| Flores-en-cs | 79.95 | 76.44 | 83.02 | 84.87 |
| Flores-cs-uk | 79.96 | 60.11 | 83.45 | 86.04 |
Table 8. COMET score for MT task of models based on different pre-training settings. E_cross is our method.
| Model | MMLU | LogiQA | Alpaca_Eval |
| (Accuracy %) | (Accuracy %) | (Win Rate %) | |
| L-3-8B-Ins | 56.77 | 36.71 | 50.25 |
| L_CL_Ins | 57.01 | 36.25 | 49.75 |
Table 9. Model Performance Comparison on MMLU, LogiQA, and Alpaca_Eval. Best scores are in bold .
| Task | Question | L-3-8B | L_NTP | L_Bi_NTP | L_CL | Remark |
| MT | Translate the following English to Chinese: "It also arguably makes reading easier, though writing is somewhat complicated by the need to find out whether a verb or adjective is used in a substantivized form." | It also arguably makes reading easier, though writing is somewhat complicated by the need to find out whether a verb or adjective is used in a substantivized form. | It also arguably makes reading easier, though writing is somewhat complicated by the need to find out whether a verb or adjective is used in a substantivized form. | 它也使阅读更容易，但由于需要确定动词或形容词是否使用了名词化形式，因此写作有些复杂。 | 它也可以说使阅读更容易，虽然写作是有点复杂的，因为需要找到一个动词或形容词是否在名词形式中使用。 | Llama-3-8B and Llama_NTP failed to translate the source sentence. |
| Translate the following English to Chinese: "According to the lawsuit, waste from the UN camp was not properly sanitized, causing bacteria to enter the tributary of the Artibonite River, one of Haiti’s largest." | 根据诉讼,来自联合国营地的废物未得到适当消毒,导致细菌进入艺术博尼特河,海地最大的河流之一。 | 根据诉讼,联合国营地的垃圾没有得到适当的消毒,使细菌进入阿尔蒂波内特河,该河是海地最大的河流之一。 | 根据诉讼,联合国营地的垃圾没有得到适当的消毒,使细菌进入阿尔蒂波尼特河,是海地最大的河流之一。 | 根据诉讼，联合国营地的废物没有得到适当的消毒，导致细菌进入阿尔蒂博尼特河的分流之一，即海地最大的分流之一。 | Llama_CL generates a more detailed, accurate translation. |
Table 10. Translation Results Comparison
| Task | Question | L-3-8B | L_NTP | L_Bi_NTP | L_CL | Remark |
| CLSum | Please read the following English text and generate a short and precise Chinese summary: "By Peter BilesBBC World Affairs Correspondent The then-prime minister only saw it was likely after getting "raw intelligence" two days before the Argentines landed. Papers released under the 30-year rule show Mrs Thatcher was acutely worried about retaking the islands. One historian said the documents were among the "most powerful material" declassified in the last three decades. In October 1982, a few months after the war ended, Mrs Thatcher gave evidence behind closed doors to the Falkland Islands Review Committee, chaired by Lord Franks. The transcript of that dramatic testimony has now been published for the first time. "I never, never expected the Argentines to invade the Falklands head-on. It was such a stupid thing to do, as events happened, such a stupid thing even to contemplate doing", Mrs Thatcher told the Franks Committee. There had been some British contingency planning in the month before the Argentine invasion of the Falklands. On 26 March 1982, Ministry of Defence officials came back to Mrs Thatcher with a plan to deter a full-scale invasion. One sentence shocked her, and she wrote it in her diary: "Moreover, if faced with Argentine occupation on arrival there would be no certainty that such a force would be able to retake the dependency." She told the committee: "You can imagine that turned a knife in my heart, that lot." However, in her oral evidence, she said she had still considered an invasion unlikely: "I again stress, I thought that they would be so absurd and ridiculous to invade the Falklands that I did not think it would happen. "Nevertheless, one has always to make contingency plans, and soon after we got the South Georgia incident happening." The picture changed on 31 March 1982 when Mrs Thatcher was shown intelligence suggesting that an invasion was on the cards: "I just say it was the worst I think moment of my life." She also told the committee: "That night no-one could tell me whether we could retake the Falklands - no-one. We did not know - we did not know." The British foreign secretary at the time, Lord Carrington, also gave evidence to the Franks Committee, where he too held the view that Argentina was not going to invade the Falklands. "I have been accused, and was accused violently after 2 April in the House of Commons, and in the press and elsewhere - of wilfully ignoring signs and statements and evidence, and just ignoring it. I can truthfully say I did not do any of these things… they (the signs of an invasion) were not there," … It also bases its claim on the islands’ proximity to the South American mainland. Some 255 British servicemen and three Falklands civilians died during the conflict. The number of Argentine dead is estimated at about 650 All document images courtesy of the National Archives. | 玛格丽特·撒切尔前英国首相在1982年福克兰群岛战争之前就有所预料，但当时的政府官员却没有意识到这一点。 | 英国前首相玛格丽特·撒切尔说，她1982年对英国失去福克兰群岛的战争时期所面临的压力是"人生中最糟糕的一刻"。 | 1982年4月2日，阿根廷突然入侵福克兰群岛，英国迅速集结军队回应。30年后，英国政府公布了关于这场战争的新档案。 | 马格丽特·撒切尔夫人当时说，她的生命从来没有经历过如此严重的危机。现在，刚公布的政府文件显示，这个英国首相在1982年阿根廷对福克兰群岛的入侵之前，并没有料到会发生这样的事。 | L_CL lists two facts but does not generate a summary statement. |
Table 11. CLSum Results Comparison
| Task | Question | L-3-8B | L_NTP | L_Bi_NTP | L_CL | Remark |
| CLQA | Answer the following questions in Czech. "Write a limerick about the joys of childhood in the style of Lewis Carroll." | Kdybys si vzpomněl na dětství, považte si to rádi!Bylo to tak plné nadšené vzrušení, plné úžasu a radosti!Nakonec to skončilo, ale jaká úžasná doba, když jsme byli malí a takoví, jaké jsme dnes. | Jako malý jsem si užíval,Až se mi v životě zatajvalo,Vždycky jsem věděl,Co jsem chtěl,Jako malé dítě, bylo to nejlepší. | Jako malý holky v lese, \nKterý nás potěšil,\nTento svět byl plný štěstí,\nS radostí a veselím, \nNikdy jsme se necítily starosti. Overpostingotázkou | Koukal jsem na svět se štěstím plný,\nA všude kolem se tají radost,\nVysoké stromy, malé holky, \nNávštěvy kamarádů, hrací kosti,\nVšechno bylo úžasné, ne? To je pravda! | While both L_CL and L_Bi_NTP employ parallel structure, a technique common in poetry, L_CL’s writing style is more poetic, with a rhythm that better aligns with the musical quality of verse. |
| Answer the following questions in Czech. "Explain to me the importance of the following quotation.’The only way to do great work is to love what you do.’" | Tento citát znamená, že pokud chcete dosáhnout úspěchu v jakékoli oblasti, musíte milovat to, co děláte. Kreativita a odhodlání jsou důležitými aspekty úspěchu, ale pokud se nebudete zajímat o to, co děláte, pravděpodobně nebudete schopni dosáhnout svých cílů. Mít opravdový zájem o své pracovní povinnosti je klíčem k tomu, abyste se dostali k tomu, kde se chcete dostat. | To love what you do je důležitý aspekt úspěšné práce. Když si člověk vybere práci, kterou opravdu miluje, je mnohem pravděpodobnější, že se mu podaří vykonávat ji s velkým nasazením a přinášet kvalitní výsledky. Milovat svou práci zajišťuje, že člověk se bude cítit spokojenějším a bude mít větší motivaci a nadšení k práci. Navíc, když se člověk skutečně zajímá o svou práci, je pravděpodobnější, že bude mít větší znalosti a schopnosti a bude lépe splňovat požadavky a očekávání zaměstnavatele. | To mít velký úspěch ve svém povolání, je třeba milovat to, co děláte. Když máte rádi to, co děláte, máte motivaci pracovat tvrdě, vyvíjet se a dosahovat cílů. Milujete své povolání, budete se cítit spokojenější a vaše práce se bude lépe projevovat v práci a životě. | To nejlepší, co můžete dělat, je milovat to, co děláte. Pokud vás vaše práce nudí, vaše výsledky budou značně omezené. Když milujete svou práci, získáte z ní větší spokojenost a z ní se vám líbí být. To vám umožní dělat svou práci s větší odvahou a odhodláním a získáte více z jejího výkonu. | Responses of L_CL and L-3-8B are relatively accurate, with the answer of L_CL answer being more relevant to the question. The response of L_NTP contains English content, while the answer of L_Bi_NTP has repetitive elements. |
Table 12. CLQA Results Comparison