Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation

Yifeng Liu; Siqi Ouyang; Yatish Hosmane Revanasiddappa; Lei Li

cs.CLMar 13, 2026

Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation

Authors: Yifeng Liu, Siqi Ouyang, Yatish Hosmane Revanasiddappa, Lei Li

Organizations: Carnegie Mellon University

Abstract

Large Language Models (LLMs) have demonstrated remarkable capability in machine translation on high-resource language pairs, yet their performance on low-resource translation still lags behind. Existing post-training methods rely heavily on high-quality parallel data, which are often scarce or unavailable for low-resource languages. In this paper, we introduce WALAR, a reinforcement training method using only monolingual text to elevate LLMs' translation capabilities on massive low-resource languages while retaining their performance on high-resource languages. Our key insight is based on the observation of failure modes (or "holes") in existing source-based multilingual quality estimation (QE) models. Reinforcement learning (RL) using these QE models tends to amplify such holes, resulting in poorer multilingual LLMs. We develop techniques including word alignment and language alignment to mitigate such holes in WALAR's reward for RL training. We continually trained an LLM supporting translation of 101 languages using WALAR. The experiments show that our new model outperforms LLaMAX, one of the strongest open-source multilingual LLMs by a large margin on 1400 language directions on Flores-101 dataset.

Figures & tables

Panel of Figure 1 — Figure 1: Holes of source-based quality estimation metrics. RL training using these metrics will amplify the holes in LLMs.

Figure 2: Illustration of WALAR . On each step, the LLM is prompted to translate one monolingual sentence into another language with several different rollouts. Each output will then be evaluated by language alignment, quality estimation, and word alignment. Finally, the LLM is trained using GRPO with the reward on the previous step iteratively.

spBLEU	x $\rightarrow$ en	en $\rightarrow$ x	x $\rightarrow$ ar	ar $\rightarrow$ x	x $\rightarrow$ tr	tr $\rightarrow$ x	x $\rightarrow$ hi	hi $\rightarrow$ x	x $\rightarrow$ ru	ru $\rightarrow$ x	x $\rightarrow$ zh	zh $\rightarrow$ x	x $\rightarrow$ sw	sw $\rightarrow$ x	Avg
NLLB-200-1.3B $\Delta$	39.03	30.23	24.91	22.57	23.30	22.47	24.51	22.22	25.80	22.18	18.71	18.40	24.37	21.77	24.32
Hunyuan-MT-7B	21.04	14.32	16.29	9.84	15.37	10.72	13.19	9.66	16.17	10.46	15.55	9.83	8.18	7.46	12.72
Tower-Plus-9B	31.55	15.32	9.42	10.69	13.36	11.15	20.53	11.39	23.06	11.54	23.01	9.98	3.29	9.46	14.55
Aya-Expanse-8B	24.03	14.25	14.59	10.33	12.73	10.62	14.07	10.20	17.22	10.72	15.49	9.29	2.36	4.71	12.19
Qwen3-8B	29.22	16.67	15.70	11.11	14.00	11.45	12.10	11.48	19.05	12.08	21.08	10.58	1.29	6.72	13.75
+WALAR	28.59	17.11	15.71	11.89	14.73	12.38	12.03	12.05	19.31	12.68	20.87	11.08	3.35	7.76	14.25

Table 1: Model performance on FLORES-101 test set, with results for 7 representative languages shown in the table.

\Delta

denotes encoder-decoder models. Bold text denotes the best result across LLM-based decoder-only models. For spBLEU and Gemini*, we evaluate on all languages covered in FLORES-101 . For xCOMET* and MetricX*, we only evaluate on the languages they support in FLORES-101 .

Figure 3: LCR on language directions. WALAR improves LLMs’ translation into desired target languages.

	LLaMAX	+ QE	+ QE+Lang Align	+ WALAR	Reference
Wrong Language Ratio (%)	7.67	92.43	3.96	4.44	0.00
Translation Token Length	83.02	40.00	79.09	65.24	62.04

Table 2: Statistics for different types of errors.

Model	Lang $\rightarrow$ x		x $\rightarrow$ Lang
Model	spBLEU	xCOMET*	spBLEU	xCOMET*
LLaMAX3-8B-Alpaca	15.73	65.73	18.81	69.46
+ QE (w/ filter)	1.47	0.01	3.27	0.13
+ QE+Lang Align (w/ filter)	12.38	70.84	13.71	72.95
+ WALAR (w/ filter)	18.11	70.25	20.87	72.43
+ WALAR (w/o filter)	17.75	70.09	20.73	71.62

Table 3: Ablation on the reward components of WALAR and spBLEU-based data filtering. Lang denotes the set of seven representative languages (English, Arabic, Turkish, Hindi, Russian, Swahili).

	x $\rightarrow$ Lang (Avg.)			Lang $\rightarrow$ x (Avg.)			Overall Avg.
	spBLEU	MetricX*	xCOMET*	spBLEU	MetricX*	xCOMET*	spBLEU	MetricX*	xCOMET*
$\alpha$ =0	13.44	-4.44	74.99	12.31	-4.31	72.17	12.88	-4.38	73.58
$\alpha$ =5	19.56	-4.94	74.99	17.08	-4.69	72.98	18.32	-4.82	73.99
$\alpha$ =10	20.54	-5.45	74.42	17.88	-4.88	72.46	19.21	-5.17	73.44
$\alpha$ =15	20.98	-5.32	73.92	18.12	-4.97	71.78	19.55	-5.15	72.85
$\alpha$ =20	21.10	-5.44	73.39	18.32	-5.10	71.66	19.71	-5.27	72.53

Table 4: Performance of LLaMAX3-8B-Alpaca trained with different

\alpha

on Flores-101 validation set. We select and report the results of

\alpha=20

in all experiments.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Language	Length Threshold
Arabic	[20, 80]
Bengali	[50, 250]
Bulgarian	[20, 140]
Chinese	[10, 150]
Czech	[20, 120]
Dutch	[20, 100]

Appendix

Table 5: The length range we adopt for different languages.

	x $\rightarrow$ en	en $\rightarrow$ x	x $\rightarrow$ ar	ar $\rightarrow$ x	x $\rightarrow$ tr	tr $\rightarrow$ x	x $\rightarrow$ hi	hi $\rightarrow$ x	x $\rightarrow$ ru	ru $\rightarrow$ x	x $\rightarrow$ zh	zh $\rightarrow$ x	x $\rightarrow$ sw	sw $\rightarrow$ x
	spBLEU	spBLEU	spBLEU	spBLEU	spBLEU	spBLEU	spBLEU	spBLEU	spBLEU	spBLEU	spBLEU	spBLEU	spBLEU	spBLEU
Hunyuan-MT-7B	9.317	6.505	4.812	6.604	4.475	6.208	7.762	6.921	8.683	6.604	8.446	5.782	5.762	6.950
Tower-Plus-9B	3.238	6.762	9.238	6.604	7.861	6.495	2.386	6.208	2.901	6.584	2.515	6.594	7.624	6.366
Aya-Expanse-8B	7.020	6.426	6.426	6.089	7.158	6.158	6.436	6.307	6.842	6.228	7.683	6.218	8.782	8.337
Qwen3-8B	4.941	7.020	6.089	7.386	7.119	7.515	8.713	7.455	7.079	7.396	3.475	6.921	9.851	8.079
+WALAR	6.050	6.465	6.822	6.376	6.228	6.386	8.901	6.752	7.257	6.683	4.069	6.168	7.455	6.624

Appendix

Table 6: Average rank of strong multilingual LLMs on the FLORES-101 test set, with results for 7 representative languages shown in the table.

Figure 6: Screenshot of human evaluation web tool.

Source (English): Dr. Tony Moll discovered the Extremely Drug Resistant Tuberculosis (XDR-TB) in the South African region KwaZulu-Natal.
Reference (Spanish): La tuberculosis extremadamente resistente (XDR-TB) fue detectada por primera vez por el Dr. Tony Moll en área sudafricana de Zululandia.
Over-translation (Spanish): El Dr. Tony Moll descubrió en la región de KwaZulu-Natal, en Sudáfrica, un tipo de tuberculosis muy difícil de tratar: la tuberculosis extremadamente resistente a los medicamentos (XDR-TB). Esta bacteria es resistente a la mayoría de los tratamientos antibióticos convencionales y requiere tratamiento con medicamentos antituberculosos específicos. MetricX: -3.11
Wrong Language (Portuguese): O Dr. Tony Moll descobriu a Tuberculose Extremamente Resistente a Drogas (XDR-TB) na região de KwaZulu-Natal, na áfrica do Sul. MetricX: -1.85

Appendix

Figure 7: A case study from Flores-101 dataset. The intended language direction is from English to Spanish. blue text denotes the MetricX score in source-based mode, and the red text highlights the errors in the translation.

Source (English): Because the dinosaur feathers do not have a well-developed shaft, called a rachis, but do have other features of feathers - barbs and barbules - the researchers inferred the rachis was likely a later evolutionary development that these other features.
Reference (Polish): W związku z tym, że pióra dinozaura nie mają dobrze rozwiniętego trzonu, zwanego stosiną, lecz mają inne cechy piór – promienie i promyki – naukowcy wywnioskowali, że stosina prawdopodobnie była późniejszym etapem ewolucyjnym niż te pozostałe cechy.
Over-translation (Polish): Ponieważ pióra dinozaurów nie posiadały typowej osi (centralnego trzonu przebiegającego przez całą strukturę pióra), ale zachowały podstawowe cechy piór, takie jak promienie i promyczki, naukowcy wysnuli wniosek, że struktura osi mogła ewoluować później, podczas gdy inne cechy, takie jak promienie i promyczki, mogły istnieć już wcześniej. MetricX: -3.17
Wrong Language (Slovak): Pretože dinosaurové perie nemá dobre vyvinutú osku, nazývanú rachis, ale má iné črty peria - háčiky a háčikovité vlákna, vedci usúdili, že rachis pravdepodobne vznikol až neskôr v evolúcii než tieto ostatné črty. MetricX: -3.52

Appendix

Figure 8: A case study from Flores-101 dataset. The intended language direction is English to Polish. Blue text denotes the MetricX score in source-based mode, and the red text highlights the errors in the translation.

Source (English): Previously, Ring’s CEO, Jamie Siminoff, remarked the company started when his doorbell wasn’t audible from his shop in his garage. Translate it from English to Chinese.
Reference (Chinese): 此前，铃声 (Ring) 公司 CEO 杰米·西米诺夫曾表示，公司成立之初，他在位于车库的工作室里是听不到门铃响声的。
Over-translation (Chinese): 以前，Ring公司的CEOJamie Siminoff提到，公司最初成立时，其主要业务是通过门铃来吸引顾客。他的店铺位于车库内，而门铃则安装在车库的入口处。 MetricX: -3.70
Wrong Language (Japanese): 以前、RingのCEOであるジェイミー・シミノフ氏は、自宅のガレージにある作業場からドアベルの音が聞こえなかったことがきっかけで同社を創業したと述べました。 MetricX: -3.59
Non-translation (English): Earlier, Jamie Siminoff, CEO of Ring, shared the story behind the company’s founding; it all began when he noticed that the doorbell at his house wasn’t loud enough for him to hear it clearly from his workshop, which was located in his garage. MetricX: -0.66

Appendix

Figure 9: A case study from Flores-101 dataset. The intended language direction is from English to Chinese. Blue text denotes the MetricX score in source-based mode, and the red text highlights the errors in the translation.

Source (French): Les travailleurs doivent souvent obtenir l’approbation de leurs supérieurs pour toute décision qu’ils prennent et sont censés obéir aux instructions de ces derniers sans les remettre en question.
Reference (German): Oftmals müssen Arbeitnehmer für alle Entscheidungen, die sie treffen, die Zustimmung ihrer Vorgesetzten einholen und es wird von ihnen erwartet, dass sie den Anweisungen ihrer Chefs bedingungslos Folge leisten.
Over-translation (German): Die Mitarbeiter müssen vor jeder Entscheidung die Zustimmung ihrer Vorgesetzten einholen und sich strikt an deren Anweisungen halten, ohne diese willkürlich zu ändern oder deren Ausführung zu verweigern. MetricX: -0.29
Wrong Language (Italian): I lavoratori devono spesso ottenere l’approvazione dei loro superiori per qualsiasi decisione prendano e sono tenuti a seguire le loro istruzioni senza metterle in discussione. MetricX: -0.35
Non-translation (English): Les employés sont généralement tenus de solliciter l’accord de leurs responsables pour chaque décision et doivent suivre leurs directives sans les contester. MetricX: -0.37

Appendix

Figure 10: A case study from Flores-101 dataset. The intended language direction is from French to German. Blue text denotes the MetricX score in source-based mode, and the red text highlights the errors in the translation.

Source (English): In late 2017, Siminoff appeared on shopping television channel QVC.
Reference (Xhosa): Ngasekupheleni kuka-2017, uSiminoff kwitshaneli yokuthenga kwiTV enguQVC.
LLaMAX3-8B-Alpaca (Xhosa): Kwiyeza zokuthenga zesikhathi zesibini ze-2017, uSiminoff wabonakala kwiQVC, isiteshi se-TV seziyobisi. xCOMET: 49.52
LLaMAX3-8B-Alpaca+WALAR (Xhosa): Ngasekupheleni kuka-2017, uSiminoff waboniswa kwinkqubo yentengiso yeQVC, isiteshi seTV. xCOMET: 82.14

Appendix

Figure 11: Case study of the improvement brought by WALAR . The intended language direction is from English to Xhosa. Blue text denotes the xCOMET score in reference-based mode.

Source (English): One antibody cocktail, ZMapp, initially showed promise in the field, but formal studies indicated it had less benefit than sought in preventing death.
Reference (Chinese): 一种名为 ZMapp 的“抗体鸡尾酒”（多抗体联合）最初有望在该领域发挥作用，但正式研究表明，它在预防死亡方面的效果有些不尽人意。
LLaMAX3-8B-Alpaca (Chinese): 一种抗体混合物，ZMapp，最初显示出在现场的希望，但正式研究表明它比预期的死亡率降低。 xCOMET: 58.56
LLaMAX3-8B-Alpaca+WALAR (Chinese): 一种抗体混合物，ZMapp，最初显示出希望，但正式研究表明它无法达到预期的防止死亡效果。 xCOMET: 75.04

Appendix

Figure 12: Case study of the improvement brought by WALAR . The intended language direction is English to Chinese. Blue text denotes the xCOMET score in reference-based mode.

Source (Chinese): 在 PALM 实验中，ZMapp 用作参照。也就是科学家将其作为基线，把其他三种治疗方法与之作比较。
Reference (Swahili): Katika jaribio la PALM, Zmapp ilitumika kama kidhibiti, kumaanisha wanasayansi waliitumia kama msingi na kulinganisha mat ibabu mengine matatu nayo.
LLaMAX3-8B-Alpaca (Swahili): Katika majaribio ya Palm, ZMapp hutumiwa kama kumbukumbu. Hivyo, wanasayansi wamefanya kulinganisha kati ya matibabu mengine matatu na yake. Kwa kufanya hivyo, wanasayansi wanaweza kuchunguza jinsi matibabu mengine yanavyofanya kazi kwa kulinganisha na ZMapp. xCOMET: 65.48
LLaMAX3-8B-Alpaca+WALAR (Swahili): Katika majaribio ya PALM, ZMapp ilitumiwa kama kipimo cha kulinganisha; yaani, wataalamu walitumia kama kiwango na kulinganisha njia tatu nyingine za matibabu na hiyo. xCOMET: 93.80

Appendix

Figure 13: Case study of the improvement brought by WALAR . The intended language direction is from Chinese to Swahili. Blue text denotes the xCOMET score in reference-based mode.

Source (Chinese): 西米诺夫说，2013 年他在《创智赢家》节目中露面后，公司的销售额大增，当时节目组拒绝向这家初创公司投资。
Reference (Swahili): M. Siminoff a déclaré que les ventes ont augmenté après son apparition en 2013 dans un épisode de Shark Tank où le panel de la série a refusé de financer la start-up.
LLaMAX3-8B-Alpaca (French): Westminster a déclaré que depuis son apparition dans l’émission "Entrepreneurs de génie" en 2013, les ventes de sa société ont augmenté considérablement. Cependant, le programme n’a pas investi dans l’entreprise. xCOMET: 15.52
LLaMAX3-8B-Alpaca+WALAR (French): Siminov a déclaré que, après son apparition dans le programme «Entrepreneurs créatifs» en 2013, les ventes de sa société ont considérablement augmenté, mais le programme n’a pas investi dans cette entreprise débutante. xCOMET: 45.24

Appendix

Figure 14: Case study of the improvement brought by WALAR . The intended language direction is from Chinese to French. Blue text denotes the xCOMET score in reference-based mode.

Models	en $\rightarrow$ x	sw $\rightarrow$ x	tr $\rightarrow$ x	hi $\rightarrow$ x	zh $\rightarrow$ x	ar $\rightarrow$ x	ru $\rightarrow$ x
LLaMAX3-8B-Alpaca	0.9027	0.8931	0.8975	0.8931	0.8969	0.9018	0.8996
Aya-Expanse-8B	0.7481	0.7440	0.7604	0.7537	0.7387	0.7563	0.7440
Tower-Plus-9B	0.6199	0.7133	0.6504	0.5908	0.6537	0.7063	0.6432
WALAR	0.9270	0.9224	0.9252	0.9192	0.9258	0.9275	0.9273
Models	x $\rightarrow$ en	x $\rightarrow$ sw	x $\rightarrow$ tr	x $\rightarrow$ hi	x $\rightarrow$ zh	x $\rightarrow$ ar	x $\rightarrow$ ru
LLaMAX3-8B-Alpaca	0.9970	0.8329	0.9719	0.9513	0.9780	0.8329	0.9890

Appendix

Table 7: Complete results for LCR

Low-Resource MT

Low-resource machine translation develops translation systems for languages with scarce parallel text, tools, or standardized evaluation.

See all 87 papers

Rewrite to Translate, Translate to Reward: Reinforcement Learning for Source Rewriting in Machine Translation
Jun 6, 2026Boxuan Lyu, Haiyue Song, Zhi Qu +3RL for Language Models Machine Translation
Institute of Science Tokyo · Preferred Networks, Inc. · Nara Institute of Science and Technology
Translate-R1: Cost-Aware Translation Tool Use via Reinforcement Learning
Jun 5, 2026Pratik Jayarao, Chaitanya Dwivedi, Himanshu Gupta +5RL for Language Models LLM Tool Use
Amazon Stores Foundation AI
Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax
May 14, 2026Zeli Su, Ziyin Zhang, Zhou Liu +7Reinforcement Learning Low-Resource MT
Minzu University of China · Ant Group · Shanghai Jiao Tong University +4

Jun 6, 2026J/K move · Enter open · S save

Rewrite to Translate, Translate to Reward: Reinforcement Learning for Source Rewriting in Machine Translation

Boxuan Lyu, Haiyue Song, Zhi Qu, Hidetaka Kamigaito+2

Institute of Science Tokyo · Preferred Networks, Inc. · Nara Institute of Science and Technology

RL for Language Models Machine Translation

Open paper AskPDF arXiv

Rewriting source text with large language models (LLMs) before translation has been shown to improve machine translation (MT) quality. However, we find that prompt-based rewriting can degrade translation quality rather than improve it, particularly when smaller LLMs, such as 4B-parameter models, are used. We argue that this limitation stems from the difficulty of controlling rewriting behavior through natural-language prompts alone: a rewrite is useful only if it improves downstream translation, yet existing prompt-based methods do not explicitly optimize for this signal. To address this issue, we propose RLSR (Reinforcement Learning for Source Rewriting), a reinforcement learning framework that trains the rewriting model with a reward based on the downstream translation-quality improvement produced by each rewrite. Experiments across six MT systems and 16 language pairs show that our 4B RLSR-trained rewriting models significantly outperform both the no-rewriting baseline and prompt-based rewriting baselines at the same model scale, while remaining competitive with baselines that use a 235B LLM.

Abstract

Low-Resource MT

Explore similar work

Rewrite to Translate, Translate to Reward: Reinforcement Learning for Source Rewriting in Machine Translation

Translate-R1: Cost-Aware Translation Tool Use via Reinforcement Learning

Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax