As machine translation (MT) systems continue to improve, standard benchmarks become less informative for exposing remaining weaknesses. Traditional methods for creating challenging test sets rely on expensive manual creation or curation, while automated approaches struggle to produce sets with the necessary translation difficulty and linguistic diversity. We propose a scalable reinforcement-learning-based approach for rewriting existing source texts into instances that are more difficult to translate for MT systems. We fine-tune a large language model with Group Relative Policy Optimization (GRPO), using reward signals based on translation difficulty together with constraints for semantic similarity, grammaticality, and approximate length preservation. On WMT25, our approach substantially reduces average COMET translation quality from 0.63 to 0.48, while preserving grammaticality and readability, whereas the base model remains at 0.64. Evaluations on the unseen WMT19-WMT24 benchmarks confirm that this behavior generalizes beyond the training data, and human evaluation further shows that the rewrites substantially lower translation quality while incurring a moderate drop in naturalness and only a small change in grammaticality. We release our code to support reproducibility.
Figures & tables
Figure 1: Translation quality (COMET score, scale 0–1; lower is harder to translate) vs. number of grammatical errors on WMT23–WMT25. The rewrites generated by our fine-tuned model (blue) successfully decrease translation quality in comparison to the non-fine-tuned base model (green) and the original dataset (black), while maintaining grammaticality.
Figure 2: During training, the LLM generates multiple rewrites that are evaluated using machine translation (MT) and quality estimation (QE) models to compute a reward reflected by the difference in translation quality. This signal is used to optimize the model via GRPO. At inference time, the model directly produces rewrites that are difficult to translate.
(a) Original Bigrams
(b) Fine-tuned Bigrams
(c) Fine-tuned Bigrams Collapsed
"go to" : 29
"go to" : 30
"high high" : 1965
"feel like" : 18
"low key" : 28
"high stake" : 242
"look place" : 18
"real deal" : 27
"stake high" : 214
"price range" : 18
"dead set" : 27
"hook high" : 46
"x thank" : 17
"get to" : 23
"high hook" : 41
Table 1: Most frequent lemmatized bigrams: (a) original text vs. (b) fine-tuned model at step 2000 vs. (c) collapsed fine-tuned model at step 3500. Primarily optimizing for specific metrics may lead the model to exploit high-reward constructions, resulting in low linguistic diversity (c). Our best performing model (b) is able to increase translation difficulty while maintaining similarly distributed word frequency to the original data (a).
Metric
Original
Base Model
Sentinel
COMET-NLLB
COMET-Helsinki
COMET-Average
Diversity
Unique topics
403
436
430
406
410
400
Pairwise Self-chrF
21.61
26.00
24.46
22.77
22.58
21.65
Grammatical Correctness
Grammatical Errors
3.45
1.06
0.88
1.74
1.33
1.36
Complexity
Table 2: Evaluation metrics on WMT25 for the original dataset, the non-fine-tuned base model, and the four GRPO fine-tuned variants (gray). COMET-Average achieves the best scores under all COMET-based evaluations and is competitive on Sentinel, while naturalness and grammaticality improve compared to the original data. All COMET scores in this table are computed for English → Italian translations.
Metric
Original
Base Model
COMET-Average
Diversity
Unique topics
381
468
346
Pairwise Self-chrF
16.51
21.19
16.85
Grammatical Correctness
Grammatical Errors
1.05
0.70
0.78
Complexity
Table 3: Evaluation metrics on WMT24, comparing the original dataset, the non-fine-tuned base model, and the COMET-Average fine-tuned model. All COMET scores in this table are computed for English → Italian translations.
Figure 3: Translation quality, grammaticality, diversity, and complexity metrics change over training steps. The red dashed line corresponds to the original dataset and the green star marks the checkpoint selected for evaluation (step 2000), which reflects the best trade-off between increased difficulty and preservation of linguistic quality.
Figure 4: Modified Pearmut annotation interface for text and translation quality annotation.
Grammaticality
Naturalness
Translation Quality
Original
69.75
67.10
56.51
(Base Model)
74.79
64.43
57.56
Ours
67.13
60.01
41.29
Table 4: Human evaluation results on a 1–100 scale. Our fine-tuned model achieves substantially lower translation quality, accompanied by a moderate drop in naturalness and only a small change in grammaticality.
Figure 5: Human-rated translation quality against grammaticality (top) and naturalness (bottom). Each point represents one source segment averaged over its annotations ( n=50 per condition). Legends report within-condition Spearman correlations ( ∗p<.05 , ∗∗∗p<.001 , two-sided). Translation quality is not significantly associated with either dimension in our condition.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Metric
Original
Base Model
COMET-Average
Diversity
Unique topics
266
353
246
Pairwise Self-chrF
16.91
21.54
16.21
Grammatical Correctness
Grammatical Errors
0.75
0.67
0.62
Complexity
Appendix
Table 5: Evaluation metrics on WMT19
Metric
Original
Base Model
COMET-Average
Diversity
Unique topics
283
380
267
Pairwise Self-chrF
16.95
22.15
16.51
Grammatical Correctness
Grammatical Errors
0.76
0.62
0.63
Complexity
Appendix
Table 6: Evaluation metrics on WMT20
Metric
Original
Base Model
COMET-Average
Diversity
Unique topics
269
350
228
Pairwise Self-chrF
17.24
22.37
16.27
Grammatical Correctness
Grammatical Errors
0.90
0.75
0.81
Complexity
Appendix
Table 7: Evaluation metrics on WMT21
Metric
Original
Base Model
COMET-Average
Diversity
Unique topics
375
470
320
Pairwise Self-chrF
16.23
21.36
15.64
Grammatical Correctness
Grammatical Errors
0.51
0.36
0.39
Complexity
Appendix
Table 8: Evaluation metrics on WMT22
Metric
Original
Base Model
COMET-Average
Diversity
Unique topics
359
425
312
Pairwise Self-chrF
16.54
21.24
15.90
Grammatical Correctness
Grammatical Errors
0.60
0.43
0.52
Complexity
Appendix
Table 9: Evaluation metrics on WMT23
Figure 6: WMT25: Evolution of translation difficulty over training using COMET-Average.
Figure 7: WMT25: Evolution of translation difficulty over training using COMET-NLLB.
Figure 8: WMT25: Evolution of translation difficulty over training using COMET-Helsinki.
Figure 9: WMT25: Evolution of translation difficulty over training using Sentinel.
Role
Model / checkpoint
Generation and evaluation
Policy, judge
meta-llama/Meta-Llama-3.1-8B-Instruct
Translation difficulty
QE metric
Unbabel/wmt22-cometkiwi-da
MT system 1
facebook/nllb-200-distilled-600M
MT system 2
Helsinki-NLP/opus-mt-en-it
Appendix
Table 10: Models used for generation, reward computation, and evaluation. Llama-3.1-8B-Instruct serves both as the rewriting policy (fine-tuned with LoRA) and, unmodified, as the LLM judge for naturalness, word rarity, syntax complexity, and topics. CometKiwi scores the output of both MT systems without a reference, while Sentinel scores the source alone. The embedding model provides the semantic similarity reward, computed as cosine similarity over sentence embeddings.
As state-of-the-art machine translation models saturate standard benchmarks, the field needs more challenging evaluations to distinguish between models of varying quality. We propose augmenting existing benchmarks to increase translation difficulty by combining adversarial optimization with a differentiable translation difficulty estimator. Our Adversarial Translation Optimization (ATO) uses gradients from a combined difficulty and fluency objective to iteratively replace tokens. Because each step branches over candidate substitutions at every position, optimization becomes a tree search problem, which we address with Beam Search. ATO offers a gradient-based alternative to LLM-based dataset creation without LLM prompting, expensive human curation, or task-specific model training. Our ATO-modified benchmark lowers average translation quality (xCOMET) from 0.93 to 0.82, compared to 0.88 for paraphrasing and 0.86 for a zero-shot baseline. Human evaluation shows the modified texts are somewhat less natural than the baselines but remain reasonably grammatical and plausible while being substantially harder to translate. We release two datasets of 350 English texts each, generated by our methods, as well as the code.
Rewriting source text with large language models (LLMs) before translation has been shown to improve machine translation (MT) quality. However, we find that prompt-based rewriting can degrade translation quality rather than improve it, particularly when smaller LLMs, such as 4B-parameter models, are used. We argue that this limitation stems from the difficulty of controlling rewriting behavior through natural-language prompts alone: a rewrite is useful only if it improves downstream translation, yet existing prompt-based methods do not explicitly optimize for this signal. To address this issue, we propose RLSR (Reinforcement Learning for Source Rewriting), a reinforcement learning framework that trains the rewriting model with a reward based on the downstream translation-quality improvement produced by each rewrite. Experiments across six MT systems and 16 language pairs show that our 4B RLSR-trained rewriting models significantly outperform both the no-rewriting baseline and prompt-based rewriting baselines at the same model scale, while remaining competitive with baselines that use a 235B LLM.
Boxuan Lyu, Haiyue Song, Zhi Qu +3
Institute of Science Tokyo · Preferred Networks, Inc. · Nara Institute of Science and Technology
While Group Relative Policy Optimization (GRPO) offers a powerful framework for LLM post-training, its effectiveness in open-ended domains like Machine Translation hinges on accurate intra-group ranking. We identify that standard Pointwise Quality Metrics (PQM) fall short in this context: candidates are evaluated in isolation, so the comparative context is missing for distinguishing fine-grained linguistic nuances. To address this, we introduce the Group Quality Metric (GQM) paradigm and its instantiation, the Group Relative Reward Model (GRRM). Unlike traditional independent scorers, GRRM jointly processes the entire candidate group, leveraging comparative analysis to rigorously resolve relative quality and adaptive granularity. Empirical evaluations confirm that GRRM achieves competitive ranking accuracy among all baselines; integrating GRRM into the GRPO training not only improves general translation quality but also unlocks reasoning capabilities comparable to state-of-the-art reasoning models. We release codes, datasets, and model checkpoints at https://github.com/NJUNLP/GRRM.
Sen Yang, Shanbo Cheng, Lu Xu +2
National Key Laboratory for Novel Software Technology, Nanjing University · Singapore University of Technology and Design