cs.CLSep 5, 2026
SaveGenerating Adversarial Texts for Machine Translation via GRPO
Organizations: ETH Zurich, Switzerland
Abstract
As machine translation (MT) systems continue to improve, standard benchmarks become less informative for exposing remaining weaknesses. Traditional methods for creating challenging test sets rely on expensive manual creation or curation, while automated approaches struggle to produce sets with the necessary translation difficulty and linguistic diversity. We propose a scalable reinforcement-learning-based approach for rewriting existing source texts into instances that are more difficult to translate for MT systems. We fine-tune a large language model with Group Relative Policy Optimization (GRPO), using reward signals based on translation difficulty together with constraints for semantic similarity, grammaticality, and approximate length preservation. On WMT25, our approach substantially reduces average COMET translation quality from 0.63 to 0.48, while preserving grammaticality and readability, whereas the base model remains at 0.64. Evaluations on the unseen WMT19-WMT24 benchmarks confirm that this behavior generalizes beyond the training data, and human evaluation further shows that the rewrites substantially lower translation quality while incurring a moderate drop in naturalness and only a small change in grammaticality. We release our code to support reproducibility.
Figures & tables
Figure 1: Translation quality (COMET score, scale 0–1; lower is harder to translate) vs. number of grammatical errors on WMT23–WMT25. The rewrites generated by our fine-tuned model (blue) successfully decrease translation quality in comparison to the non-fine-tuned base model (green) and the original dataset (black), while maintaining grammaticality.
Figure 2: During training, the LLM generates multiple rewrites that are evaluated using machine translation (MT) and quality estimation (QE) models to compute a reward reflected by the difference in translation quality. This signal is used to optimize the model via GRPO. At inference time, the model directly produces rewrites that are difficult to translate.
| (a) Original Bigrams | (b) Fine-tuned Bigrams | (c) Fine-tuned Bigrams Collapsed |
|---|---|---|
| "go to" : 29 | "go to" : 30 | "high high" : 1965 |
| "feel like" : 18 | "low key" : 28 | "high stake" : 242 |
| "look place" : 18 | "real deal" : 27 | "stake high" : 214 |
| "price range" : 18 | "dead set" : 27 | "hook high" : 46 |
| "x thank" : 17 | "get to" : 23 | "high hook" : 41 |
Table 1: Most frequent lemmatized bigrams: (a) original text vs. (b) fine-tuned model at step 2000 vs. (c) collapsed fine-tuned model at step 3500. Primarily optimizing for specific metrics may lead the model to exploit high-reward constructions, resulting in low linguistic diversity (c). Our best performing model (b) is able to increase translation difficulty while maintaining similarly distributed word frequency to the original data (a).
| Metric | Original | Base Model | Sentinel | COMET-NLLB | COMET-Helsinki | COMET-Average |
|---|---|---|---|---|---|---|
| Diversity | ||||||
| Unique topics | 403 | 436 | 430 | 406 | 410 | 400 |
| Pairwise Self-chrF | 21.61 | 26.00 | 24.46 | 22.77 | 22.58 | 21.65 |
| Grammatical Correctness | ||||||
| Grammatical Errors | 3.45 | 1.06 | 0.88 | 1.74 | 1.33 | 1.36 |
| Complexity | ||||||
Table 2: Evaluation metrics on WMT25 for the original dataset, the non-fine-tuned base model, and the four GRPO fine-tuned variants (gray). COMET-Average achieves the best scores under all COMET-based evaluations and is competitive on Sentinel, while naturalness and grammaticality improve compared to the original data. All COMET scores in this table are computed for English Italian translations.
| Metric | Original | Base Model | COMET-Average |
|---|---|---|---|
| Diversity | |||
| Unique topics | 381 | 468 | 346 |
| Pairwise Self-chrF | 16.51 | 21.19 | 16.85 |
| Grammatical Correctness | |||
| Grammatical Errors | 1.05 | 0.70 | 0.78 |
| Complexity | |||
Table 3: Evaluation metrics on WMT24, comparing the original dataset, the non-fine-tuned base model, and the COMET-Average fine-tuned model. All COMET scores in this table are computed for English Italian translations.
Figure 3: Translation quality, grammaticality, diversity, and complexity metrics change over training steps. The red dashed line corresponds to the original dataset and the green star marks the checkpoint selected for evaluation (step 2000), which reflects the best trade-off between increased difficulty and preservation of linguistic quality.
Figure 4: Modified Pearmut annotation interface for text and translation quality annotation.
| Grammaticality | Naturalness | Translation Quality | |
|---|---|---|---|
| Original | 69.75 | 67.10 | 56.51 |
| (Base Model) | 74.79 | 64.43 | 57.56 |
| Ours | 67.13 | 60.01 | 41.29 |
Table 4: Human evaluation results on a 1–100 scale. Our fine-tuned model achieves substantially lower translation quality, accompanied by a moderate drop in naturalness and only a small change in grammaticality.
Figure 5: Human-rated translation quality against grammaticality (top) and naturalness (bottom). Each point represents one source segment averaged over its annotations ( per condition). Legends report within-condition Spearman correlations ( , , two-sided). Translation quality is not significantly associated with either dimension in our condition.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Metric | Original | Base Model | COMET-Average |
|---|---|---|---|
| Diversity | |||
| Unique topics | 266 | 353 | 246 |
| Pairwise Self-chrF | 16.91 | 21.54 | 16.21 |
| Grammatical Correctness | |||
| Grammatical Errors | 0.75 | 0.67 | 0.62 |
| Complexity | |||
Table 5: Evaluation metrics on WMT19
| Metric | Original | Base Model | COMET-Average |
|---|---|---|---|
| Diversity | |||
| Unique topics | 283 | 380 | 267 |
| Pairwise Self-chrF | 16.95 | 22.15 | 16.51 |
| Grammatical Correctness | |||
| Grammatical Errors | 0.76 | 0.62 | 0.63 |
| Complexity | |||
Table 6: Evaluation metrics on WMT20
| Metric | Original | Base Model | COMET-Average |
|---|---|---|---|
| Diversity | |||
| Unique topics | 269 | 350 | 228 |
| Pairwise Self-chrF | 17.24 | 22.37 | 16.27 |
| Grammatical Correctness | |||
| Grammatical Errors | 0.90 | 0.75 | 0.81 |
| Complexity | |||
Table 7: Evaluation metrics on WMT21
| Metric | Original | Base Model | COMET-Average |
|---|---|---|---|
| Diversity | |||
| Unique topics | 375 | 470 | 320 |
| Pairwise Self-chrF | 16.23 | 21.36 | 15.64 |
| Grammatical Correctness | |||
| Grammatical Errors | 0.51 | 0.36 | 0.39 |
| Complexity | |||
Table 8: Evaluation metrics on WMT22
| Metric | Original | Base Model | COMET-Average |
|---|---|---|---|
| Diversity | |||
| Unique topics | 359 | 425 | 312 |
| Pairwise Self-chrF | 16.54 | 21.24 | 15.90 |
| Grammatical Correctness | |||
| Grammatical Errors | 0.60 | 0.43 | 0.52 |
| Complexity | |||
Table 9: Evaluation metrics on WMT23
Figure 6: WMT25: Evolution of translation difficulty over training using COMET-Average.
Figure 7: WMT25: Evolution of translation difficulty over training using COMET-NLLB.
Figure 8: WMT25: Evolution of translation difficulty over training using COMET-Helsinki.
Figure 9: WMT25: Evolution of translation difficulty over training using Sentinel.
| Role | Model / checkpoint |
|---|---|
| Generation and evaluation | |
| Policy, judge | meta-llama/Meta-Llama-3.1-8B-Instruct |
| Translation difficulty | |
| QE metric | Unbabel/wmt22-cometkiwi-da |
| MT system 1 | facebook/nllb-200-distilled-600M |
| MT system 2 | Helsinki-NLP/opus-mt-en-it |
Table 10: Models used for generation, reward computation, and evaluation. Llama-3.1-8B-Instruct serves both as the rewriting policy (fine-tuned with LoRA) and, unmodified, as the LLM judge for naturalness, word rarity, syntax complexity, and topics. CometKiwi scores the output of both MT systems without a reference, while Sentinel scores the source alone. The embedding model provides the semantic similarity reward, computed as cosine similarity over sentence embeddings.