Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varieties

Jannis Vamvas; Ignacio Pérez Prat; Angela Heldstab; Dominic P. Fischer; Sina Ahmadi; Rico Sennrich

cs.CLMar 26, 2026

Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varieties

Authors: Jannis Vamvas, Ignacio Pérez Prat, Angela Heldstab, Dominic P. Fischer, Sina Ahmadi, Rico Sennrich

Organizations: University of Zurich · Lia Rumantscha

Abstract

Recent strategies for low-resource machine translation rely on LLMs to generate synthetic data based on text in higher-resource languages. We revisit this idea for Romansh, a language with 6 distinct varieties. LLMs tend to confuse these varieties when translating into Romansh, but they are quite good at translating out of Romansh into a high-resource language such as German. Due to this asymmetry, the direction of data augmentation is a crucial choice. We find that contrary to recent strategies, creating synthetic translations into the higher-resource language is the superior approach, and only this approach allows us to surpass a Gemini 3 Pro baseline on German-Romansh translation (+23 BLEU over Gemini in the lowest-resource variety). A human evaluation confirms that our experiments yield the first model that generates fluent translations in the individual Romansh varieties.

Figures & tables

Figure 1: LLMs have asymmetric translation capabilities regarding low-resource or multi-variety languages like Romansh. In the case of Romansh, they demonstrate a general understanding of all varieties when translating out of the language, but they fail to adhere to a specific target variety when translating into the language. This asymmetry is relevant for data augmentation.


Rumantsch Grischun (RG)	Quai n’hai jau betg ditg.
Sursilvan	Quei hai jeu buca detg.
Sutsilvan	Quegl ve jou betga getg.
Surmiran	Chegl vaia betg detg.
Puter	Que nu d’heja dit.
Vallader	Quai nun haja dit.

Figure 2: The varieties of Romansh are highly diverse, as shown in this example from the WMT24++ benchmark Deutsch et al. (2025) ; Vamvas et al. (2025) .

Figure 3: When translating from German into Romansh, Gemini 2.5 Flash often produces the wrong target variety, which can be visualized through a confusion matrix that evaluates the system translations against the references for the individual varieties. Our fine-tuned NLLB model adheres to the target varieties and achieves higher BLEU.

	DE $\rightarrow$ RM	RM $\rightarrow$ DE
Approach	BLEU	BLEU / COMET
Baseline LLMs
Llama 3.3 (70B)	11.1	27.1 / 70.1
Apertus (70B)	13.7	35.6 / 84.3
GPT-4o	22.7	48.3 / 89.4
Gemini 2.5 Flash

Table 1: Validation results for multiple LLMs and data augmentation strategies, based on our validation split of the WMT24++ benchmark and averaged across the six varieties of Romansh. The results for baseline LLMs are reported for comparison, and only Gemini 2.5 Flash is used for the experiments below.

Variety	# Raters	# Fluency ratings	# Document accuracy ratings	# Segment accuracy preferences
RG	2	1,117 6 (51 overlap)	168 (16 overlap)	531 1, (159 overlap)
Sursilvan	5	1,015 6 (64 overlap)	204 (21 overlap)	684 1, (271 overlap)
Sutsilvan	3	987 (139 overlap)	166 (37 overlap)	704 1, (341 overlap)
Surmiran	2	1,017 (238 overlap)	168 (23 overlap)	668 1, (303 overlap)
Puter	3	929 (160 overlap)	158 9 (0 overlap)	425 1,11 (0 overlap)
Vallader	2	827 65 (0 overlap)	166 9 (0 overlap)	484 1,11 (0 overlap)

Table 2: Human evaluation statistics: number of raters and annotations by target variety. The overlap counts in parentheses refer to items annotated by more than one rater, which we use for inter-rater agreement analysis.

Metric	Fluency	Document accuracy	Segment accuracy
Item-level Pearson correlation between raters	67.6	31.7	42.3
Item-level pairwise agreement rate between raters	68.5	59.0	57.8
System-level Spearman correlation between raters	92.0	75.0	90.0

Table 3: Inter-rater agreement metrics for the human evaluation, averaged across the 4–5 target varieties in which overlapping annotations have been collected.

System	WMT24++						BOUQuET	Avg.
	RG	Sursilvan	Sutsilvan	Surmiran	Puter	Vallader	RG	Avg.
Gemini 2.5 Flash	40.3	28.0	10.6	16.9	22.0	27.3	27.2	24.6
Gemini 3 Pro (preview)	45.3	37.1	17.3	27.1	34.7	36.1	31.3	32.7
Gemini 3.7 Flash	40.9	34.2	14.8	22.1	30.4	29.8	29.8	28.9
Fine-tuned NLLB
No data augmentation	35.9	29.4	29.2	25.3	31.0	26.0	27.9	29.2

Table 4: BLEU scores for German

\rightarrow

Romansh translation on the second half of the WMT24++ benchmark, together with BOUQuET results for Rumantsch Grischun. Bold denotes the best system; underline indicates the best variant among the fine-tuned NLLB models; the last column reports the arithmetic mean of the seven columns.

System	WMT24++						BOUQuET	Avg.
	RG	Sursilvan	Sutsilvan	Surmiran	Puter	Vallader	RG	Avg.
Gemini 2.5 Flash	93.7	93.1	89.8	91.7	92.3	92.7	97.6	93.0
Gemini 3 Pro (preview)	93.8	93.8	92.5	93.4	93.0	93.8	97.8	94.0
Gemini 3.7 Flash	94.3	94.0	93.3	93.9	93.4	94.1	97.9	94.4
Fine-tuned NLLB
No data augmentation	81.9	80.9	80.2	79.1	79.0	79.3	93.3	82.0

Table 5: COMET scores for Romansh

\rightarrow

German translation on the second half of the WMT24++ benchmark, together with BOUQuET results for Rumantsch Grischun. Bold denotes the best system; underline indicates the best variant among the fine-tuned NLLB models; the last column reports the arithmetic mean of the seven columns.

Panel of Figure 4 — Figure 4: Human ratings for German $\rightarrow$ Romansh translation, averaged across the six target varieties. The error bars show 95% bootstrap confidence intervals.

Panel of Figure 5 — Figure 5: Comparison of Romansh data (authentic / synthetic) with respect to the included domains (top row) and the proportion of Romansh varieties according to LID classification (bottom row).

HR $\rightarrow$ LR synthetic datasets:
Dataset	LID agreement							Dictionary coverage
	RG	Surs.	Suts.	Surm.	Puter	Vall.	Avg.	RG	Surs.	Suts.	Surm.	Puter	Vall.	Avg.
Authentic Romansh text	-	-	-	-	-	-	-	90.5	87.9	88.5	87.7	83.9	87.1	87.6
–based on Europarl	100.0	40.3	17.3	80.9	33.1	95.8	61.2	93.5	80.3	58.9	71.5	77.9	83.0	77.5
– Newscrawl / FineWeb2	99.2	41.5	17.0	61.6	47.0	86.9	58.9	89.2	74.9	57.5	66.0	72.9	76.6	72.9

Table 6: Computational analysis of Romansh data (authentic / synthetic) with regard to adherence to variety-specific features. LID agreement quantifies the proportion of synthetic samples where the requested target variety matches the prediction of an LID system. Dictionary coverage is the proportion of words in the text that overlap with variety-specific Romansh dictionaries.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Base model	NLLB-200-Distilled 1.3B
Learning rate	$2\times 10^{-4}$
Warmup updates	150
Max epochs	10
Stopping criterion	Validation loss on a held-out split of the training data
Optimizer	Adam ( $\beta_{1}=0.9$ , $\beta_{2}=0.98$ , $\epsilon=10^{-6}$ )

Appendix

Table 7: Fine-tuning hyperparameters.

Panel of Table 8 — Table 8: Parallel datasets included in the MT training. Tokens are counted based on whitespace tokenization. † Counting each pairwise combination of varieties separately.

Panel of Table 9 — Table 9: Monolingual German datasets used for the two HR $\rightarrow$ LR augmentation variants. Token counts are approximate. Note that we sample a subset from these datasets, independently for each Romansh target variety.

Panel of Table 10 — Table 10: Monolingual Romansh datasets used for LR $\rightarrow$ HR augmentation . Tokens are counted based on whitespace tokenization.

HR $\rightarrow$ LR augmentation
	Requests	Total input tokens	Total output tokens	Approx. API cost
– Europarl	251,432	283,562,126	252,592,957	350 USD
– Newscrawl / FineWeb2	2,615,424	1,163,299,838	267,593,106	500 USD
LR $\rightarrow$ HR augmentation
– Baseline prompt	765,908	530,210,202	194,726,402	315 USD
– Dictionary prompting	765,908	8,292,343,080	196,047,179	1260 USD

Appendix

Table 11: Statistics of LLM requests for different data augmentation strategies. Tokens are counted in terms of the tokenizer of Gemini 2.5 Flash. The variants required a different number of requests (and consequently, few-shot tokens) because the average document length can vary. Furthermore, different tokenizer fertilities for German and Romansh lead to different input/output token ratios.

Language	Europarl		Newscrawl / FineWeb2
	# Segments	# Tokens	# Segments	# Tokens
German	0 7.8M (70% synthetic)	121.0M (94% synthetic)	0 0 6.5M (64% synthetic)	118.7M (94% synthetic)
RG	0 2.7M (71% synthetic)	0 49.8M (92% synthetic)	0 0 2.7M (70% synthetic)	0 62.7M (94% synthetic)
Sursilvan	0 0.7M (60% synthetic)	0 11.1M (85% synthetic)	0 0 0.6M (55% synthetic)	0 12.0M (86% synthetic)
Sutsilvan	0 0.6M (25% synthetic)	0 0 5.0M (66% synthetic)	0 0 0.6M (24% synthetic)	0 0 6.1M (72% synthetic)
Surmiran	0 1.4M (83% synthetic)	0 26.8M (97% synthetic)	0 0 0.8M (70% synthetic)	0 17.8M (96% synthetic)

Appendix

Table 12: Training data statistics for the HR

\rightarrow

LR direction.

Language	Baseline prompt		Dictionary prompting
	# Segments	# Tokens	# Segments	# Tokens
German	0 6.3M (63% synthetic)	115.3M (94% synthetic)	0 6.4M (64% synthetic)	117.5M (94% synthetic)
RG	0 3.3M (76% synthetic)	0 67.1M (94% synthetic)	0 3.4M (76% synthetic)	0 68.9M (94% synthetic)
Sursilvan	0 1.1M (74% synthetic)	33.1M (95% synthetic)	0 1.1M (75% synthetic)	0 33.9M (95% synthetic)
Sutsilvan	0 0.5M (14% synthetic)	0 0 4.4M (61% synthetic)	0 0.5M (14% synthetic)	0 0 4.5M (61% synthetic)
Surmiran	0 0.4M (37% synthetic)	0 0 6.7M (89% synthetic)	0 0.4M (38% synthetic)	0 0 6.8M (89% synthetic)

Appendix

Table 13: Training data statistics for the LR

\rightarrow

HR direction.

System	WMT24++						BOUQuET	Avg.
	RG	Sursilvan	Sutsilvan	Surmiran	Puter	Vallader	RG	Avg.
Gemini 2.5 Flash	55.7	50.0	44.9	49.9	52.0	58.9	33.9	49.3
Gemini 3 Pro (preview)	55.8	50.8	48.7	52.2	53.8	60.8	33.8	50.8
Gemini 3.7 Flash	55.2	50.3	48.2	52.9	53.3	60.6	33.6	50.6
Fine-tuned NLLB
No data augmentation	38.3	33.9	33.3	33.7	34.6	37.2	26.5	33.9

Appendix

Table 14: Per-variety BLEU scores for Romansh

\rightarrow

Figure 6: Confusion matrices similar to Figure 3 illustrating the target variety adherence in German

\rightarrow

Romansh translation. Results are based on BLEU.

System	RG	Surs.	Suts.	Surm.	Puter	Vall.
Human reference	0.24 $\pm$ 0.10	0.66 $\pm$ 0.07	0.62 $\pm$ 0.07	0.60 $\pm$ 0.08	0.50 $\pm$ 0.07	0.23 $\pm$ 0.12
Gemini 3 Pro (preview)	-0.30 $\pm$ 0.14	-0.75 $\pm$ 0.11	-1.35 $\pm$ 0.08	-0.98 $\pm$ 0.11	-0.92 $\pm$ 0.15	-0.65 $\pm$ 0.13
LR $\rightarrow$ HR augmented NLLB	0.03 $\pm$ 0.11	0.02 $\pm$ 0.12	0.36 $\pm$ 0.09	0.17 $\pm$ 0.10	0.05 $\pm$ 0.10	0.19 $\pm$ 0.12
+ Dictionary prompting	0.04 $\pm$ 0.11	0.10 $\pm$ 0.12	0.26 $\pm$ 0.09	0.28 $\pm$ 0.11	0.15 $\pm$ 0.12	0.20 $\pm$ 0.14

Appendix

Table 15: Per-variety fluency scores for German

\rightarrow

Romansh translation. Ratings were made on the segment level and then averaged across segments.

System	RG	Surs.	Suts.	Surm.	Puter	Vall.
Human reference	0.40 $\pm$ 0.27	0.74 $\pm$ 0.25	0.62 $\pm$ 0.26	0.64 $\pm$ 0.22	0.31 $\pm$ 0.31	0.55 $\pm$ 0.21
Gemini 3 Pro (preview)	-0.25 $\pm$ 0.34	-0.85 $\pm$ 0.17	-0.75 $\pm$ 0.18	-1.16 $\pm$ 0.31	-0.92 $\pm$ 0.23	-0.57 $\pm$ 0.33
LR $\rightarrow$ HR augmented NLLB	-0.02 $\pm$ 0.24	-0.08 $\pm$ 0.27	-0.16 $\pm$ 0.27	-0.01 $\pm$ 0.27	0.40 $\pm$ 0.26	-0.15 $\pm$ 0.30
+ Dictionary prompting	-0.03 $\pm$ 0.33	0.21 $\pm$ 0.24	0.36 $\pm$ 0.29	0.03 $\pm$ 0.26	0.18 $\pm$ 0.28	0.19 $\pm$ 0.27

Appendix

Table 16: Per-variety accuracy scores for German

\rightarrow

Romansh translation. Ratings were made on the document level and then averaged across documents.

	Human reference	Gemini 3 Pro (preview)	LR $\rightarrow$ HR augmented NLLB	+ Dictionary prompting
Human reference	-	0 72 $\pm$ 0 4	0 53 $\pm$ 0 4	0 56 $\pm$ 0 5
Gemini 3 Pro (preview)	00 9 $\pm$ 0 2	-	0 15 $\pm$ 0 3	0 11 $\pm$ 0 3
LR $\rightarrow$ HR augmented NLLB	0 23 $\pm$ 0 3	0 63 $\pm$ 0 4	-	0 20 $\pm$ 0 4
+ Dictionary prompting	0 22 $\pm$ 0 4	0 66 $\pm$ 0 4	0 22 $\pm$ 0 4	-

Appendix

Table 17: Win-rates of systems in segment-level fluency evaluation, on average across varieties. The values indicate the percentage of segments where the system in the row was rated higher than the system in the column.

	Human reference	Gemini 3 Pro (preview)	LR $\rightarrow$ HR augmented NLLB	+ Dictionary prompting
Human reference	-	0 80 $\pm$ 10	0 66 $\pm$ 10	0 50 $\pm$ 14
Gemini 3 Pro (preview)	00 5 $\pm$ 0 4	-	00 4 $\pm$ 0 4	0 11 $\pm$ 0 5
LR $\rightarrow$ HR augmented NLLB	0 11 $\pm$ 0 7	0 64 $\pm$ 10	-	0 26 $\pm$ 0 9
+ Dictionary prompting	0 15 $\pm$ 0 8	0 76 $\pm$ 12	0 25 $\pm$ 0 9	-

Appendix

Table 18: Win-rates of systems in document-level accuracy ranking, on average across varieties.

	Human reference	Gemini 3 Pro (preview)	LR $\rightarrow$ HR augmented NLLB	+ Dictionary prompting
Human reference	-	0 60 $\pm$ 0 5	0 51 $\pm$ 0 4	0 51 $\pm$ 0 7
Gemini 3 Pro (preview)	0 10 $\pm$ 0 3	-	0 14 $\pm$ 0 3	0 14 $\pm$ 0 3
LR $\rightarrow$ HR augmented NLLB	0 20 $\pm$ 0 3	0 54 $\pm$ 0 4	-	0 19 $\pm$ 0 3
+ Dictionary prompting	0 24 $\pm$ 0 6	0 56 $\pm$ 0 4	0 22 $\pm$ 0 3	-

Appendix

Table 19: Win-rates of systems in segment-level accuracy evaluation, on average across varieties.

Variety	Human reference	Gemini 3 Pro (preview)	LR $\rightarrow$ HR augmentation	+ Dictionary prompting
Rumantsch Grischun	266 / 38 / 0 92	280 / 54 / 149	292 / 40 / 166	279 / 36 / 124
Rater 1	243 / 33 / 0 59	249 / 42 / 0 96	268 / 30 / 114	258 / 31 / 102
Rater 2	0 23 / 0 5 / 0 33	0 31 / 12 / 0 53	0 24 / 10 / 0 52	0 21 / 0 5 / 0 22
Sursilvan	250 / 46 / 188	260 / 49 / 176	245 / 54 / 179	260 / 55 / 140
Rater 3	208 / 0 0 / 00 0	210 / 0 0 / 00 0	190 / 0 0 / 00 0	204 / 0 0 / 00 0
Rater 4	00 0 / 37 / 112	00 0 / 40 / 0 96	00 0 / 42 / 118	00 0 / 43 / 0 87

Appendix

Table 20: Statistics for human evaluation: number of fluency ratings of segments / number of accuracy ratings of documents / number of accuracy preferences of segment pairs. Pairwise win-rates based on the latter are reported in Appendix M .

Metric	RG	Surs.	Suts.	Surm.	Puter	Vall.
Number of overlapping segments	51	64	139	238	160	(0)
Item-level Pearson correlation	52.5	73.2	82.4	65.9	64.0	—
Pairwise comparison agreement rate	63.8	66.7	77.9	68.2	65.8	—
System-level Spearman correlation	80.0	100.0	100.0	100.0	80.0	—

Appendix

Table 21: Inter-rater agreement metrics for segment-level fluency ratings.

Metric	RG	Surs.	Suts.	Surm.	Puter	Vall.
Number of overlapping documents	16	21	37	23	(0)	(0)
Item-level Pearson correlation	23.5	35.4	17.4	50.4	—	—
Pairwise comparison agreement rate	56.2	52.4	62.2	65.2	—	—
System-level Spearman correlation	80.0	40.0	100.0	80.0	—	—

Appendix

Table 22: Inter-rater agreement metrics for document-level accuracy ratings.

Metric	RG	Surs.	Suts.	Surm.	Puter	Vall.
Number of overlapping segments	159	271	341	303	(0)	(0)
Item-level Pearson correlation	43.0	45.3	36.1	44.6	—	—
Pairwise comparison agreement rate	56.0	63.5	52.2	59.4	—	—
System-level Spearman correlation	80.0	100.0	100.0	80.0	—	—

Appendix

Table 23: Inter-rater agreement metrics for segment-level accuracy ratings.

Machine Translation

Machine translation develops computational methods for converting text, speech, and visual content between languages while preserving meaning and purpose.

See all 259 papers

A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books
Jul 24, 2026Varun Ghat Ravikumar, Sina Ahmadi, Lena Jäger +1LLM Fine-Tuning Synthetic Data Generation
Department of Informatics, University of Zurich · Department of Computational Linguistics, University of Zurich
Can Linguistically Related Languages Guide LLM Translation in Low-Resource Settings?
Mar 17, 2026Aishwarya Ramasethu, Niyathi Allu, Rohin Garg +2Machine Translation Low-Resource MT
Prediction Guard · Independent · Scale AI +1
Less is More: Adapting Text Embeddings for Low-Resource Languages with Small Scale Noisy Synthetic Data
Mar 4, 2026Zaruhi Navasardyan, Spartak Bughdaryan, Bagrat Minasyan +1Text Embeddings Low-Resource Language Processing
Metric AI Lab

Jul 24, 2026J/K move · Enter open · S save

A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books

Varun Ghat Ravikumar, Sina Ahmadi, Lena Jäger, Rico Sennrich

Department of Informatics, University of Zurich · Department of Computational Linguistics, University of Zurich

LLM Fine-Tuning Synthetic Data Generation Low-Resource MT Synthetic Data Generation for Language Models Machine Translation

Open paper AskPDF arXiv

Most endangered languages lack the parallel data required for machine translation, despite the existence of descriptive grammar books. We introduce a pipeline that uses large language models to extract grammatical rules, example sentences, and lexicons from grammar books and generate synthetic parallel corpora for fine-tuning-rather than feeding grammar content into prompts at inference time, as in prior work. Validated on three typologically diverse low-resource languages-Kalamang (Papuan), Tuatschin (Romance), and Mandan (Siouan)-we show that fine-tuning on synthetic data improves over seed-data baselines in 75% of configurations for Kalamang and 59% for Tuatschin, with best-case ChrF++ gains of +8.8, +5.3, and +3.3 respectively. Through a systematic factorial study across 96 configurations varying target part-of-speech, retrieval granularity, and sample volume, we identify which factor combinations drive gains and where they break down. Our results demonstrate that static linguistic documentation can be repurposed for machine translation fine-tuning, offering a practical path towards translation tools for severely under-resourced languages.

Abstract

Machine Translation

Explore similar work

A Factorial Study of Synthetic Data Generation for Low-Resource Machine Translation using Grammar Books

Can Linguistically Related Languages Guide LLM Translation in Low-Resource Settings?

Less is More: Adapting Text Embeddings for Low-Resource Languages with Small Scale Noisy Synthetic Data