Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography

Gianluca Vico; Jindřich Libovický

cs.CLFeb 16, 2026

Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography

Authors: Gianluca Vico, Jindřich Libovický

Organizations: Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Malostranské náměstí 25, 118 00 Praha, Czech Republic

Abstract

We present a crowdsourced dataset for Piedmontese, an endangered Romance language of northwestern Italy. The dataset comprises 145 Italian-Piedmontese parallel sentences derived from Flores+, with translations produced by speakers writing in their natural orthographic style rather than adhering to standardized conventions, along with manual word alignment. We use this resource to benchmark several large language models on tokenization parity, topic classification, and machine translation. Our analysis reveals that Piedmontese incurs a tokenization penalty relative to higher-resource Romance languages, yet LLMs achieve classification performance approaching that of Italian, French, and English. Machine translation results are asymmetric: models translate adequately from Piedmontese into high-resource languages, but generation into Piedmontese remains challenging. The dataset and code are publicly released.

Figures & tables

Flores+ dev 114
Ita	Si tratta della maggiore acquisizione nella storia di eBay.
Pms	A l’è la pi granda aquisission ënt la stòria d’ebay
Eng	It is the biggest acquisition in eBay’s history.
Fra	C’est la plus grande acquisition de l’histoire d’eBay.

Table 1: Sample from parallel sentences for evaluating machine translation. Annotators translated the Italian sample into Piedmontese. The Italian, French and English samples are originally from Flores+.

Figure 1: On the left, the main language used by the annotators; Icelandic is included in Other . On the right, the self-reported proficiency in Piedmontese. The majority of people uses Italian and self-reports perfect or fair proficiency in Piedmontese.

Figure 2: Age distribution of the annotators. Most annotators are 20-30 years old, while older people are more likely to know Piedmontese, but we did not reach them.

Figure 3: Parity scores with respect to English and Italian. Piedmontese has worse parity compared to the other languages; however, it is closer to one when compared to Italian.

Figure 4: This is a sample alignment. The gray background is the reference alignment, while eflomal alignment is represented by the blue circles and SimAlign one by the green crosses. The English translation of the sentence is This seems sensible, because the Earth doesn’t feel as if it’s moving, does it?

Model	F 1 $\uparrow$	Precision $\uparrow$	Recall $\uparrow$
Eflomal	.774	.817	.735
SimAlign	.589	.726	.496

Table 2: Alignment scores of eflomal and SimAlign.

Figure 5: Comparison on the F1 scores of the models in the topic classification task. We perform bootstrapping to compute the confidence interval of the scores.

Figure 6: chrF++ scores of the different models. Each subplot shows the target language, while the sources languages are on the x-axis. The dotted horizontal lines indicate the scores obtained when using the reference text in a given language as if it were the translation.

Dev		Target
		Pms	Ita	Fra	Eng
Source	Pms	-	58.80	42.48	45.95
	Ita	32.62	-	57.83	60.98
	Fra	26.45	54.68	-	70.89
	Eng	27.02	57.60	72.25	-
	Pms Pivot	-	-	44.62	46.91

Table 3: Average chrF++ scores of the models on the two sets and all directions. Note that only columns are comparable. Pivot refers to the experiments that use Italian as a pivot for the translation.

Flores+ dev 114
It is the biggest acquisition in eBay’s history.
Pms	A l’è la pi granda aquisission ënt la stòria d’ebay
Ita	Si tratta della maggiore acquisizione nella storia di eBay.
Pms $\rightarrow$ Ita
EuroLLM	È la più grande acquisizione nella storia di eBay
Gemini	È la più grande acquisizione nella storia di eBay

Table 4: Translation examples. The Ita

\rightarrow

Pms translations are understandable, but have different spellings than the reference and across models. The Pms

\rightarrow

Ita translations are correct, although, phrased differently than the reference translation.

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Metric: F 1
Model	Pms	Ita	Fra	Eng
Llama	0.778	0.849	0.822	0.849
Llama	(0.047)	(0.041)	(0.044)	(0.040)
Qwen	0.671	0.772	0.778	0.750
Qwen	(0.054)	(0.047)	(0.046)	(0.051)
Gemma	0.767	0.849	0.877	0.863

Appendix

Table 5: F 1 of the different models on the classification task. In parenthesis the STD of the score.

Metric: Precision
Model	Pms	Ita	Fra	Eng
Llama	0.789	0.849	0.822	0.849
Llama	(0.047)	(0.041)	(0.044)	(0.040)
Qwen	0.686	0.778	0.789	0.761
Qwen	(0.054)	(0.047)	(0.046)	(0.051)
Gemma	0.767	0.849	0.877	0.863

Appendix

Table 6: Precision of the different models on the classification task. In parenthesis the STD of the score.

Metric: Recall
Model	Pms	Ita	Fra	Eng
Llama	0.767	0.849	0.822	0.849
Llama	(0.048)	(0.041)	(0.044)	(0.040)
Qwen	0.658	0.767	0.767	0.740
Qwen	(0.054)	(0.047)	(0.047)	(0.052)
Gemma	0.767	0.849	0.877	0.863

Appendix

Table 7: Recall of the different models on the classification task. In parenthesis the STD of the score.

Metric: Accuracy
Model	Pms	Ita	Fra	Eng
Llama	0.767	0.849	0.822	0.849
Llama	(0.048)	(0.041)	(0.044)	(0.040)
Qwen	0.658	0.767	0.767	0.740
Qwen	(0.054)	(0.047)	(0.047)	(0.052)
Gemma	0.767	0.849	0.877	0.863

Appendix

Table 8: Accuracy of the different models on the classification task. In parenthesis the STD of the score.

Pms → Ita
Model	BLEU	chrF++	TER	COMET
Llama	42.910	63.800	44.080	0.933
Llama	(3.720)	(2.890)	(3.650)	(0.003)
Qwen	37.490	59.360	48.890	0.932
Qwen	(3.730)	(2.850)	(3.810)	(0.003)
Gemma	42.090	62.920	45.960	0.935

Appendix

Table 9: Translation results from Pms to Ita.

Pms → Fra
Model	BLEU	chrF++	TER	COMET
Llama	21.700	47.070	70.330	0.908
Llama	(2.400)	(2.160)	(3.470)	(0.005)
Qwen	20.420	45.510	71.000	0.905
Qwen	(2.500)	(2.160)	(2.920)	(0.005)
Gemma	21.550	47.170	70.680	0.908

Appendix

Table 10: Translation results from Pms to Fra.

Pms → Eng
Model	BLEU	chrF++	TER	COMET
Llama	20.270	48.640	71.780	0.901
Llama	(2.760)	(2.490)	(4.460)	(0.004)
Qwen	19.180	46.060	74.230	0.900
Qwen	(2.690)	(2.380)	(3.790)	(0.004)
Gemma	19.970	48.200	73.510	0.900

Appendix

Table 11: Translation results from Pms to Eng.

Ita → Pms
Model	BLEU	chrF++	TER	COMET
Llama	6.980	34.780	83.220	0.842
Llama	(1.290)	(1.640)	(2.550)	(0.008)
Qwen	5.250	31.100	93.290	0.843
Qwen	(1.540)	(1.760)	(7.820)	(0.007)
Gemma	6.270	34.090	84.580	0.841

Appendix

Table 12: Translation results from Ita to Pms.

Ita → Fra
Model	BLEU	chrF++	TER	COMET
Llama	33.500	58.890	54.750	0.943
Llama	(2.550)	(1.840)	(2.930)	(0.003)
Qwen	32.380	58.060	55.300	0.943
Qwen	(2.510)	(1.750)	(2.940)	(0.003)
Gemma	34.260	59.610	53.520	0.942

Appendix

Table 13: Translation results from Ita to Fra.

Ita → Eng
Model	BLEU	chrF++	TER	COMET
Llama	32.520	60.930	52.970	0.941
Llama	(2.910)	(2.050)	(3.430)	(0.003)
Qwen	32.050	60.750	52.360	0.942
Qwen	(2.790)	(1.920)	(3.110)	(0.003)
Gemma	33.410	61.440	51.580	0.941

Appendix

Table 14: Translation results from Ita to Eng.

Fra → Pms
Model	BLEU	chrF++	TER	COMET
Llama	3.000	27.160	95.790	0.835
Llama	(1.120)	(1.360)	(2.510)	(0.008)
Qwen	2.410	24.890	101.020	0.838
Qwen	(1.110)	(1.400)	(6.610)	(0.007)
Gemma	2.960	27.450	94.370	0.831

Appendix

Table 15: Translation results from Fra to Pms.

Fra → Ita
Model	BLEU	chrF++	TER	COMET
Llama	27.890	55.010	58.320	0.952
Llama	(2.270)	(1.660)	(2.660)	(0.003)
Qwen	26.810	54.430	59.620	0.952
Qwen	(2.450)	(1.750)	(2.710)	(0.003)
Gemma	29.270	55.710	57.120	0.952

Appendix

Table 16: Translation results from Fra to Ita.

Fra → Eng
Model	BLEU	chrF++	TER	COMET
Llama	46.510	69.120	37.150	0.947
Llama	(3.530)	(2.250)	(3.270)	(0.002)
Qwen	45.320	68.170	38.340	0.947
Qwen	(3.510)	(2.130)	(3.400)	(0.002)
Gemma	46.410	69.290	36.160	0.947

Appendix

Table 17: Translation results from Fra to Eng.

Eng → Pms
Model	BLEU	chrF++	TER	COMET
Llama	3.170	28.010	94.430	0.824
Llama	(1.140)	(1.390)	(2.370)	(0.009)
Qwen	2.340	24.970	98.110	0.828
Qwen	(1.200)	(1.280)	(3.870)	(0.008)
Gemma	2.460	27.470	94.370	0.825

Appendix

Table 18: Translation results from Eng to Pms.

Eng → Ita
Model	BLEU	chrF++	TER	COMET
Llama	29.850	56.230	54.900	0.953
Llama	(2.110)	(1.650)	(2.390)	(0.003)
Qwen	29.920	56.340	56.200	0.952
Qwen	(2.560)	(1.930)	(2.790)	(0.003)
Gemma	30.770	57.210	54.380	0.953

Appendix

Table 19: Translation results from Eng to Ita.

Eng → Fra
Model	BLEU	chrF++	TER	COMET
Llama	52.800	71.170	33.280	0.949
Llama	(3.640)	(2.420)	(2.970)	(0.002)
Qwen	48.010	68.150	38.380	0.948
Qwen	(3.130)	(1.980)	(2.820)	(0.002)
Gemma	52.420	71.090	33.750	0.947

Appendix

Table 20: Translation results from Eng to Fra.

Pms → Fra, Pivot
Model	BLEU	chrF++	TER	COMET
Llama	23.580	49.090	68.000	0.927
Llama	(2.450)	(2.220)	(3.290)	(0.002)
Qwen	19.890	45.140	72.630	0.927
Qwen	(2.460)	(2.180)	(3.190)	(0.002)
Gemma	23.740	49.450	68.140	0.927

Appendix

Table 21: Translation results with pivoting from Pms to Fra.

Pms → Eng, Pivot
Model	BLEU	chrF++	TER	COMET
Llama	21.780	49.550	68.830	0.917
Llama	(2.910)	(2.450)	(3.870)	(0.002)
Qwen	19.340	47.160	70.800	0.917
Qwen	(2.730)	(2.400)	(3.610)	(0.002)
Gemma	23.630	50.770	66.010	0.917

Appendix

Table 22: Translation results with pivoting from Pms to Eng.

Fra → Pms, Pivot
Model	BLEU	chrF++	TER	COMET
Llama	3.990	28.780	92.600	0.836
Llama	(1.180)	(1.430)	(2.440)	(0.008)
Qwen	2.720	26.220	99.710	0.838
Qwen	(1.120)	(1.430)	(7.940)	(0.007)
Gemma	3.760	28.640	92.540	0.834

Appendix

Table 23: Translation results with pivoting from Fra to Pms.

Fra → Eng, Pivot
Model	BLEU	chrF++	TER	COMET
Llama	42.730	66.800	39.970	0.952
Llama	(3.390)	(2.190)	(3.580)	(0.001)
Qwen	42.940	65.990	40.340	0.952
Qwen	(3.490)	(2.240)	(3.410)	(0.001)
Gemma	42.950	66.630	39.080	0.952

Appendix

Table 24: Translation results with pivoting from Fra to Eng.

Eng → Pms, Pivot
Model	BLEU	chrF++	TER	COMET
Llama	3.770	28.910	92.680	0.826
Llama	(1.050)	(1.410)	(2.490)	(0.008)
Qwen	2.200	26.200	96.920	0.828
Qwen	(0.980)	(1.370)	(3.500)	(0.008)
Gemma	3.560	29.210	91.640	0.826

Appendix

Table 25: Translation results with pivoting from Eng to Pms.

Eng → Fra, Pivot
Model	BLEU	chrF++	TER	COMET
Llama	47.630	67.830	37.390	0.956
Llama	(3.130)	(2.170)	(2.810)	(0.001)
Qwen	42.230	64.330	42.110	0.956
Qwen	(2.970)	(1.970)	(2.750)	(0.001)
Gemma	47.460	67.640	38.090	0.956

Appendix

Table 26: Translation results with pivoting from Eng to Fra.

Parity w.r.t Pms
Model	Ita	Fra	Eng
Llama	0.905	0.898	0.569
Qwen	0.909	0.897	0.576
Gemma	0.816	0.850	0.609
Tower	0.806	0.830	0.616
Eurollm	0.784	0.858	0.639

Appendix

Table 27: Parity scores with respect to Piedmontese.

Parity w.r.t Ita
Model	Pms	Fra	Eng
Llama	1.140	0.998	0.634
Qwen	1.136	0.993	0.638
Gemma	1.267	1.041	0.749
Tower	1.284	1.033	0.767
Eurollm	1.324	1.097	0.818

Appendix

Table 28: Parity scores with respect to Italian.

Parity w.r.t Fra
Model	Pms	Ita	Eng
Llama	1.164	1.028	0.640
Qwen	1.167	1.034	0.649
Gemma	1.243	0.983	0.726
Tower	1.269	0.993	0.751
Eurollm	1.232	0.935	0.753

Appendix

Table 29: Parity scores with respect to French.

Parity w.r.t Eng
Model	Pms	Ita	Fra
Llama	1.848	1.633	1.602
Qwen	1.831	1.622	1.584
Gemma	1.740	1.380	1.416
Tower	1.721	1.347	1.371
Eurollm	1.653	1.255	1.357

Appendix

Table 30: Parity scores with respect to English.

Tokenization in Language Models

See all 46 papers

"Chi nas dal soch el sent de legn" -- Auditing Text Corpora for Lombard
Jun 4, 2026Edoardo Signoroni, Pavel RychlýTraining Data Curation Low-Resource Language Processing
NLP Centre, Faculty of Informatics
Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varieties
Mar 26, 2026Jannis Vamvas, Ignacio Pérez Prat, Angela Heldstab +3Machine Translation Low-Resource MT
University of Zurich · Lia Rumantscha
LLiMba: Sardinian on a Single GPU -- Adapting a 3B Language Model to a Vanishing Romance Language
May 9, 2026Luca BalloreLanguage Model Pretraining Adapter Tuning

Jun 4, 2026J/K move · Enter open · S save

"Chi nas dal soch el sent de legn" -- Auditing Text Corpora for Lombard

Edoardo Signoroni, Pavel Rychlý

NLP Centre, Faculty of Informatics

Training Data Curation Low-Resource Language Processing Corpus Linguistics

Open paper AskPDF arXiv

Several of the world's languages are still under-resourced in terms of Natural Language Processing (NLP) tools. This is mostly due to the lack of high-quality datasets to train, develop, and evaluate systems and models for several tasks, such as Machine Translation (MT). We conduct a manual audit of the parallel and monolingual corpora available for Lombard, an under-resourced language continuum from Italy. Our analysis reveals that the perceived abundance of web-scraped data is an illusion, with massive datasets plagued by severe language misidentification, boilerplate text, and non-linguistic noise. Furthermore, we analyze the orthographic composition of the valid Lombard portions across web-scraped datasets, curated corpora, and benchmarks. Our findings show conflicting orthographical systems and severe representational bias across all corpora: high-quality data is heavily skewed towards Western Lombard varieties, with Eastern ones left on the margins. This underscores the need for variety-aware, community-driven data curation rather than purely quantity-driven scraping.

Abstract

Tokenization in Language Models

Explore similar work

"Chi nas dal soch el sent de legn" -- Auditing Text Corpora for Lombard

Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varieties

LLiMba: Sardinian on a Single GPU -- Adapting a 3B Language Model to a Vanishing Romance Language