Precision over Scale: A Polish-Silesian Benchmark and a Translation System Outperforming Open-Source and Commercial Models

Grzegorz Kulik; Mikołaj Pokrywka; Adam Jatowt; Wojciech Kusa

cs.CLOct 1, 2026

Precision over Scale: A Polish-Silesian Benchmark and a Translation System Outperforming Open-Source and Commercial Models

Authors: Grzegorz Kulik, Mikołaj Pokrywka, Adam Jatowt, Wojciech Kusa

Organizations: NASK National Research Institute, Poland · University of Innsbruck, Austria

Abstract

Dialectal machine translation remains challenging due to limited data and strong linguistic variation not captured by standard benchmarks, which often assume standardized and well-edited text. We study Polish-Silesian MT using neural and rule-based systems, evaluating on SiLTT - a new Pol-Szl testset, alongside established BOUQuET and FLORES benchmarks. Results show our rule-based system is consistently strongest on SiLTT and BOUQuET datasets and that TranslateGemma fine-tuned on a curated dataset improves over strong neural baselines but does not surpass the rule-based system in dialectal settings. We release SiLTT and our best neural model to support further research.

Figures & tables

Panel of Figure 1 — Figure 1: The eight modules that build the assembly line of the shallow-transfer machine translation system.

Statistic	Polish	Silesian
Avg sentence length (chars)	93.1	79.4
Avg sentence length (words)	14.8	13.6
Max sentence length (chars)	542	445
Max sentence length (words)	82	73
Vocabulary size	7673	5393

Table 1: Dataset statistics of SiLTT.

Model	SiLTT		BOUQuET
Model	BLEU	chrF	BLEU	chrF
PLLuM-12B-nc-chat	1.7	22.3	3.4	28.4
Bielik-PL-11B-v3.0-IT	3.4	26.3	7.8	35.5
MADLAD-400-10B-MT	2.1	21.3	7.5	32.0
NLLB 3.3B	3.8	26.8	12.2	39.2
NLLB 54B MOE	3.2	26.0	9.9	38.3

Table 2: Comparison of POL

\rightarrow

SZL results on SiLTT and BOUQuET benchmarks. Bold indicates the best result, and underlining indicates the second-best result.

Model	BLEU	chrF
PLLuM-12B-nc-chat	0.1	12.1
Bielik-PL-11B-v3.0-IT	5.7	35.1
MADLAD-400-10B-MT	9.8	37.2
NLLB 3.3B	7.4	34.8
NLLB 54B MOE	7.4	37.5
TransGemma	5.1	32.1

Table 3: Results on the FLORES benchmark for POL

\rightarrow

SZL translation. Google Translate achieves the best performance overall, followed by the rule-based system. TranslateGemma fine-tuned on CBT-22k improves over the base model but remains below the strongest systems.

	System output
Source (PL)	Rule-based	TransGemma-FT (CBT-22k)	Google Translate
Chłopcy i dziewczynki byli bardzo szczęśliwi podczas zabawy. Boys and girls were very happy during play.	Synki i dziouszki byli fest szczyńśliwi w czasie graczki. ✓ Correct.	Chłopcy i dziouchy byli bardzo szczyńśliwi podczas zabawy . ✗ Polish retained: chłopcy $\rightarrow$ synki , zabawy $\rightarrow$ graczki . Wrong diminutive: dziouchy $\rightarrow$ dziouszki .	Syny i dziouchki byli barzo szczyńśliwi w czasie szpilu . ✗ Morphology: syny $\rightarrow$ synki , dziouchki $\rightarrow$ dziouszki . Lexical: szpilu (‘game’) $\rightarrow$ graczki .
Możesz też skontaktować się z nami przez telefon. Alternatively, you can reach out to us by phone.	Możesz tyż skōntaktować sie z nami bez telefōn. ✓ Correct.	Możesz tyż skōntaktować sie z nami przez telefōn. ✗ Polish retained: przez $\rightarrow$ bez .	Możesz tyż sie z niymi spokopić bez telefōn. ✗ Person: niymi (‘them’) $\rightarrow$ nami . Lexical: spokopić (‘understand’) $\rightarrow$ skōntaktować .
I przyprowadzę dzieci. And I’ll bring the kids.	I przikludza dzieci. ✓ Correct.	I przikludza dzieci. ✓ Correct.	A ja prziwiozōm dzieci. ✗ Redundant Polish ja $\rightarrow$ jo . Person and lexis: prziwiozōm (‘they will bring by car’) $\rightarrow$ przikludza / prziwiyda .
Twój ojciec za bardzo się martwi. Your father worries too much.	Twōj fater za fest sie umrzici . ✗ martwi read as ‘the dead’ rather than ‘worries’; the two are homographs in Polish.	Twōj ôjciec za bardzo sie martwi . ✗ Polish retained: martwi $\rightarrow$ tropi / staro .	Twōj fater sie za moc tropi. ✓ Correct.
Rzeczy, które chciałbym wiedzieć wcześniej. Things I wish I had known earlier.	Rzeczy, co ôna chcioł bych wiedzieć wczaśnij. ✗ Case of rzeczy misread: the singular genitive and plural nominative are homographs in both Polish and Silesian, giving the wrong gender and case in ôna $\rightarrow$ je .	Rzeczy, kere chcioł bych wiedzieć wczaśnij. ✓ Correct.	Rzeczy, kere chciołbych wczasnij . ✗ Misspelling: wczasnij $\rightarrow$ wczaśnij .

Table 4: Qualitative comparison on the BOUQuET dataset. Source sentences are given with English glosses. Within each system output, tokens marked in red are erroneous; the expected Silesian form is given in green in the accompanying note.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	BLEU	chrF
NLLB 3.3B	18.0	32.5
TransGemma	24.8	48.4
GPT-5.4	41.1	63.2
Google Translate	38.3	61.7
ours
Rule-Based	31.7	57.4

Appendix

Table 5: Evaluation results on the BOUQuET SZL

\rightarrow

POL translation task.

Explore similar work

Unified Multi-Dialectal Neural Machine Translation for Bangla Using the Dwadash Benchmark Corpus
Aug 12, 2026Rakib Ullah, Md. Ruhul Islam, Tanbir Ahmed +1Dialects Neural Machine Translation
Department of Computer Science and Engineering SYLHET ENGINEERING COLLEGE, SYLHET Sylhet, Bangladesh
Last Translation Benchmark
Sep 3, 2026Vilém Zouhar, Niyati Bafna, Mukund Choudhary +257Machine Translation Quality Reproducibility
Benchmarking Bengali Dialectal Bias: A Multi-Stage Framework Integrating RAG-Based Translation and Human-Augmented RLAIF
Mar 22, 2026K. M. Jubair Sami, Dipto Sumit, Ariyan Hossain +1Dialects Large Language Model Bias
Department of Computer Science and Engineering BRAC University, Dhaka, Bangladesh

Aug 12, 2026, cs.CLJ/K move · Enter open · S save

Unified Multi-Dialectal Neural Machine Translation for Bangla Using the Dwadash Benchmark Corpus

Rakib Ullah, Md. Ruhul Islam, Tanbir Ahmed, Nayan Kumar Nath

Department of Computer Science and Engineering SYLHET ENGINEERING COLLEGE, SYLHET Sylhet, Bangladesh

Dialects Neural Machine Translation Bangla Text Corpora

Open paper AskPDF arXiv

Neural Machine Translation (NMT) and Large Language Models (LLMs) excel at cross-lingual tasks but often fail to capture intra-lingual morphological variation, marginalizing dialectal speakers. In Bangla, existing translation frameworks commonly rely on Standard Colloquial Bangla (SCB) as an intermediate pivot, which can compound errors and reduce cross-dialectal nuance. To address this gap, we introduce a unified, multi-directional NMT system for direct translation between SCB and eleven regional variants. We first review prior dialectal NLP resources to identify existing technological gaps. As a foundational contribution, we construct and release a large multi-dialect parallel corpus for Bangla, comprising 14,562 aligned rows and 51,541 non-null sentence pairs through the integration of seven prior datasets and native-speaker-verified manual augmentation. Using this corpus, we benchmark state-of-the-art sequence-to-sequence architectures with parameter-efficient Weight-Decomposed Low-Rank Adaptation (DoRA). Results show that deep monolingual pre-training is more effective than large multilingual capacity for this task. The compact BanglaT5 model outperforms NLLB-200 and mBART-50 by up to 13.96 BLEU, achieving 29.26 BLEU, 57.26 chrF++, and 49.68 METEOR. A dataset scaling study shows diminishing returns beyond 3,000 parallel pairs and indicates that linguistic proximity to Standard Bangla is more important than raw data volume for translation quality. Finally, we deploy the optimized model as an INT8-quantized web application, providing a scalable, open-source framework for inclusive language technology and equitable digital access.

Abstract

Explore similar work

Unified Multi-Dialectal Neural Machine Translation for Bangla Using the Dwadash Benchmark Corpus

Last Translation Benchmark

Benchmarking Bengali Dialectal Bias: A Multi-Stage Framework Integrating RAG-Based Translation and Human-Augmented RLAIF