cs.CLFeb 18, 2026

Meenz bleibt Meenz, but Large Language Models Do Not Speak Its Dialect

Authors: Minh Duc Bui, Manuel Mager, Peter Herbert Kann, Katharina von der Wense

Organizations: Johannes Gutenberg University Mainz, Germany · Marburg University, Germany · University of Colorado Boulder, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Meenzerisch, the dialect spoken in the German city of Mainz, is also the traditional language of the Mainz carnival, a yearly celebration well known throughout Germany. However, Meenzerisch is on the verge of dying out-a fate it shares with many other German dialects. Natural language processing (NLP) has the potential to help with the preservation and revival efforts of languages and dialects. However, so far no NLP research has looked at Meenzerisch. This work presents the first research in the field of NLP that is explicitly focused on the dialect of Mainz. We introduce a digital dictionary-an NLP-ready dataset derived from an existing resource (Schramm, 1966)-to support researchers in modeling and benchmarking the language. It contains 2,351 words in the dialect paired with their meanings described in Standard German. We then use this dataset to answer the following research questions: (1) Can state-of-the-art large language models (LLMs) generate definitions for dialect words? (2) Can LLMs generate words in Meenzerisch, given their definitions? Our experiments show that LLMs can do neither: the best model for definitions reaches only 6.27% accuracy and the best word generation model's accuracy is 1.51%. We then conduct two additional experiments in order to see if accuracy is improved by few-shot learning and by extracting rules from the training set, which are then passed to the LLM. While those approaches are able to improve the results, accuracy remains below 10%. This highlights that additional resources and an intensification of research efforts focused on German dialects are desperately needed.

Figures & tables

Figure 1: Dataset Creation Pipeline. Overview of the semi-automatic five-step process used to create the Mainz Dialect Dataset.
CategoryValue
Dataset Size
Total extracted pairs2,471
Extracted pairs after clean-up2,351
Word Length
Median length (characters)8
Definition Length
Table 1: Overview of the Mainz Dialect Dataset. Summary of the extracted and cleaned dataset.
Word in MeenzerischStandard German Definition
AaweiderworschdSalzgurke (English: pickled cucumber )
BitzelwasserMineralwasser mit Kohlensäuregehalt (English: Carbonated mineral water )
Mauldabbeziererscherzhaftes Lob für einen guten Wein […] ( English: playful compliment for a good wine )
Rachebutzersaurer Wein (English: sour wine )
Schlobb1. Schleife; 2. Knoten; 3. […] (English: 1. bow; 2. knot )
SchimmesHunger (English: hunger )
Table 2: Examples from Our Dataset. Samples illustrating the mapping between dialect words and their corresponding Standard German definitions.
ModelDialect of MainzEnglish
 Dev SetTest SetTest Set
Large Models
GPT-OSS 120B (Thinking)5.60%4.92%91.19%
Llama-3.3 70B8.80%6.27%91.69%
Leo-HessianAI 70B4.40%4.81%81.88%
Qwen-2.5 72B4.80%5.35%90.19%
Table 3: Accuracy of Definition Generation. Development and test set accuracy of LLMs for the Mainz dialect, and test set accuracy for the English baseline.
ModelDialect of MainzEnglish
 Dev SetTest SetTest Set
Large Models
GPT-OSS 120B (Thinking)2.40%1.51%82.50%
Llama-3.3 70B0.80%1.03%69.00%
Leo-HessianAI 70B0.80%0.92%47.10%
Qwen-2.5 72B0.80%0.43%66.90%
Table 4: Accuracy of Word Generation given Explanation. Development and test set accuracy of LLMs in generating the correct word for the Mainz dialect, and test set accuracy for the English baseline.
ShotsDefinitionWord
 GenerationGeneration
Random Selection
16.00%1.60%
59.60%2.00%
1011.20%1.60%
257.60%1.20%
Table 5: Different Numbers of Few-shot Examples on the Development Set. Comparison of Llama-3.3 70B Instruct across different numbers of in-context examples.
Selection MethodAccuracy
Definition Generation
Baseline: Zero-Shot6.27%
Random Selection ( k=10k=10 )7.67%
Edit Distance-Based Selection ( k=50k=50 )9.24%*
Word Generation
Baseline: Zero-Shot1.03%
Table 6: Few-shot Results for Llama-3.3 70B. Asterisks (*) mark significant improvements (McNemar’s test, p<.05p<.05 ); bolding highlights the best significant score (or multiple scores where differences are not significant).
Figure 2: Part of our File with Automatically Extracted Rules. The final rules are fed into Llama-3.3 70B for definition and word generation.
Selection MethodAccuracy
Definition Generation
Baseline: Zero-Shot6.27%
Injected Extracted Rules8.37%*
Word Generation
Baseline: Zero-Shot1.03%
Injected Extracted Rules0.76%
Table 7: Results for Automatically Extracted Rules Injected into Llama-3.3 70B. Asterisks (*) mark significant improvements (McNemar’s test, p<.05p<.05 ); bolding highlights the best significant score (or multiple scores where differences are not significant).
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Prompt used for extracting dictionary definitions. English translation: “You receive an unstructured or faulty definition of the word ’{Word}’ from an old dictionary. Your task is to extract only the actual meaning of the word from the text, without comments, reformulations, or explanations. Rules: (1) If multiple meanings are present, number them consecutively (1., 2., 3., …) and separate them by line breaks. (2) If the definition refers exclusively to another word, output ’[SEE] <word>’. (3) Do not alter the original text; output only the relevant excerpt. (4) If there is no definition in the text, output ’No definition’.”
Figure 4: Prompt used for cleaning dictionary definitions English translation: “You are a careful linguistic assistant. Your task is to clean dictionary definitions without changing their meaning. Remove only unnecessary special characters such as hyphens, double spaces, or similar OCR artifacts. Leave references in the form [SEE] unchanged and preserve numbering of definitions. Here is the definition for the word ’{Word}’: {cleaned}. Clean the definition according to the instructions. If it is already clean, return it unchanged. Output only the cleaned definition.”
Figure 5: Prompt used for generating dictionary definitions. English translation: “You are a precise and reliable linguistic assistant. Your task is to create dictionary definitions. Provide exclusively a single, short, and concise meaning of the requested word. Use the meaning commonly used in the Mainzer dialect, but formulate the definition in Standard German. Create exactly one short dictionary definition for the word ’{Word}’. Use the meaning commonly used in the Mainzer dialect, without providing additional information, examples, or alternative meanings.”
Figure 6: Prompt used for evaluating semantic equivalence. English translation: “You are a precise and neutral evaluator of semantic similarity between dictionary definitions. Your task is to assess whether two definitions express the same meaning, even if they are phrased differently. Consider synonyms, paraphrases, or stylistic differences. Respond exclusively with ’SAME’ if the two definitions convey the same content, or with ’DIFFERENT’ if their meaning differs substantially. Dictionary entries may contain multiple definitions. If at least one of these definitions matches the one generated by the LLM, the result counts as ’SAME’. Do not provide any additional explanations or justifications. Compare the following two definitions. Assess whether they are semantically equivalent. Definition A (LLM): ’{response_generator}’. Definition B (Dictionary): ’{definition_final}’.”
Figure 7: Prompt used for generating a Mainzer dialect word . English translation: “You are a precise and reliable linguistic assistant. Your task is to find the appropriate word in the Mainzer dialect for a given meaning. Provide exclusively a single word that expresses this meaning in the Mainzer dialect. Do not provide explanations, translations, or additional information. Find the Mainzer dialect word that expresses the following meaning: ’{definition_final_generation}’. Respond only with the dialect word.”
Figure 8: Two-turn prompting strategy for automatic rule extraction. In the first turn, the model generates Standard German candidates. In the second turn, it induces mapping rules between dialect and Standard German forms.
Figure 9: Automatic extracted rules for mapping Mainzer dialect words to Standard German forms. This is generated by DeepSeek-R1.
More in this topic

Low-Resource Language Processing

Low-resource language processing builds and evaluates language technologies for languages with limited data, tools, or digital representation.

See all 565 papers

Explore similar work

CardsList
Apr 26, 2026

Resource-Lean Lexicon Induction for German Dialects

Automatic induction of high-quality dictionaries is essential for building lexical resources, yet low-resource languages and dialects pose several challenges: limited access to annotators, high degree of spelling variations, and poor performance of large language models (LLMs). We empirically show that statistical models (random forests) trained on string similarity features are surprisingly effective for inducing German dialect lexicons. They outperform LLMs, enable cross-dialect transfer, and offer a lightweight data-driven alternative. We evaluate our models intrinsically on bilingual lexicon induction (BLI) and extrinsically on dialect information retrieval (IR). On BLI, random forests outperform Mistral-123b while being more resource-lean. On dialect IR with BM25, using our dialect dictionaries for query expansion yields relative improvements of up to 28.9% in nDCG@10 and 50.7% in Recall@100. Motivated by the resource scarcity in dialects, we further investigate the extent to which models transfer across different German dialects, and their performance under varying amounts of training data.

Robert Litschko, Barbara Plank, Diego Frassinelli

MaiNLP, Center for Information and Language Processing, LMU Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany

Low-Resource Language Processing
AskPDF
Feb 16, 2026

Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography

We present a crowdsourced dataset for Piedmontese, an endangered Romance language of northwestern Italy. The dataset comprises 145 Italian-Piedmontese parallel sentences derived from Flores+, with translations produced by speakers writing in their natural orthographic style rather than adhering to standardized conventions, along with manual word alignment. We use this resource to benchmark several large language models on tokenization parity, topic classification, and machine translation. Our analysis reveals that Piedmontese incurs a tokenization penalty relative to higher-resource Romance languages, yet LLMs achieve classification performance approaching that of Italian, French, and English. Machine translation results are asymmetric: models translate adequately from Piedmontese into high-resource languages, but generation into Piedmontese remains challenging. The dataset and code are publicly released.

Gianluca Vico, Jindřich Libovický

Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Malostranské náměstí 25, 118 00 Praha, Czech Republic

Tokenization in Language ModelsLow-Resource Language ProcessingLow-Resource MTMultilingual Language Model EvaluationMultilingual Language Models
AskPDF
Jul 8, 2026

DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

Large language models increasingly understand dialectal English, yet still produce only standard, US-leaning English, leaving dialectal generation, the harder half of the problem, largely unaddressed. We introduce DiaLLM, which continually pretrains three open-weight language model families on the International Corpus of English and applies implicit and explicit post-training paradigms, each combined with three model alignment strategies, giving the first controlled comparison of these components across Australian, Indian, and Northern British English. Our results reveal a robustness-generation gap: benchmarks are shaped by continual pretraining and SFT, while alignment visibly reshapes generation in ways benchmarks do not capture. Explicit variety-targeted adaptation produces output reliably recognised as dialectal and judged more dialectal than broad alignment, yet where human judgement was directly assessed, the method that most aggressively optimises the dialectal reward is not the one judged most dialectal. Independent linguistic analysis corroborates this reward-quality gap, most clearly on two of the three families. No single alignment method dominates, and closing the gap will require richer reward designs and continued investment in dialectal resources. We release all code, checkpoints, and preference datasets.

Jordan Painter, Dipankar Srirag, Adarsh Kappiyath +3

Institute for People-Centred AI, University of Surrey, Surrey, United Kingdom · University of New South Wales, Sydney, Australia

Language ModelingLLM AlignmentText GenerationLanguage Model PretrainingLanguage Model Post-TrainingLanguage Model Robustness+1
AskPDF