Meenzerisch, the dialect spoken in the German city of Mainz, is also the traditional language of the Mainz carnival, a yearly celebration well known throughout Germany. However, Meenzerisch is on the verge of dying out-a fate it shares with many other German dialects. Natural language processing (NLP) has the potential to help with the preservation and revival efforts of languages and dialects. However, so far no NLP research has looked at Meenzerisch. This work presents the first research in the field of NLP that is explicitly focused on the dialect of Mainz. We introduce a digital dictionary-an NLP-ready dataset derived from an existing resource (Schramm, 1966)-to support researchers in modeling and benchmarking the language. It contains 2,351 words in the dialect paired with their meanings described in Standard German. We then use this dataset to answer the following research questions: (1) Can state-of-the-art large language models (LLMs) generate definitions for dialect words? (2) Can LLMs generate words in Meenzerisch, given their definitions? Our experiments show that LLMs can do neither: the best model for definitions reaches only 6.27% accuracy and the best word generation model's accuracy is 1.51%. We then conduct two additional experiments in order to see if accuracy is improved by few-shot learning and by extracting rules from the training set, which are then passed to the LLM. While those approaches are able to improve the results, accuracy remains below 10%. This highlights that additional resources and an intensification of research efforts focused on German dialects are desperately needed.
Figures & tables
Figure 1: Dataset Creation Pipeline. Overview of the semi-automatic five-step process used to create the Mainz Dialect Dataset.
Category
Value
Dataset Size
Total extracted pairs
2,471
Extracted pairs after clean-up
2,351
Word Length
Median length (characters)
8
Definition Length
Table 1: Overview of the Mainz Dialect Dataset. Summary of the extracted and cleaned dataset.
Word in Meenzerisch
Standard German Definition
Aaweiderworschd
Salzgurke (English: pickled cucumber )
Bitzelwasser
Mineralwasser mit Kohlensäuregehalt (English: Carbonated mineral water )
Mauldabbezierer
scherzhaftes Lob für einen guten Wein […] ( English: playful compliment for a good wine )
Table 2: Examples from Our Dataset. Samples illustrating the mapping between dialect words and their corresponding Standard German definitions.
Model
Dialect of Mainz
English
Dev Set
Test Set
Test Set
Large Models
GPT-OSS 120B (Thinking)
5.60%
4.92%
91.19%
Llama-3.3 70B
8.80%
6.27%
91.69%
Leo-HessianAI 70B
4.40%
4.81%
81.88%
Qwen-2.5 72B
4.80%
5.35%
90.19%
Table 3: Accuracy of Definition Generation. Development and test set accuracy of LLMs for the Mainz dialect, and test set accuracy for the English baseline.
Model
Dialect of Mainz
English
Dev Set
Test Set
Test Set
Large Models
GPT-OSS 120B (Thinking)
2.40%
1.51%
82.50%
Llama-3.3 70B
0.80%
1.03%
69.00%
Leo-HessianAI 70B
0.80%
0.92%
47.10%
Qwen-2.5 72B
0.80%
0.43%
66.90%
Table 4: Accuracy of Word Generation given Explanation. Development and test set accuracy of LLMs in generating the correct word for the Mainz dialect, and test set accuracy for the English baseline.
Shots
Definition
Word
Generation
Generation
Random Selection
1
6.00%
1.60%
5
9.60%
2.00%
10
11.20%
1.60%
25
7.60%
1.20%
Table 5: Different Numbers of Few-shot Examples on the Development Set. Comparison of Llama-3.3 70B Instruct across different numbers of in-context examples.
Selection Method
Accuracy
Definition Generation
Baseline: Zero-Shot
6.27%
Random Selection ( k=10 )
7.67%
Edit Distance-Based Selection ( k=50 )
9.24%*
Word Generation
Baseline: Zero-Shot
1.03%
Table 6: Few-shot Results for Llama-3.3 70B. Asterisks (*) mark significant improvements (McNemar’s test, p<.05 ); bolding highlights the best significant score (or multiple scores where differences are not significant).
Figure 2: Part of our File with Automatically Extracted Rules. The final rules are fed into Llama-3.3 70B for definition and word generation.
Selection Method
Accuracy
Definition Generation
Baseline: Zero-Shot
6.27%
Injected Extracted Rules
8.37%*
Word Generation
Baseline: Zero-Shot
1.03%
Injected Extracted Rules
0.76%
Table 7: Results for Automatically Extracted Rules Injected into Llama-3.3 70B. Asterisks (*) mark significant improvements (McNemar’s test, p<.05 ); bolding highlights the best significant score (or multiple scores where differences are not significant).
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 3: Prompt used for extracting dictionary definitions. English translation: “You receive an unstructured or faulty definition of the word ’{Word}’ from an old dictionary. Your task is to extract only the actual meaning of the word from the text, without comments, reformulations, or explanations. Rules: (1) If multiple meanings are present, number them consecutively (1., 2., 3., …) and separate them by line breaks. (2) If the definition refers exclusively to another word, output ’[SEE] <word>’. (3) Do not alter the original text; output only the relevant excerpt. (4) If there is no definition in the text, output ’No definition’.”
Figure 4: Prompt used for cleaning dictionary definitions English translation: “You are a careful linguistic assistant. Your task is to clean dictionary definitions without changing their meaning. Remove only unnecessary special characters such as hyphens, double spaces, or similar OCR artifacts. Leave references in the form [SEE] unchanged and preserve numbering of definitions. Here is the definition for the word ’{Word}’: {cleaned}. Clean the definition according to the instructions. If it is already clean, return it unchanged. Output only the cleaned definition.”
Figure 5: Prompt used for generating dictionary definitions. English translation: “You are a precise and reliable linguistic assistant. Your task is to create dictionary definitions. Provide exclusively a single, short, and concise meaning of the requested word. Use the meaning commonly used in the Mainzer dialect, but formulate the definition in Standard German. Create exactly one short dictionary definition for the word ’{Word}’. Use the meaning commonly used in the Mainzer dialect, without providing additional information, examples, or alternative meanings.”
Figure 6: Prompt used for evaluating semantic equivalence. English translation: “You are a precise and neutral evaluator of semantic similarity between dictionary definitions. Your task is to assess whether two definitions express the same meaning, even if they are phrased differently. Consider synonyms, paraphrases, or stylistic differences. Respond exclusively with ’SAME’ if the two definitions convey the same content, or with ’DIFFERENT’ if their meaning differs substantially. Dictionary entries may contain multiple definitions. If at least one of these definitions matches the one generated by the LLM, the result counts as ’SAME’. Do not provide any additional explanations or justifications. Compare the following two definitions. Assess whether they are semantically equivalent. Definition A (LLM): ’{response_generator}’. Definition B (Dictionary): ’{definition_final}’.”
Figure 7: Prompt used for generating a Mainzer dialect word . English translation: “You are a precise and reliable linguistic assistant. Your task is to find the appropriate word in the Mainzer dialect for a given meaning. Provide exclusively a single word that expresses this meaning in the Mainzer dialect. Do not provide explanations, translations, or additional information. Find the Mainzer dialect word that expresses the following meaning: ’{definition_final_generation}’. Respond only with the dialect word.”
Figure 8: Two-turn prompting strategy for automatic rule extraction. In the first turn, the model generates Standard German candidates. In the second turn, it induces mapping rules between dialect and Standard German forms.
Figure 9: Automatic extracted rules for mapping Mainzer dialect words to Standard German forms. This is generated by DeepSeek-R1.
Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Malostranské náměstí 25, 118 00 Praha, Czech Republic