cs.CLFeb 18, 2026
SaveEvaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmark
Organizations: Industrial Management and Information Systems Lab, MEAD, University of Patras, Rio Patras, Greece
Abstract
Recent advancements in Natural Language Processing and Deep Learning have enabled the development of Large Language Models (LLMs), which have significantly advanced the state-of-the-art across a wide range of tasks, including Question Answering (QA). Despite these advancements, research on LLMs has primarily targeted high-resourced languages (e.g., English), and only recently has attention shifted toward multilingual models. However, these models demonstrate a training data bias towards a small number of popular languages or rely on transfer learning from high- to under-resourced languages; this may lead to a misrepresentation of social, cultural, and historical aspects. To address this challenge, monolingual LLMs have been developed for under-resourced languages; however, their effectiveness remains less studied when compared to multilingual counterparts on language-specific tasks. In this study, we address this research gap in Greek QA by contributing: (i) DemosQA, a novel dataset, which is constructed using social media user questions and community-reviewed answers to better capture the Greek social and cultural zeitgeist; (ii) a memory-efficient LLM evaluation framework adaptable to diverse QA datasets and languages; and (iii) an extensive evaluation of 11 monolingual and multilingual LLMs on 6 human-curated Greek QA datasets using 3 different prompting strategies. We release our code and data to facilitate reproducibility.
Figures & tables
| Dataset | # Docs | Domain | Type | P5 | P25 | P50 | Mean | P75 | P95 | P99 |
|---|---|---|---|---|---|---|---|---|---|---|
| DemosQA | 600 | Social | Question | 26 | 53 | 84.5 | 103.04 | 132.25 | 243 | 347.7 |
| Answer | 11 | 31 | 54.5 | 80.22 | 105 | 222 | 362.04 | |||
| BELEBELE (Greek) | 900 | General | Question | 58 | 77 | 98 | 100.33 | 121 | 147 | 181 |
| Answer | 1 | 3 | 4 | 4.96 | 7 | 11 | 15.01 | |||
| Greek Medical MCQA | 432 | Medical | Question | 3 | 6 | 9 | 9.96 | 12 | 18 | 31 |
| Answer | 1 | 2 | 3.5 | 4.67 | 6 | 12 | 17.69 |
Table 1: Number of evaluation documents and domain per dataset, with a statistical overview of their question and answer word counts.
| LLM | Full Model Name | Greek Adapted | Open-Weights |
|---|---|---|---|
| GPT-4o mini | gpt-4o-mini-2024-07-18 | - | - |
| Gemma 2 9B | google/gemma-2-9b-it | - | D |
| Llama Krikri 8B | ilsp/Llama-Krikri-8B-Instruct | D | D |
| Meltemi 7B v1.5 | ilsp/Meltemi-7B-Instruct-v1.5 | D | D |
| Llama 3.1 8B | meta-llama/Llama-3.1-8B-Instruct | - | D |
| EuroLLM 9B v1 | utter-project/EuroLLM-9B-Instruct | - | D |
Table 2: Considered LLMs for the evaluation
| Acc (%) | DemosQA | Greek Truthful QA | BELEBELE (Greek) | Greek Medical MCQA | Greek ASEP MCQA | INCLUDE (Greek) |
|---|---|---|---|---|---|---|
| GPT-4o mini | 57.17 | 61.69 | 89.44 | 69.21 | 76.25 | 66.49 |
| Gemma 2 9B | 54.83 | 59.00 | 88.89 | 46.06 | 65.75 | 51.99 |
| Llama Krikri 8B | 57.17 | 37.82 | 77.33 | 44.44 | 58.92 | 49.82 |
| Meltemi 7B v1.5 | 42.67 | 36.35 | 64.78 | 36.81 | 57.50 | 43.84 |
| Llama 3.1 8B | 47.17 | 37.21 | 64.89 | 25.23 | 43.58 | 29.53 |
| EuroLLM 9B v1 | 41.67 | 35.01 | 52.89 | 39.81 | 53.17 | 38.41 |
Table 3: Experimental results for the instruction prompt. Acc (%) denotes the macro model accuracy. The best and second-best results are highlighted in bold and underline respectively.
| Acc (%) | DemosQA | Greek Truthful QA | BELEBELE (Greek) | Greek Medical MCQA | Greek ASEP MCQA | INCLUDE (Greek) |
|---|---|---|---|---|---|---|
| GPT-4o mini | 55.17 | 54.96 | 89.11 | 65.74 | 75.00 | 64.67 |
| Gemma 2 9B | 56.17 | 59.61 | 89.22 | 45.14 | 65.42 | 52.90 |
| Llama Krikri 8B | 56.33 | 53.98 | 81.89 | 46.99 | 65.92 | 53.08 |
| Meltemi 7B v1.5 | 50.17 | 37.45 | 61.11 | 31.71 | 58.08 | 41.12 |
| Llama 3.1 8B | 52.50 | 38.56 | 69.22 | 26.16 | 51.58 | 36.78 |
| EuroLLM 9B v1 | 41.83 | 35.86 | 62.78 | 38.43 | 56.25 | 44.38 |
Table 4: Experimental results for the role prompt. Acc (%) denotes the macro model accuracy. The best and second-best results are highlighted in bold and underline respectively.
| Acc (%) | DemosQA | Greek Truthful QA | BELEBELE (Greek) | Greek Medical MCQA | Greek ASEP MCQA | INCLUDE (Greek) |
|---|---|---|---|---|---|---|
| GPT-4o mini | 54.33 | 54.96 | 88.56 | 68.06 | 75.17 | 64.67 |
| Gemma 2 9B | 53.67 | 56.18 | 82.33 | 46.3 | 58.75 | 50.36 |
| Llama Krikri 8B | 56.00 | 54.22 | 82.33 | 46.06 | 66.25 | 51.81 |
| Meltemi 7B v1.5 | 48.17 | 32.80 | 57.67 | 27.55 | 49.83 | 34.96 |
| Llama 3.1 8B | 55.00 | 40.88 | 74.89 | 25.46 | 52.17 | 39.31 |
| EuroLLM 9B v1 | 40.17 | 36.47 | 67.67 | 41.67 | 56.83 | 48.91 |
Table 5: Experimental results for the CoT prompt. Acc (%) denotes the macro model accuracy. The best and second-best results are highlighted in bold and underline respectively.
| Acc (%) | DemosQA | Greek Truthful QA | BELEBELE (Greek) | Greek Medical MCQA | Greek ASEP MCQA | INCLUDE (Greek) |
|---|---|---|---|---|---|---|
| GPT-4o mini | 55.56 | 57.20 | 89.04 | 67.67 | 75.47 | 65.28 |
| Gemma 2 9B | 54.89 | 58.26 | 86.81 | 45.83 | 63.31 | 51.75 |
| Llama Krikri 8B | 56.50 | 48.67 | 80.52 | 45.83 | 63.70 | 51.57 |
| Meltemi 7B v1.5 | 47.00 | 35.53 | 61.19 | 32.02 | 55.14 | 39.97 |
| Llama 3.1 8B | 51.56 | 38.88 | 69.67 | 25.62 | 49.11 | 35.21 |
| EuroLLM 9B v1 | 41.22 | 35.78 | 61.11 | 39.97 | 55.42 | 43.90 |
Table 6: Experimental results across all prompts (mean accuracy). Acc (%) denotes the macro model accuracy. The best and second-best results are highlighted in bold and underline respectively.
| Prompt Type | Greek | English (Translated) |
|---|---|---|
| Instruction | \acctonos \acctonos \acctonos ς ς \acctonos \acctonos ς \acctonos ς µ \acctonos µ \acctonos µµ (, , \acctonos ). | Select the best answer to the following question and answer only with the letter (A, B, C or D). |
| Role | \acctonos ς \acctonos ςς \acctonos µ \acctonos \acctonos \acctonos ςς. \acctonos µ \acctonos \acctonos \acctonos ς \acctonos \acctonos ςµ \acctonos ς ς \acctonos \acctonos ς. \acctonos \acctonos µ µ \acctonos \acctonos ς. | You are a language model for the Greek language. Select only the best answer from the available answers to the following question. Write the text of the selected answer. |
| Chain-of-Thought (CoT) | \acctonos ς \acctonos ςς \acctonos µ \acctonos \acctonos \acctonos ςς. \acctonos µ \acctonos \acctonos \acctonos ς \acctonos \acctonos ςµ \acctonos ς ς \acctonos \acctonos ς. \acctonos \acctonos µ µ \acctonos \acctonos ς. \acctonos ς \acctonos \acctonos µ \acctonos µ. | You are a language model for the Greek language. Select only the best answer from the available answers to the following question. Write the text of the selected answer. Please think step-by-step. |
Table 7: Multiple-choice QA model prompts used in our study alongside their English translations.
| Question | Disregarded Answer | Reason | Selected Answer |
|---|---|---|---|
| \acctonos ς µ \acctonos ; µ \acctonos \acctonos \acctonos \acctonos \acctonos \acctonos \acctonos µ \acctonos \acctonos µ \acctonos ς \acctonos \acctonos ςς ς \acctonos , µ \acctonos ς ςµµ \acctonos ς µ \acctonos ; Can I go to the Olympics? Objectively, if a common person chose a sport that doesn’t require terrible physical condition e.g. shooting, could they participate? | \acctonos ς \acctonos \acctonos \acctonos ς break dancer. µ \acctonos ς. Go to Australia and say you are a break dancer. You’ll go immediately. | Sarcasm | \acctonos \acctonos , \acctonos \acctonos ς \acctonos \acctonos ς \acctonos ς ς µ \acctonos \acctonos . It is not impossible, but it requires hard training and passing qualifying matches. |
| \acctonos µ \acctonos ςµ \acctonos µς \acctonos \acctonos ς ς \acctonos \acctonos ς \acctonos ; \acctonos \acctonos ς \acctonos µ µ \acctonos ς ς ς \acctonos \acctonos \acctonos ς \acctonos µ \acctonos ς Guys any advice on what home workout to do to lose belly fat? If I’m going to waste my time at home at least let me do good for my health | \acctonos · \acctonos . For belly? Diet. | Unhelpful Comment | \acctonos ς µ \acctonos µ \acctonos (…) \acctonos , \acctonos ς ς \acctonos \acctonos , µς \acctonos ς \acctonos \acctonos ς \acctonos \acctonos \acctonos (…) Reduce your daily calories (…) Now, to answer what you ask, the workout that helps you lose fat is what they call (…) |
| \acctonos \acctonos µµ ς \acctonos ς \acctonos \acctonos µς \acctonos ς \acctonos µ \acctonos ς \acctonos ; ς \acctonos ς \acctonos , µ \acctonos ς \acctonos \acctonos µ µ \acctonos . (…) Is it legal to deduct money from your salary for misconduct during work hours? Good evening everyone, to make a long story short yesterday at work I made a big blunder. (…) | [deleted] | Deleted Comment | \acctonos . \acctonos ς \acctonos \acctonos \acctonos ς \acctonos ς. \acctonos µ \acctonos µ \acctonos \acctonos \acctonos ς µ \acctonos \acctonos ς. (…) No. If they want, let them sue you or fire you. Mistakes are part of the job and the risk the business undertakes. (…) |
| \acctonos \acctonos µµ ς \acctonos ς \acctonos \acctonos µς \acctonos ς \acctonos µ \acctonos ς \acctonos ; ς \acctonos ς \acctonos , µ \acctonos ς \acctonos \acctonos µ µ \acctonos . (…) Is it legal to deduct money from your salary for misconduct during work hours? Good evening everyone, to make a long story short yesterday at work I made a big blunder. (…) | \acctonos , * \acctonos \acctonos , µ \acctonos \acctonos . \acctonos µ \acctonos \acctonos \acctonos \acctonos \acctonos \acctonos Come on, say what bullsh*t you did, you intrigued us. Besides, only those who do nothing make no mistakes | Offensive Language | \acctonos . \acctonos ς \acctonos \acctonos \acctonos ς \acctonos ς. \acctonos µ \acctonos µ \acctonos \acctonos \acctonos ς µ \acctonos \acctonos ς. (…) No. If they want, let them sue you or fire you. Mistakes are part of the job and the risk the business undertakes. (…) |
Table 8: Examples of the manual curation process (English translations are provided below the original Greek text).
More in this topic
Question Answering
Question answering studies how AI systems interpret questions and produce accurate, evidence-grounded answers across text, dialogue, and multimodal data.
See all 260 papers