Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarks

Panagiota Kyriazi; Eleni Kasoura; Prokopis Prokopidis

cs.CLSep 28, 2026

Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarks

Authors: Panagiota Kyriazi, Eleni Kasoura, Prokopis Prokopidis

Organizations: Institute for Language and Speech Processing / Athena RC

Abstract

The rapid advancement of Large Language Models (LLMs) imposes a thorough evaluation of their linguistic and analytical capabilities as well as constraints, particularly for a language with limited benchmark coverage such as Greek. To address the limited availability of comprehensive benchmarks in this domain, we introduce Prot-Ex and Pan-Ex, two benchmarks consisting of questions from entrance exams for Greek Model and Experimental schools as well as the Panhellenic exams (the Greek national university entrance examinations). These benchmarks are employed to assess the performance of text-only LLMs-including the Greek-adapted KriKri-8B-Instruct, Llama-3.1-8B, Gemma-4-26B, and Qwen-3-32B-across diverse academic disciplines (Modern Greek, Mathematics, Physics, etc.) and task formats (closed, structured, and open-ended), including textualized visual context (i.e., image descriptions). Our findings indicate the localized KriKri-8B significantly outperforms its base model, successfully rivalling much larger LLMs in linguistically demanding humanities tasks. By leveraging an LLM-as-a-Judge methodology, we expose the inadequacy of traditional lexical metrics for evaluating complex reasoning. Crucially, we uncover a few-shot prompting paradox: while synthetic examples improve accuracy in closed-ended questions, they severely overload the context window of 8B models in structured tasks, causing significant performance degradation. Ultimately, this study suggests targeted linguistic adaptation offsets lower parameter counts in specialized domains, despite the fragility of smaller models to prompt verbosity.

Figures & tables

Subject	KriKri-8B			Llama-8B			Gemma-4-26B			Qwen3-32B
Subject	Closed	Structured	Open-ended	Closed	Structured	Open-ended	Closed	Structured	Open-ended	Closed	Structured	Open-ended
Greek Language	56.45	21.67	77.54	51.08	7.08	56.15	75.94	35.28	75.90	68.28	30.35	74.75
Mathematics	30.88	–	66.89	30.72	–	41.06	56.26	–	87.02	52.09	–	88.91
Physics	–	–	69.44	–	–	47.22	–	–	75.00	–	–	69.44
Religious Studies	77.78	–	–	72.22	–	–	78.89	–	–	80.00	–	–
Aggregate	47.10	21.67	69.93	43.89	7.08	45.48	67.90	35.28	83.46	62.25	30.35	84.21

Table 1: Zero-shot performance comparison (%) for the Prot-Ex benchmark. We use the LM-Eval framework for the closed and structured question formats, and Inspect AI for open-ended questions. Note: The aggregate scores for Structured tasks reflect only the Greek Language subject, as other subjects do not contain questions in this format.

Subject	KriKri-8B			Llama-8B			Gemma-4-26B			Qwen3-32B
Subject	Closed	Structured	Open-ended	Closed	Structured	Open-ended	Closed	Structured	Open-ended	Closed	Structured	Open-ended
Ancient Greek	51.11	21.04	57.06	51.11	20.00	35.31	68.89	46.04	65.53	75.56	28.75	64.62
Economics	54.76	–	56.71	40.48	–	24.32	78.57	–	76.78	59.52	–	74.59
Physics	48.65	–	51.37	41.89	–	21.23	66.22	–	70.55	55.41	–	74.66
History	63.33	–	68.98	53.33	–	44.44	50.00	–	66.20	53.33	–	63.33
Latin	73.33	12.31	51.68	66.67	6.15	32.72	66.67	15.38	64.03	73.33	15.38	66.14

Table 2: Zero-shot performance comparison (%) for the Pan-Ex benchmark. We use the LM-Eval framework for the closed and structured question formats, and Inspect AI for open-ended questions.

Subject	KriKri-8B		Llama-8B		Gemma-4-26B		Qwen3-32B
Subject	BERTScore	LLM-Judge	BERTScore	LLM-Judge	BERTScore	LLM-Judge	BERTScore	LLM-Judge
Greek Language	68.17	77.54	69.56	56.15	72.26	75.90	70.57	74.75
Mathematics	68.15	66.89	65.65	41.06	67.56	87.02	71.22	88.91
Physics	71.83	69.44	75.11	47.22	79.82	75.00	77.55	69.44
Aggregate	68.31	69.93	67.11	45.48	69.36	83.46	71.29	84.21

Table 3: Comparison of evaluation metrics (%) for zero-shot open-ended tasks in the Prot-Ex benchmark. The LLM-as-a-Judge approach utilizes Gemma-3-27B via the Inspect AI framework.

Subject	KriKri-8B		Llama-8B		Gemma-4-26B		Qwen3-32B
Subject	BERTScore	LLM-Judge	BERTScore	LLM-Judge	BERTScore	LLM-Judge	BERTScore	LLM-Judge
Ancient Greek	64.37	57.06	67.71	35.31	67.69	65.53	61.89	64.62
Economics	65.24	56.71	63.70	24.32	66.13	76.78	63.34	74.59
Physics	67.06	51.37	65.77	21.23	67.96	70.55	69.38	74.66
History	70.17	68.98	69.61	44.44	70.33	66.20	70.43	63.33
Latin	55.20	51.68	62.11	32.72	62.63	64.03	53.29	66.14

Table 4: Comparison of evaluation metrics (%) for zero-shot open-ended tasks in the Pan-Ex benchmark. The LLM-as-a-Judge approach utilizes Gemma-3-27B via the Inspect AI framework.

Model	Closed		Structured		Open-ended
Model	Zero-Shot	Few-Shot	Zero-Shot	Few-Shot	Zero-Shot	Few-Shot
KriKri-8B	47.10	52.20	21.67	8.61	69.93	68.03
Llama-8B	43.89	49.06	7.08	8.61	45.48	44.30
Gemma-4-26B	67.90	69.43	35.28	36.94	83.46	83.64
Qwen3-32B	62.25	62.81	30.35	32.85	84.21	84.05

Table 5: Aggregate impact of few-shot prompting on model performance (%) across task types in the Prot-Ex benchmark. Note: The aggregate scores for Structured tasks reflect only the Greek Language subject, as other subjects do not contain questions in this format.

Model	Closed		Structured		Open-ended
Model	Zero-Shot	Few-Shot	Zero-Shot	Few-Shot	Zero-Shot	Few-Shot
KriKri-8B	56.74	65.28	15.57	4.88	59.33	60.13
Llama-8B	49.48	56.22	11.89	0.41	30.81	32.49
Gemma-4-26B	72.54	78.76	28.70	30.77	74.01	70.34
Qwen3-32B	66.06	71.76	23.86	22.11	75.49	75.71

Table 6: Aggregate impact of few-shot prompting on model performance (%) across task types in the Pan-Ex benchmark.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 1 — Figure 1: Examples with LLM-generated image descriptions and transcriptions from the Pan-Ex (Physics 2022) and Prot-Ex (Mathematics 2025) benchmarks.

Prot-Ex
Subject	CEQ	SQ	OEQ	Total Items
Greek Language	735	57	61	853
Mathematics	599	0	151	750
Religious Studies	90	0	0	90
Physics	0	0	9	9
Total	1,424	57	221	1,702

Appendix

Table 7: Subject-wise distribution across Closed-Ended Questions (CEQ), Structured Questions (SQ), and Open-Ended Questions (OEQ) for the public Prot-Ex and Pan-Ex benchmarks.

Prot-Ex Private
Subject	CEQ	SQ	OEQ	Total Items
Greek Language	35	5	0	40
Mathematics	16	0	8	24
Total	51	5	8	64
Pan-Ex Private
Ancient Greek	7	0	23	30

Appendix

Table 8: Subject-wise distribution across Closed-Ended Questions (CEQ), Structured Questions (SQ), and Open-Ended Questions (OEQ) for the private Prot-Ex and Pan-Ex test sets.

Model	Setup	Closed	Struct.	Open
KriKri-8B	Zero-Shot	64.71	36.00	56.25
KriKri-8B	Few-Shot	60.78	56.00	84.38
Llama-8B	Zero-Shot	54.90	32.00	31.25
Llama-8B	Few-Shot	56.86	56.00	43.75
Gemma-4-26B	Zero-Shot	68.63	60.00	90.62
Gemma-4-26B	Few-Shot	64.71	80.00	96.88

Appendix

Table 9: Evaluation results for the private Prot-Ex (2019) test set. Scores represent aggregate accuracy for Closed and Structured question formats under the LM-Eval framework, alongside Inspect AI evaluations for Open-Ended questions.

Model	Setup	Closed	Struct.	Open
KriKri-8B	Zero-Shot	72.06	13.00	57.26
KriKri-8B	Few-Shot	79.41	0.00	56.18
Llama-8B	Zero-Shot	57.35	6.50	31.08
Llama-8B	Few-Shot	63.24	0.00	29.34
Gemma-4-26B	Zero-Shot	80.88	23.50	73.02
Gemma-4-26B	Few-Shot	79.41	26.00	68.68

Appendix

Table 10: Evaluation results for the private Pan-Ex (2026) test set. Scores represent aggregate accuracy for Closed and Structured question formats under the LM-Eval framework, alongside Inspect AI evaluations for Open-Ended questions.

Explore similar work

Maistros: A Greek Large Language Model Adapted Through Knowledge Distillation From Large Reasoning Models
May 3, 2026Nikolaos Giarelis, Charalampos Mastrokostas, Nikos KaracapilidisGreek Large Language Model Training
Industrial Management and Information Systems Lab, MEAD University of Patras, Rio Patras, Greece
KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding
Jul 19, 2026Timur Turatali, Aida Turdubaeva, Rustem Izmailov +2Multilingual Benchmark
1The Cramer Project · Institute of IT, Kyrgyz State Technical University named after I. Razzakov · School of Computer Science, University of Windsor +2
BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams
Jun 21, 2026João Guilherme Alves Santos, Giovana Kerche Bonás, Thiago Laitz +2Open-Ended Responses Portuguese
University of Campinas (UNICAMP), Campinas-SP, Brazil · Tropic AI, Campinas-SP, Brazil · Maritaca AI, Campinas-SP, Brazil

May 3, 2026, cs.CLJ/K move · Enter open · S save

Maistros: A Greek Large Language Model Adapted Through Knowledge Distillation From Large Reasoning Models

Nikolaos Giarelis, Charalampos Mastrokostas, Nikos Karacapilidis

Industrial Management and Information Systems Lab, MEAD University of Patras, Rio Patras, Greece

Greek Large Language Model Training Large Reasoning Models Natural Language Processing Knowledge Distillation

Open paper AskPDF arXiv

Large Language Models (LLMs) have substantially advanced the field of Natural Language Processing (NLP), achieving state-of-the-art performance across a wide range of tasks. These improvements have been attributed, in part, to their emerging reasoning capabilities, which are enabled by large-scale training and increased model capacity. However, existing LLMs can generate erroneous responses when addressing complex queries that fall outside their training distribution, due to limited internal knowledge or the need for multi-step reasoning. To address these limitations, recent work has introduced large reasoning models (LRMs), which incorporate explicit internal reasoning processes to improve response accuracy. Additionally, state-of-the-art LRMs often comprise hundreds of billions of parameters and require several seconds per inference, even on advanced multi-GPU systems. These characteristics limit their practicality for deployment in conventional computing environments. Meanwhile, NLP research on multilingual LLMs continues to prioritize high-resource languages. However, these models exhibit limited performance in under-resourced languages, primarily due to insufficient language- and culture-specific training data. In this paper, we focus on Modern Greek, for which only a limited number of question answering (QA) datasets have been proposed, most of which are intended for model evaluation. To address this research gap in Greek QA, we make the following contributions: (i) CulturaQA, a high-quality LRM-generated and human-curated dataset, for Greek LLM training and evaluation; (ii) a memory-efficient LLM evaluation framework adaptable to diverse languages and QA tasks; (iii) Maistros 8B, a state-of-the-art open-weights Greek LLM developed via knowledge distillation and fine-tuning on CulturaQA; and (iv) a comprehensive evaluation of nine LLMs across nine human-curated Greek QA datasets.

Abstract

Explore similar work

Maistros: A Greek Large Language Model Adapted Through Knowledge Distillation From Large Reasoning Models

KyrgyzLLM-Bench: Benchmarking Kyrgyz Language Understanding

BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams