cs.CLNov 6, 2026

Compact Greek Language Models from Scratch: Pretraining 500M- and 1B-Parameter Decoder-Only Transformers

Authors: Konstantinos Perifanos, Anne Laure Poite, Giannis Tsilikas, Ilias Katis, Stavros Deligiannidis, Dionysis Goutsos

Organizations: National and Kapodistrian University of Athens (NKUA)

Abstract

We present two compact decoder-only language models trained from scratch exclusively on Greek text: LexisLM-1B (1.09B parameters) and LexisLM-500M (502M parameters). Both use a dedicated 32k byte-level BPE tokenizer and a 22B-token pretraining corpus drawn primarily from Greek Wikipedia and FineWeb2. Both models complete the 22B-token schedule, with best held-out perplexities of 9.77 for LexisLM-1B and 11.31 for LexisLM-500M. Evaluation of both models on Greek-translated ARC, MMLU, and TruthfulQA shows above-chance performance on ARC-Easy, while ARC-Challenge and MMLU remain near chance. We also perform supervised fine-tuning of both models on Greek instruction data, primarily EuroAlpaca, reaching held-out instruction perplexities of 7.12 for LexisLM-500M and 6.37 for LexisLM-1B. The report describes the tokenizer, data preparation, training, and evaluation methods, providing a foundation for further work on compact Greek language models.

Figures & tables

Explore similar work

CardsList
  1. Maistros: A Greek Large Language Model Adapted Through Knowledge Distillation From Large Reasoning Models

    May 3, 2026Nikolaos Giarelis, Charalampos Mastrokostas, Nikos KaracapilidisLLM Fine-TuningQuestion Answering

  2. MORFES: A Benchmark for Productive Inflectional Competence in Modern Greek

    Jul 30, 2026Ioakeim Perros, Cleopatra Papadopoulou, Ayoub Kirouane +1Multilingual Language Model EvaluationLLM Evaluation

  3. Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmark

    Feb 18, 2026Charalampos Mastrokostas, Nikolaos Giarelis, Nikos KaracapilidisQuestion AnsweringMultilingual Language Model Evaluation