Paper ID: 2307.14134

Developing and Evaluating Tiny to Medium-Sized Turkish BERT Models

Himmet Toprak Kesgin, Muzaffer Kaan Yuce, Mehmet Fatih Amasyali

This study introduces and evaluates tiny, mini, small, and medium-sized uncased Turkish BERT models, aiming to bridge the research gap in less-resourced languages. We trained these models on a diverse dataset encompassing over 75GB of text from multiple sources and tested them on several tasks, including mask prediction, sentiment analysis, news classification, and, zero-shot classification. Despite their smaller size, our models exhibited robust performance, including zero-shot task, while ensuring computational efficiency and faster execution times. Our findings provide valuable insights into the development and application of smaller language models, especially in the context of the Turkish language.

Submitted: Jul 26, 2023