MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum

Furkan Yilmaz; Habibe Aleyna Tasdemir; Muhammed Faruk Gozay

cs.CLAug 26, 2026

MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM-to-MLM Curriculum

Authors: Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk Gozay

Abstract

Turkish encoder models have adopted modern architectures while leaving the pretraining objective fixed at masked language modelling. This paper introduces MoganBert-TR, a 149M-parameter Turkish encoder foundation model trained from scratch on a language-specifically filtered corpus, together with an embedding model derived from it (MoganBert-Embed). MoganBert-TR is trained over 237.3B tokens with a two-stage CLM-to-MLM curriculum: causal language modelling first, masked language modelling for the remainder, with the transition made inside the stable phase of a WSD schedule. In a controlled ablation under an equal step budget, this design outperforms pure MLM by 2.7-3.7x on Turkish MS MARCO retrieval; the measured mechanism is embedding geometry, where a single direction absorbs 28.1% of the variance under pure MLM against 11.9% under the curriculum. Long-context extension and learning-rate decay are then split into two branches after a shared prefix: running the final portion of decay at 1024 context improves the TrGLUE average by 0.49 +/- 0.26 points across five paired seeds (p = 0.013) and beats a model-soup alternative by 0.75 points at ~4.3% additional cost. MoganBert-TR attains 78.41 on TrGLUE, the best among the Turkish ModernBERT models compared, and 77.73 on TabiBench, where it leads two of the eight categories with the largest margin on code retrieval (+3.62 points over TabiBERT). MoganBert-Embed, produced through teacher distillation and multi-signal contrastive fine-tuning, ranks first among student models on the MTEB(Turkish) overall average with 68.30 and reaches 99.5% of its 7.57B-parameter teacher's score with a 51x smaller backbone. The accompanying 50,048-token tokenizer outperforms all compared Turkish tokenizers on compression and fertility across two independent test sets. Weights, tokenizer, embedding model and evaluation code: https://huggingface.co/moganai

Figures & tables

Panel of Figure 1 — Figure 1: The data pipeline: from FineWeb2 and Common Crawl input through a single-pass processing chain to the deduplicated output. Quality classification (step 6) is not a separate pass but part of the same process.

Tokenizer	Vocab	Fertility $\downarrow$	Chars/Token $\uparrow$	TR% $\uparrow$	Pure% $\uparrow$
Mogan-64K	64,000	1.5268	5.0245	84.71	40.83
Mogan-50K	50,000	1.5682	4.8918	84.42	42.67
Mürşit	59,008	1.6326	4.6989	77.00	35.81
MBERT-tr	32,000	1.7037	4.5029	78.73	35.65
BERTurk	32,000	1.8008	4.2600	81.06	43.22
TabiBERT	50,176	1.8501	4.1464	59.84	30.46

Table 1: Tokenizer efficiency on TR-MMLU.

Figure 2: Fertility ( $x$ , lower is better) vs Pure% ( $y$ , higher is better); bubble size indicates vocabulary size.

Test set	Tokenizer	Vocab	Chars/Token $\uparrow$	Fertility $\downarrow$	UNK
Wikipedia	Mogan-64K	64,000	4.705	1.640	0.00%
Wikipedia	BERTurk	32,000	4.391	1.757	0.04%
Wikipedia	Kumru-2B	50,176	4.010	1.924	0.00%
Wikipedia	TabiBERT	50,176	4.010	1.924	0.00%
FLORES-200	Mogan-64K	64,000	5.277	1.460	0.00%
FLORES-200	BERTurk	32,000	4.932	1.563	0.002%

Table 2: Tokenizer efficiency on independent test sets.

Tokenizer	Code compression	Lossless roundtrip
Mogan (final)	2.790	100%
Mürşit	2.454	96%
TabiBERT	2.196	49%
BERTurk	2.184	0%

Table 3: Tokenizer comparison on a code corpus (18 languages, 540 files). Lossless roundtrip is the rate at which a model can reproduce its own output.

Panel of Figure 3 — Figure 3: Decision flow for the CLM $\rightarrow$ MLM curriculum. The first ablation (§ 5.3 ) selects the training objective; the masking ablation was designed but not run on the basis of corpus statistics (dashed branch, § 5.4 ). Training hyperparameters are given in Table 5 .

Linear probe (accuracy %)
Metric	MLM-only	CLM25+MLM75	$\Delta$
POS (BOUN), 15 classes	87.62	87.42	$-$ 0.20
News category, 5 classes	97.39	97.39	0.00
NLI, 3 classes	48.53	50.22	+1.69
Product review, 2 classes	80.75	81.80	+1.05
Average	78.57	79.21	+0.64

Table 4: Ablation results. The probes are linear classifiers trained on frozen representations; MS MARCO was run without contrastive training, on the raw pretraining representations. In the embedding geometry block, lower is better on the first two rows.

Item	Value
Parameter count	149.4M (22 layers, hidden 768, local_attention =128)
Vocabulary	50,048
Hardware	4 $\times$ H100 (DDP)
Attention	FlashAttention-3 [ 22 ] , varlen ( cu_seqlens )
Precision	bf16
Optimizer	StableAdamW, betas (0.90, 0.98), eps 1e-6, weight decay 1e-5

Table 5: Training configuration.

Phase	Tokens	Steps	Ctx	Mask	LR
CLM	36.0B	1 – 17,167	1024	—	warmup + stable
MLM	180.7B	17,168 – 103,342	1024	30%	8e-4 stable
Annealing — shared prefix
Anneal (shared)	10.5B	1 – 5,000	8192	10%	decay, upper half
Annealing — two branches
Branch A: anneal	10.1B	5,001 – 9,810	8192	10%	decay, lower half

Table 6: Phase plan. 237.3B tokens in total for each branch.

Group	Target	Realised	$\Delta$
CommonCrawl	52.6%	52.60%	0.00
High-quality Turkish	20.7%	20.83%	+0.13
fineweb_edu_eng	13.9%	13.84%	$-$ 0.06
coding_gold	10.0%	9.97%	$-$ 0.03
eng_math_gold	2.8%	2.77%	$-$ 0.03

Table 7: Realised group ratios in the main run. Within-step deviation is 0.055 points.

Candidate	TrGLUE AVG	Rank (6 models)
anneal1k (decay at 1024)	78.41 $\pm$ 0.32	2
anneal (decay at 8192)	77.92 $\pm$ 0.48	3
soup	77.66 $\pm$ 0.37	5

Table 8: TrGLUE average of the three candidates (five official seeds).

Task	BERTurk	MBERT-TR	TabiBERT	Mogan-anneal	Mogan-anneal1k
cola	41.71 $\pm$ 1.87	26.62 $\pm$ 2.36	31.89 $\pm$ 1.76	35.92 $\pm$ 2.29	37.63 $\pm$ 1.97
sst2	87.62 $\pm$ 0.19	86.47 $\pm$ 0.44	85.72 $\pm$ 0.27	85.60 $\pm$ 0.13	85.83 $\pm$ 0.11
mrpc	71.00 $\pm$ 0.94	70.45 $\pm$ 0.63	68.91 $\pm$ 0.77	68.39 $\pm$ 1.55	69.18 $\pm$ 0.56
stsb	72.08 $\pm$ 0.60	70.51 $\pm$ 0.98	69.18 $\pm$ 5.38	68.96 $\pm$ 1.28	69.59 $\pm$ 1.42
qqp	95.04 $\pm$ 0.07	95.17 $\pm$ 0.03	95.08 $\pm$ 0.08	94.46 $\pm$ 0.06	94.54 $\pm$ 0.08
mnli	89.11 $\pm$ 0.11	88.13 $\pm$ 0.30	88.00 $\pm$ 0.21	88.10 $\pm$ 0.14	88.24 $\pm$ 0.19

Table 9: TrGLUE eight tasks, average

\pm

standard deviation over the five official seeds. Bold: best in row.

Model	Text	Token	STS	NLI	QA	Acad.	IR	Code	Total
MoganBert-TR	83.71	91.20	85.43	84.14	68.37	72.62	75.78	60.57	77.73
ModernBERT-TR	85.21	90.78	86.08	84.74	67.03	71.83	77.51	60.16	77.92
TabiBERT	83.44	93.42	84.74	84.51	69.71	72.44	75.44	56.95	77.58
BERTurk	83.42	93.67	85.33	84.33	60.16	71.40	74.84	54.54	75.96
YTU-Cosmos-BERT [ 10 ]	84.25	93.60	84.68	84.16	31.50	71.78	74.29	53.80	72.26
TurkishBERTweet [ 16 ]	79.71	92.02	75.86	79.10	38.13	63.12	68.40	43.49	67.48

Table 10: TabiBench category scores. Metrics by category: macro-F1 (text classification, NLI, academic), micro-F1 at word level (token classification), Pearson (STS), F1 (QA), NDCG@10 (both retrieval categories). Bold: best in column among monolingual Turkish models. Reference scores for the four models below the rule are taken from Türker et al. [26] , Table 4; the ModernBERT-TR row is read from the figures of its release article [ 1 ] , which does not publish a numeric table. mmBERT is a 307M-parameter multilingual model trained on 3T tokens and is listed as a reference point, not as a comparable system.

Model	cos_raw	eff_rank	top1_var
MoganBert-TR	0.9841	117.9	0.0824
TabiBERT	0.9405	125.1	0.0682
ModernBERT-TR	0.9234	125.5	0.0924
BERTurk	0.7848	126.8	0.0632

Table 11: Anisotropy comparison. cos_raw is the mean cosine similarity of random sentence pairs; proximity to 1 means all vectors are collected within a narrow cone.

Task	Metric	theta 160k	theta 10k	Difference
stsb	pearson	0.6933	0.6893	$-$ 0.0040
stsb	spearman	0.6710	0.6698	$-$ 0.0012
mrpc	accuracy	0.7150	0.7060	$-$ 0.0090
mrpc	f1	0.6570	0.6397	$-$ 0.0173

Table 12: Effect of changing RoPE theta at inference time.

Metric	Start (MLM)	After Phase 1
cos_raw (anisotropy)	0.9841	0.0851
eff_rank	117.9	157.2
top1_var	0.0824	0.0355
Zero-shot IR (10 sets, NDCG@10)	0.2361	0.5927

Table 13: Effect of Phase 1 on geometry.

Version	OVERALL	Retr.	Pair	STS	Class.	Clust.	Bitext
phase1	63.53	56.13	56.76	73.95	70.61	64.35	96.55
phase2_v2	64.56	57.95	57.56	70.59	71.34	64.18	98.72
phase2_v4	66.88	58.22	70.89	80.02	73.17	62.42	95.68
phase2_v3	66.97	57.77	71.39	80.94	73.78	62.32	95.66

Table 14: Phase 2 version results, MTEB(Turkish). v5 was not evaluated individually.

Soup	Components	Weights	Phase 1 share	OVERALL
f34-heavy	phase1+v3+v4	0.4 / 1 / 1	0.167	68.30
f34-06	phase1+v3+v4	0.6 / 1 / 1	0.231	68.22
f-v3	phase1+v3	0.5 / 1	0.333	68.22
f34-05	phase1+v3+v4	0.5 / 1 / 1	0.200	68.19
345 (no Phase 1)	v3+v4+v5	equal	0.000	67.08
fisher-f34	phase1+v3+v4	0.4 / 1 / 1	—	67.19

Table 15: Selected soups. “Phase 1 share” = the ratio of the Phase 1 weight to the total weight.

Model	Param	Ctx	OVERALL	Retr.	Pair	STS	Class.	Clust.	Bitext
MoganBert-Embed ∗	149M	8192	68.30	59.58	69.32	79.49	75.36	66.46	97.24
ModernBERT-TR	149M	8192	68.13	59.41	69.21	77.61	76.51	63.20	94.04
mE5-large-instruct	560M	512 †	67.47	61.05	63.03	81.23	73.70	61.87	98.99
BGE-M3	568M	8192	67.36	60.26	68.72	79.60	72.79	60.80	98.99
Mursit-Large-TR	404M	2048	62.00	55.70	55.19	74.60	68.66	61.53	86.72
trmteb-tr-embed	111M	512 †	59.71	52.54	56.81	74.96	70.62	63.17	37.75 ‡

Table 16: MTEB(Turkish) category averages. “Ctx” = the model’s position limit. Bold: best in column among student models. The last row is the teacher model and is not included in the ranking.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 4 — Figure 4: The pre-tokenizer stage: separate but parallel normalization chains for natural language and code data. Shaded boxes occur in the code branch only.

Text classification
Task	Test size	Mogan	Tabi	BERTurk	YTU	TRBw	mmBERT	MB-TR
NewsCat	250	97.22	95.20	95.60	97.21	91.98	94.80	?
BilTweetNews	150	53.32	50.11	57.87	53.39	53.16	49.06	?
GenderHateSpeech	2,000	66.57	69.01	68.25	71.02	68.58	66.45	?
ProductReviews	35,275	84.72	84.32	84.30	85.04	80.37	83.51	?
Weighted avg		83.71	83.44	83.42	84.25	79.71	82.54	85.21

Appendix

Table 17: TabiBench, all 28 tasks and the eight category averages. Scores for TabiBERT, BERTurk, YTU-Cosmos-BERT, TurkishBERTweet and mmBERT are taken from Türker et al. [26] , Tables 6–13. Bold: best in row among monolingual Turkish models; mmBERT is excluded from the ranking, following the source work. MB-TR denotes ModernBERT-TR, which publishes TabiBench results only at category level [ 1 ] ; its per-task scores are marked “?”. Category averages are weighted by test-set size; the overall score is the unweighted macro average of the eight categories. Metrics are as listed in Table 10 .

Retrieval
Task	Mogan	MBERT-TR	mE5	BGE-M3	Mursit	Qwen3 (teach.)
ArguAnaTR	51.99	49.93	49.07	50.43	45.70	61.84
CQADupstackGamingTR	54.96	56.36	61.29	58.41	53.60	67.08
FiQA2018TR	47.85	46.20	47.82	45.00	40.65	53.35
MSMarcoTRRetrieval	58.09	57.87	58.35	57.93	58.95	58.34
NFCorpusTR	9.96	9.45	9.62	9.41	9.50	12.00

Appendix

Table 18: MTEB(Turkish), all 26 tasks. Bold: best in row among student models. The rightmost column is the teacher model.

Explore similar work

Dec 28, 2025

TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish

The introduction of BERT established encoder-only transformer models as a foundational paradigm in natural language processing. Encoder-only models remain the standard tool for classification, tagging and retrieval, where contextual representations and low inference cost matter more than text generation, yet Turkish lacks a monolingual encoder trained from scratch with the advances consolidated in ModernBERT (rotary positional embeddings, FlashAttention, refined normalization). We introduce TabiBERT, a monolingual Turkish encoder based on the ModernBERT architecture, pretrained from scratch for one trillion tokens sampled from an 86.58B-token multi-domain corpus of web text (72%), scientific publications (19%), source code (6%) and mathematical content (0.3%). The model supports a context length of 8,192 tokens, sixteen times that of existing Turkish BERT models, and inherits the ModernBERT architecture's efficiency at long context. For rigorous and reproducible evaluation we introduce TabiBench, a benchmark of 27 datasets across eight task categories with standardized splits and evaluation protocols, summarized as a GLUE-style macro-average on a 0-100 scale. TabiBERT leads the Turkish models in five of eight categories and BERTurk, the previous best, in six of eight; the gains concentrate on question answering (+9.55 F1) and code retrieval (+2.41 NDCG@10), while the four short-text categories are near saturation. Its average of 77.28 exceeds BERTurk's 75.66; the multilingual mmBERT reaches 78.98 with twice the parameters and three times the training tokens, at 41% more tokens per Turkish input. We release model weights, training configurations and evaluation code as a transparent and reproducible foundation for future Turkish encoder research.

Melikşah Türker, A. Ebrar Kızıloğlu, Onur Güngör +1

Department of Computer Engineering, Boğaziçi University, Türkiye · VNGRS, Istanbul, Türkiye · Technical University of Munich (TUM), Munich, Germany

Language Model Pretraining Long-Context Language Modeling Language Modeling

Ask PDF

Aug 26, 2026

MoganColBERT-TR: A Late-Interaction Multi-Vector Retrieval Model for Turkish

We previously reported a ModernBERT encoder trained from scratch for Turkish (MoganBERT-TR) and a single-vector embedding model built on top of it (MoganBERT-embed). This work introduces the third model in that lineage: MoganColBERT-TR, a multi-vector retrieval model that, instead of compressing a query or a document into a single vector, represents it at the token level through a 768->128 projection and scores it with MaxSim late interaction. The model is not trained from scratch: the embedding model's encoder is taken as the starting point and adapted to the ColBERT objective with a single-epoch distillation phase. Training data is produced from two sources - title-to-passage pairs carved out of our own pretraining corpus in the character domain and at sentence boundaries, and two Turkish question-based retrieval sets - and is distilled from the soft scores of a cross-encoder teacher (bge-reranker-v2-m3) over one positive and seven mined negatives. We show that in hard negative mining, rank-based skipping alone is insufficient and must be combined with a group mask and a cosine ceiling. Evaluation is carried out with the official pipeline of TurkColBERT, a benchmark built for Turkish late-interaction retrieval (PLAID index, exact MaxSim), on five Turkish BEIR datasets; none of them appears in our training pool, so all five results are clean zero-shot. With 148.9M parameters, MoganColBERT-TR reaches an overall score of 37.36 (35.53 nDCG@100, 31.81 nDCG@10) averaged over the five datasets and finishes second among the five models compared: it outperforms the twice-as-large ColmmBERT-base-TR on four of five datasets and by +3.05 overall, and the benchmark's largest model by +12.30. The gap to the leading model (mLateOn) is concentrated on ArguAna-TR, the dataset with by far the longest queries.

Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk Gozay

Information Retrieval Late Interaction Retrieval Multi-Vector Retrieval

Ask PDF

Sep 18, 2026

ufakzeka-1: Building and Evaluating a 151M-Parameter Turkish Language Model from Scratch

We describe ufakzeka-1, a 151M-parameter (182M with embeddings) decoder-only Turkish language model pretrained from scratch on 13.5B tokens of openly licensed text and instruction-tuned for chat, at a total cost of about $286 in cloud GPU, API and notebook time. The contribution is not the model's capability, which is what a model this size can be expected to have, but the record of building and measuring it: a Turkish byte-level tokenizer at 1.77 tokens per word, a three-stage pretraining schedule, a post-training mixture of openly licensed and generated data, and an evaluation battery of release gates, a rule-checked sweep of 5,508 conversations, judged conversations and hand tests, all with prompts held out from the training data, enforced by decontamination inside the data build and by a checked-in invariant script we run before each build. We report three findings that we believe transfer to other small-model efforts: a safety gate that had been "fixed" with training data written from its own questions read 64/64 while the honest figure was 34/64; training-seed variance was as large as the spread across every recipe we tried, so single-seed comparisons at this scale are uninformative; and data rounds repaired only what was absent from the data, while identity tracking over long context and multi-turn arithmetic did not move across any data change we tried, which we read as limits of the model size rather than gaps in the data, a reading the next, larger model will test. Weights, the data recipe, the evaluation code and the spend ledger are released under Apache-2.0.

Sait Furkan Teke

ufak AI

Language Modeling Language Model Pretraining Small Language Models LLM Evaluation Language Model Safety Evaluation Byte-Level Language Model

Ask PDF