KazByte: Adapting Qwen models to Kazakh via Byte-level Adapter

Rauan Akylzhanov

cs.CLMar 29, 2026

KazByte: Adapting Qwen models to Kazakh via Byte-level Adapter

Authors: Rauan Akylzhanov

Organizations: Independent Researcher, Almaty, Kazakhstan

Abstract

Large language models fragment Kazakh text into many more tokens than equivalent English text, because their tokenizers were built for high-resource languages. This tokenizer tax inflates compute, shortens the effective context window, and weakens the model's grip on Kazakh morphology. We propose to bypass the tokenizer entirely by feeding raw bytes through a small adapter that learns to speak the internal language of a frozen Qwen2.5-7B. Once the adapter is trained, we freeze it and fine-tune only the attention layers of Qwen on Kazakh text. Our central hypothesis is that this two-stage process -- first teach the interface, then adapt the model -- should match or exceed the accuracy of the original Qwen2.5-7B on standard Kazakh benchmarks. This report describes the ByteKaz architecture and training protocol. Empirical validation is ongoing; this version stakes the design and hypotheses for the record.

Figures & tables

Panel of Figure 1 — Figure 1: The ByteKaz architecture. Blue modules are trained in Stage A. The Qwen2.5-7B stack is frozen during adapter training; in Stage B the adapter is frozen and attention sublayers in Qwen are updated on Kazakh data. Bytes are grouped into entropy-based patches, projected into Qwen’s space, and processed by the transformer; the local decoder predicts bytes autoregressively.

Component	Role	Dim	Params	Status
BLT Local Encoder	Bytes $\to$ patches	512	$\approx$ 150M	Trained
Projection $W_{\mathrm{enc}}$	Patches $\to$ Qwen space	$512{\to}4096$	$\approx$ 2M	Trained
Qwen2.5-7B body	Sequence modelling	4096	$\approx$ 7B	Stage A: frozen; Stage B: attention trainable
Projection $W_{\mathrm{dec}}$	Qwen space $\to$ patches	$4096{\to}512$	$\approx$ 2M	Trained
BLT Local Decoder	Patches $\to$ bytes	512	$\approx$ 150M	Trained
Trainable total			$\approx$ 304M

Table 1: Components of ByteKaz , their sizes, and training status.

	LLaVA	ByteKaz
Input modality	Image patches (CLIP)	Byte patches (BLT encoder)
Output modality	Text tokens (BPE LM head)	Byte sequence (BLT decoder)
Frozen body	LLaMA / Vicuna	Qwen2.5-7B (Stage A)
Trainable components	CLIP encoder + projection	Encoder + decoder + projections
Bidirectional interface	Input only	Input and output
Target application	Vision-language tasks	Agglutinative language NLP

Table 2: Comparison of LLaVA and ByteKaz .

Stage	Tokens	Hardware	Est. Time
A — Adapter alignment (Qwen frozen)	1–2B	4 $\times$ A100 80 GB	1–2 weeks
B — Kazakh LM, attention-only Qwen	5–10B	4 $\times$ A100 80 GB	2–4 weeks
C — Task SFT	50–200M	2 $\times$ A100 80 GB	2–3 days

Table 3: Training stages and resource estimates (indicative).

Benchmark	Task	Metric	Random
MC QA ( Tukenov, 2026a )	4-choice QA (7,111 items)	Accuracy	25%
Belebele ( Bandarkar et al., 2023 )	Reading comprehension	Accuracy	25%
SIB-200 ( Adelani et al., 2024 )	Topic classification (7-class)	Accuracy	14.3%

Table 4: Kazakh NLP benchmarks (SozKZ suite ( Tukenov, 2026c ) ).

Dataset ID	Kazakh indicator	License / access
statmt/cc100	Config kk : $\approx$ 889M chars listed	No IP claim on corpus prep.; bound by Common Crawl ToU.
wikimedia/wikipedia	Config e.g. 20231101.kk	GFDL + CC BY-SA 3.0 (Wikimedia dumps legal page).
HPLT/HPLT2.0_cleaned	Subset kaz_Cyrl ; card lists cc0-1.0 ; not gated	Prefer HPLT/HPLT3.0 if starting fresh.
kurumikz/Cleaned-Kazakh-Wikipedia	228,810 articles; Cyrillic	ODC-BY (dataset card).
legacy-datasets/mc4	Language kk listed; deprecated in favor of allenai/c4	ODC-BY; use allenai/c4 for new work.

Table 5: Verified Kazakh-capable corpora on Hugging Face (open access).

Method	MC QA	Belebele	SIB-200
SozKZ-600M (published)	30.3	27.0	25.5
Qwen-2.5-0.5B (published)	31.5	30.0	19.1
Qwen2.5-7B + BPE	—	—	—
ByteKaz after Stage B	—	—	—

Table 6: Reference accuracies (%) from Tukenov (2026c) on the SozKZ suite; ByteKaz and Qwen2.5-7B rows to be measured.

Explore similar work

SozKZ: Training Efficient Small Language Models for Kazakh from Scratch
Mar 21, 2026Saken TukenovLanguage Model Pretraining Small Language Models
Independent Researcher
Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir
May 6, 2026Mullosharaf K. Arabov, Svetlana S. KhaybullinaLLM Fine-Tuning Low-Resource Language Processing
Institute of Computational Mathematics and Information Technologies, Kazan Federal University, Kazan, Russia
ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
Mar 3, 2026Chunyuan Deng, Sanket Lokegaonkar, Colin Lockard +3Language Modeling Byte-Level Language Model
Rice University · Amazon Science

Mar 21, 2026J/K move · Enter open · S save

SozKZ: Training Efficient Small Language Models for Kazakh from Scratch

Saken Tukenov

Independent Researcher

Language Model Pretraining Small Language Models Low-Resource Language Processing Efficient Language Model Training

Open paper AskPDF arXiv

Kazakh, a Turkic language spoken by over 22 million people, remains underserved by existing multilingual language models, which allocate minimal capacity to low-resource languages and employ tokenizers ill-suited to agglutinative morphology. We present SozKZ, a family of Llama-architecture language models (50M-600M parameters) trained entirely from scratch on 9 billion tokens of Kazakh text with a dedicated 50K BPE tokenizer. We evaluate all models on three Kazakh benchmarks -- multiple-choice cultural QA, reading comprehension (Belebele), and topic classification (SIB-200) -- alongside five multilingual baselines ranging from 500M to 3B parameters. Our 600M model achieves 30.3% accuracy on Kazakh cultural QA, approaching the 32.0% of Llama-3.2-1B (2x larger), and 25.5% on SIB-200 topic classification, surpassing all evaluated multilingual models up to 2B parameters. We observe consistent scaling from 50M to 600M, with MC QA accuracy rising from 22.8% to 30.3%, suggesting that further scaling remains beneficial. These results demonstrate that small, dedicated models trained from scratch with a language-appropriate tokenizer offer a viable path for low-resource language technology, achieving competitive performance at a fraction of the computational cost. All models and the tokenizer are released under open licenses.

Abstract

Explore similar work

SozKZ: Training Efficient Small Language Models for Kazakh from Scratch

Adapting Large Language Models to a Low-Resource Agglutinative Language: A Comparative Study of LoRA and QLoRA for Bashkir

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer