Diffutron: A Masked Diffusion Language Model for Turkish Language

Şuayp Talha Kocabay; Talha Rüzgar Akkuş

cs.CLMar 20, 2026

Diffutron: A Masked Diffusion Language Model for Turkish Language

Authors: Şuayp Talha Kocabay, Talha Rüzgar Akkuş

Organizations: Hugging Face: huggingface.co/suayptalha · Hugging Face: huggingface.co/Q-bert

Abstract

Masked Diffusion Language Models (MDLMs) have emerged as a compelling non-autoregressive alternative to standard large language models; however, their application to morphologically rich languages remains limited. In this paper, we introduce $\textit{Diffutron}$ , a masked diffusion language model specifically designed for Turkish. Our approach leverages a resource-efficient training pipeline, starting with LoRA-based continual pre-training of a multilingual encoder on a large-scale corpus. To enable generative capabilities, we employ a progressive instruction-tuning strategy, sequentially adapting the model on general and task-specific instruction sets. Experimental results across comprehensive benchmarks demonstrate that, despite its compact size, our model achieves competitive performance compared to existing multi-billion-parameter baselines. These findings validate the effectiveness of masked diffusion modeling combined with multi-stage tuning for non-autoregressive text generation in Turkish.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of the reverse diffusion step. The model predicts masked tokens (e.g., $x_{t}\to x_{0}$ ) by attending to visible context bi-directionally (dashed lines).

Hyperparameter	Configuration
Base Architecture	mmBERT-base
Adapter Rank ( $r$ ) / Alpha ( $\alpha$ )	256 / 256
Target Modules	Attn ( $W_{q},W_{k},W_{v},W_{o}$ ) MLP ( $W_{up},W_{down}$ )
Trainable Parameters	$\approx$ 14.94%
Maximum sequence length	512
Optimizer	Paged AdamW (8-bit)

Table 1: Hyperparameters and training statistics for the Continual Pre-training stage.

Figure 2: Pretraining loss curve for the model.

Hyperparameter	Configuration
Maximum sequence length	256
Training epochs	20
Batch size (train/eval)	16
Optimizer	AdamW (lr=1e-4)
Training hardware	A100 40 GB GPU
Runtime	1h 40m

Table 2: First stage fine-tuning configuration

Figure 3: Training loss for the first stage of instruction fine-tuning.

Hyperparameter	Configuration
Maximum sequence length	256
Training epochs	8
Batch size (train)	96
Optimizer	AdamW (lr=1e-4)
Training hardware	2xA100 80 GB GPUs
Runtime	9h 53m

Table 3: Second stage fine-tuning configuration

Figure 4: Training loss for the second stage of instruction fine-tuning.

Text Generation

Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.

See all 422 papers

Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes
Mar 4, 2026Fangyu Ding, Ding Ding, Sijin Chen +8Diffusion Language Models Language Modeling
HKUST · Huawei Foundation Model Dept · CUHK
Backdooring Masked Diffusion Language Models
May 19, 2026Daniel Yiming Cao, Chengzhong Wang, Sheng-Yen Chou +3LLM Backdoor Attacks Masked Diffusion Models
Cornell University · Virginia Tech · IBM Research
Machine Unlearning for Masked Diffusion Language Models
May 18, 2026Georu Lee, Seungwon Jeong, Hoki Kim +2Masked Diffusion Models Diffusion Model Unlearning
1Dongguk University-Seoul · 2Chung-Ang University · 3Korea Institute for Advanced Study

Mar 4, 2026J/K move · Enter open · S save

Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes

Fangyu Ding, Ding Ding, Sijin Chen, Kaibo Wang+7

HKUST · Huawei Foundation Model Dept · CUHK

Diffusion Language Models Language Modeling Masked Diffusion Models Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

While Masked Diffusion Language Models (MDLMs) relying on token masking and unmasking have shown promise in language modeling, their computational efficiency and generation flexibility remain constrained by the masking paradigm. In this paper, we propose Deletion-Insertion Diffusion language models (DID) that rigorously formulate token deletion and insertion as discrete diffusion processes, replacing the masking and unmasking processes in current MDLMs. DID improves training and inference efficiency by eliminating two major sources of computational overhead in MDLMs: the computations on non-informative 1) <MASK> tokens inherent to the paradigm, and 2) <PAD> tokens introduced in variable-length settings. Furthermore, DID offers greater flexibility by: 1) natively supporting variable-length sequences without requiring fixed-length padding, and 2) an intrinsic self-correction mechanism during generation due to insertion that dynamically adjusts token positions. To train DID, we design a score-based approach that assigns scores to token insertion operations and derive appropriate training objectives. The objectives involve subsequence counting problems, which we efficiently solve via a parallelized dynamic programming algorithm. Our experiments across fixed and variable-length settings demonstrate the advantage of DID over baselines of MDLMs and existing insertion-based LMs, in terms of modeling performance, sampling quality, and training/inference speed, without any hyperparameter tuning.

Abstract

Text Generation

Explore similar work

Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes

Backdooring Masked Diffusion Language Models

Machine Unlearning for Masked Diffusion Language Models