SelFusion: Self-distillation for Diffusion Language Models

Hyeongsoo Lim; Jinyoung Kim; Eunseo Seo; Minho Jang; Jiwon Yoon

cs.CLAug 24, 2026

SelFusion: Self-distillation for Diffusion Language Models

Authors: Hyeongsoo Lim, Jinyoung Kim, Eunseo Seo, Minho Jang, Jiwon Yoon

Organizations: Department of Artificial Intelligence, Chung-Ang University

Abstract

Diffusion language models (DLMs) alleviate the inherent latency bottleneck of autoregressive (AR) large language models (LLMs), but their degraded generation quality limits practical applicability. Although knowledge distillation (KD) can be a promising direction for improving performance, we empirically find that naively applying conventional KD yields only marginal gains, or even degrades generation quality. Based on these observations, we propose a novel self-distillation framework for DLMs, namely SelFusion. To enable effective KD without an external teacher model, SelFusion performs two forward passes with different masking levels, defining the hard mode with a larger masking probability and the easy mode with a smaller masking probability. However, the easy mode is not always more accurate than the hard mode and can be overconfident on incorrect tokens. Thus, we introduce bidirectional KD between the two modes, which can dynamically determine the distillation direction based on token-level correctness. Experimental results on instruction-following tasks show that the proposed self-distillation substantially outperforms other KD methods with external LLM and DLM teachers. In many configurations, the student trained with SelFusion even surpasses the performance of the LLM teacher, providing a practical path toward improving DLM generation quality. Source code can be found at https://github.com/scai-research/SelFusion_official

Figures & tables

Figure 1: Rouge-L scores on the Dolly dataset. Existing KD methods for DLMs often underperform the SFT baseline, regardless of whether the teacher is DLM or LLM. In contrast, SelFusion significantly outperforms all baselines at the same model size.

Figure 2: Overall architecture of SelFusion. The framework comprises easy and hard modes that generate tokens under less and more masked context, respectively. Bidirectional KD determines the distillation direction based on token-level correctness.

Panel of Figure 3 — Figure 3: Comparison of average prediction probabilities for masked tokens in the Dolly dataset for LLM and DLM. For a fair comparison, we compute token probabilities using the same token prediction procedure as in training. Unlike AR models, DLMs exhibit flatter distributions, especially at higher MR.


Method	Model	Size	Dolly	Self-inst	Vicuna	Sinst	Uinst

SFT	$\text{LLM}_{tea}$	1476M	24.4765	11.0382	14.9436	23.2118	27.1273
	$\text{DLM}_{tea}$ (8 Steps)		17.2631	9.7284	12.4577	22.6308	21.9190
	$\text{DLM}_{tea}$ (16 Steps)		18.6108	10.5172	14.7385	24.3427	23.9995
	$\text{LLM}_{stu}$	472M	25.5660	11.1115	14.6017	22.4033	25.2105

Table 1: Performance comparison on multiple evaluation datasets. The first block reports SFT results of the teacher and student baselines, where

\mathrm{LLM}_{\mathrm{tea}}

and

\mathrm{DLM}_{\mathrm{tea}}

denote the teacher models and

\mathrm{LLM}_{\mathrm{stu}}

and

\mathrm{DLM}_{\mathrm{stu}}

denote the student baseline models. The second block reports KD results, where the KD column indicates the distillation direction. Bold indicates the best result.


Method	KD	Student training	Teacher training	Total

SFT	—	$2.8\times 10^{2}$	—	$2.8\times 10^{2}$
Seq-level KD	$\mathrm{LLM}_{\mathrm{tea}}\rightarrow\mathrm{DLM}_{\mathrm{stu}}$	$2.8\times 10^{2}$	$7.5\times 10^{2}$	$1.03\times 10^{3}$
	$\mathrm{DLM}_{\mathrm{tea}}\rightarrow\mathrm{DLM}_{\mathrm{stu}}$	$2.8\times 10^{2}$	$7.5\times 10^{2}$	$1.03\times 10^{3}$
Logit-level KD	$\mathrm{DLM}_{\mathrm{tea}}\rightarrow\mathrm{DLM}_{\mathrm{stu}}$	$5.4\times 10^{2}$	$7.5\times 10^{2}$	$1.29\times 10^{3}$

Table 2: Training cost analysis measured in TFLOPs. Although SelFusion requires higher per-iteration computation than SFT, it eliminates teacher training and thus reduces total training computation compared to KD methods that rely on a separately trained teacher. All methods are compared under the same training setup with 485 iterations.


Method	Steps	Dolly	Self-inst	Vicuna	Sinst	Uinst

SelFusion	16	21.6317	12.8707	17.0788	27.0745	27.8595

w/o bidirectional KD (easy $\rightarrow$ hard)		13.6755	8.5558	10.6129	21.8953	20.9376
		(-7.9562)	(-4.3149)	(-6.4659)	(-5.1792)	(-6.9219)

Table 3: Ablation results of SelFusion. We perform ablation studies by individually removing bidirectional KD and RMSNorm. For bidirectional KD, we further examine each unidirectional variant (easy→hard and hard→easy) to isolate the contribution of each direction. Across all benchmarks, removing either component leads to consistent performance degradation, and neither unidirectional variant matches the bidirectional setting

Panel of Figure 4 — Figure 4: Training accuracy comparison between SelFusion and the naive KD baseline. Naive KD uses a fixed one-way distillation direction between the two modes, Easy $\rightarrow$ Hard. Accuracy is evaluated only on the masked positions for each of the Easy and Hard modes, which use different masking ratios.

Panel of Figure 5 — Figure 5: Step-wise comparison of the mean probability assigned to the ground truth tokens by the easy and hard modes during training.

Panel of Figure 6 — Figure 6: Analysis of RMSNorm effect on SelFusion. We analyze tokens that are masked in both modes. (a) shows the probability distribution when both modes are correct, while (b) shows the case when both modes are incorrect. Easy mode uses 50% masking ratio (MR), and hard mode uses 75% MR.

Panel of Figure 7 — Figure 7: Mean top- $k$ token overlap between LLM and DLM fine-tuned on the Dolly dataset. For each position, overlap is computed as $|S_{k}^{\mathrm{LLM}}\cap S_{k}^{\mathrm{DLM}}|/k$ , where $S_{k}$ denotes the set of top- $k$ predicted tokens.


Method	KD	Steps	Dolly	Self-inst	Vicuna	Sinst	Uinst

Logit-level KD	$\text{LLM}_{tea}$ $\rightarrow$ $\text{DLM}_{stu}$	32	14.7424	8.9812	15.2372	15.6568	17.1595
	$\text{DLM}_{tea}$ $\rightarrow$ $\text{DLM}_{stu}$		18.8043	11.6876	16.0992	26.8568	26.1251
Seq-level KD	$\text{LLM}_{tea}$ $\rightarrow$ $\text{DLM}_{stu}$		20.1426	11.7443	16.6505	24.1393	25.1063
	$\text{DLM}_{tea}$ $\rightarrow$ $\text{DLM}_{stu}$		15.9906	9.8436	17.2801	16.9256	18.6831

Table 4: Ablation on larger diffusion steps. We evaluate DLMs with 32 diffusion steps to assess generalization beyond the main 8 and 16 step settings. Bold indicates the best result.

	Step	Latency (ms)	Speed-up
AR	–	2240	1.00 $\times$
DLM	1	61.4	36.5 $\times$
	2	109.7	20.4 $\times$
	4	199.1	11.3 $\times$
	8	378.4	5.9 $\times$
	16	745.4	3.0 $\times$

Table 5: Per-sample inference latency and speed-up for AR and DLM with varying diffusion steps, measured on the Dolly validation set.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Stage	Setting	Value
SFT	DLM (472M)	$5\times 10^{-5}$
	DLM (1.476B)	$1\times 10^{-5}$
	LLM (472M)	$2\times 10^{-4}$
	LLM (1.476B)	$1\times 10^{-4}$
KD	DLM $\rightarrow$ DLM (logit KD)	$5\times 10^{-5}$
	DLM $\rightarrow$ DLM (seq KD)	$5\times 10^{-5}$

Appendix

Table 6: Selected hyperparameters from grid search and shared training settings.

Panel of Figure 8 — Figure 8: Masked-token prediction accuracy comparison between the easy and hard modes. The left plot evaluates both modes on hard mode masked positions, and the right plot evaluates both modes on easy mode masked positions. Note that a token masked in one mode may remain visible in the other mode. Thus, for a token masked in one mode, the counterpart mode may observe the ground truth token at that position.


Method	KD	8 Steps	16 Steps

SFT	—	27.9175	29.3172
Seq-level KD	$\text{LLM}_{tea}\rightarrow\text{DLM}_{stu}$	27.7835	29.2686
	$\text{DLM}_{tea}\rightarrow\text{DLM}_{stu}$	21.9711	23.4053
Logit-level KD	$\text{LLM}_{tea}\rightarrow\text{DLM}_{stu}$	26.3727	26.6891

Appendix

Table 7: Results on SAMSum measured by ROUGE-L across different diffusion steps. SelFusion consistently outperforms strong baselines, with larger gains under the more efficient 8-step setting. Bold indicates the best result.

Diffusion Language Model Training

See all 40 papers

Few-Step Diffusion Language Models via Trajectory Self-Distillation
Feb 12, 2026Tunyu Zhang, Xinxi Zhang, Ligong Han +10Diffusion Model Distillation Few-Step Diffusion Sampling
Rutgers University · Red Hat AI Innovation · MIT-IBM Watson AI Lab
Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Models
Sep 14, 2026Shijian Xu, Andrea Miele, Metod Jazbec +3Diffusion Model Distillation Diffusion Language Model Inference
University of Basel · University of Amsterdam · Johns Hopkins University
Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation
Jun 4, 2026Xingyu Su, Jacob Helwig, Shubham Parashar +6Language Model Distillation Diffusion Language Models
Department of Computer Science and Engineering, Texas A&M University · Department of Bioinformatics and Systems Medicine,2026 University of Texas Health Science Center at Houston · Department of Electrical and Computer Engineering, Texas A&M UniversityJun 4

Feb 12, 2026J/K move · Enter open · S save

Few-Step Diffusion Language Models via Trajectory Self-Distillation

Tunyu Zhang, Xinxi Zhang, Ligong Han, Haizhou Shi+9

Rutgers University · Red Hat AI Innovation · MIT-IBM Watson AI Lab

Diffusion Model Distillation Few-Step Diffusion Sampling Diffusion Language Model Decoding Diffusion Language Models Diffusion Language Model Distillation

Open paper AskPDF arXiv

Diffusion large language models (DLLMs) have emerged as powerful generative models with the promise of fast text generation through parallel decoding. However, realizing this potential in practice remains challenging: reducing the number of decoding steps, typically causes a substantial degradation in output quality due to token factorization error. To alleviate this, we propose a self-distillation framework that trains a few-step student to match the generative trajectory of a full-step teacher. We theoretically and empirically show that trajectory-level supervision mitigates this factorization error, thereby enabling effective few-step decoding. We further incorporate Direct Discriminative Optimization (DDO), a reverse-KL objective that encourages mode-seeking toward the teacher's modes, yielding stronger performance on challenging reasoning tasks. Across reasoning and code-generation benchmarks, our method substantially narrows the gap between few-step and full-step decoding. The source code is available at https://github.com/Tyrion58/T3D.

Abstract

Diffusion Language Model Training

Explore similar work

Few-Step Diffusion Language Models via Trajectory Self-Distillation

Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Models

Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation