Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

Caio Vicentino

cs.CLMar 23, 2026

Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

Authors: Caio Vicentino

Organizations: Independent Researcher

Abstract

We present a controlled empirical comparison between autoregressive (AR) and masked diffusion (MDLM) language models. Both models are trained on identical data (50M tokens from TinyStories), identical compute budget (20,000 steps, batch size 32, sequence length 512), and identical hardware (NVIDIA H100 80GB), isolating the generation paradigm as the sole variable. We report three findings. First, both paradigms achieve comparable training throughput (~50K tokens/second), with MDLM requiring only 4.7% more wall-clock time. Second, AR converges faster and begins overfitting by step 14,000, while MDLM converges more slowly and is still improving at step 20,000, suggesting different compute-optimal training regimes. Third, quantitative diversity analysis over 1,000 generated samples reveals a structural diversity-fluency trade-off: AR produces fluent but repetitive outputs (99.8% begin with the same word), while MDLM generates more diverse narratives (93.4% unique 5-word openings, higher Distinct-n, lower Self-BLEU), at the cost of occasional grammatical inconsistencies. All code, trained checkpoints, and data pipelines are released for reproducibility.

Figures & tables

Shared (identical)
Variable	AR	MDLM
Data	TinyStories ( Eldan & Li, 2023 ) , 50M tokens
Steps	20,000
Batch size	32
Sequence length	512
$d_{\text{model}}$	768

Table 1: Controlled experimental setup. All shared variables are identical; differences arise solely from the paradigm.

Val losses use different objectives and are not cross-comparable.
Metric	AR	MDLM
Parameters	123.6M	162.7M
Training time	107.9 min	113.0 min
Throughput	50,620 tok/s	48,343 tok/s
Best val loss	1.589	3.412
Best checkpoint	Step 14,000	Step 20,000

Table 2: AR vs. MDLM training comparison on H100 80GB (20K steps, TinyStories 50M tokens).

Step	AR Val Loss	MDLM Val Loss
2,000	2.660	3.952
4,000	2.047	3.841
6,000	1.837	3.585
8,000	1.726	3.528
10,000	1.665	3.509
12,000	1.633	3.527

Table 3: Validation loss at selected checkpoints. Each column measures a different objective; read vertically for convergence trends, not horizontally.

Figure 1: Convergence comparison. Dual

y

-axes reflect different loss functions; the shape of each curve (not the absolute value) is the relevant comparison. AR overfits after step 14K (blue shaded region); MDLM continues improving monotonically.

Figure 2: Training throughput and wall-clock time. MDLM achieves 95.5% of AR throughput, requiring only 5.1 additional minutes over 20K steps.

Metric	AR	MDLM	More diverse
Distinct-1	0.053	0.058	MDLM
Distinct-2	0.275	0.297	MDLM
Distinct-3	0.531	0.542	MDLM
Distinct-4	0.701	0.690	AR
Self-BLEU ( $\downarrow$ )	0.341	0.334	MDLM
Vocabulary used	5,360	5,754	MDLM

Table 4: Generation diversity metrics over 1,000 samples (128 tokens each). Distinct-

n

measures the ratio of unique

n

-grams to total

n

-grams (higher = more diverse). Self-BLEU measures average BLEU of each sample against 50 random others (lower = more diverse). Unique first-5grams measures the fraction of samples with a unique 5-word opening.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Hardware	Time	Cost
AR (123.6M, 20K steps)	H100 80GB (Colab)	107.9 min	${\sim}$ $35
MDLM (162.7M, 20K steps)	H100 80GB (Colab)	113.0 min	${\sim}$ $35
Total		220.9 min	$\boldsymbol{\sim}$ $70

Appendix

Table 5: Compute budget for all experiments reported in this paper.

Explore similar work

Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models
Jan 21, 2026Injin Kong, Hyoungjoon Lee, Yohan JoMasked Diffusion Models Autoregressive Diffusion
Graduate School of Data Science, Seoul National University · Department of Biosystems & Biomaterials Science and Engineering, Seoul National University
Understanding and Accelerating the Training of Masked Diffusion Language Models
May 13, 2026Chunsan Hong, Sanghyun Lee, Chieh-Hsin Lai +5Masked Diffusion Models Diffusion Model Acceleration
1KAIST · 2Sony AI · University of Tokyo +1
Scaling Beyond Masked Diffusion Language Models
Feb 16, 2026Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang +4Diffusion Language Models Diffusion Models
NVIDIA, Santa Clara, USA · Department of Computer Science, Cornell Tech, NYC, USA · Department of Computer Science, Cornell University, Ithaca, USA +2

Jan 21, 2026J/K move · Enter open · S save

Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models

Injin Kong, Hyoungjoon Lee, Yohan Jo

Graduate School of Data Science, Seoul National University · Department of Biosystems & Biomaterials Science and Engineering, Seoul National University

Masked Diffusion Models Autoregressive Diffusion Autoregressive Language Modeling Mechanistic Interpretability Diffusion Model Fine-Tuning Language Modeling+2

Open paper AskPDF arXiv

Post-training pretrained autoregressive models (ARMs) into masked diffusion models (MDMs) provides an efficient route to diffusion language modeling, but it remains unclear whether the resulting models reuse inherited autoregressive computation or reorganize it for non-autoregressive generation. We compare two 7B ARM-MDM families across four controlled diagnostic tasks and find a task-dependent mechanism shift. On prefix-dominant tasks, MDMs largely preserve inherited high-attribution pathways or exhibit only modest changes in where computation occurs. On globally constrained tasks, the reorganization is substantially stronger, with task-relevant computation shifting toward earlier layers. This depth-wise pattern persists across prompt resampling, circuit budgets, and tested inference budgets, while targeted ablations support the functional importance of the identified structures under the tested intervention protocols. At the component level, diagnostic probes suggest that ARMs rely more strongly on sharply specialized components, whereas MDMs exhibit weaker single-component specialization and more diffuse output-space alignment. Together, these results suggest that diffusion post-training selectively preserves or reorganizes inherited computation according to task structure, rather than uniformly replacing autoregressive mechanisms.

Abstract

Explore similar work

Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models

Understanding and Accelerating the Training of Masked Diffusion Language Models

Scaling Beyond Masked Diffusion Language Models