Differences in Text Generated by Diffusion and Autoregressive Language Models

Zeyang Zhang; Chengwei Liang; Xingyan Chen; Meiqi Gu; Minrui Luo; Jingzhao Zhang; Tianxing He

cs.CLApr 4, 2026

Differences in Text Generated by Diffusion and Autoregressive Language Models

Authors: Zeyang Zhang, Chengwei Liang, Xingyan Chen, Meiqi Gu, Minrui Luo, Jingzhao Zhang, Tianxing He

Organizations: Shanghai Qi Zhi Institute · Institute for Interdisciplinary Information Sciences, Tsinghua University · Xiongan AI Institute

Abstract

Diffusion language models (DLMs) are promising alternatives to autoregressive language models (ARMs), yet the intrinsic differences in their generated text remain underexplored. We first find empirically that off-the-shelf DLMs exhibit lower $n$ -gram entropy, higher semantic coherence, and higher semantic diversity. To understand the cause, we conduct controlled experiments that decouple the effects of training objectives and decoding algorithms. Results suggest that the DLM training objective contributes to the increases in semantic coherence and semantic diversity, but has a minor influence on entropy. These differences are primarily driven by the bidirectional context; other components in the training objective, such as input masking, label masking, and the weighting function, have a much weaker influence. Further, our experiments demonstrate that the reduction in entropy stems from DLMs' decoding algorithms, particularly confidence-based remasking strategies. We provide a theoretical understanding for this entropy reduction phenomenon. Together, our work uncovers key mechanisms underlying the differences between DLMs and ARMs in text generation, and informs future design of training objectives and decoding algorithms in DLMs.

Figures & tables

Figure 1: Comparison of three metrics between 20 off-the-shelf DLMs and ARMs. DLMs tend to exhibit lower trigram entropy alongside higher semantic coherence and semantic diversity compared to ARMs.

Figure 2: Definition and evaluation of eight interpolated training objectives. Abbreviations denote the incorporated components: unidirectional context ( uc ), bidirectional context ( bc ), input masking ( im ), label masking ( lm ), and weighting function ( wf ). Curves are averaged over random seeds. A clear clustering emerges, demonstrating that the bidirectional context in the DLM training objective markedly increases semantic coherence and semantic diversity, while entropy stays approximately the same across all eight models.

Figure 3:

n

-gram entropy and cross-entropy (defined in Eq.( 9 ) ) across different DLM remasking strategies and block lengths. The labels denote low-confidence remasking (Confidence), dynamic low-confidence remasking (D-Confidence), high-entropy remasking (Entropy), random remasking (Random). The horizontal dashed line in the last subplot indicates

\mathcal{H}(p_{\text{seq}})

. The combined height of the bars for the confidence‑based strategies, including the hatched areas stacked on top, represents the results of the bias elimination experiments (refer to § • ‣ 5.2 ). This figure shows results for a single seed; see Appendix C.3 for other seeds.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Model	Count
ARM	DeepSeek-R1-0528-Qwen3-8B ( Guo et al., 2025 )	946
	Gemma-3-4B-it ( Team et al., 2025 )	996
	Llama-2-7B-chat ( Touvron et al., 2023b )	997
	Llama-3.1-8B-Instruct ( Grattafiori et al., 2024 )	999
	Llama-3.2-3B-Instruct ( Grattafiori et al., 2024 )	996
	Mistral-7B-Instruct-v0.3 ( Jiang et al., 2023 )	995

Appendix

Table 1: List of evaluated models and the number of remaining samples for each model.

Type	Model	$n$ -gram entropy			Semantic coherence	Semantic diversity
Type	Model	Unigram	Bigram	Trigram	Semantic coherence	Semantic diversity
ARM	DeepSeek-R1-0528-Qwen3-8B	7.579	11.385	12.655	0.575	0.569
	Gemma-3-4B-it	7.411	11.241	12.639	0.557	0.572
	Llama-2-7B-chat	7.013	10.612	12.027	0.634	0.604
	Llama-3.1-8B-Instruct	7.129	10.816	12.316	0.626	0.604
	Llama-3.2-3B-Instruct	7.059	10.723	12.271	0.632	0.599

Appendix

Table 2: Evaluation results for 20 off-the-shelf ARMs and DLMs. “Avg.” denotes the average over models within each type.

Figure 4: Scatter plot visualization of the off-the-shelf evaluation results in Table 2 . Each point denotes one model, and the horizontal bars indicate the mean over models within each type.

Panel of Figure 5 — Figure 5: Controlled experiments on interpolated training objectives with three different random seeds, presented in the same format as Figure 2 .

Figure 6: Controlled experiments on interpolated training objectives using the Qwen2 architecture, presented in the same format as Figure 2 .

Figure 7: Controlled experiments on interpolated training objectives using the TinyStories dataset, presented in the same format as Figure 2 .

Panel of Figure 8 — Figure 8: $n$ -gram entropy and cross-entropy (defined in Eq.( 9 ) ) across different remasking strategies with other seeds. The format follows Figure 3 , which displays the seed = 0 results.

Figure 9:

n

-gram entropy and cross-entropy (defined in Eq.( 9 ) ) across different remasking strategies for the Qwen2 architecture experiment, presented in the same format as Figure 3 .

Figure 10:

n

-gram entropy and cross-entropy (defined in Eq.( 9 ) ) across different remasking strategies for the TinyStories dataset experiment, presented in the same format as Figure 3 .

Panel of Figure 11 — Figure 11: Semantic coherence and semantic diversity across different remasking strategies with three random seeds.

Figure 12: Semantic coherence and semantic diversity across different remasking strategies for the Qwen2 architecture experiment.

Figure 13: Semantic coherence and semantic diversity across different remasking strategies for the TinyStories dataset experiment.

Text Generation

Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.

See all 422 papers

Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison
Mar 23, 2026Caio VicentinoDiffusion Language Models Autoregressive Language Modeling
Independent Researcher
Scaling and Distilling Text Embeddings for Better Diffusibility
Oct 1, 2026Zekai Zhang, Yunjie Tian, Yanjin He +4Word Embeddings Language Model Distillation
University of Michigan
CoBit: Language Modeling with Bitstream Diffusion
May 7, 2026Georgios Batzolis, Mark Girolami, Luca AmbrogioniDiffusion Language Model Decoding Diffusion Language Models
Department of Engineering, University of Cambridge. · Devotion AI Labs Ltd. · Donders Institute for Brain, Cognition, and Behaviour, Radboud University.

Mar 23, 2026J/K move · Enter open · S save

Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

Caio Vicentino

Independent Researcher

Diffusion Language Models Autoregressive Language Modeling Diffusion Language Model Benchmarking LLM Training Diffusion Language Model Training Language Model Generation Evaluation

Open paper AskPDF arXiv

We present a controlled empirical comparison between autoregressive (AR) and masked diffusion (MDLM) language models. Both models are trained on identical data (50M tokens from TinyStories), identical compute budget (20,000 steps, batch size 32, sequence length 512), and identical hardware (NVIDIA H100 80GB), isolating the generation paradigm as the sole variable. We report three findings. First, both paradigms achieve comparable training throughput (~50K tokens/second), with MDLM requiring only 4.7% more wall-clock time. Second, AR converges faster and begins overfitting by step 14,000, while MDLM converges more slowly and is still improving at step 20,000, suggesting different compute-optimal training regimes. Third, quantitative diversity analysis over 1,000 generated samples reveals a structural diversity-fluency trade-off: AR produces fluent but repetitive outputs (99.8% begin with the same word), while MDLM generates more diverse narratives (93.4% unique 5-word openings, higher Distinct-n, lower Self-BLEU), at the cost of occasional grammatical inconsistencies. All code, trained checkpoints, and data pipelines are released for reproducibility.

Abstract

Text Generation

Explore similar work

Autoregressive vs. Masked Diffusion Language Models: A Controlled Comparison

Scaling and Distilling Text Embeddings for Better Diffusibility

CoBit: Language Modeling with Bitstream Diffusion