Hierarchical Masked Diffusion Language Models

Parul Gupta; Satwik; Avni Aggarwal; Bhavesh Gurnani; Arnab Kumar Mondal; Dinesh Khandelwal; Dinesh Garg; Parag Singla

cs.CLOct 21, 2026

Hierarchical Masked Diffusion Language Models

Authors: Parul Gupta, Satwik, Avni Aggarwal, Bhavesh Gurnani, Arnab Kumar Mondal, Dinesh Khandelwal, Dinesh Garg, Parag Singla

Organizations: Indian Institute of Technology Delhi · IBM Research

Abstract

Masked diffusion language models have emerged as an alternative to autoregressive models, enabling parallel token generation. However, standard masked diffusion language models use a shared noise schedule across positions without explicitly encoding which positions should be resolved first. Many tasks exhibit a natural priority among positions: document titles and section headings can guide the generation of body text, while Sudoku cells determined directly from the initial clues can help resolve more difficult cells. To incorporate this notion of priority (or hierarchy) among token positions, we introduce the Hierarchical Masked Diffusion Language Model (HMDLM), which augments each token position with a hierarchy level available as part of the training data, and requires zero or minimal additional annotations. It factors generation into two passes: the first predicts a hierarchy distribution over masked positions, the second predicts token distributions conditioned on it. Giving each hierarchy level its own noise schedule induces a soft, controllable preference for revealing higher-priority positions earlier while keeping decoding fully parallel. Our formulation is mathematically grounded, and we re-derive the forward and reverse processes and the NELBO for this factorization. Our evaluation shows that for the task of language generation on LM1B, HMDLM reduces perplexity by more than 1.5 points over Anchored Diffusion Language Model (ADLM), a recently proposed strong diffusion baseline; on WikiHow, we see a drop in perplexity of up to 4.4 pts on ADLM and up to 7 pts against MDLM. An LLM judge on WikiHow prefers HMDLM generations (the two variants combined) more than $2\times$ as often as the baseline. On the task of solving Sudoku puzzles requiring multi-step reasoning, it improves solution accuracy by up to $8.6\%$ absolute points over existing diffusion baselines.

Figures & tables

Figure 1: HMDLM inference example. Positions $i=0,3,5$ are masked in $(x_{t},h_{t})$ . Step $t$ : Stage 1 reveals $h_{t-1}^{5}=1$ , then Stage 2 predicts $x_{t-1}^{5}=\text{ball}$ . Step $t-1$ : Stage 1 reveals $h_{t-2}^{0}=0$ and $h_{t-2}^{3}=0$ , then Stage 2 predicts $x_{t-2}^{0}=\text{The}$ and $x_{t-2}^{3}=\text{the}$ .

33B tokens
Model	PPL ( $\downarrow$ )
Diffusion BERT ( He et al., 2023 )	$\leq 63.78$
SEDD ( Lou et al., 2024 )	$\leq 32.79$
MDLM ( Sahoo et al., 2024 )	$\leq 27.04$
ADLM ( Rout et al., 2025 )	$\leq 26.40$
HMDLM	$\leq\mathbf{24.30}$

Table 1: Test perplexities (PPL

\downarrow

) on LM1B. Best value is bolded.

Model	Gen. PPL ( $\downarrow$ )	MAUVE ⋆ ( $\uparrow$ )	Entropy
MDLM	$177.94{\scriptstyle\pm 0.82}$	$0.66{\scriptstyle\pm 0.03}$	$6.022{\scriptstyle\pm 0.003}$
HMDLM	$\mathbf{83.35}{\scriptstyle\pm 0.83}$	$\mathbf{0.76}{\scriptstyle\pm 0.06}$	$5.876{\scriptstyle\pm 0.014}$

Table 2: LM1B generative evaluation on

65

B checkpoints. Mean

\pm

std over 3 seeds.

$L$	MDLM	HMDLM-P †	ADLM	HMDLM-F ‡
$256$	$24.60\pm 0.22$	$23.89\pm 0.11$	$22.05\pm 0.21$	$\mathbf{17.62\pm 0.12}$
$512$	$21.04\pm 0.06$	$21.76\pm 0.15$	$20.15\pm 0.10$	$\mathbf{16.64\pm 0.06}$
$1024$	$22.65\pm 0.04$	$22.58\pm 0.04$	$20.92\pm 0.05$	$\mathbf{16.90\pm 0.09}$

Table 3: Model PPL (

\downarrow

) across sequence lengths on WikiHow. Best value is in bold.

	Gen. PPL			MAUVE ( $\uparrow$ )			Entropy
Model	256	512	1024	256	512	1024	256	512	1024
Gold	25.2	21.0	20.8	1.00	1.00	1.00	5.86	6.42	6.70
MDLM	76.0	67.4	62.7	0.33	0.11	0.12	6.47	7.10	7.60
ADLM	84.0	82.1	93.81	0.40	0.21	0.08	6.49	7.17	7.77
HMDLM-P	23.1	17.6	14.6	0.38	0.13	0.07	5.91	6.26	6.45
HMDLM-F	29.2	25.4	$19.92$	0.52	0.41	0.20	5.98	6.60	$6.83$

Table 4: Generative evaluation on WikiHow across sequence lengths

L\in\{256,512,1024\}

. HMDLM-P/F: position-/frequency-based hierarchy. Gold is the reference data.

Metric	Model	stochastic	Greedy
Board Acc.	MDLM	$0.81$	$0.83$
Board Acc.	HMDLM	$\mathbf{0.86}$	$\mathbf{0.88}$
Cell Acc.	MDLM	$0.94$	$0.94$
Cell Acc.	HMDLM	$\mathbf{0.96}$	$\mathbf{0.97}$

Table 5: Sudoku-hard results: 25M parameter models under stochastic & greedy decoding.

Model	Board Acc.	Cell Acc.
MGDM	$63.07\pm 0.18$	$85.79\pm 0.10$
HMDLM	$\mathbf{71.69\pm 0.26}$	$\mathbf{91.13\pm 0.07}$

Table 6: Sudoku-hard test set (

18

k puzzles). Results are from

6

M-parameter models trained for

100

k steps with

T=81

and confidence-based sampling. Mean

\pm

std over

3

sampling seeds.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	$g_{1}$	$\rho=1-g_{1}/3$	Steps	Extra steps (epochs)
$L=512$
MDLM	–	1.000	28,650	–
HMDLM (summary)	22.34%	0.926	30,955	$+2{,}305$ ( $+2.40$ )
HMDLM ( $\tau{=}1$ )	30.82%	0.897	31,930	$+3{,}280$ ( $+3.41$ )
$L=1024$
MDLM	–	1.000	41,137	–

Appendix

Table 7: Step-matched training budgets on WikiHow at context lengths 512 and 1024.

g_{1}

is the fraction of tokens at level 1. Under the summary rule, level 1 is the article summary. Under the count rule (

\tau{=}1

), level 1 is any token occurring at most once in its own sample.

Level	Cells	MDLM	HMDLM
5	givens + round 0	99.76	100.00
4	round 1	99.59	99.99
3	round 2	99.29	99.98
2	round 3	98.73	99.92
1	rounds 4–5	96.77	99.45
0	round 6+ / never forced	78.78	86.57

Appendix

Table 8: Cell accuracy (%) by hierarchy level.

Masked Diffusion Models

Masked diffusion models generate sequences by repeatedly predicting and revising masked tokens, using bidirectional context for parallel sampling.

See all 183 papers

Hierarchical Continuous Diffusion Language Models
Oct 1, 2026Hui Ren, Zihan Li, Chang Liu +2Structured Reasoning in Language Models Diffusion Language Model Decoding
University of Illinois Urbana-Champaign · Amazon.com, Inc.
Mask-Aware Policy Gradients for Diffusion Language Models
Jul 16, 2026Haran Raajesh, Kulin Shah, Adam Klivans +1Masked Diffusion Models RL for Language Model Reasoning
Department of Computer Science The University of Texas at Austin
Edit-Based Refinement for Parallel Masked Diffusion Language Models
May 10, 2026Houxing Ren, Mingjie Zhan, Zimu Lu +5Masked Diffusion Models Parallel Decoding
CUHK MMLab · SenseTime Research · Shenzhen Loop Area Institute +1

Oct 1, 2026J/K move · Enter open · S save

Hierarchical Continuous Diffusion Language Models

Hui Ren, Zihan Li, Chang Liu, Huidong Liu+1

University of Illinois Urbana-Champaign · Amazon.com, Inc.

Structured Reasoning in Language Models Diffusion Language Model Decoding Diffusion Language Models Continuous Diffusion Language Models

Open paper AskPDF arXiv

Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently from its marginal, severing the statistical dependencies among the tokens decoded together. Continuous diffusion language models avoid this by denoising a shared continuous state, but their denoiser sees only that state, so nothing ties it to a valid token configuration until it is finally decoded. To address this, we propose Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood. In contrast to recent methods that attach continuous context to a self-contained discrete chain, HC-DLM makes the latent the only persistent generative state: tokens are read out from it at every step and feed back as a scaffold for the next latent update. On structured reasoning (Sudoku), mathematical planning (Countdown) and language modeling (LM1B), HC-DLM improves over discrete and continuous diffusion baselines at matched model size, in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B. Project page: https://hc-dlm.github.io/.

Abstract

Masked Diffusion Models

Explore similar work

Hierarchical Continuous Diffusion Language Models

Mask-Aware Policy Gradients for Diffusion Language Models

Edit-Based Refinement for Parallel Masked Diffusion Language Models