Large Language Continuous Diffusion Models

Zhihan Yang; Wei Guo; Jean-Marie Lemercier; Simon Welker; Yonggan Fu; Mohammad Mahdi Kamani; Sajad Norouzi; Julius Berner

cs.CLOct 2, 2026

Large Language Continuous Diffusion Models

Authors: Zhihan Yang, Wei Guo, Jean-Marie Lemercier, Simon Welker, Yonggan Fu, Mohammad Mahdi Kamani, Sajad Norouzi, Julius Berner,

Organizations: NVIDIA · Cornell University · Georgia Institute of Technology

Abstract

Despite the success of discrete diffusion language models (dLMs) for fast parallel decoding, their non-smooth, high-dimensional space hinders trajectory steering for reasoning and inference acceleration. To overcome this, we present Sigma, the first large-scale (3B/8B) continuous dLM built on steerable, low-dimensional ODE/SDE latent trajectories. Trained blockwise via likelihood optimization, Sigma jointly denoises Gaussian-corrupted token embeddings while learning an optimal embedding geometry. To accelerate training, Sigma leverages pre-trained weights from autoregressive (AR) models for warm-starting. During inference, we identify classifier-free guidance and score temperature as essential for high-fidelity reasoning and coding. Across comprehensive math reasoning and coding evaluations against state-of-the-art discrete counterparts (masked dLMs and AR baselines), Sigma achieves competitive performance with discrete models on standard benchmarks (e.g., GSM8K, Minerva, HumanEval, MBPP) after pre-training and on challenging reasoning tasks (e.g., MATH-500, AIME) after supervised fine-tuning. Beyond performance parity, we uncover key structural properties unique to continuous dLMs: (i) embedding-space steering effectively governs the quality-diversity trade-off, yielding strong pass@k performance and (ii) continuous trajectories enable graceful degradation for low NFEs and efficient distillation. These establish continuous dLMs as a promising paradigm for efficient language generation.

Figures & tables

Figure 1: Left: Inference pipeline. Sigma generates text blockwise via iterative diffusion denoising over 16D continuous latents (

\mathbf{z}_{t}

) conditioned on a KV-cached prefix (

\mathbf{c}

), followed by categorical readout. Predicted vocabulary probabilities

\mathbf{x}_{\theta}(\mathbf{z}_{t},t;\mathbf{c})

map to mean predictions

\mathbf{e}_{\theta}(\mathbf{z}_{t},t;\mathbf{c})

in 16D space via categorical reparameterization. Clean embeddings (dots) lie on the 16D unit sphere, while

\mathbf{e}_{\theta}

lies on or within it. Middle: Smooth trajectory enables low-NFE reasoning. GSM8K accuracy vs. NFEs using the optimal sampler per step budget (PDD: Parallel Decoding Distillation). Right: Scaling inference-time compute. Pass@

k

vs.

k

on AIME 2025 (solid curves: fixed config across

k

; thin curves: alternative tested configs).

Panel of Figure 2 — Figure 2: Training procedure of Sigma (a) and its transformer attention mask (b) for an example sequence ${\mathbf{x}}$ of length $L=6$ with block size $L^{\prime}=2$ . During pre-training, we feed the clean discrete tokens $\mathbf{x}$ to the AR stream (a, left part) and the noisy continuous embeddings $\mathbf{z}$ to the diffusion stream (a, right part). For the diffusion stream, each block is randomly chosen to either compute the reconstruction loss (1st block, $t=0$ ) or the diffusion loss (2nd & 3rd blocks, $t\sim\operatorname{Unif}[0,1]$ ); see 5 . Reconstruction loss is the cross entropy between the logits and the ground truth tokens, while for diffusion loss we use categorical reparameterization to obtain the mean embedding prediction and compute the mean squared error against the ground truth diffusion embeddings.

Panel of Figure 3 — Figure 3: Pre-training Sigma -3B from scratch (a), from AR initialization (b), and from AR initialization with AR regularization (c). PPL for AR and diffusion streams is computed via $\mathcal{L}_{\text{ar}}({\mathbf{x}})$ 6 and $\mathcal{L}_{\text{BD}}({\mathbf{x}})$ 3 , respectively. Early investigations (a, b) used block size 64; our flagship 3B model (c) used block size 32 to match the inference block size of NLD. Proper AR initialization/regularization leads to the best Sigma model.

Panel of Figure 4 — Figure 4: Inference for blockwise continuous diffusion: packed CFG attention (a) and one reverse step (b).

Figure 5: CFG and ST tuning on Sigma -3B for DDPM sampling. Combining both techniques yields larger gains over applying either in isolation, and markedly outperforms sampling without CFG or ST. See Fig. 10 for the full tuning grid.

	Math		Coding
Method	GSM8K	Minerva Math	HumanEval	HumanEval+	MBPP	MBPP+	Avg.
AR initialization	83.70	66.28	51.22	48.78	60.60	77.78	64.73
AR mode
Nemotron-Labs-Diffusion-3B	83.78	65.06	53.05	47.56	60.80	78.84	64.85
Sigma -3B (Ours)	83.40	65.10	57.32	51.83	59.00	76.19	65.47
Diffusion mode

Table 1: Performance of Sigma -3B (base model; block size: 32) on math reasoning and coding benchmarks. Accuracy (%) is pass@1. The Sigma diffusion rows are reported as mean (over 10 seeds)

\pm

two standard errors. Bold : the best value across all methods; underline : the best value within each mode.

	Math		Coding
Method	GSM8K	Minerva Math	HumanEval	HumanEval+	MBPP	MBPP+	Avg.
AR initialization	85.90	69.18	59.15	53.05	68.20	83.33	69.80
AR mode
Llama-3.1-8B	54.06	18.22	35.37	28.66	48.80	61.90	41.17
Ministral3-8B	80.21	44.58	42.68	38.41	61.60	76.98	57.41
Qwen3-8B	86.73	52.94	64.63	56.71	69.40	83.07	68.91

Table 2: Performance of Sigma -8B (base model; block size: 32) on math reasoning and coding benchmarks. Accuracy (%) is pass@1. The Sigma diffusion rows are reported as mean (over 10 seeds)

\pm

two standard errors. Bold : the best value across all methods; underline : the best value within each mode.

Figure 6: Pass@

k

accuracy on MBPP. As

k

grows, Sigma prefers higher ST and overtakes NLD after

k=32

Figure 7: Performance vs. diffusion steps of Sigma -3B and Sigma -SUSReg-3B (block size: 32). All base models use DDIM sampling with uniform-

\gamma

time discretization, with best per-NFE (CFG, ST) shared across the three tasks. PDD models use their own DDIM-based sampling and a common (CFG, ST) setting inherited at distillation, with self-conditioning disabled in both the student and teacher. Results on all six base tasks: Tab. 8 .

	Math				Coding			Science & Knowledge		IF
Method	GSM8K	Math500	AIME24	AIME25	HumanEval	MBPP	LCB-CPP	MMLU	GPQA	IFEval	Avg.
Llama-3.2-3B-Instruct	62.09	32.60	0.00	0.00	65.85	66.93	9.25	62.45	27.78	69.69	39.66
Phi-4-mini-Instruct	76.57	70.80	10.00	6.67	77.44	75.13	9.91	53.37	40.91	71.16	49.20
Ministral3-3B-Instruct-2512	87.41	71.90	12.50	12.71	64.79	68.39	12.50	63.21	28.79	57.55	47.98
Qwen3-4B	92.19	68.40	10.63	10.63	75.91	72.49	15.58	77.05	37.18	72.20	53.23
AR mode

Table 3: Performance of Sigma -3B-Instruct (block size: 32) . Accuracy (%) is pass@1. The Sigma diffusion rows are reported as mean (over 10 seeds)

\pm

two SEs. The four reference models at the top are AR models. Baseline numbers are from Fu et al. (2026) . In diffusion mode , underline marks best value per task.

	Math				Coding			Science & Knowledge		IF
Method	GSM8K	Math500	AIME24	AIME25	HumanEval	MBPP	LCB-CPP	MMLU	GPQA	IFEval	Avg.
Qwen2.5-7B-Instruct	91.89	75.10	13.75	6.88	77.44	81.55	12.33	74.86	37.12	74.58	54.55
Qwen3-8B	92.42	84.80	30.21	22.08	81.71	81.88	21.09	76.66	49.24	87.38	62.75
Ministral3-8B-Instruct-2512	92.42	83.60	27.71	24.58	71.04	78.97	20.76	73.90	42.87	64.31	58.02
AR mode
Nemotron-Labs-Diffusion-8B-Instruct	94.01	88.00	33.33	33.33	80.49	85.19	28.85	79.85	44.44	68.65	63.61

Table 4: Performance of Sigma -8B-Instruct (block size: 32) . Accuracy (%) is pass@1. The Sigma diffusion rows are reported as mean (over 10 seeds)

\pm

two SEs. The three reference models at the top are AR models. NLD-8B-Instruct’s diffusion mode results are reproduced under aligned protocol, while other baseline numbers are from Fu et al. (2026, Tab. 5) . In diffusion mode , underline marks best value per task.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Density plot of a random $\mathbb{S}^{15}\to\mathbb{S}^{2}$ Mollweide projection of all learned embeddings living on $\mathbb{S}^{15}$ .

Panel of Figure 9 — Figure 9: Pre-training Sigma -3B with block size 32 and AR regularization from AR initialization (AR init) and from MDLM initialization (MDLM init). PPL for AR stream (a) and diffusion stream (b) are computed using $\mathcal{L}_{\text{ar}}({\mathbf{x}})$ 6 and $\mathcal{L}_{\text{BD}}({\mathbf{x}})$ 3 , respectively. MDLM initialization has an initial advantage that dies off during learning rate decay.

Sampler	Init	GSM8K	Minerva Math	HumanEval	HumanEval+	MBPP	MBPP+	Avg.
	AR	84.50 $\pm$ 0.31	64.60 $\pm$ 0.28	59.45 $\pm$ 1.30	53.84 $\pm$ 1.29	52.32 $\pm$ 0.73	71.32 $\pm$ 0.91	64.34
DDPM ( $256$ steps, $w{=}2,\tau{=}0.1$ )	MDLM	84.45 $\pm$ 0.35	65.37 $\pm$ 0.12	57.13 $\pm$ 1.05	52.87 $\pm$ 1.35	55.56 $\pm$ 0.65	72.62 $\pm$ 0.56	64.67
	$\Delta$	-0.05	+0.77	-2.32	-0.97	+3.24	+1.30	+0.33
	AR	81.91 $\pm$ 0.32	62.76 $\pm$ 0.18	56.59 $\pm$ 1.31	50.67 $\pm$ 1.00	47.26 $\pm$ 0.46	63.49 $\pm$ 0.77	60.45
DDIM ( $32$ steps, $w{=}3,\tau{=}0.3$ )	MDLM	83.71 $\pm$ 0.17	64.23 $\pm$ 0.28	52.99 $\pm$ 0.59	50.18 $\pm$ 0.68	51.24 $\pm$ 0.51	67.46 $\pm$ 0.58	61.64
	$\Delta$	+1.80	+1.47	-3.60	-0.49	+3.98	+3.97	+1.19

Appendix

Table 5: Diffusion initialization ablation for Sigma -3B pre-training. Same six generative reasoning tasks as Tab. 1 . The AR init rows are copied verbatim from Tab. 1 . The MDLM init rows use the same

300

B-token training recipe but warm-started from the Nemotron-Labs-Diffusion-3B checkpoint of Fu et al. (2026) .

\Delta=

MDLM

-

AR. All metrics are mean

\pm

two standard errors over

10

sampling lanes. Avg. is the unweighted mean over the six tasks.

Method	ARC-e	ARC-c	HellaSwag	PIQA	WinoGrande	MMLU	Avg.
Chance	25.00	25.00	25.00	50.00	50.00	25.00	33.33
AR initialization	80.39	57.42	71.74	78.18	70.64	70.74	71.52
AR mode
Nemotron-Labs-Diffusion-3B	78.70	55.46	70.39	78.24	70.32	69.61	70.46
Sigma -3B (Ours)	79.29	55.29	69.98	77.64	69.46	68.50	70.03
Diffusion mode

Appendix

Table 6: Likelihood-based multiple-choice benchmarks (3B Base). Accuracy (%) per task. AR mode ranks candidates with the AR likelihood from the AR stream; diffusion mode ranks them by the variational bound of the diffusion stream: 17 for Sigma , the count-based masked diffusion NELBO 19 for Nemotron-Labs-Diffusion. Underline : the best within each mode. Bold : the best value across all methods.

Method	ARC-e	ARC-c	HellaSwag	PIQA	WinoGrande	MMLU	Avg.
Chance	25.00	25.00	25.00	50.00	50.00	25.00	33.33
AR initialization	83.80	61.52	77.05	80.03	76.48	76.84	75.95
AR mode
Nemotron-Labs-Diffusion-8B	82.32	59.73	76.09	79.76	75.93	74.88	74.78
Sigma -8B (Ours)	82.28	60.67	75.70	80.41	76.48	74.58	75.02
Diffusion mode

Appendix

Table 7: Likelihood-based multiple-choice benchmarks (8B Base). Accuracy (%) per task. AR mode ranks candidates with the AR likelihood from the AR stream; diffusion mode ranks them by the variational bound of the diffusion stream: 17 for Sigma , the count-based masked diffusion NELBO 19 for Nemotron-Labs-Diffusion. Underline : the best within each mode. Bold : the best value across all methods.

	Math		Coding
Method (Diffusion Mode)	GSM8K	Minerva Math	HumanEval	HumanEval+	MBPP	MBPP+	Avg.	$\Delta$ Avg.
Nemotron-Labs-Diffusion-3B	83.40	63.96	56.10	53.05	57.60	74.34	64.74
DDIM 256 steps / block
Sigma -3B ( $w{=}3,\tau{=}0.2$ )	84.18 $\pm$ 0.18	64.00 $\pm$ 0.14	60.24 $\pm$ 0.84	54.02 $\pm$ 0.80	51.02 $\pm$ 0.42	69.79 $\pm$ 0.16	63.88
+ SUSReg ( $w{=}3,\tau{=}0.2$ )	84.12 $\pm$ 0.14	63.68 $\pm$ 0.18	57.32 $\pm$ 0.67	52.20 $\pm$ 0.74	56.08 $\pm$ 0.39	72.65 $\pm$ 0.31	64.34	+0.46
DDIM 32 steps / block

Appendix

Table 8: Low-NFE performance of Sigma -3B (base model; block size 32) with DDIM and SUSReg embedding regularization on math reasoning and coding benchmarks. Accuracy (%) is Pass@1. The Sigma rows are reported as mean (over 10 seeds)

\pm

two standard errors. All results use the uniform-

\gamma

schedule. Bold marks the best value per task; underline marks the better of Sigma -3B and

+

SUSReg within each step budget. For DDIM 256 steps, guidance settings were tuned according to the best 6-task average accuracy. For other runs (except PDD), settings were selected by tuning on 3-task average using GSM8K, HumanEval and MBPP. Within this table, all DDIM runs use SC; PDD does not use SC due to limitations discussed in Sec. 5.3 .

Figure 10: Sweeping CFG and ST for Sigma -3B trained with AR regularization. Accuracies are averaged over 10 random seeds. Red : per-task best; orange : within 1 point of per-task best; green : best average across 6 tasks. For this model, per-task best and best average align on 5 out of 6 tasks. In the exception of Minerva Math (subset; random $10\%$ from each subtask), per-task best is still within 1 point from best average. We report best average ( $\text{CFG}=2$ , $\text{ST}=0.1$ ) in Fig. 5 and Tab. 1 .

Figure 11: Sweeping CFG and ST for Sigma -3B trained without AR regularization. Accuracies are averaged over 10 random seeds. Red : per-task best; orange : within 1 point of per-task best; green : best average across 6 tasks. Both per-task best and best average are worse than Sigma -3B trained AR regularization ( Fig. 10 ).

Setting	Guidance space	$w=1$	$w=2$	$w=3$	$w=4$	$w=5$	Tuned best
GSM8K, DDIM $64$ steps, $\tau=0.3$	Logit	68.13 $\pm$ 0.38	83.49 $\pm$ 0.22	81.55 $\pm$ 0.41	77.08 $\pm$ 0.41	67.78 $\pm$ 0.53	83.49 $\pm$ 0.22 ( $w=2$ )
GSM8K, DDIM $64$ steps, $\tau=0.3$	Embedding (ours)	68.13 $\pm$ 0.38	82.52 $\pm$ 0.21	82.84 $\pm$ 0.28	83.04 $\pm$ 0.20	83.19 $\pm$ 0.18	83.19 $\pm$ 0.18 ( $w=5$ )
GSM8K, DDPM $256$ steps, $\tau=1$	Logit	53.09 $\pm$ 0.85	76.76 $\pm$ 0.50	75.16 $\pm$ 0.79	69.48 $\pm$ 0.59	60.77 $\pm$ 0.99	76.76 $\pm$ 0.50 ( $w=2$ )
GSM8K, DDPM $256$ steps, $\tau=1$	Embedding (ours)	53.09 $\pm$ 0.85	77.12 $\pm$ 0.66	80.04 $\pm$ 0.39	80.99 $\pm$ 0.48	81.30 $\pm$ 0.41	82.43 $\pm$ 0.50 ( $w=10$ )
HumanEval, DDIM $64$ steps, $\tau=0.3$	Logit	42.93 $\pm$ 0.66	53.78 $\pm$ 1.09	39.15 $\pm$ 1.13	20.67 $\pm$ 0.84	12.68 $\pm$ 0.47	53.78 $\pm$ 1.09 ( $w=2$ )
HumanEval, DDIM $64$ steps, $\tau=0.3$	Embedding (ours)	42.93 $\pm$ 0.66	53.96 $\pm$ 0.95	56.04 $\pm$ 1.16	56.22 $\pm$ 0.83	54.09 $\pm$ 1.22	56.22 $\pm$ 0.83 ( $w=4$ )

Appendix

Table 9: Logit-space versus embedding-space CFG. GSM8K is accuracy (%) and HumanEval is execution-based pass@1 (%). Entries are mean

\pm

two standard errors over the 10 generations per prompt, an approximate

95\%

confidence interval for the mean, and bold marks the best entry in each row. The last column is the best value over the full sweep of Tabs. 12 and 13 , which continues past

w=5

for the embedding rows. The logit rows were swept only to

w=5

as they already peak at

w=2

and fall monotonically after that.

$T$	$1$	$\frac{1}{2}$	$\frac{1}{3}$	$\frac{1}{4}$	$\frac{1}{5}$	$\frac{1}{10}$	$0$ (MAP)
LT at $T$	52.44 $\pm$ 1.00	76.88 $\pm$ 0.62	79.07 $\pm$ 0.38	78.97 $\pm$ 0.50	79.11 $\pm$ 0.32	78.74 $\pm$ 0.23	78.61 $\pm$ 0.33
LCFG at $w=\frac{1}{T}$	52.44 $\pm$ 1.00	76.76 $\pm$ 0.50	75.16 $\pm$ 0.79	69.48 $\pm$ 0.59	60.77 $\pm$ 0.99	—	—
$\Delta$ (LT $-$ LCFG)	0.00	+0.12	+3.91	+9.49	+18.34	—	—

Appendix

Table 10: Logit temperature, GSM8K accuracy (%) with DDPM,

256

steps/block, EDM

\rho=7

schedule, no CFG. Score entries are mean

\pm

two standard errors over the 10 generations per prompt, an approximate

95\%

confidence interval for the mean.

T=1

is the unsharpened MMSE denoiser and is the same configuration as the

w=1

logit-space entry of Tab. 9 . The bottom row is LCFG at the matched weight

w=\frac{1}{T}

, repeated from Tab. 9 ; the two are the same operation if the unconditional logits are uniform over the vocabulary, so the gap isolates what logit-space guidance adds over plain sharpening.

$T$	$1$	$\frac{1}{2}$	$\frac{1}{3}$	$\frac{1}{5}$	$0$ (MAP)
No guidance ( $w=1$ )	68.18 $\pm$ 0.33	77.60 $\pm$ 0.23	77.42 $\pm$ 0.35	76.13 $\pm$ 0.31	71.83 $\pm$ 0.24
Embedding CFG at $w=5$	83.19 $\pm$ 0.18	81.44 $\pm$ 0.19	81.49 $\pm$ 0.25	79.96 $\pm$ 0.31	76.84 $\pm$ 0.44

Appendix

Table 11: Composing logit temperature with the other two knobs: GSM8K accuracy (%) with DDIM,

64

steps/block, uniform-

\gamma

schedule, ST

\tau=0.3

. Entries are mean

\pm

two standard errors over the 10 generations per prompt, an approximate

95\%

confidence interval for the mean. Bold marks the best entry in each row.

Sampler	Nom. NFE	Steps	Stride	GSM8K	Minerva Math	HumanEval	HumanEval+	MBPP	MBPP+	Avg.
DDPM ( $w{=}2,\tau{=}0.1$ )	256	256	32	84.50 $\pm$ 0.31	64.60 $\pm$ 0.28	59.45 $\pm$ 1.30	53.84 $\pm$ 1.29	52.32 $\pm$ 0.73	71.32 $\pm$ 0.91	64.34
DDIM ( $w{=}3,\tau{=}0.2$ )	256	128	16	85.70 $\pm$ 0.15	64.68 $\pm$ 0.20	60.67 $\pm$ 0.80	54.88 $\pm$ 0.84	55.52 $\pm$ 0.36	72.14 $\pm$ 0.34	65.60
DDIM ( $w{=}3,\tau{=}0.2$ )	256	64	8	84.94 $\pm$ 0.17	64.73 $\pm$ 0.18	61.52 $\pm$ 0.76	56.10 $\pm$ 0.62	58.86 $\pm$ 0.35	75.87 $\pm$ 0.40	67.00
DDIM ( $w{=}3,\tau{=}0.2$ )	128	128	32	83.78 $\pm$ 0.11	63.89 $\pm$ 0.13	59.15 $\pm$ 0.68	53.17 $\pm$ 0.87	50.48 $\pm$ 0.21	68.12 $\pm$ 0.51	63.10
DDIM ( $w{=}3,\tau{=}0.2$ )	128	64	16	85.82 $\pm$ 0.09	64.24 $\pm$ 0.14	60.73 $\pm$ 0.83	54.63 $\pm$ 0.85	54.06 $\pm$ 0.30	71.56 $\pm$ 0.44	65.17
DDIM ( $w{=}3,\tau{=}0.3$ )	128	32	8	84.17 $\pm$ 0.29	64.44 $\pm$ 0.22	62.13 $\pm$ 0.63	56.34 $\pm$ 0.49	57.56 $\pm$ 0.61	74.68 $\pm$ 0.50	66.55

Appendix

Table 14: Commit-stride ablation on Sigma -3B. Same six generative reasoning tasks as Tab. 1 . Nom. NFE is the nominal number of forward passes per

32

committed tokens (i.e.,

\text{steps}/\text{stride}\times 32

). All DDIM rows use the uniform-

\gamma

schedule; the DDPM row uses the EDM

\rho=7

schedule. Rows within each nominal-NFE bucket are directly compute-comparable. Mean

\pm

two standard errors across

10

sampling lanes. Bold marks the best value per column across the stride sweep.

Sampler	Nom. NFE	Steps	Stride	GSM8K	Minerva Math	HumanEval	HumanEval+	MBPP	MBPP+	Avg.
DDPM ( $w{=}2,\tau{=}0.1$ )	256	256	32	89.07 $\pm$ 0.28	67.47 $\pm$ 0.20	61.40 $\pm$ 1.73	57.44 $\pm$ 1.62	64.78 $\pm$ 0.54	77.83 $\pm$ 0.72	69.66
DDIM ( $w{=}3,\tau{=}0.2$ )	256	128	16	88.64 $\pm$ 0.11	67.59 $\pm$ 0.10	60.91 $\pm$ 0.64	55.67 $\pm$ 0.60	67.44 $\pm$ 0.35	81.75 $\pm$ 0.33	70.33
DDIM ( $w{=}3,\tau{=}0.2$ )	256	64	8	90.17 $\pm$ 0.15	68.82 $\pm$ 0.20	60.49 $\pm$ 0.98	53.96 $\pm$ 0.93	67.72 $\pm$ 0.21	81.35 $\pm$ 0.28	70.42
DDIM ( $w{=}3,\tau{=}0.2$ )	128	128	32	88.70 $\pm$ 0.14	67.28 $\pm$ 0.09	58.48 $\pm$ 1.00	52.68 $\pm$ 1.00	63.22 $\pm$ 0.28	76.85 $\pm$ 0.31	67.87
DDIM ( $w{=}3,\tau{=}0.2$ )	128	64	16	88.76 $\pm$ 0.17	67.63 $\pm$ 0.17	59.33 $\pm$ 1.06	53.54 $\pm$ 0.94	66.62 $\pm$ 0.42	80.90 $\pm$ 0.31	69.46
DDIM ( $w{=}3,\tau{=}0.3$ )	128	32	8	89.61 $\pm$ 0.24	68.11 $\pm$ 0.15	61.22 $\pm$ 0.55	55.18 $\pm$ 0.69	66.22 $\pm$ 0.49	80.19 $\pm$ 0.34	70.09

Appendix

Table 15: Commit-stride ablation on Sigma -8B. Same six tasks as Tab. 2 ; conventions match Tab. 14 . Bold marks the best value per column across the stride sweep.

Figure 12: Left: the learned noise schedule

\gamma(t)

for Sigma -3B. Right: Different time discretization schemes for 16-step generation. Note that an extra decoding step is needed at

t=0

Figure 13: DDIM substantially outperforms DDPM at low NFEs, while matching DDPM at high NFEs.

Figure 14: Best CFG-ST configuration vs. diffusion sampling steps for the best average score of GSM8K & HumanEval under uniform-

\gamma

discretization.

Task	Steps	NFE	DDIM	DPM-Solver++(2M)	$\rho$ AB1-DEIS	$\rho$ AB2-DEIS	$\rho$ RK-DEIS
GSM8K	8	9	66.49 $\pm$ 0.52	22.77 $\pm$ 0.92	61.78 $\pm$ 0.46	48.95 $\pm$ 1.04	2.88 $\pm$ 0.20
GSM8K	16	17	78.08 $\pm$ 0.34	33.18 $\pm$ 0.60	48.96 $\pm$ 0.98	2.65 $\pm$ 0.37	0.86 $\pm$ 0.15
HumanEval	8	9	29.09 $\pm$ 1.59	12.07 $\pm$ 1.57	24.45 $\pm$ 1.66	20.24 $\pm$ 1.37	3.72 $\pm$ 1.10
HumanEval	16	17	47.50 $\pm$ 1.14	22.01 $\pm$ 1.49	33.48 $\pm$ 1.39	1.58 $\pm$ 0.58	1.77 $\pm$ 0.56
MBPP	8	9	22.96 $\pm$ 1.06	0.48 $\pm$ 0.17	9.48 $\pm$ 0.61	4.66 $\pm$ 0.46	0.00 $\pm$ 0.00
MBPP	16	17	42.36 $\pm$ 0.66	0.36 $\pm$ 0.18	6.58 $\pm$ 0.52	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00

Appendix

Table 16: Higher-order solver comparison on Sigma -3B. “Steps” is the number of denoiser calls in the sampling loop, and NFE

=

Steps

+1

is the total denoiser count including the final

\argmax

decoding; both are matched across methods within each row. Each entry is mean

\pm

two standard errors over 10 responses per question, forming an approximate

95\%

confidence interval. Bold marks the best method in each row.

Figure 15: Endpoint geometry under low-step ODE solvers. (a) Terminal distance from the unit-sphere vocabulary manifold. (b) Fifth-percentile cosine similarity to the nearest vocabulary embedding. At

8

steps, higher-order extrapolation leaves endpoints far off-manifold and between vocabulary vertices, whereas these differences largely disappear by

32

steps.

Figure 16: Pass@

k

on MBPP. Left: at each

k

, each family uses the best configuration in its tested sampler grid; Sigma overtakes NLD at

k=32

and reaches

0.918

versus

0.876

k=256

. Right: one fixed high-coverage configuration per family, selected by the one reaching the highest pass@256 accuracy: Sigma

(w=2,\tau=1.0)

and NLD

(\vartheta=0.9,T=1.0)

. The

+4.2

point improvement at

k=256

has a paired

95\%

CI of

[+1.7,+6.7]

, with the interval excluding zero from

k=64

onward.

Figure 17: Pass@

k

on Minerva Geometry. Left: the best tested configuration in each family at every

k

; Sigma leads from

k=8

onward and reaches

0.954

versus

0.942

k=512

. Right: fixed high-coverage configurations Sigma

(w=2,\tau=1.0)

and NLD

(\vartheta=0.9,T=1.0)

. Their

+1.3

point difference at

k=512

is directional but unresolved, with a paired 95% CI of

[-0.3,+2.8]

Sigma sampling configuration	$k=1$	$k=8$	$k=32$	$k=64$	$k=128$	$k=256$	$k=512$
$w=3,\tau=0.7$	$-0.0080$	$-0.002$	$+0.001$	$0.000$	$-0.003$	$-0.002$	$+0.006$
$w=4,\tau=0.7$	$-0.0025$	$-0.001$	$+0.002$	$+0.001$	$-0.001$	$-0.004$	$-0.004$
$w=5,\tau=0.7$	$+0.0005$	$+0.001$	$+0.005$	$+0.004$	$-0.001$	$-0.005$	$-0.004$

Appendix

Table 17: Sigma ties the best-tuned NLD sampler

(\vartheta=0.9,T=1)

at matched pass@1 on Minerva Geometry. Entries are paired differences in pass@

k

( Sigma

-

NLD) with 512 independent generations per problem for every setting.

Sigma vs. NLD	$\Delta$ pass@1	$\Delta$ pass@256	95% CI
$w=4,\tau=0.7$ vs. $\vartheta=0.9,T=0.3$	$-0.0103$	$\mathbf{+0.066}$	$[+0.044,+0.088]$
$w=5,\tau=0.7$ vs. $\vartheta=0.9,T=0.3$	$-0.0073$	$\mathbf{+0.065}$	$[+0.042,+0.088]$
$w=3,\tau=0.7$ vs. $\vartheta=0.9,T=0.6$	$-0.0115$	$\mathbf{+0.024}$	$[+0.008,+0.041]$
$w=4,\tau=0.7$ vs. $\vartheta=0.9,T=0.6$	$-0.0064$	$\mathbf{+0.020}$	$[+0.003,+0.037]$
$w=5,\tau=0.7$ vs. $\vartheta=0.9,T=0.6$	$-0.0034$	$\mathbf{+0.019}$	$[+0.002,+0.036]$

Appendix

Table 18: Sigma outperforms lower-temperature NLD samplers at matched pass@1 on Minerva Geometry. The final columns report the paired pass@256 difference ( Sigma

-

NLD) and its 95% confidence interval.

Model	Configuration	Pass@1	Pass@8	Pass@32	Pass@64	Pass@256	Solved
Sigma	$w{=}2,\tau{=}1.0$	0.651	0.942	0.980	0.988	0.996	1314
NLD	$\vartheta{=}0.9,T{=}1.0$	0.823	0.950	0.973	0.981	0.989	1305
Sigma	$w{=}3,\tau{=}0.7$	0.806	0.938	0.965	0.974	0.987	1302
NLD	$\vartheta{=}0.9,T{=}0.3$	0.836	0.913	0.937	0.946	0.964	1271
Sigma	$w{=}2,\tau{=}0.4$	0.799	0.878	0.906	0.916	0.931	1228

Appendix

Table 19: Full pass@

k

results on GSM8K , sorted by pass@256. “Solved” is the number of problems answered correctly at least once in 256 generations.

Figure 18: Sigma overtakes NLD in the GSM8K hard tail. Best-of-family fixed configuration: Sigma

(w{=}2,\tau{=}1.0)

versus NLD’s best sampler

(\vartheta{=}0.9,T{=}1.0)

. Left: the pass@

k

curves; GSM8K saturates high, so the curves cross near

k=16

and the gap is small. Right: the paired per-problem difference ( Sigma

-

NLD) with its

95\%

CI, where the effect is legible: negative at

k\leq 8

, then positive with the interval excluding zero at every

k\geq 32

Figure 19: Sigma leads NLD at every

k

on AIME 2025, with a margin that grows with

k

. Left: all eight tested configurations at

n=256

; the bold curve of each family is its best setting by pass@256. Four Sigma settings exceed every NLD setting at

k=256

, and the fifth ties NLD’s best (

\vartheta=0.9,T=0.2

). Right: paired per-problem differences against each NLD temperature, with the

95\%

interval of the highlighted pair shaded. The lead is consistent in direction at every

k

against all three NLD samplers.

Figure 20: Sigma tend to commit early positions of a block later than MDLM and the final positions earlier. The sampler used is 256-step DDPM (

w=2,\tau=0.1

) as in Tab. 1 . Generation order averaged across multiple documents and blocks. For Sigma , the commitment step

t_{i}

of position

i

is defined as the smallest

t

such that for every step

t^{\prime}\geq t

up to the final readout, (i) the argmax of

\mathbf{x}_{\theta}

equals the finally decoded token, and (ii) that argmax is assigned probability at least

0.9

. The commitment order of position

i

is the rank of

t_{i}

among

{t_{1},\dots,t_{32}}

within its block, in ascending order. Positions sharing a commitment step share the lowest rank of the tie.

Block Size	Stage	No CFG (ms)	Packed CFG (ms)	$\Delta$	$\Delta\ (\%)$
$L^{\prime}=32$	Transformer forward	$18.06$	$18.38$	$+0.322\pm 0.181$	$+1.8$
	Logit head + cat. reparam.	$1.02$	$1.50$	$+0.481\pm 0.008$	$+47.2$
	Total per NFE	$19.08$	$19.89$	$+0.804\pm 0.186$	$+4.2$
$L^{\prime}=64$	Transformer forward	$18.23$	$18.21$	$-0.021\pm 0.159$	$-0.1$
	Logit head + cat. reparam.	$1.08$	$1.60$	$+0.519\pm 0.006$	$+48.3$
	Total per NFE	$19.31$	$19.80$	$+0.498\pm 0.162$	$+2.6$

Appendix

Table 20: Wall-clock cost per denoiser evaluation (NFE) of packing the conditional and unconditional CFG branches into one forward pass, at batch size

1

. Times are in milliseconds on H100.

\Delta

is the time cost of packed conditional and unconditional predictions (packed CFG)

-

only conditional prediction (no CFG), reported as mean

\pm

two standard errors over the

16

questions, an approximate

95\%

confidence interval for the mean. The transformer forward row shows that doubling the query length from

L^{\prime}

2L^{\prime}

leaves both the KV cache and the weight traffic unchanged, so the extra work hides entirely under the memory-bandwidth floor. The residual cost is when passing through the logit head and computing the categorical reparameterization

{\mathbf{e}}_{\theta}=\operatorname{softmax}({\bm{\ell}}_{\theta}){\mathbf{E}}

, which is independent of how the two branches are batched.

Setting	Value
Sequence length / block length	$4096$ / $32$
Global / micro batch size	$256$ / $1$ sequence
Training duration	$8000$ steps ( $\approx 2.4$ epochs)
Optimizer	AdamW, $(\beta_{1},\beta_{2})=(0.9,0.98)$ , $\epsilon=10^{-5}$
Learning rate / weight decay	$2.5\times 10^{-6}$ / $0.1$
Schedule	$50$ -step linear warmup, then constant

Appendix

Table 21: SFT optimization settings for Sigma .

	Math				Coding			Multiple Choice		IF
Configuration	GSM8K	Math500	AIME24	AIME25	HumanEval	MBPP	LCB-CPP	MMLU	GPQA	IFEval	Avg.
DDPM ( $256$ steps, $w{=}2,\tau{=}0.1$ )
Frozen (default)	89.18 $\pm$ 0.24	70.44 $\pm$ 0.79	17.00 $\pm$ 3.05	11.00 $\pm$ 2.12	77.62 $\pm$ 1.31	73.10 $\pm$ 0.75	11.17 $\pm$ 0.28	71.28 $\pm$ 0.08	34.70 $\pm$ 1.14	66.04 $\pm$ 0.60	52.15
Unfrozen ( $1000\times$ LR)	88.39 $\pm$ 0.21	72.62 $\pm$ 0.88	13.00 $\pm$ 2.57	11.67 $\pm$ 2.36	75.61 $\pm$ 1.42	72.33 $\pm$ 0.92	11.37 $\pm$ 0.40	72.00 $\pm$ 0.15	33.28 $\pm$ 1.27	66.22 $\pm$ 0.37	51.65
DDIM ( $32$ steps, $w{=}3,\tau{=}0.3$ )
Frozen (default)	87.86 $\pm$ 0.29	68.30 $\pm$ 0.67	9.67 $\pm$ 2.20	14.00 $\pm$ 2.46	72.74 $\pm$ 1.22	68.84 $\pm$ 0.78	10.11 $\pm$ 0.31	69.15 $\pm$ 0.11	31.82 $\pm$ 1.82	66.09 $\pm$ 0.22	49.86

Appendix

Table 22: Freezing the continuous-diffusion geometry during SFT. 3B instruct performance, mean

\pm

two standard errors over

10

sampling lanes, under both sampler operating points of Tab. 3 . Frozen is the default of Tab. 3 : the noise schedule, its endpoints, and the diffusion embedding matrix are held fixed. Unfrozen trains all three at a

1000\times

learning-rate multiplier. Avg. is the unweighted mean over the ten tasks. The frozen/unfrozen ordering of the average is not stable across the two samplers, and in general the gap is within the seed interval.

Continuous Diffusion Language Models

Continuous diffusion language models generate text by gradually denoising continuous token embeddings or latent representations.

See all 48 papers

Oct 1, 2026

Hierarchical Continuous Diffusion Language Models

Discrete diffusion language models offer a compelling alternative to autoregressive generation for tasks demanding bidirectional reasoning and global constraint satisfaction. Yet they share a structural bottleneck: when decoding in parallel, each token is sampled independently from its marginal, severing the statistical dependencies among the tokens decoded together. Continuous diffusion language models avoid this by denoising a shared continuous state, but their denoiser sees only that state, so nothing ties it to a valid token configuration until it is finally decoded. To address this, we propose Hierarchical Continuous Diffusion Language Models (HC-DLM), which couple discrete token generation with a continuous latent trajectory in a single, principled denoising process, whose training objective is derived from a variational bound on the token likelihood. In contrast to recent methods that attach continuous context to a self-contained discrete chain, HC-DLM makes the latent the only persistent generative state: tokens are read out from it at every step and feed back as a scaffold for the next latent update. On structured reasoning (Sudoku), mathematical planning (Countdown) and language modeling (LM1B), HC-DLM improves over discrete and continuous diffusion baselines at matched model size, in puzzle accuracy on Sudoku and Countdown and in generative perplexity on LM1B. Project page: https://hc-dlm.github.io/.

Hui Ren, Zihan Li, Chang Liu +2

University of Illinois Urbana-Champaign · Amazon.com, Inc.

Structured Reasoning in Language Models Diffusion Language Model Decoding Diffusion Language Models Continuous Diffusion Language Models

Ask PDF

Apr 13, 2026

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow

Yuxin Chen, Chumeng Liang, Hangke Sui +4

UIUC

Self-Conditioning Diffusion Language Models Continuous Diffusion Language Models Diffusion Language Model Training Flow Matching

Ask PDF

Mar 3, 2026

CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think

We study why continuous diffusion language models (DLMs) have lagged behind discrete diffusion approaches despite their appealing continuous generative dynamics. Under a controlled token--recovery study, we identify token rounding, the final projection from denoised embeddings to tokens, as a primary bottleneck. Building on these insights, we propose CoDAR (Continuous Diffusion with Contextual AutoRegressive Decoder), a two--stage framework that keeps diffusion entirely continuous in an embedding space while learning a strong, context--conditional discretizer: an autoregressive Transformer decoder that cross--attends to the denoised embedding sequence and performs contextualized rounding to tokens. Experiments on LM1B and OpenWebText demonstrate that CoDAR substantially improves generation quality over latent diffusion and becomes competitive with strong discrete DLMs, while exposing a simple decoder--temperature knob to navigate the fluency--diversity trade off.

Junzhe Shen, Jieru Zhao, Ziwei He +1

LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · School of Computer Science, Shanghai Jiao Tong University · Shanghai Innovation Institute +1

Continuous Diffusion Language Models Diffusion Language Model Decoding

Ask PDF

Abstract

Continuous Diffusion Language Models

Explore similar work

Hierarchical Continuous Diffusion Language Models

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think