Simplex Diffusion Models

Justin Deschenaux; Alexandre Galashov; Andrew Campbell; Li Kevin Wenliang; James Thornton; Arnaud Doucet; Valentin De Bortoli

cs.LGSep 28, 2026

Simplex Diffusion Models

Authors: Justin Deschenaux, Alexandre Galashov, Andrew Campbell, Li Kevin Wenliang, James Thornton, Arnaud Doucet, Valentin De Bortoli

Organizations: Google DeepMind · EPFL · UCL Gatsby

Abstract

Diffusion models have revolutionized generative modeling for continuous data through the gradual refinement of a belief state. This iterative refinement has not yet carried over to discrete diffusion models, which discard uncertainty at intermediate steps through categorical sampling (information collapse). We propose Simplex Diffusion Models (SDMs), a framework that lifts the diffusion process to the probability simplex to represent beliefs over categories. SDMs admit probability paths with closed-form reverse transitions and can be trained with a simple cross-entropy loss. Contrary to earlier proposals such as Dirichlet Flow Matching which requires integrating an ordinary differential equation, we introduce a DDIM-like sampler with a tunable level of stochasticity. Because SDMs operate on samples on the simplex, they can carry uncertainty across denoising steps, which mitigates information collapse. On OpenWebText, SDMs are competitive with strong Discrete Diffusion baselines, achieving $17.0$ GenPPL at $5.46$ unigram entropy in 64 sampling steps, close to real validation data. Even without Self-Conditioning (SC), SDMs outperform masked and uniform diffusion (with SC or predictor-corrector sampling) on code generation (TinyGSM, $T=0.1$ ; $49.0\%$ vs. $45.8\%$ ). Distilled down to 8 steps, SDMs solve $32.1\%$ of GSM8K problems, more than distilled Discrete Diffusion models with 128 steps ( $21.4\%$ ).

Figures & tables

Figure 1: SDMs outperform Discrete Diffusion on TinyGSM. Left: Accuracy as a function of the number of function evaluation (NFE) at low temperature (

T=0.1

). Right: Accuracy against NFEs at

T=1

. Without Self-Conditioning (SC), SDMs with the expected embedding outperform masked diffusion with SC (MDM + SC) and uniform diffusion with predictor-corrector sampling (UDM + PC). At 512 NFE, SDMs reach

49.0\%

T=0.1

, vs.

45.8\%

for MDM ‡ + SC, and

45.8\%

T=1

, vs.

36.8\%

for UDM † + PC. With SC, SDM (Argmax + SC) is the best diffusion variant at

T=0.1

from 32 NFE onwards, and solves

57\%

of problems with 8k NFEs. Autoregressive models with greedy decoding is best (62.6%). All SDMs use churn

\kappa=1

and the adaptive sampling time grid ( Section J.1 ), and we compare with the best baseline configurations ( Section 5.2 ). † / ‡ : trained with the uniform/adaptive time sampler. The accuracy drop for UDM at 16k is consistent with typical floating-point rounding errors, which can accumulate at very small step sizes ( Karras et al., 2022 ) .

Figure 2: Left to right: different stages of a backward step during inference . We assume that at the current step,

P_{0}=e_{x_{0}}

where

x_{0}\sim\hat{P}_{\theta}(t,P_{t})

is such that

P_{0}=e_{1}

. The top row corresponds to a low churn

\kappa=0.15

and the bottom row corresponds to a high churn of

\kappa=0.65

. We assume an interpolation schedule

\alpha_{t}=1-t

and concentration schedule

c_{t}=\varepsilon/(1-\alpha_{t})

with

\varepsilon=4.0

. Additionally,

s=0.40

and

t=0.70

. The dots with colors yellow, orange, red correspond to three different samples from the current step. In the first step , using the churn

\kappa\in[0,1]

, we sample

{{\color[rgb]{0.0508,0.2773,0.6328}P_{s,t}^{\kappa}}}

. This corresponds to a multiplicative noising step around

P_{t}

, with a higher churn corresponding to a higher noise. In the second step , we sample the innovation variable

{{\color[rgb]{0.7188,0.1094,0.1094}V_{s,t}^{\kappa}}}

, a Dirichlet random variable concentrated around the prediction

P_{0}

. In the third step we sample from a

\mathrm{Beta}

random variable

{{\color[rgb]{0.1055,0.3672,0.125}W_{s,t}^{\kappa}}}

corresponding to the mixing weight in the interpolation between the thinned variable

{{\color[rgb]{0.0508,0.2773,0.6328}P_{s,t}^{\kappa}}}

and the innovation

{{\color[rgb]{0.7188,0.1094,0.1094}V_{s,t}^{\kappa}}}

. For high churns, the mixing weight

{{\color[rgb]{0.1055,0.3672,0.125}W_{s,t}^{\kappa}}}

is close to

0

, meaning that we favor the innovation random variable

{{\color[rgb]{0.7188,0.1094,0.1094}V_{s,t}^{\kappa}}}

. Finally in the fourth step , we visualize the interpolation between the innovation

{{\color[rgb]{0.7188,0.1094,0.1094}V_{s,t}^{\kappa}}}

and the thinned variable

{{\color[rgb]{0.0508,0.2773,0.6328}P_{s,t}^{\kappa}}}

using the sampled

\mathrm{Beta}

mixing weight

{{\color[rgb]{0.1055,0.3672,0.125}W_{s,t}^{\kappa}}}

Discrete
Model	Linear	Cosine
AR (Greedy)	$\hphantom{0}3.1_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.5}}$	–
MDM ‡	$69.4_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4.8}}$	$69.7_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4.0}}$
MDM ‡ (+SC)	$94.7_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.8}}$	$\underline{99.3}_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.2}}$
UDM ‡	$82.1_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.7}}$	$82.4_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.7}}$
UDM † (+SC)	$97.8_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.7}}$	$98.1_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.2}}$

Table 1: Accuracy (%) on Sudoku in 180 steps, varying the sampling schedule. † trained with uniform time sampler, ‡ trained with adaptive time sampler. For each column, we underline the best and bold the highest accuracy for continuous methods. We show SDM with normalized variance

\nu_{t}^{\text{cst.-lin.}}

, with

\nu_{0}=0.4

\nu_{1}=0.75

\ell=0.8

. Values are mean

\pm

std over 5 seeds.

Figure 3: OWT Pareto Frontiers of Entropy

H_{1}

vs. GPT-2 GenPPL. Dotted crosshairs and (

\bigstar

) denote measured OpenWebText validation data (

H_{1}=5.46\text{ nats}

\text{Gen PPL}=15.2

). Left: Head-to-head Pareto frontiers comparing 64-step SDM , MDM , and UDM against 1024-step AR after sampler interventions. Right: Progressive ablation on 64-step SDM showing gains from Stage A (nucleus

\text{top-}p

)

\to

Stage B (power-law logit temperature annealing)

\to

Stage C (sequence frequency penalty)

\to

Stage D (local frequency penalty).

Appendix figures & tables51 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Visualization of the Self-Conditioning weights

L_{i,j}

from Section E.4 . Each panel shows the exact mean profile

\mathbb{E}[L_{i,j}]

(solid line) together with a

\pm 1

standard-deviation band for the Dirichlet law in Section E.4 , for

\varepsilon\in\{0.01,1,100\}

and

i\in\{0,5,10,15\}

. The mean curve is independent of

\varepsilon

Figure 6: Each figure corresponds to the density of $p_{t|0}(P_{t}|P_{0})=\mathrm{Dir}(c_{t}(\alpha_{t}P_{0}+(1-\alpha_{t})\pi))$ , with $\pi$ the uniform distribution and $c_{t}=\varepsilon/(1-\alpha_{t})$ for different values of $t\in[0,1]$ and $\varepsilon>0$ . We also display the mean (red star) of $p_{t|0}$ , i.e., $\mathbb{E}[P_{t}|P_{0}]=\alpha_{t}P_{0}+(1-\alpha_{t})\pi$ and observe that this quantity is independent of $\varepsilon$ . Finally, we also display 5 samples (white dots) of $p_{t|0}$ . Note that as $\varepsilon\rightarrow 0$ the samples concentrate on the vertices of the simplex, thereby illustrating the convergence of SDMs to their Discrete Diffusion counterpart.

Panel of Figure 7 — Figure 7: The design space of Simplex Diffusion Models (SDMs). The forward process is controlled by the inverse temperature $\varepsilon$ : as $\varepsilon\to 0$ , the simplex mass concentrates on vertices and recovers discrete-diffusion behavior; as $\varepsilon\to\infty$ , the forward process concentrates around the deterministic interpolation $\bar{P}_{t}=\alpha_{t}P_{0}+(1-\alpha_{t})\pi$ . The reverse process is controlled by the churn parameter $\kappa$ : $\kappa=0$ fully trusts the current belief $P_{t}$ , while $\kappa=1$ yields the independent bridge $p_{s\mid 0,t}=p_{s\mid 0}$ . SDMs operate in the intermediate regime, maintaining a continuous belief state throughout sampling.

Model	Linear
Continuous
DFM (reimpl.)	$76.7_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.75}}$
SDM ( $\kappa=1.0$ )	$88.4_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 2.2}}$
SDM (+SC; $\kappa=1.0$ )	$\underline{\textbf{99.1}}_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.2}}$

Appendix

Table 2: Accuracy (%) on Sudoku in 180 steps comparing Dirichlet Flow Matching and our SDM method. SDMs use

\nu_{t}^{\text{cst.-lin.}}

with

\nu_{0}=0.4

\nu_{1}=0.75

\ell=0.8

, as in Table 1 .

Figure 8: Churn

\kappa

vs. TinyGSM accuracy at

T=1.0

with 512 steps for the Linear (left), Cosine (center) and Adaptive (right) sampling schedules. We compare two concentration schedules, Const.-Lin. (

\nu_{0}=0.2,\nu_{1}=0.5,\ell=0.2

, orange) and Constant (

\nu=0.5

, indigo), and three input variants: Argmax + SC (

\star

), Argmax (

\blacksquare

) and Expectation

P_{t}

(

\bullet

). Horizontal lines show AR decoding (greedy:

62.6\%

; sampling:

52.6\%

) and UDM with the Predictor-Corrector (PC) sampler. Accuracy generally increases with

\kappa

. With the adaptive schedule, Const.-Lin. with the expectation input rises from

12.6\%

\kappa=0

\mathbf{45.8\%}

\kappa=1

9.0

points above the best UDM with PC (

36.8\%

). Exact values for

\kappa\in\{0,0.2,1\}

are in Table 29 .

Figure 9: Churn

\kappa

vs. TinyGSM accuracy at

T=0.1

with 512 steps for the Linear (left), Cosine (center) and Adaptive (right) sampling schedules. We compare two concentration schedules, Const.-Lin. (

\nu_{0}=0.2,\nu_{1}=0.5,\ell=0.2

, orange) and Constant (

\nu=0.5

, indigo), and three input variants: Argmax + SC (

\star

), Argmax (

\blacksquare

) and Expectation

P_{t}

(

\bullet

). Horizontal lines show AR decoding (greedy:

62.6\%

; sampling:

52.6\%

) and UDM with the Predictor-Corrector (PC) sampler. Accuracy generally increases with

\kappa

, and Argmax + SC is the strongest input under the linear and cosine schedules. At

\kappa=1

, Constant with Argmax + SC reaches

\mathbf{50.9\%}

(cosine), above the best UDM with PC (

45.5\%

). With the adaptive schedule, Const.-Lin. with the expectation input rises from

19.3\%

\kappa=0

49.0\%

\kappa=1

. Exact values for

\kappa\in\{0,0.2,1\}

are in Table 31 .

Figure 10: Churn

\kappa

vs. TinyGSM accuracy at

T=1.0

with 64 steps for the Linear (left), Cosine (center) and Adaptive (right) sampling schedules. We compare two concentration schedules, Const.-Lin. (

\nu_{0}=0.2,\nu_{1}=0.5,\ell=0.2

, orange) and Constant (

\nu=0.5

, indigo), and three input variants: Argmax + SC (

\star

), Argmax (

\blacksquare

) and Expectation

P_{t}

(

\bullet

). Horizontal lines show AR decoding (greedy:

62.6\%

; sampling:

52.6\%

) and UDM with the Predictor-Corrector (PC) sampler. With few steps and high temperature, the expectation input combined with the adaptive schedule and

\kappa=1

performs best: Const.-Lin. rises from

10.6\%

\kappa=0

\mathbf{32.7\%}

\kappa=1

and Constant from

11.2\%

31.8\%

, compared with

14.7\%

and

13.8\%

under the linear schedule and

23.7\%

for the best UDM with PC. Exact values for

\kappa\in\{0,0.2,1\}

are in Table 30 .

Figure 11: Churn

\kappa

vs. TinyGSM accuracy at

T=0.1

with 64 steps for the Linear (left), Cosine (center) and Adaptive (right) sampling schedules. We compare two concentration schedules, Const.-Lin. (

\nu_{0}=0.2,\nu_{1}=0.5,\ell=0.2

, orange) and Constant (

\nu=0.5

, indigo), and three input variants: Argmax + SC (

\star

), Argmax (

\blacksquare

) and Expectation

P_{t}

(

\bullet

). Horizontal lines show AR decoding (greedy:

62.6\%

; sampling:

52.6\%

) and UDM with the Predictor-Corrector (PC) sampler. At

\kappa=1

, Argmax + SC reaches up to

\mathbf{44.5\%}

( Constant , adaptive), above the best UDM with PC (

38.3\%

). With the adaptive schedule, the expectation input also improves strongly with churn, from

21.8\%

40.1\%

( Constant ) and from

18.9\%

39.5\%

( Const.-Lin. ). Exact values for

\kappa\in\{0,0.2,1\}

are in Table 32 .

Figure 12: Full-sequence vs. block-wise SDM generation on TinyGSM. Accuracy vs. total NFE at

T=0.1

(left) and

T=1

(right) for block-wise generation with block size 32 under the linear, cosine and adaptive grids, compared with full-sequence SDMs (expected-embedding and Argmax + SC inputs,

\kappa=1

). Block-wise generation is less accurate at every budget.

$T=1.0$
Model	64	512
DFM (reimpl.)	5.4	6.1
SDM ( $\kappa=1.0$ )	32.7	45.8
$T=0.1$
DFM (reimpl.)	12.1	13.3
SDM ( $\kappa=1.0$ )	39.5	49.0

Appendix

Table 3: Accuracy (%) on TinyGSM (Adaptive schedule) comparing Dirichlet Flow Matching and our SDM method across step budgets and logit temperature

T

. For SDM, we use the expectation input,

\nu_{t}^{\text{cst.-lin.}}

(

\nu_{0}=0.2

\nu_{1}=0.5

\ell=0.2

) and

\kappa=1

( Tables 29 , 30 , 31 and 32 ).

Spherical Flow
Model	64 NFE	512 NFE
ODE	6.1	6.4
PC	26.9	32.4
PC + SC	35.6	41.7
SDMs
Expectation ( $\kappa=0$ )	10.6	12.6

Appendix

Table 4: Accuracy (%) of SDMs and Spherical Flow on TinyGSM at

T=1

. Spherical Flow numbers come from Chemseddine et al. (2026) (Table 20), who train with the same architecture, tokenizer, context length and training budget as ours. Each PC entry uses their best sampler configuration for that budget. For SDMs, we always use

\nu_{t}^{\text{cst.-lin.}}

(

\nu_{0}=0.2

\nu_{1}=0.5

\ell=0.2

) and the adaptive grid, as in Figure 1 .

Model	8	32	64	128
IDLM-MDLM ( Li et al., 2026 )	–	12.8	14.9	19.9
IDLM-Duo ( Li et al., 2026 )	–	15.4	19.0	21.4
SDM (undistilled)	6.3	25.4	32.7	35.5
SDM (distilled)	32.1	37.6	37.8	39.4

Appendix

Table 5: Accuracy (%) of distilled models on TinyGSM across sampling steps (

T=1

). SDMs use the concentration schedule defined in terms of the total variance with

\nu_{t}^{\text{cst.-lin.}}

(

\nu_{0}=0.2

\nu_{1}=0.5

\ell=0.2

lr	$\beta_{1}$	$\beta_{2}$	$\lambda_{\text{gen}}$	pass@1 (%) $\uparrow$	pass@2 (%) $\uparrow$	pass@5 (%) $\uparrow$	AST Div. $\uparrow$	AST Div. (Correct) $\uparrow$
1e-05	0.0	0.95	1.0	31.9	40.0	48.7	17.5	7.5
1e-05	0.0	0.95	2.0	31.9	40.0	48.7	17.5	7.5
1e-05	0.0	0.95	5.0	31.9	40.0	48.7	17.5	7.5
1e-05	0.9	0.999	1.0	31.8	40.0	49.6	17.4	7.6
1e-05	0.9	0.999	2.0	31.8	40.0	49.6	17.4	7.6
1e-05	0.9	0.999	5.0	31.7	40.0	49.6	17.4	7.6

Appendix

Table 6: Distilled SDMs on TinyGSM (8 steps,

\kappa=1

, adaptive grid,

T=1

K=5

samples per problem). We sweep the learning rate

\in\{3\times 10^{-6},10^{-5},3\times 10^{-5}\}

, the Adam parameters

(\beta_{1},\beta_{2})\in\{0,0.9\}\times\{0.95,0.999\}

and the generator loss weight

\lambda_{\text{gen}}\in\{1,2,5\}

(36 configurations), and report the 25 configurations with highest pass@1.

Figure 13: Distilled vs. undistilled SDMs on TinyGSM across sampling steps (NFE

\in[1,2^{14}]

\kappa=1

, adaptive grid) at

T=0.1

(left) and

T=1

(right). At

T=1

, the distilled SDM solves

32.1\%

of the problems with 8 NFEs, against

6.3\%

for the undistilled SDM with the expected embedding, and

39.4\%

with 128 NFEs. Beyond 128 NFEs it plateaus at

40

–

41\%

, whereas the undistilled SDM keeps improving (

45.8\%

at 512 and

49.7\%

at 16k NFEs).

Figure 14: TinyGSM accuracy (%) vs. Auto-Guidance checkpoint step

K

for different churn regimes (

\kappa\in\{0.0,0.2,1.0\}

), with 64 sampling steps. We compare logit temperatures

T=0.01

(green),

T=0.1

(orange) and

T=1.0

(blue) for different guidance weights

w\in\{0.1,0.25,0.5,1.0\}

against the corresponding un-guided baselines (

w=0

, dashed horizontal lines). Error bars denote a variation of one standard deviation across 3 evaluation seeds. Early-to-intermediate checkpoints (

K\in[20\text{k},50\text{k}]

) consistently achieve peak accuracy.

Figure 15: Logit temperature sensitivity and Auto-Guidance gain on TinyGSM (

T\in\{0.01,0.1,1.0\}

). Left: Validation accuracy (%) of the best auto-guided model (

w^{\star}

, solid lines with stars) vs. the un-guided baseline (

w=0

, dashed lines with circles) for

\kappa\in\{0.0,0.2,1.0\}

. Right: Accuracy gain (

\Delta\%

) from Auto-Guidance. Error bars denote a variation of one standard deviation across 3 evaluation seeds.

Figure 16: OpenWebText (

L=1024

) Pareto frontiers across generative families. Best cumulative Pareto frontiers of Generative Perplexity (scored by GPT-2 Large; top is lower/better ) versus sequence token diversity ( right is higher/better ) for Simplex Diffusion Models ( SDM , orange), Masked Diffusion ( MDM , blue), Uniform Discrete Diffusion ( UDM , green), and the Autoregressive baseline ( AR , black). Left: Unigram token entropy

H_{1}

(nats). Right: Bigram token entropy

H_{2}

(nats). Dotted reference lines and the gold star (

\star

) denote the measured OpenWebText validation distribution (

H_{1}=5.46\text{ nats}

H_{2}=6.57\text{ nats}

\text{Gen PPL}=15.19

Panel of Figure 17 — Figure 17: Progressive intervention ablations on Unigram Entropy $H_{1}$ (nats) vs. Generative Perplexity. Cumulative Pareto frontiers per family across the intervention stack: Stage A (nucleus truncation $p\in\{0.92,0.96\}$ + initial temperature $T_{\text{start}}$ ), Stage B ( $+$ power-law temperature annealing $T_{\text{start}}=0.80\,T_{\text{end}}$ , exponent $1.5$ ), Stage C ( $+$ sequence-level frequency penalty $\lambda\in\{1.5,3.0,4.5,6.0,8.0\}$ ), and Stage D ( $+$ local frequency penalty $\gamma$ , SDM only).

Panel of Figure 18 — Figure 18: Progressive intervention ablations on Bigram Entropy $H_{2}$ (nats) vs. Generative Perplexity. Order-sensitive bigram diversity control ( $H_{2}$ ) across the same nested intervention stages ( Stage A $\to$ Stage D ) for SDM , MDM , UDM , and AR . Both the sequence frequency penalty ( Stage C ) and the local frequency penalty ( Stage D ) preserve gains along the bigram entropy $H_{2}$ axis.

Threshold ( $H_{1}\geq\tau$ )	SDM (Simplex)	MDM (Masked)	UDM (Uniform)	AR (1024 steps)
$H_{1}\geq 5.20$	14.0	14.7	13.5	8.9
Hyperparameters $(p,T_{\text{start}},\lambda,\gamma)$	$(.96,.40,3.0,2.0)$	$(.96,.70,1.5,0.0)$	$(.92,.60,1.5,0.0)$	$(.96,.65,3.0,0.0)$
$H_{1}\geq 5.35$	15.8	16.3	15.2	8.9
Hyperparameters $(p,T_{\text{start}},\lambda,\gamma)$	$(.92,.30,6.0,2.5)$	$(.92,.60,3.0,0.0)$	$(.96,.60,1.5,0.0)$	$(.96,.65,3.0,0.0)$
$\mathbf{H_{1}\geq 5.46}$ ( $\star$ OWT = 15.2)	17.0	16.9	17.3	9.0
Hyperparameters $(p,T_{\text{start}},\lambda,\gamma)$	$(.92,.30,6.0,2.5)$	$(.92,.50,4.5,0.0)$	$(.92,.70,1.5,0.0)$	$(.96,.65,3.0,0.0)$

Appendix

Table 7: Vertical Cuts on Unigram Entropy (

H_{1}\geq\tau

): Minimum Generative Perplexity (

\text{GenPPL}\downarrow

) on OpenWebText. AR is our model with the same architecture as the diffusion models, trained on OWT (1024 steps). The best diffusion method is bold , second best is underlined . The associated hyperparameters

(p,\,T_{\text{start}},\,\lambda,\,\gamma)

are shown in grey below each entry. The real OWT validation data (

\star

) has

H_{1}=5.46

and

\text{GenPPL}=15.2

Budget ( $\text{GenPPL}\leq\tau_{\text{PPL}}$ )	SDM (Simplex)	MDM (Masked)	UDM (Uniform)	AR (1024 steps)
$\text{GenPPL}\leq 12.5$	5.09	5.17	5.13	6.18
Hyperparameters $(p,T_{\text{start}},\lambda,\gamma)$	$(.92,.30,4.5,2.5)$	$(.96,.50,3.0,0.0)$	$(.96,.50,1.5,0.0)$	$(.96,.40,8.0,0.0)$
$\text{GenPPL}\leq 14.0$	5.20	5.18	5.23	6.20
Hyperparameters $(p,T_{\text{start}},\lambda,\gamma)$	$(.92,.40,3.0,2.0)$	$(.96,.40,4.5,0.0)$	$(.96,.50,1.5,0.0)$	$(.96,.40,8.0,0.0)$
$\mathbf{\text{GenPPL}\leq 15.2}$ ( $\star$ OWT $H_{1}=5.46$ )	5.29	5.22	5.32	6.21
Hyperparameters $(p,T_{\text{start}},\lambda,\gamma)$	$(.96,.40,3.0,2.0)$	$(.92,.70,1.5,0.0)$	$(.96,.50,1.5,0.0)$	$(.92,.65,4.5,0.0)$

Appendix

Table 8: Horizontal Cuts on Generative Perplexity (

\text{GenPPL}\leq\tau_{\text{PPL}}

): Maximum Unigram Entropy (

H_{1}\uparrow

(p,\,T_{\text{start}},\,\lambda,\,\gamma)

are shown in grey below each entry. The real OWT validation data (

\star

) has

\text{GenPPL}=15.2

and

H_{1}=5.46

Figure 19: We present a sweep on the Pointwise Mutual Information (PMI) weight

w

in ( 178 ). For each of the three tasks that we investigate HellaSwag ( Zellers et al., 2019 ) , ARC-easy ( Clark et al., 2018 ) and PIQA ( Bisk et al., 2020 ) , we report the influence of

w

on the final results. Note that

w=0

corresponds to the setup which is most comparable to the MDM scoring ( 177 ) (even though MDM noise is applied on the whole sequence while ours focus on the continuation). We denote

w^{\star}

the best weighting for each task. The optimal weight differs across tasks (

w^{\star}=0.1

for PIQA,

0.6

for ARC-Easy and

0.55

for HellaSwag) and is selected on the evaluation split. The results are averaged over 5 different seeds. In red we report the results for the baseline out of distribution OWT checkpoint. In blue we report the results after the short adaptation run.

Autoregressive baselines
Model	PIQA	ARC-Easy	HellaSwag
LLaMA	62.7	40.5	33.1
GPT-2	62.9	43.8	28.9
Discrete Diffusion models
MDM	54.1	31.0	31.1
PGM	58.9	40.4	33.2

Appendix

Table 9: Zero-shot accuracy (%) on language understanding benchmarks. We evaluate SDMs by sampling 5 times with a different random seed, and report the

\text{mean}_{\pm\text{std}}

. The best score per column is bolded and the second best is underlined . The LLaMA baseline is taken from von Rütte et al. (2025) .

		Standard Sampler		Confidence-Based Sampler
Schedule	Steps ( $N$ )	Temperature ( $T$ )	Churn ( $\kappa$ )	Temperature ( $T$ )	Churn ( $\kappa$ )	Power ( $p$ )
Linear	32	$0.01$	$1.0$	$0.01$	$1.0$	$1.0$
	64	$0.005$	$1.0$	$0.05$	$1.0$	$1.0$
	256	$0.05$	$0.2$	$0.05$	$1.0$	$1.0$
	1024	$0.001$	$0.2$	$0.05$	$1.0$	$1.0$
Cosine	32	$0.03$	$1.0$	$0.03$	$1.0$	$1.0$

Appendix

Table 10: Selected hyperparameters for Simplex Diffusion (with time conditioning, SDM_TE ) across noise schedules and sampling steps (molecular generation;

N\in\{32,64,256,1024\}

) corresponding to Figure 20 .

T

: logit sampling temperature;

\kappa

: Simplex churn parameter;

p

: top-fraction power for

\alpha_{s}^{p}

to select the most confident subset.

Method	Schedule	Steps ( $N$ )	Temperature ( $T$ )	Churn ( $\kappa$ )
Masked Diffusion (MDM)	Cosine	32	$0.05$	—
		64	$0.01$	—
		256	$0.005$	—
		1024	$0.03$	—
Uniform Diffusion (UDM)	Cosine	32	$0.05$	—
		64	$0.001$	—

Appendix

Table 11: Selected hyperparameters for diffusion methods under standard sampling across steps (

N\in\{32,64,256,1024\}

) corresponding to Figure 21 . Simplex methods use the adaptive schedule; Masked and Uniform use the cosine schedule.

T

: logit sampling temperature;

\kappa

: Simplex churn parameter.

Method	Schedule	Steps ( $N$ )	Temperature ( $T$ )	Churn ( $\kappa$ )	Power ( $p$ )
Masked Diffusion (MDM)	Cosine	32	$0.01$	—	$1.5$
		64	$0.01$	—	$1.5$
		256	$0.01$	—	$1.0$
		1024	$0.01$	—	$2.0$
Uniform Diffusion (UDM)	Cosine	32	$0.10$	—	$2.0$
		64	$0.03$	—	$1.5$

Appendix

Table 12: Selected hyperparameters for diffusion methods under confidence-based sampling across steps (molecular generation;

N\in\{32,64,256,1024\}

) corresponding to Figure 22 . Simplex methods use the adaptive schedule; Masked and Uniform diffusion use the cosine schedule.

T

: logit sampling temperature;

\kappa

: Simplex churn parameter;

p

: top-fraction power for

\alpha_{s}^{p}

to select the most confident subset.

Method	Schedule	Standard Sampling Hypers	Confidence ( conf. ) Sampling Hypers
Autoregressive	—	$T=0.7$	—
Simplex	Linear	$T=0.03,\;\kappa=1.0$	$T=0.05,\;\kappa=1.0,\;p=1.0$
Simplex	Cosine	$T=0.01,\;\kappa=1.0$	$T=0.05,\;\kappa=1.0,\;p=1.0$
Simplex	Adaptive	$T=0.10,\;\kappa=1.0$	$T=0.05,\;\kappa=1.0,\;p=1.0$
Simplex (time cond.)	Linear	$T=0.05,\;\kappa=0.2$	$T=0.05,\;\kappa=1.0,\;p=1.0$
Simplex (time cond.)	Cosine	$T=0.10,\;\kappa=0.0$	$T=0.05,\;\kappa=1.0,\;p=1.0$

Appendix

Table 13: Optimal sampling hyperparameters for each method and schedule (molecular generation) in Table 14 (standard sampling) and Table 15 (confidence-based sampling, conf. ) using

256

sampling steps.

T

: logit temperature;

\kappa

: Simplex churn ( Section 3.1 );

p

: top-fraction power for

\alpha_{s}^{p}

to select the most confident subset.

Figure 20: Sampling-budget and sampling time schedule comparison for time-conditioned SDM. Quality and Diversity (molecular generation) are shown for standard sampling (solid) and confidence-based sampling (dashed), using linear (blue), cosine (orange), and adaptive (green) inference grids. Increasing the sampling budget improves Quality under confidence-based sampling while reducing Diversity. Standard sampling retains higher Diversity, with smaller and nonmonotonic changes in Quality. Error bars show one standard deviation across three sampling seeds.

Figure 21: Quality and Diversity versus sampling steps under standard sampling (molecular generation). Comparison across all five diffusion methods, using adaptive time schedule for Simplex diffusion and cosine for the rest. Error bars show one standard deviation across three sampling seeds.

Figure 22: Quality and Diversity versus sampling steps under confidence-based sampling (molecular generation). Comparison across all five diffusion methods, using adaptive time schedule for Simplex diffusion and cosine for the rest. Error bars show one standard deviation across three sampling seeds.

Figure 23: Empirical Quality–Diversity Pareto frontiers across diffusion methods with confidence based sampling and hyperparameter sweeps. Molecule Quality (%),

\uparrow

, versus Diversity (

\uparrow

) across all evaluated sampling hyperparameter configurations. All markers denote individual exploration runs sweeping sampling temperatures, number of sampling steps, top-fraction power for selecting the most confident subset and churn parameters, see Section J.8 for more details. Solid curves trace the empirical Pareto frontier of non-dominated configurations for each diffusion family, where we use adaptive time schedule for Simplex diffusion and cosine for the rest. Distinct stars mark external and autoregressive baselines: GenMol (gold star), the autoregressive AR model (black star), and SAFE-GPT (gray star). Simplex Diffusion Pareto frontier leans more towards top right, with many points located in high diversity regime.

Published Reference ( Lee et al., 2025 )
Method	Schedule	Quality (%) $\uparrow$	Validity (%) $\uparrow$	Uniqueness (%) $\uparrow$	Diversity $\uparrow$
SAFE-GPT	—	$54.7\pm 0.3$	$94.0\pm 0.4$	$100.0\pm 0\phantom{.0}$	$0.879\pm 0.001$
GenMol	—	$84.6\pm 0.8$	$100.0\pm 0.0$	$99.7\pm 0.1$	$0.818\pm 0.001$
Our Run Comparison
Autoregressive (AR)	—	$63.97\pm 1.67$	$96.37\pm 0.21$	$99.76\pm 0.06$	$\underline{0.883\pm 0.002}$
Simplex	Linear	$73.57\pm 1.86$	$99.63\pm 0.12$	$99.83\pm 0.12$	$0.879\pm 0.001$

Appendix

Table 14: Unconditional de novo molecular generation on SAFE-GPT with standard sampling following the GenMol evaluation protocol ( Lee et al., 2025 ) (

1{,}000

molecules/seed, mean

\pm

std over 3 seeds). Bold : best; underline : second best among our runs. All our runs use

256

sampling steps.

Published Reference ( Lee et al., 2025 )
Method	Schedule	Quality (%) $\uparrow$	Validity (%) $\uparrow$	Uniqueness (%) $\uparrow$	Diversity $\uparrow$
SAFE-GPT	—	$54.7\pm 0.3$	$94.0\pm 0.4$	$100.0\pm 0\phantom{.0}$	$0.879\pm 0.001$
GenMol	—	$84.6\pm 0.8$	$100.0\pm 0.0$	$99.7\pm 0.1$	$0.818\pm 0.001$
Our Run Comparison
Autoregressive (AR)	—	$63.97\pm 1.67$	$96.37\pm 0.21$	$99.76\pm 0.06$	$\mathbf{0.883\pm 0.002}$
Simplex	Linear	$83.87\pm 1.36$	$99.80\pm 0.10$	$\underline{99.87\pm 0.12}$	$0.859\pm 0.000$

Appendix

Table 15: Unconditional de novo molecular generation on SAFE-GPT with confidence-based ( conf. ) sampling following the GenMol evaluation protocol ( Lee et al., 2025 ) (

1{,}000

molecules/seed, mean

\pm

std over 3 seeds). Bold : best; underline : second best among our runs. All our runs use

256

sampling steps.

Sample 1 (1024 tokens) GenPPL (GPT-2 Large): 14.15 — Unigram Entropy: 5.27 — Distinct-2: 0.724

bring peace and stability Middle East,” he told reporters. “We’re all going to be hopeful if we’re part of an agreement that is retroactively … And I think the Democratic Party will be up for grabs if she takes it seriously.” Republican strategist Steve Bannon also criticized Trump’s comments earlier this week, saying that he had a “waste for American diplomacy” by “trying to burn out sanctions relief with Iran. New York Times columnist Hugh Hewitt, also a former State Department official and a member of the Republican National Committee, says Trump is trying to forge a deal with Iran without ever breaching international sanctions. “I have a lot of miscalculations because I think they’ve succeeded in a bad engagement” Hewitt told The New York Times, adding that the administration needs to understand the terms of the deal. “I think I don’t think this deal will be bad, but it’s hard — which is very hard, very hard — to fix it,” he added. “What can we do? Can we depolarize the rest of the Middle East?” ___ 10 p.m. North Korea’s government says it backs Republican presidential candidate Donald Trump for setting a new tone with his criticism of North Korea’s state-controlled media, in a move at odds with Washington rhetoric over its pursuit of nuclear weapons. South Korean Foreign Minister Kim Kyung-seo responded in a statement calling Trump “reckless and reckless.” He criticizes Trump on North Korea, calling him a “highly dangerous man for the President of the United States.” He also said he would not talk to North Korea without doubting its nuclear weapons abroad. “Without such a strong leadership, any threat from the North Korean regime would be rejected,” he said. South Korea has said it banned Pyongyang’s nuclear programs in the past because it would stop North Koreans from leaving the country without their weapons. ___ 9:30 p.m. North Korean military leader Jang Song-thaek says Trump “totally impoundately” the U.S. nuclear weapons program because it was “totally fragile” after the United States dropped an H-bomb on the city of Nagasaki, according to an Air Force One video released Tuesday. He also said he hoped that the U.S. would make a good return solely to the negotiating table. He also called for North Korea to get serious about halting its program, although the U.S. has not sanctioned or formally approved its nuclear-weapons program since August 1950. He also urged the United States to learn a lesson from history. Trump said on Tuesday that the U.S. must getting ready to restart its nuclear program. He said restarting its work with North Korea could mean more years to begin in the coming months. “It will be very gradual,” he said. “And I think that will not be abrupt — and it will will be gradual — until the United States can get back on the table.” ___ 8:30 p.m. Yuri Yuri, South Korea’s former deputy prime minister, says Donald Trump believes the U.S. should continue to unravel its nuclear program but is a “preaching and grave mistake.” His remarks followed a recent high-level meeting between Japan and South Korea at an annual summit in Seoul. Yuri, who is South Korea’s first president, has criticized the U.S. should apologize for dropping two atomic bombs on Hiroshima during World War II. He said the U.S. should take steps necessary to try to halt its nuclear program. ___ 8:45 p.m. Former Republican presidential candidate Donald Trump said on Tuesday that his country’s nuclear and missile programs should hold a meeting to discuss “a peaceful solution.” Trump made similar comments about his country’s relationship with the United States, which has long labeled its nuclear weapons and other NATO allies are “obsolete.” North Korean leader, Kim Il Jong-un, had earlier this month accused China of cutting off off the country’s nuclear program during talks with other parties for peace talks. Trump’s comments have been seen as provocative by some over his tough stance toward China and his controversial unification policy. But South Korean Foreign Minister Jang Song-thaek issued a statement Tuesday saying the issue was non-negotiable and adding that nuclear talks could not be reached unless both sides reached a peaceful political solution. _____ 7:30 p.m. The United

Appendix

Table 16: Unconditional 1024-token generation from Simplex Diffusion Model trained on OWT (Sample 1).

Sample 2 (1024 tokens) GenPPL (GPT-2 Large): 19.52 — Unigram Entropy: 5.59 — Distinct-2: 0.804

after it was advertised as a “gender-only” service run by a non-binary woman. The ad came less than two hours after a video featuring a YouTube user called Avoid Open Door Wicked, which featured 12 women, 14 men, 11 men and 10 women was posted online on its website. “I have been harassed 400+ times because I am surrounded by a hostile environment directed at me,” the 23-year-old woman wrote. “I can’t believe it,” read one of the ads, which read “Divorce is in my veil!” Another added: “You are not a misogynist, and you cannot use your own operating system to escape your own travails.” “If you were a non-binary woman would you be captive for your own sexual desires? If you’d you were a woman woman would you be captive for your desires?” asked ACLU attorney Jennifer Partridge, who filed the case through the Justice Department’s nonprofit Philanthropy Law Project. “It’s reflective of what we kind of social network is about and whether it is a real issue,” Partridge added. Courtney ACLU attorneys argued that the company’s discrimination against harassment and gender-based bias could violate her legal rights. “We do not substantially suggest that gender discrimination is not a real issue,” she wrote her brief. “This ad does not fit the context of our social Facebook or Google ads.” Partridge noted that the company’s policy — which requires customers to purchase or sell their products online — suggests more for women than it does for men. The company also sells products in the same retail department stores. “One could argue that this product based on gender-based disrespect is far more sanctimonious what people might buy from a store online,” Partridge said in an email. “This is not an abstract issue.” The company’s justification is that its site must engage users to ‘disconnect’ amongst psychological issues. “I don’t think it should,” Partridge said. “I don’t think another person should be discriminated against.” The company has also said that its decision to remove names or corporate logos from all ads on the site because it doesn’t want to deter any potential trespassing. Story continues below advertisement — Hillary Clinton has maintained her commanding lead in the presidential race over Republican presidential nominee Donald Trump. Clinton says Trump leads her by 39%, according to a new NBC News/Wall Street Journal Journal poll that shows Republican presidential nominee Donald Trump with the largest-ever lead in any presidential election. The poll, conducted by Public Opinion Strategies, a long-time polling firm, surveyed 1,000 likely voters. It has a margin of error at 3.0 points with the total sample of 1,000. Clinton gets 43 percent of the vote behind Green Green Party candidate Jill Stein (38 percent), while Trump gets 41 (36 percent) and Jill Stein (35 percent). Under the case for both candidates, Stein would get 33.9 percent of the vote. The poll predicts Clinton would win over Trump (34 percent), while Stein would get 9.7 percent ahead of Stein (8 percent) Stein/Garrabee (6 percent). Despite last week’s debate announcing her bid for the presidency, Clinton has implied that she would support any future Republican presidential candidate. In November, when she joined Barack Obama’s national security team, she said she would need to cast her first female vote in the U.S. Senate to serve as president. The poll also shows Trump leads all other Republican candidates with 34% support, while Texas Sen. Ted Cruz leads the presumptive GOP nominee with just 15%. Trump has previously abandoned former Democratic Secretary of State Hillary Clinton after launching an unsuccessful bid to revive his campaign. He himself, however, has endorsed the presumptive GOP nominee, admitting that he still had no intention of securing support. He also suggested that he might be tempted to endorse Clinton if his party didn’t back him up. Story continues below advertisement. The polls are conducted by landline and automated landline telephone interviews with 1,000 likely voters nationwide and were conducted between Oct. 19 through Nov. 20, 2012, with Barack Obama and Mitt Romney registered among likely voters. The margin of error is plus or minus 3.5 points. Also on HuffPost: Two new skyscraper plans will bring some of Manhattan’s tallest skyline to the rest of the world, after Madison Square Park was knocked out of Manhattan’s tallest buildings earlier this year. Construction on the first three-story tower has been delayed in time for a full moon. The city’s skyscraper straddled between New York, New York, Chicago and

Appendix

Table 17: Unconditional 1024-token generation from Simplex Diffusion Model trained on OWT (Sample 2).

Sample 3 (1024 tokens) GenPPL (GPT-2 Large): 19.53 — Unigram Entropy: 5.23 — Distinct-2: 0.645

Related Articles Section 491. Related Articles Section 492 Miscellaneous Related Articles Section 493 Miscellaneous. Section 493. Related Section 494 Miscellaneous Related Articles Section 494 Miscellaneous. Section 493 Miscellaneous Articles Section 494 Miscellaneous Articles Sections 495 Miscellaneous Sections 487 Miscellaneous Sections 488 Miscellaneous Sections 489 24/24 Sections 4811 Miscellaneous Sections 4812 24/24 4813 Miscellaneous Sections 4812 24/24 4813 Miscellaneous Section 4812. Sec. 501 of Hawaii Revised, Hawaii Revised Code as follows: (a) It shall be unlawful as a citizen of the United States or a foreign territory of the United States; (a) may conduct, including but not not limited to as a citizen of the United States; (b) as an ex-concitizen of the State of Hawaii or a foreign territory; or (b) as a non-citizen of Hawaii or a former citizen of the United States; (ii) perform other activities as defined in this title. If such an individual does not wish to sign up for annual free membership or holiday free walks in an area outside Hawaii, it shall be unlawful to use internet services, advertisements newspaper articles, or witherbecoming to advertise such activities. Any person who knowingly giving birth to another person under this same title (or any other person under this title or regulations) shall be penalized. If a person who is denied health care coverage has been notified in advance that a person who does not give birth birth at his birthplace will he become a resident of a foreign territory, or a Territory of another state or vice versa, there is no penalty for violation of this above law. If it also is found that a person who denies giving birth occurs at a site of birth will he is a resident of a foreign territory, or Foreign Territory of the State. To prohibit any person visiting a foreign territory or from becoming a resident of any territory outside the United States, I am hereby directing that such person knowingly violates the above law. The incurring any violation of the above law or regulations may be taken pursuant to the provisions of the Rawled “Hawaii” Virgin Islands Act of 2018. Hawaiian and native Hawai’i people: Hawaiian is a Hawaiian language which is a native Hawaiian language that has been traditionally associated with the Hawai’i people. It is nevertheless traditional Hawaiian language spoken by native Hawai’i people. The Native Hawaiian people are descendants of Native Hawaiian people who are descended from Native Hawaiian tribes, and since then they have been interceded by tribes from other Native Hawaiian tribes. A permanent Hawaiian resident or permanent Hawaiian resident who resides in Hawaii becomes a permanent Hawaiian resident shall continue to form a liaison with all Hawaiian community and volunteer volunteers to assist with navig land being navigated navigated within Hawaii. A permanent Hawaiian resident shall be required surrender reside in Hawaii court for specified time period, and if granted permanent Hawaiian resident order obtain permission from his residence in Hawaii for a specified time period. Permanent Hawaiian residents shall be required to reside before a Hawaii court. A permanent Hawaiian resident shall continue to become legal citizens of Poly Hawaiians because they are native Hawai’i residents who are non-institutionalized employment and who operate their own agricultural enterprises. Article XV01 of the Civil Rights Act of Samoa: I am proposing to amend section article XV01 of the Civil Rights Act of Samoa and shall be amended by sections VII, II and Article XII, the Constitution of the Republic on January 1, 2016. Article XV01 U.S. Constitution of Samoa — The Pacific Ocean Rights Act of Samoa Act: This act was performed on January 1, 2016 while facilitating the Civil Rights Act of Samoa. This Act shall be amended in section I, II, sections XVI, and XVI, the Constitution of Samoa as amended by section VI. The American Samoa Act: This Act shall be amended in section Article XIII, the Pacific Ocean Rights Act as amended by section XXVIII. This Act shall be amended in section Article XIV, the Constitution of the United States as amended by section VI. Section 18017 U.S. Constitution of the United States of America: Nothing shall be made construed unlawful for any county, state, political subdivision of any country or any other State whatsoever to employ any person in the United States of America. Section 18017 U.S. Constitution of the United States of America includes persons: Native American, African American, Baltic American, Mexican-American Indians non-Indigenous Indian Americans, immigrants, citizens of the United States. Corrections / Proions: Section 17:20 Sec. 2 — the Constitution 1/3 dated January 1, 2017. Section 17:20 Sec. 3 — the text of the Constitution 1/3, dated January 1, 2017. (Proclamation). Section 17:20 Sec. 3

Appendix

Table 18: Unconditional 1024-token generation from Simplex Diffusion Model trained on OWT (Sample 3).

Sample 4 (1024 tokens) GenPPL (GPT-2 Large): 20.85 — Unigram Entropy: 5.60 — Distinct-2: 0.848

efficient price for carbon emitted from renewable energy rests with the European Commission, spokesman Richard Ritter said. Commissions are come under a special agreement, which allows member states to be analysed and then sell all of the emissions emitted from other EU states. Under this agreement member states would buy emission-free allowances. The member states could then decide to sell them only if they couldn’t afford to buy additional items. For emissions-free under the new agreement, those allowances from Germany would have to give up their market share, said Ritter. “The Commission will henceforth refuse to use emission-free allowances as an exercise for Germany or for Jean Juniet,” Ritter said. “There are other alternatives I would advise that we should be able to bring them into the domestic market, but not at all.” Ritter added that the EU would need to remove emission-free allowances from its domestic market before sending them back to emission-free markets. He added that such plans were approved by the German parliament. German MEP Bart Schoto told reporters on Sunday that there is no need for a new deal. “It’s just a simple legislative process,” he said. He suggested emission-free allowances should be sold. “I don’t think so much but but leaving is not really a problem,” he said. “If we think we will be able to keep up the cost of our growing economy then I think this pathway will be very difficult.” Other measures must be taken through the new emissions agreement, Ritter said. The measures could include building incentives to build nuclear power plants, which could cost about $1 billion a year plus subsidies for nuclear power plants in South Florida, which could raise fuel costs. “We will have to expand our capacity if we can’t get our policies back into place,” he said. “It’s going to be a very tough decision and it will be taken out as soon as possible,” he said. Merkel’s action ‘unemptive’: Juncker, who was arrested in Brussels in connection with his meeting between the two leaders and German Chancellor Angela Merkel, met privately with former Chancellor Gerhard Schroeder, the former Bavarian prime minister and medical doctor Reinforce Eisenberg at a press conference in Berlin last week. The commission’s decision has been fraught with political tensions. However, Germany has appealed to Brussels against a form of one-party rule imposed by the European Union. On Friday, Germany’s state energy company (SDF) and its state utility EDF — which has been the main target for calls for divesting coal, said it would not move forward. Eisenberg said it was highly unlikely that Merkel would make any effort to change her position. “This is a step in a right direction,” said Marguerte Schweiko, the head of the German daily newspaper Vorb Mauschaftung in an interview with Der Spiegel. “The chancellor has clearly changed position… She has already shown herself capable of exerting power without the popular support of her friends, who control the world’s most valuable fossil fuel reserves.” “Merkel is an anti-nuclear campaigner… She has made it clear that she did not actually participate in any concrete action.” Schweiko also said Merkel her actions are “unemptive”, adding that “the German government played a principal role in cultivation of tools for climate change”. She added that Merkel had pressure on many Germans. “Mrs Merkel seems to have lost her credibility in many ways she cannot stand trusted,” she said. Earlier this week, the German Green Party, Christian Saar, issued a statement condemning “polarly polarized” climate negotiations. “It is inconceivable that all parties can agree on a legally binding treaty—even within the framework of the parliament—but it documents highly polarised,” the statement read. The EU has started undertaking its own voluntary emissions trading scheme, making matters more pressing among some European countries that are realistic about climate change, as well as other countries that have warned against taking climate action. Germany may have already signed up to reduce carbon emissions, meaning that still does not have the resources to do so. But it may also have an opportunity to kick-start talks in Paris on climate change. Later this year, Europe and the major United States are expected to host talks on agriculture and climate change. Their involvement The Paris talks is scheduled to begin at the end of 2012, months after the Kyoto Protocol took place in Copenhagen in December 2009. British Prime Minister David Cameron, who is in Copenhagen, and is due to discuss negotiations on climate change, had told reporters the Paris talks would “have no end in sight yet”. Cameron told reporters earlier this week that Britain was “not ruling out of Paris”, nor was an organiser of the negotiations. G20 leaders will hold a summit in Hamburg later next month

Appendix

Table 19: Unconditional 1024-token generation from Simplex Diffusion Model trained on OWT (Sample 4).

Sample 5 (1024 tokens) GenPPL (GPT-2 Large): 4.04 — Unigram Entropy: 3.56 — Distinct-2: 0.197

Commandments. 300. 325 325. The Son of God, His Son, His Father and the Commandments. 300. 325 325. The Son of God, The Son of the Holy and Father and the Commandments. 300. 325 325 Elijah Elijah is not commanded by The Lord of Christ, His Father and the Commandments. 300. 325 325 Elijah is not commanded by Jesus Christ, His Father, the Commandments. 300. 325 325. Of The Lord of Christ, His Father and the Commandments. 300. 325. The Lord of Christ, His Son and the Commandments. 300. 325 325. The Lord of Christ, His Father and the Commandments. 301. 325 The Lord of Christ, His Father and the Commandments. 301. 325. The Lord of Christ, His Father, the Commandments. 301. 325 325. Of The Lord’s son, His Father, the Commandments. 301. the Lord Elijah is not commanded by The Lord of son, His Father, the Commandments. 301. the Lord Elijah is not commanded by The Lord’s son, Father, the Commandments. 301. the Lord Elijah is not being commanded by the Lord’s Christ, His Son, His Father and the Commandments. 301. the Lord Elijah is not being commanded by The Lord’s son, His Father and the Commandments. 302. 304 308 308. The Son of God to His Father and the Commandment 351. 303. 308 308 The Son of God, Maker Commandments. 304. 328 309. Jesus God to His Father and the Commandment 351. 303. 328 309. Jesus Lord to be Father, Maker Commandments. 303. 328 309. Jesus Lord to be Stronger. 305. 328 309. The Son of God. Maker Commandment. 306. 328 309 309. The Son of Christ, His Father, the Commandments. 305. 328 309. The Lord to be Strong, Maker Commandments. 307. 328. The Son of Christ, His Father, the Commandments. 308. 328 328. The Son of Christ, His Father, the Commandments. 309. 328 328. The Lord to be Stronger. 322. 329 330. The Son of Christ, His Father, the Commandments. 323. 329 330. Jesus The Son of God, Maker Command. 324. 329 330. Jesus Lord to be Strong. 323. 329 330. The Son of God, Maker Commandments. 324. 329 331 Lose! Of The Son’s God, His Father and the Commandments. 326. 329 331. The Son of Christ, His Father and the Commandments. 325. 320 315 315. Jesus The Son of God, His Exalted Father and the Commandments. 326. 320 320. Jesus the Son of Christ, His Father and the Commandments. 325 320 320. Jesus Christ to the Father, the Commandment 351. 325 320 320. The Son of Christ to the Father and the Commandment 351. 327. 320 320. The Son of Christ to the Father and the Commandments. 349. 321 321. The Son of God, Maker Command. 350. 321 321. Jesus Christ to He Father, Maker Commandment. 351. 321 321. Jesus the Son of God to His Father, the Commandment 351. 352 321 321. The Son of Christ to His Father, the Commandment 351. 352. 321 321. The Son’s God, His Exalted Father and the Commandment 351. 353. 321 321. The Son of Christ to His Father, the Commandment 351. 352. 412 355. The Son of Christ to His Father, the Commandment 351. 352. 412 355. The Son of God, Maker Commandment. 353. 412 355. The Lord to be Strongerful. 354. 412 355. The Lord of his Father, Maker Commandments. 355. 412 412. Jesus The Son of Christ, the Father and the Commandments. 355. 412 412. The Son of Christ, the Father and the Commandments. 356. 412 412. Jesus Christ to the Father, Maker Commandments. 357. 412 410. The Son of Christ, His God, the Commandment 351. 356 412 412 410. Jesus Christ to the Father, Maker Commandments. 357. 412 410. The Son of God, the Father and the Commandments. 358. 412 412 410. The Lord’s God, the Father and Commanders. 358. 412 410. The Son of God, Maker Commandments. 357. 415 414. Jesus Christ to the Father, the Commandments. 357. 412 414. The Son of Christ, the Father and the Commandments. 358. 412 415. It Happen to

Appendix

Table 20: Unconditional 1024-token generation from Simplex Diffusion Model trained on OWT (Sample 5).

Concentration Schedule	Linear	Cosine	Adaptive
Uniform time sampler
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.5,\ell=0.2$ )	$62.8_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 5.0}}$	$62.4_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4.7}}$	–
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.75,\ell=0.2$ )	$74.0_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4.4}}$	$73.7_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4.3}}$	–
Const.-Lin. ( $\nu_{0}=0.4,\nu_{1}=0.75,\ell=0.8$ )	$73.9_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4.7}}$	$74.0_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4.7}}$	–
Constant ( $\nu=0.5$ )	$64.9_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 3.7}}$	$64.6_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 3.1}}$	–
Constant ( $\nu=0.25$ )	$56.0_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 6.2}}$	$55.6_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 5.8}}$	–

Appendix

Table 21: Accuracy (%) on Sudoku in 180 steps for Simplex Diffusion Models with Churn

\kappa=0.0

across sampling schedules. Values are reported as

\text{mean}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{std}}}

over 5 seeds. Column-wise top scores are bolded , and overall best for

\kappa=0.0

is underlined

Concentration Schedule	Linear	Cosine	Adaptive
Uniform time sampler
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.5,\ell=0.2$ )	$81.1_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 3.2}}$	$81.4_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 3.3}}$	–
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.75,\ell=0.2$ )	$85.3_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 3.2}}$	$85.4_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 3.3}}$	–
Const.-Lin. ( $\nu_{0}=0.4,\nu_{1}=0.75,\ell=0.8$ )	$80.4_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 3.7}}$	$79.1_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 3.5}}$	–
Constant ( $\nu=0.5$ )	$83.5_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 2.0}}$	$80.0_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.8}}$	–
Constant ( $\nu=0.25$ )	$76.6_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 3.5}}$	$77.6_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 3.5}}$	–

Appendix

Table 22: Accuracy (%) on Sudoku in 180 steps for Simplex Diffusion Models with Churn

\kappa=0.2

across sampling schedules. Values are reported as

\text{mean}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{std}}}

over 5 seeds. Column-wise top scores are bolded , and overall best for

\kappa=0.2

is underlined

Concentration Schedule	Linear	Cosine	Adaptive
Uniform time sampler
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.5,\ell=0.2$ )	$89.3_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 2.3}}$	$89.7_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 2.3}}$	–
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.75,\ell=0.2$ )	$91.5_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.9}}$	$91.6_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.6}}$	–
Const.-Lin. ( $\nu_{0}=0.4,\nu_{1}=0.75,\ell=0.8$ )	$88.4_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 2.2}}$	$88.9_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 2.4}}$	–
Constant ( $\nu=0.5$ )	$90.5_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.1}}$	$90.6_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 1.2}}$	–
Constant ( $\nu=0.25$ )	$86.9_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 2.1}}$	$87.8_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 2.5}}$	–

Appendix

Table 23: Accuracy (%) on Sudoku in 180 steps for Simplex Diffusion Models with Churn

\kappa=1.0

across sampling schedules. Values are reported as

\text{mean}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{std}}}

over 5 seeds. Column-wise top scores are bolded , and overall best for

\kappa=1.0

is underlined

Model	Linear	Cosine	Adaptive
Autoregressive
AR (Sample)	$\hphantom{0}2.9_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.6}}$	–	–
AR (Greedy)	$\hphantom{0}3.1_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 0.5}}$	–	–
Discrete (Absorbing)
Ancestral † (CE)	$60.1_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 3.8}}$	$61.1_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4.5}}$	–
Ancestral ‡ (CE)	$69.4_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4.8}}$	$69.7_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4.0}}$	$70.0_{{\color[rgb]{0.5,0.5,0.5}\scriptscriptstyle\pm 4.4}}$

Appendix

Table 24: Baseline Models Accuracy (%) on Sudoku in 180 steps across sampling schedules for the Appendix. We evaluate Autoregressive (AR), Discrete Absorbing (MDM with CE and ELBO losses), Discrete Uniform (UDM), Predictor-Corrector (PC), Self-Conditioning (SC / loopholing), Flow Matching over one-hot encoding (FLM with uniform, Gauss-Hermite LUT, and adaptive time samplers), and Hyperspherical Flows (

\mathbb{S}

-FLM). † trained with uniform time sampler, ‡ trained with adaptive time sampler. Values are reported as

\text{mean}_{{\color[rgb]{0.5,0.5,0.5}\pm\text{std}}}

over 5 seeds. We bold the best result and underline the second best in each column.

Autoregressive
Model	Linear	Cosine	Adaptive
AR (Sample)	52.6	–	–
AR (Greedy)	62.6	–	–
Discrete (Absorbing)
Ancestral † (CE)	12.9	14.8	–
Ancestral ‡ (CE)	13.2	13.4	13.1

Appendix

Table 25: Baseline Models Accuracy (%) on TinyGSM (

T=1.0

, 512 steps) across sampling schedules. We evaluate Autoregressive (AR), Discrete Absorbing (CE and ELBO losses), Discrete Uniform (ELBO loss), Predictor-Corrector, Self-Conditioning (Ancestral + SC), FLM, and

\mathbb{S}

-FLM. † trained with the uniform time sampler, ‡ trained with the adaptive time sampler. We bold the best result in each column and underline the best result within each section (both if achieving both).

Discrete (Absorbing)
Model	Linear	Cosine	Adaptive
Ancestral † (CE)	9.7	11.1	–
Ancestral ‡ (CE)	8.3	11.7	11.2
Ancestral † (ELBO)	10.0	13.3	–
Ancestral ‡ (ELBO)	8.5	11.8	12.6
Predictor-Corrector †	10.7	10.2	–

Appendix

Table 26: Baseline Models Accuracy (%) on TinyGSM (

T=1.0

, 64 steps) across sampling schedules. We evaluate Autoregressive (AR), Discrete Absorbing (CE and ELBO losses), Discrete Uniform (ELBO loss), Predictor-Corrector, Self-Conditioning (Ancestral + SC), FLM, and

\mathbb{S}

Autoregressive
Model	Linear	Cosine	Adaptive
AR (Sample)	52.6	–	–
AR (Greedy)	62.6	–	–
Discrete (Absorbing)
Ancestral † (CE)	31.0	31.1	–
Ancestral ‡ (CE)	30.4	30.0	30.3

Appendix

Table 27: Baseline Models Accuracy (%) on TinyGSM (

T=0.1

\mathbb{S}

\mathbb{S}

-FLM sampler is deterministic given the initial noise and does not depend on the temperature, hence identical rows at

T=1

and

T=0.1

Discrete (Absorbing)
Model	Linear	Cosine	Adaptive
Ancestral † (CE)	25.0	28.2	–
Ancestral ‡ (CE)	24.9	29.3	27.3
Ancestral † (ELBO)	27.4	29.5	–
Ancestral ‡ (ELBO)	29.6	30.7	29.9
Predictor-Corrector †	31.1	31.8	–

Appendix

Table 28: Baseline Models Accuracy (%) on TinyGSM (

T=0.1

\mathbb{S}

\mathbb{S}

-FLM sampler is deterministic given the initial noise and does not depend on the temperature, hence identical rows at

T=1

and

T=0.1

Expectation Input Processing ( $P_{t}$ )
Concentration Schedule	Linear			Cosine			Adaptive
Concentration Schedule	$\kappa=0$	$\kappa=0.2$	$\kappa=1.0$	$\kappa=0$	$\kappa=0.2$	$\kappa=1.0$	$\kappa=0$	$\kappa=0.2$	$\kappa=1.0$
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.5,\ell=0.2$ )	15.4	25.1	34.7	19.8	29.1	38.8	12.6	35.7	45.8
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.75,\ell=0.2$ )	15.1	24.5	31.7	17.9	28.4	36.8	11.5	29.2	38.8
Constant ( $\nu=0.5$ )	15.7	23.1	33.6	17.5	24.4	38.2	15.0	29.1	41.7
Constant ( $\nu=0.25$ )	12.1	19.4	26.6	14.2	25.8	32.5	10.2	33.9	43.3

Appendix

Table 29: TinyGSM Simplex Diffusion (SDM) Accuracy (%) (

T=1.0

, 512 steps) across noise concentration schedules, sampling schedules (Linear, Cosine, Adaptive), and churn (

\kappa\in\{0.0,0.2,1.0\}

). We compare Expectation (

P_{t}

), Argmax, and Argmax with Self-Conditioning (+ SC). All SDMs are trained with the adaptive time sampler. We bold the best result in each column and underline the best result within each section (both if achieving both).

Expectation Input Processing ( $P_{t}$ )
Concentration Schedule	Linear			Cosine			Adaptive
Concentration Schedule	$\kappa=0$	$\kappa=0.2$	$\kappa=1.0$	$\kappa=0$	$\kappa=0.2$	$\kappa=1.0$	$\kappa=0$	$\kappa=0.2$	$\kappa=1.0$
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.5,\ell=0.2$ )	6.7	9.8	14.7	14.3	16.8	22.8	10.6	23.1	32.7
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.75,\ell=0.2$ )	4.9	12.0	16.1	14.1	16.1	20.4	10.3	15.3	25.5
Constant ( $\nu=0.5$ )	7.4	9.4	13.8	15.0	14.2	24.0	11.2	21.1	31.8
Constant ( $\nu=0.25$ )	4.2	5.9	8.6	11.4	14.6	18.9	8.0	18.2	30.1

Appendix

Table 30: TinyGSM Simplex Diffusion (SDM) Accuracy (%) (

T=1.0

, 64 steps) across noise concentration schedules, sampling schedules (Linear, Cosine, Adaptive), and churn (

\kappa\in\{0.0,0.2,1.0\}

). We compare Expectation (

P_{t}

Expectation Input Processing ( $P_{t}$ )
Concentration Schedule	Linear			Cosine			Adaptive
Concentration Schedule	$\kappa=0$	$\kappa=0.2$	$\kappa=1.0$	$\kappa=0$	$\kappa=0.2$	$\kappa=1.0$	$\kappa=0$	$\kappa=0.2$	$\kappa=1.0$
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.5,\ell=0.2$ )	27.3	36.0	40.4	30.8	38.5	42.4	19.3	43.9	49.0
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.75,\ell=0.2$ )	27.7	36.4	38.2	30.8	39.3	42.5	19.8	40.8	44.3
Constant ( $\nu=0.5$ )	27.0	30.5	41.7	29.5	33.3	43.3	22.2	35.0	44.8
Constant ( $\nu=0.25$ )	18.3	25.0	30.1	22.5	31.2	38.5	11.3	38.3	42.5

Appendix

Table 31: TinyGSM Simplex Diffusion (SDM) Accuracy (%) (

T=0.1

, 512 steps) across noise concentration schedules, sampling schedules (Linear, Cosine, Adaptive), and churn (

\kappa\in\{0.0,0.2,1.0\}

). We compare Expectation (

P_{t}

Expectation Input Processing ( $P_{t}$ )
Concentration Schedule	Linear			Cosine			Adaptive
Concentration Schedule	$\kappa=0$	$\kappa=0.2$	$\kappa=1.0$	$\kappa=0$	$\kappa=0.2$	$\kappa=1.0$	$\kappa=0$	$\kappa=0.2$	$\kappa=1.0$
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.5,\ell=0.2$ )	11.3	16.7	20.5	25.7	28.7	32.0	18.9	33.0	39.5
Const.-Lin. ( $\nu_{0}=0.2,\nu_{1}=0.75,\ell=0.2$ )	11.7	22.5	28.8	25.8	29.2	33.6	19.0	32.1	38.8
Constant ( $\nu=0.5$ )	15.3	16.2	20.0	26.8	21.1	33.3	21.8	31.9	40.1
Constant ( $\nu=0.25$ )	6.2	7.1	9.5	18.0	19.9	25.1	12.1	24.0	31.0

Appendix

Table 32: TinyGSM Simplex Diffusion (SDM) Accuracy (%) (

T=0.1

, 64 steps) across noise concentration schedules, sampling schedules (Linear, Cosine, Adaptive), and churn (

\kappa\in\{0.0,0.2,1.0\}

). We compare Expectation (

P_{t}

Autoregressive
Model	Temp. ( $T$ )	Steps (NFE)	pass@1 (%) $\uparrow$	pass@2 (%) $\uparrow$	pass@5 (%) $\uparrow$	AST Div. $\uparrow$	AST Div. (Correct) $\uparrow$
	0.1	512	62.6	66.3	69.6	9.6	6.2
	1.0	512	52.6	66.2	77.6	36.7	29.5
Absorbing Diffusion (MDM † ; ELBO; Linear Schedule)
	0.1	8	4.5	7.7	13.3	40.3	8.0
	0.1	64	27.4	37.7	52.3	33.4	19.2

Appendix

Table 33: Multi-Sample Accuracy and AST Structural Diversity (

K=5

samples per problem) on TinyGSM across sampling steps

\text{NFE}\in\{8,64,512\}

and temperatures

T\in\{0.1,1.0\}

. † trained with uniform time sampler (evaluated with linear sampling schedule), ‡ trained with adaptive time sampler (evaluated with adaptive sampling schedule). “ELBO” denotes the evidence lower bound objective, “SC” denotes Self-Conditioning, and “PC” denotes the Predictor-Corrector sampler. Arrows (

\uparrow

\downarrow

) indicate whether higher or lower values are better. For each column, we bold the overall best result and underline the best result within each section. For 64 and 512 steps, pass@1 is the single-sample accuracy from Tables 25 , 26 , 27 and 28 (baselines), Tables 29 , 30 , 31 and 32 (SDMs,

\nu_{0}=0.2

\nu_{1}=0.5

\ell=0.2

) and Figure 13 (distilled SDM); for 8 steps it is the accuracy of the first of the

K

samples. For AR, this is greedy decoding in the

T=0.1

row and sampling in the

T=1.0

row.

Explore similar work

Aug 11, 2026cs.CL

Simplex Relaxation for Discrete Diffusion

Discrete diffusion models for categorical generation are defined by a corruption kernel, which determines the intermediate state space and the associated reverse prediction problem. We study uniform discrete diffusion and ask whether its training objective and reverse transitions can be enriched without changing the underlying categorical corruption process. We introduce Simplax, an exact Dirichlet--categorical augmentation that couples each corrupted categorical state with an auxiliary simplex-valued variable while preserving the original uniform diffusion process as its categorical marginal. This augmentation yields a tractable Rao--Blackwellized reverse-bridge objective and a corresponding stochastic reverse sampler, while retaining the corrupted categorical state as the denoiser input. Empirically, Simplax improves the generative perplexity--entropy tradeoff on unconditional OpenWebText generation. On Sudoku, a model trained exclusively on

30

-clue puzzles achieves the highest accuracy among the compared methods across all evaluated clue densities, including the minimum uniquely solvable

17

-clue regime, and also achieves the highest validity in unconditional generation.

Jinya Sakurai, Patrick Pynadath, Satoshi Hayakawa +4

1NTU Singapore · 2The University of Tokyo · Institute for Advanced Intelligence and Computing (IAIC), A*STAR +2

Diffusion Models Data Generation Data Augmentation Convex Relaxation

Ask PDF

Sep 30, 2026cs.LG

Distribution Matching Distillation for Continuous Diffusion Language Models

Continuous diffusion language models generate all tokens in parallel, yet high-quality generation can still require hundreds of network evaluations (NFEs). We study how distributional distillation can reduce this cost by exploiting the student's probabilistic token outputs. Our unified formulation connects the student's output parameterization to the resulting gradient estimators and yields two methods with the same student architecture and reverse-KL matching objective: Simplex-DMD uses continuous token relaxations and pathwise gradients, while Reinforce-DMD uses categorical sampling and REINFORCE with a learned density ratio. We develop both methods for multi-step generation and investigate the training and sampling choices associated with each parameterization. On OpenWebText, for sequences of 1,024 tokens, Simplex-DMD achieves a generative perplexity of 45.6 at a unigram entropy of 5.44 nats in just 4 NFEs, a 49% reduction relative to the strongest evaluated diffusion baseline at matched entropy and sampling budget. Reinforce-DMD improves the frontier at larger budgets, reaching a generative perplexity of 14.9 at an entropy of 5.00 nats with 256 NFEs, a 20% reduction under the same comparison protocol.

Paul Le Van Kiem, Dario Shariatian, Umut Simsekli +1

Inria, PSL Research University · Cohere · CMAP, Ecole Polytechnique

Diffusion Language Models Diffusion Models Dataset Distillation Distribution Matching Distillation Single-Token Output Distributions Simplex

Ask PDF

Jun 1, 2026cs.LG

BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers

Is the uniform-state diffusion framework a more powerful paradigm for discrete diffusion? Recent studies indicate that this may be the case. In combination with predictor-corrector samplers, uniform-state diffusion models (USDMs) produce samples of higher-quality than masked diffusion models (MDMs), and USDMs equal or outperform MDMs in downstream tasks, even though they exhibit greater perplexity. Two issues remain unresolved. First, existing work compares uniform and masked diffusion with un-informed correctors that re-inject noise at random positions, rather than targeting tokens most likely to be wrong. Second, prior work compares full-sequence diffusion models, so we do not know whether the same conclusion holds when tokens are generated block by block. To address these issues, we introduce BlockGen, a blockwise sequence model that we instantiate with both masked and uniform diffusion. BlockGen trains on a mixture of block sizes and its likelihood interpolates between AR and pure diffusion more finely than models with a fixed block size. BlockGen enables AR-informed predictor-corrector sampling (ARPC), which combines AR and diffusion predictions to re-generate unlikely tokens without an auxiliary verifier. Under ancestral sampling, uniform outperforms masked in the block-by-block setting, especially in the few-step regime. Under ARPC, the gap closes and reverses at high NFE. With block size

16

on GSM8K, MDMs reach slightly higher accuracy than USDMs, and we observe a similar trend in Generative Perplexity on OpenWebText. Find our code at https://github.com/jdeschena/blockgen.

Justin Deschenaux, Caglar Gulcehre

EPFL Lausanne, Switzerland · EPFL, Lausanne, Switzerland · Microsoft AI

Diffusion Models Diffusion Language Models Diffusion Dynamics Accelerated Sampling Sequence Modeling Motion Generation

Ask PDF