Masked diffusion language models (dLLMs) decode by iteratively denoising a full canvas of tokens, so every forward pass runs a per-token sampling and stopping pipeline over the entire canvas-by-vocabulary logits tensor. We profile single-user decoding of a 26B-parameter diffusion Mixture-of-Experts model on a bandwidth-bound edge accelerator (NVIDIA GB10) and report a result that is easy to miss: after weight quantisation, this logits pipeline, not the expert GEMMs, is the single largest consumer of decode time. Unlike autoregressive decoding, where sampling touches one position's logits per committed token, diffusion decoding processes S times as many logit entries per output token, where S is the number of denoising steps. We show this pipeline is largely redundant or fusable and remove it in two lossless tiers: a bit-exact tier whose output is byte-identical to the reference decoder, and an accuracy-parity tier using a fused single-pass entropy kernel and Gumbel-max sampling. Combined with a W4A16 expert kernel, the full stack reaches 2.4-2.7 times end-to-end speedup over bf16 at unchanged benchmark accuracy on GSM8K, HumanEval, and MATH, on a single 273 GB/s device. We confirm the finding on a second, independently built diffusion MoE (LLaDA2.0-mini: pipeline 28.8% of decode, fused W4A16 3.3 times at accuracy parity) and show it disappears on a dense diffusion model (3.6%). The cost is specific to MoE diffusion, where quantisation makes the backbone cheap. Code and quantised weights are released.
Figures & tables
Component
GSM8K (%)
HumanEval (%)
ms/fwd
Logits / sampling pipeline + host glue
39.8 ± 7.4
34.4 ± 2.7
87.0
Expert GEMMs (W4A16 MoE)
25.3 ± 3.0
25.8 ± 0.5
57.9
Attention
16.3 ± 1.9
17.1 ± 0.4
37.9
Encoder (prompt KV, amortised)
5.9 ± 2.1
9.8 ± 2.3
17.9
RMSNorms
5.9 ± 0.7
5.9 ± 0.2
13.3
Shared MLP
3.5 ± 0.4
3.8 ± 0.1
8.3
Table 1: Decode-time breakdown of the W4A16 model (GB10, canvas 256): share of wall-clock time per bucket, mean ± std over n=20 queries per benchmark, and mean ms per forward over all 40. Buckets are disjoint. The logits/sampling pipeline (with host glue) is the largest bucket on every query.
Configuration
GSM8K
HumanEval
MATH
bfloat16 baseline
70.4
84.3
69.7
W4A16 experts (reference loop)
118.1 (1.68 × )
140.6 (1.67 × )
110.2 (1.58 × )
+ bit-exact pipeline †
134.4 (1.91 × )
167.6 (1.99 × )
123.9 (1.78 × )
+ turbo pipeline
186.2 (2.64 × )
224.9 (2.67 × )
168.4 (2.42 × )
Table 2: End-to-end decode throughput on GB10, single user, reference model.generate ; each cell is tok/s ( speedup over the bfloat16 baseline). Quantising the experts alone gives ∼1.6 – 1.7× ; the two logits-pipeline tiers of Section 3 contribute the rest, reaching 2.4 – 2.7× . Accuracy is preserved at every step : on a same-session n=100 comparison, bfloat16 scores 96/89/88 and turbo scores 97/88/87 (GSM8K / HumanEval / MATH, exact-match or pass@1); the bit-exact tier is token-identical to the W4A16 decoder, and W4A16 quantisation is accuracy-neutral versus bfloat16 ( 96→96 , 87→87 on GSM8K / MATH).
Configuration
tok/s
tier
W4A16 experts (reference loop)
117.5
—
+ entropy dedup
132.4
bit-exact
+ fused entropy & Gumbel-max
165.1
accuracy-parity
+ fused RMSNorm (turbo)
200.3
accuracy-parity
Table 3: Attribution of pipeline optimisations, GSM8K n=20 , same load. Accuracy is stable ( 19 – 20/20 ) throughout. The bit-exact tier’s outputs are byte-identical to the quantised reference (texts and step counts match 20/20 on GSM8K and HumanEval).
Model
Arch.
Pipeline share
DiffusionGemma-26B (W4)
MoE
37.1%
LLaDA2.0-mini (bf16)
MoE
28.8%
LLaDA-8B (bf16)
dense
3.6%
Table 4: Logits-pipeline share of per-forward decode across three masked-diffusion LMs (GB10). It is first-order on MoE models, negligible on the dense one. Below: porting the fused W4A16 MoE kernel to LLaDA2.0-mini gives 3.3 – 3.4× over its bfloat16 reference at accuracy parity ( n=50 ; Δ within run-to-run sampling noise).
Masked diffusion models (MDMs) generate sequences by progressively unmasking several tokens per denoising step, but their reverse process is typically factorized over positions, limiting sample quality in the few-step regime where diffusion's speed advantage over autoregressive decoding matters most. A recent line of work introduces a continuous Gaussian latent, trained as a variational autoencoder, to capture correlations across positions, but such approaches are prone to posterior collapse, where the latent is silently ignored. We propose Enhanced Mixture-of-Experts (E-MoE), which builds the reverse process as a mixture of factorized distributions over a discrete shared latent given by the expert-routing decisions of a Mixture-of-Experts (MoE) backbone, without increasing active parameters over the factorized baseline. Across synthetic multi-modal benchmarks, binarized MNIST, and LM1B, E-MoE improves few-step generation over factorized baselines.
Arseny Ivanov, Alexander Kolesov, Alexander Korotin +2
AXXX, Moscow, Russia · Applied AI Institute, Moscow, Russia
Masked diffusion language models enable parallel token decoding, providing a promising alternative to the sequential nature of autoregressive generation. However, their iterative denoising process remains computationally expensive because it repeatedly processes the entire sequence at every step. We observe that across these diffusion steps, most token representations remain stable; only a small subset, which we term salient tokens, contributes meaningfully to the next update. Leveraging this temporal sparsity, we present DyLLM, a training-free inference framework that accelerates decoding by selectively computing only these salient tokens. DyLLM identifies saliency by measuring the cosine similarity of attention contexts between adjacent denoising steps. It recomputes feed-forward and attention operations only for salient tokens while reusing cached activations for the remainder. Across diverse reasoning and code-generation benchmarks, DyLLM achieves up to 9.6x higher throughput while largely preserving the baseline accuracy of representative open-source diffusion LLMs, LLaDA, and Dream.
Younjoo Lee, Seungkyun Dan, Junghoo Lee +2
Seoul National University, Seoul, Republic of Korea
Diffusion Large Language Models (dLLMs) have recently emerged as a promising alternative to autoregressive models, offering competitive performance while naturally supporting parallel decoding. However, as dLLMs are increasingly integrated with Mixture-of-Experts (MoE) architectures to scale model capacity, a fundamental mismatch arises between block parallel decoding and token-level expert selection. Specifically, each dLLM forward pass processes multiple tokens with bidirectional dependencies, whereas conventional MoE layers route each token independently. This mismatch substantially increases the number of uniquely activated experts, making inference increasingly memory-bound. To address this, we propose dMoE, a simple yet effective block-level MoE framework. The central idea of dMoE is to aggregate token-level expert distributions within each block into a unified block-level expert distribution, which is then used to guide block expert routing in a coherent manner. In this way, dMoE substantially reduces the number of uniquely activated experts during inference without sacrificing performance, thereby mitigating the memory-bound bottleneck. Extensive experiments across a variety of benchmarks demonstrate the effectiveness of dMoE. On average, dMoE reduces the number of uniquely activated experts from 69.5 to 16.6 while retaining 99.55% of the original performance. Meanwhile, it reduces memory usage by 74.36% to 76.78% and achieves 1.36× to 2.04× end-to-end latency speedup. Code is available at: https://github.com/fscdc/dMoE.