Masked diffusion language models (dLLMs) can in principle generate text faster than autoregressive (AR) models, since they denoise many tokens at once. Recent systems have begun building serving infrastructure for dLLMs, but none first measure how these models behave under real, concurrent serving load. Serving systems built without this grounding risk carrying over assumptions from AR serving that may not hold for dLLMs. We characterize dLLM serving to close this gap, using LLaDA-8B-Instruct with a D2F (Discrete Diffusion Forcing) LoRA adapter on a single NVIDIA H200 GPU, evaluated on GSM8K and HumanEval. We report three findings. First, request difficulty, the number of denoising steps a request needs, is discrete rather than continuous: requests fall into 11 fixed step-count levels (178 + 29k), and no signal we test predicts the level before generation starts (best R2 = 0.150). Second, benchmarks with short generation budgets below 320 tokens understate serving variance, since requests are cut off before the latency spread appears. Third, only 24% of single-request wall-clock time is GPU computation; the rest is CPU-side dispatch overhead. Batching mainly helps by amortizing this overhead: sharing one forward pass per denoising step improves throughput by 16.0x at batch size 16 over a per-request-dispatch baseline. We also argue structurally that output quality should not degrade with batch size, stating three assumptions this rests on; we measure 74 to 76% GSM8K accuracy at single-request scale. Finally, we derive a batch-timeout rule for fixed-fill synchronized batching under Poisson arrivals. Together, these results show that serving diffusion language models needs parallelism at the level of each denoising step, which differs from AR serving in how admission and eviction interact with an already shared forward pass.
Figures & tables
Fig. 1: The block-threshold mechanism that induces discrete serving behavior in our test system. A new block starts only after its predecessor crosses τadd , and tokens in the active block unmask only after confidence crosses τdecode . We use this mechanism to explain serving-level phenomena rather than study the mechanism in isolation.
Fig. 2: Denoising step counts for 100 GSM8K requests fall into exactly 11 discrete levels, spaced 29 steps apart.
Fig. 3: Denoising step count predicts end-to-end latency across the 11 discrete difficulty levels ( r=0.9997 ).
Fig. 4: D2F increases serving-latency variance by roughly 2.7 × relative to the base model ( CV=0.272 vs. CV=0.100 ) on identical GSM8K prompts under a 512-token generation ceiling.
Fig. 5: None of the tested early signals predict the eventual difficulty level; all R2 values remain far below the R2=0.5 heuristic we use as a scheduling-usefulness floor.
Fig. 6: Short generation budgets systematically understate serving variance (100 GSM8K requests per budget). (a) CV rises with budget, with the sharpest increase between 256 and 512 tokens once truncation no longer masks variance. (b) Truncation falls from 100% to 1%. (c) Accuracy saturates by 512 tokens. (d) Tail-to-median latency peaks at 512 tokens and decreases slightly by 1024 once typical completion length is consistently reached.
gen_len
CV
P50
P90
P99
Trunc.
Acc.
128
0.067
2.84s
3.12s
3.48s
100%
6.0%
256
0.080
5.38s
5.70s
5.92s
83%
48.0%
512
0.283
6.46s
9.72s
11.48s
3%
76.0%
1024
0.360
6.41s
9.05s
10.66s
1%
73.0%
TABLE I: Generation-length scaling (100 GSM8K requests per budget)
Fig. 7: Synchronized batching scales close to linearly up to batch size 16 (16.0 × over single-request throughput). In contrast, naive per-request dispatch (one forward pass per request) levels off far below linear scaling.
Fig. 8: The overhead of naive per-request dispatch relative to synchronized batching tracks batch size, consistent with one forward pass per request versus one forward pass per step for the batch.
Fig. 9: Single-request wall-clock breakdown from kernel-trace summation: 76% CPU-side dispatch overhead in the per-block control loop versus 24% GPU kernel time.
Fig. 10: GSM8K exact-match accuracy under single-request inference at gen_length =512 (100 problems).
Symbol
Meaning
B
Fixed batch size (synchronized batching), B=8 throughout
λ
Request arrival rate (requests/s), Poisson process
S
Mean service time per batch, measured as S=8.74 s
T
Batch timeout: max wait before dispatching an underfull batch
Minimum safe timeout; if T<Tmin , the queue becomes unstable
TABLE II: Notation used in the scheduling model (§ V )
Fig. 11: Mean latency under fixed-fill synchronized batching is U-shaped in utilization ρ : batch-fill wait dominates at low ρ , while queueing behind other batches dominates at high ρ .
ρ
λ (req/s)
P50
P90
P99
Tmin
SLA viol. b
0.30
0.275
19.6s
36.2s
52.3s
5.1s
33%
0.50
0.458
15.8s
25.9s
34.7s
6.6s
12%
0.70
0.641
14.6s
22.2s
29.2s
7.2s
5%
0.85
0.778
16.1s
23.9s
30.9s
7.5s
8%
0.95
0.870
27.2s
78.4s
87.9s
7.6s
55%
b SLA violation defined as end-to-end latency exceeding 25s.
TABLE III: Latency percentiles under Poisson arrivals from a Monte Carlo simulation over n=8 measured batch service times (batch size B=8 )
Efficient serving of diffusion large language models (dLLMs) is hindered by convergence heterogeneity: when batching multiple requests, different sequences converge at different rates, causing faster requests to stall behind slower stragglers and introducing compute bubbles and tail latency. We present BlockServe, a continuous batching framework that integrates block-grained scheduling -- immediately evicting completed requests at block boundaries -- with mixed-state execution that extends dual cache and parallel decoding to heterogeneous batches via gather-scatter indexing. Furthermore, a compute-aware admission controller expands effective batch capacity through token-budgeted refill. On Dream and LLaDA across five benchmarks, BlockServe achieves 1.9--10.6× throughput over Fast-dLLM with comparable generation quality, establishing block-grained scheduling as a foundation for high-throughput offline dLLM inference.
Diffusion language models (dLLMs) generate text by iteratively denoising a masked response and can commit multiple output positions per model invocation. Their bidirectional attention prevents exact autoregressive-style KV caching, since committing one position shifts the KV activations of all others. Approximate caching techniques such as Fast-dLLM and dKV-Cache refresh KV activations repeatedly and reuse them across intervening decodes, inducing a repeated prefill/decode structure. This makes AR serving mechanisms relevant to dLLMs, but not directly applicable. dLLM decodes are block-sized rather than token-sized, prefills recur, and bidirectional attention precludes the chunked prefill mechanism used for stall-free colocated serving. We present Sangam, a serving system for cached dLLM inference. Sangam introduces a deficit token-budget scheduler that admits in-flight decodes first, admits whole indivisible prefills only when the accumulated token budget allows, and carries unused budget forward. This achieves amortized stall-free scheduling. Disaggregated serving avoids prefill-decode interference but suffers from prefill/decode resource partitioning problem. Sangam adopts a hybrid serving strategy, overflowing prefills onto decode workers to relieve prefill under-provisioning, and uses the same deficit-budget scheduler to protect those workers' decodes from the overflow. We show that like AR serving, dLLM serving design space is governed by prefill-decode interference and prefill/decode partitioning. Colocated serving is most effective on decode-heavy workloads, cutting mean latency by 9-20% over hybrid execution on LLaDA-8B ShareGPT; while hybrid execution is most effective on prefill-heavy workloads, cutting mean latency by 8-20% over colocated execution on Dream-7B arXiv. Sangam is available at https://github.com/UT-InfraAI/sangam.
Nitin Kedia, Saurabh Agarwal, Myungjin Lee +1
The University of Texas at Austin Austin, United States · Cisco Research Bellevue, United States
A popular selling point of diffusion large language models (dLLMs) is their capacity for parallelism: the ability to generate sequences of text far more efficiently than autoregressive models, which require one forward pass per token. Yet among the many competing paradigms for dLLMs, from masked to uniform to Gaussian diffusion, principled understanding of how these different proposals compare in parallelism remains limited. In this work, we initiate a fine-grained comparison of the capacity for parallelism among these three leading approaches and prove the following: - Uniform and Gaussian diffusion can sample in a number of forward passes which scales with the dual total correlation of the underlying distribution, a measure of intrinsic complexity which can be much smaller than the context length. Previously, it was only known how to achieve this using masked diffusion. - For a certain family of random empirical measures, we show that Θ(d) forward passes are necessary and sufficient to sample using uniform or Gaussian diffusion, yet there exist approximate score oracles for which Ω(d) forward passes are needed for masked diffusion. This establishes the first provable separation in parallelism between the three prevailing dLLM paradigms. Contrary to popular intuition that masked diffusions are harder to parallelize because they must commit to token values, the latter separation instead comes from the fact that the critical windows in masked diffusion sampling are asymptotically narrower than those in uniform and Gaussian diffusion sampling.