Organizations: Polyadic Research Group · School of Computer Science, Peking University · SAGA Lab, The Chinese University of Hong Kong, Shenzhen · The University of California, Los Angeles · Dots Studio, Xiaohongshu
Continuous diffusion language models (CDLMs) and speculative drafters offer efficient language generation but are typically evaluated differently: CDLMs generate independently through iterative refinement, whereas drafters propose text for target-model verification and correction. We compare their architectures without verification, using LangFlow for CDLMs and adapting DSpark's Markov and recurrent-head architectures for standalone generation. All models are trained from scratch for unconditional, single-block OpenWebText generation at lengths 16, 128, and 1024, without target features or verification, using common preprocessing, backbone dimensions, and training-update budgets. We evaluate generation perplexity (GenPPL) with an external language model and distributional similarity to reference text with MAUVE. With default denoising steps equal to sequence length, LangFlow outperforms both DSpark variants on both metrics at lengths 128 and 1024, while DSpark-RNN leads at 16. A conditional KL analysis characterizes dependencies excluded by the Markov head's previous-token conditioning. Recurrent memory removes this restriction and improves both metrics, but the longer-sequence gap persists. Teacher probability matching further lowers Markov GenPPL without observed MAUVE gains. Increasing LangFlow's denoising steps at fixed weights improves both metrics at lengths 16 and 128, changing the length-128 ordering relative to DSpark-RNN. These results establish LangFlow's longer-sequence advantage under the evaluated budgets and highlight iterative full-sequence refinement as a route to improving standalone generation beyond gains from recurrent head memory.
Figures & tables
Figure 1: Which information can change during generation? (a) For fixed context C , the backbone produces the same position-specific logits Ui(C) on every run. At step i≥2 , only the previous sampled token Xi−1 joins Ui(C) at the Markov head; sampled tokens do not feed back into the backbone. The first token is sampled separately from qθ,1 . DSpark-RNN adds persistent head memory and is not shown. (b) A continuous generator instead updates a sample-dependent full-sequence state, allowing information to pass across positions before decoding. Arrows show information access, not performance; Fθ abstracts a state update rather than prescribing one sampler.
Figure 2: Standalone generation with Markov, recurrent, and diffusion models. Raw metrics at the 100,000-update EMA checkpoint. Lower GenPPL and higher MAUVE are better. Each length uses separately trained models, with LangFlow NFE =L ; lines guide the eye across lengths. RNN improves on Markov at every length and leads at length 16; LangFlow leads both drafters at lengths 128 and 1024.
Figure 3: Teacher L1 supervision and the remaining gap to LangFlow. Markov and Markov+L1 share the student architecture and update budget; LangFlow uses NFE =L . Lower GenPPL and higher MAUVE are better. L1 lowers GenPPL at every length without an observed MAUVE gain. LangFlow leads both Markov variants on both metrics at lengths 128 and 1024; length 16 has a mixed ordering. Bars are grouped by length with zero baselines; each length uses separately trained models.
Figure 4: At length 128, additional iterations overtake DSpark-RNN. Blue curves show LangFlow’s raw GenPPL and MAUVE at fixed weights; orange dashed lines show DSpark-RNN. The first tested lead occurs at NFE64 for GenPPL and NFE32 for MAUVE. Thus, NFE64 is the first tested budget with a lead on both. Lower GenPPL and higher MAUVE are better. Values are point estimates; lines guide the eye.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
C
All pre-block information available to the generator.
c0
Fixed conditioning input for unconditional, single-block generation.
B , L
Generation block size and total output length, respectively.
p , q
Reference and generator distributions over token blocks.
Ui(C)
Fixed base-logit vector at position i .
Q1
All position-dependent first-order conditional block distributions.
Appendix
Table 1: Notation used in the architectural analysis.
(X1,XB)
Reference p
Projection q⋆
Endpoint consistency
(0,0)
1/2
1/4
Yes
(0,1)
0
1/4
No
(1,0)
0
1/4
No
(1,1)
1/2
1/4
Yes
Appendix
Table 2: Analytic probabilities, not experimental results. Every interior token is # . The optimal first-order projection preserves endpoint marginals but introduces two inconsistent combinations.
Model
Stored parameters
Excluding unused rows
DSpark-Markov / DSpark-Markov+L1
396,316,800
279,631,488
DSpark-RNN
397,300,608
280,615,296
LangFlow
327,085,955
327,085,955
Appendix
Table 3: Student parameter accounting. The last column subtracts only unused scaffold-embedding rows; it is not a measure of effective statistical capacity. The teacher is excluded from student counts.
L=16
L=128
L=1024
Model
GenPPL ↓
MAUVE ↑
GenPPL ↓
MAUVE ↑
GenPPL ↓
MAUVE ↑
DSpark-Markov
1167.77
0.3409
774.39
0.0221
524.79
0.0104
DSpark-RNN
488.28
0.6606
303.52
0.2682
216.93
0.0780
LangFlow
1297.25
0.4655
214.60
0.6392
50.16
0.8370
DSpark-Markov+L1
668.48
0.3404
419.82
0.0179
262.55
0.0092
Appendix
Table 4: OpenWebText unconditional generation at the 100,000-update EMA checkpoint. Bold marks the best value within each length. LangFlow uses NFE =L ; drafters use one backbone pass and L head steps. GenPPL and MAUVE use 1,024 and 5,120 generated samples, respectively.
Figure 5: Quality along training under the common evaluation protocol and default sampling budgets. All 84 recorded points are shown, sorted by update count. DSpark-Markov+L1 at every length and LangFlow at L=1024 have only a 100,000-update point under this protocol; no intermediate curve is inferred. Outlined L1 diamonds keep the nearly coincident Markov endpoints visible at their measured coordinates. The curves are checkpoint trajectories, not averages over training seeds.
L=16
L=128
NFE
GenPPL ↓
MAUVE ↑
GenPPL ↓
MAUVE ↑
8
2296.18
0.2842
992.48
0.0591
16
1297.25
0.4655
511.70
0.2265
32
939.76
0.5326
328.63
0.4622
64
807.12
0.5502
248.90
0.5601
128
731.62
0.5787
214.60
0.6392
Appendix
Table 5: Fixed-checkpoint LangFlow generation quality across denoising budgets. All settings use 1,024 GenPPL and 5,120 MAUVE samples; values are point estimates.
Figure 6: At length 16, iteration improves quality without overtaking DSpark-RNN. Raw LangFlow GenPPL and MAUVE improve across all five tested budgets, while the recurrent drafter retains the better value on both metrics. At NFE128, LangFlow reaches GenPPL 731.62 and MAUVE 0.5787, versus RNN’s 488.28 and 0.6606. Weights are fixed; values are point estimates, and connecting lines guide the eye.
Figure 7: Lexical diagnostics on 1,024 full source-token sequences per model and length. Entropy measures corpus-level lexical spread; repeated 4-grams measure within-sequence duplication. Neither is a standalone measure of text quality. Points at different lengths use different trained models and corpus token counts.
L=16
L=128
L=1024
Model
H
R4
H
R4
H
R4
DSpark-Markov
7.071
0.000
7.509
0.009
7.648
0.034
DSpark-RNN
7.035
0.030
7.445
0.029
7.577
0.237
LangFlow
7.214
0.008
7.418
0.445
7.455
7.332
DSpark-Markov+L1
6.774
0.068
6.849
0.270
6.840
1.153
Appendix
Table 6: Corpus unigram entropy H (nats) and repeated 4-grams R4 (percent) on 1,024 full source-token sequences. Values are rounded independently.
Model
Prefix
Full text
95% bootstrap interval
DSpark-Markov
524.79
523.26
[519.88,526.66]
DSpark-RNN
216.93
215.69
[212.27,219.17]
LangFlow
50.16
50.15
[48.89,51.42]
DSpark-Markov+L1
262.55
262.41
[260.49,264.24]
Appendix
Table 7: GenPPL at L=1024 : original prefix and complete-output scoring.
L
RNN − Markov
LangFlow − RNN
16
[0.2585,0.3298]
[−0.2460,−0.1541]
128
[0.2161,0.2540]
[0.3405,0.4295]
1024
[0.0561,0.0834]
[0.7282,0.7987]
Appendix
Table 8: MAUVE score-difference ranges over 20 clustering seeds.
Metric
Markov
Markov+L1
Difference
95% interval
GenPPL
1152.99
677.18
−475.81
[−491.95,−459.68]
MAUVE
0.3424
0.3347
−0.0078
[−0.0168,0.0013]
Appendix
Table 9: Mean scores from 20 independent L=16 batches per method. Differences are Markov+L1 minus Markov, with approximate 95% Welch intervals.
Continuous diffusion has served as a foundation for high-fidelity, controllable, and few-step generation across continuous data modalities such as images, videos, and molecular structures. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts. Existing categorical and simplex-based approaches operate over extremely large and sparse language spaces, while prior embedding-space approaches avoid this sparsity but lack a well-selected design space. In this work, we close this gap with LangFlow by connecting embedding-space DLMs to Flow Matching, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, which improves both likelihood and sample quality for embedding-space DLMs and behaves differently from its use in discrete diffusion. Putting everything together, LangFlow is competitive with top discrete DLMs on generative quality, reaching a generative perplexity of 92.3 on LM1B and 36.5 on OpenWebText. It also outperforms discrete counterparts on reasoning tasks like Sudoku. As the first continuous DLM shown to rival discrete diffusion in generative quality, LangFlow provides clear evidence that continuous diffusion is a promising paradigm for language modeling. Code: https://github.com/nealchen2003/LangFlow.
Continuous diffusion language models (CDLMs) offer an alternative approach to parallel language generation; however, most existing studies directly use transformer architectures from either causal large language models (LLMs) or Diffusion Transformers (DiTs) for image generation, which were not optimized for CDLMs. In this paper, we conduct a systematic study on Transformer architectures for CDLMs and introduce Language Diffusion Transformer (LangDiT), combining feature and query-key normalization, wider noise conditioning, likelihood-shortcut gradient routing and calibration, and confidence-gated training self-conditioning. LangDiT significantly reduces generation perplexity in matched LangFlow comparisons, establishing a new state of the art in continuous language generation across different model formulations. Using LangFlow as a representative CDLM, LangDiT as a plug-and-play backbone leads to a new state-of-the-art model with a generation perplexity (GenPPL) of 21.72 using 1024 NFEs and 44.72 GenPPL using 128 NFEs on the unconditional generation of 1024-token sequences on the OpenWebText dataset. When adapted to Plaid-style CDLM formulations, LangDiT results in a new leading GenPPL of 11.33 using 1024 denoising NFEs, and 27.51 GenPPL using 128 denoising NFEs. These gains are achieved while retaining each model's underlying representation, training objective, and inference sampler, isolating transformer architecture as a fundamental source of improvement. Our study identifies architectural design as an important source of improvement in continuous diffusion language models.
Weijian Luo, Yujian Chen, Hao Jiang +2
Dots Studio, Xiaohongshu · Polyadic Research Group · Peking University +2
Language models based on discrete diffusion have attracted widespread interest for their potential to provide faster generation than autoregressive models. Despite their promise, these models typically produce samples whose quality sharply degrades in the few-step regime, preventing a dramatic speedup in practice. Here, we show that language models based on continuous flows over one-hot token embeddings can outperform discrete diffusion in both quality and speed. Importantly, our continuous formulation defines a unique flow map that can be learned directly for efficient few-step inference, a structure we show is unavailable to discrete methods. In this setting, we show that both the flow and its associated flow map can be learned with simple cross-entropy objectives that respect the simplex geometry of the data, and we identify three distinct choices for flow map distillation whose performance we compare in practice. Using these insights, we build a flow language model (FLM), a continuous flow that matches state-of-the-art discrete diffusion baselines on the One Billion Words (LM1B) and OpenWebText (OWT) datasets. We then distill FLM into a flow map language model (FMLM), whose one-step generation exceeds the 8-step quality of recent few-step discrete diffusion language models. Our work challenges the widely-held hypothesis that discrete noising processes are necessary for generative modeling over discrete modalities and paves the way toward accelerated language modeling at scale. Code is available at https://github.com/david3684/flm.