Continuous diffusion language models show promising potential in text generation tasks, but are prone to problems such as text repetition, semantic drift, and generation quality degradation when the generation length exceeds the training length. To address these challenges, we propose RangeSWA, a training-free length extrapolation method. The method combines local sliding-window attention, remote block summaries, and dynamic content retrieval to preserve local details and support long-range context access. In addition, generation degradation caused by length extrapolation is mitigated through positional remapping, conditioning attention correction, and denoising-stage-adaptive window scheduling. We conducted experiments on multiple continuous diffusion language models, and the results show that RangeSWA can effectively reduce generative perplexity and text repetition rates in long-text generation.
Figures & tables
Figure 1: Context access of different attention mechanisms under length extrapolation. (a) Full attention accesses all tokens but may exceed the trained position range. (b) Sliding-window attention attends only to nearby tokens and ignores distant information. (c) RangeSWA uses both local and distant information and maps distant positions back into the trained range.
Figure 2: Overview of the RangeSWA attention mechanism. Each body query attends to a local window, summaries of remote blocks and the top- K retrieved blocks, together with control tokens, in a single softmax after positional remapping. The right panel shows the attention masks.
1k
2k
4k
8k
Method
PPL ↓
H
R4↓
PPL ↓
H
R4↓
PPL ↓
H
R4↓
PPL ↓
H
R4↓
ELF-B Full
24.42
5.15
4.4
21.10
4.97
5.1
3.49
1.91
43.1
1.36
0.94
42.6
→ Full + PI
24.42
5.15
4.4
78.28
5.13
13.2
10.77
3.83
49.8
1.62
2.05
76.1
→ Full + NTK
24.42
5.15
4.4
16.68
4.23
12.5
8.67
2.36
57.9
1.72
0.55
15.3
→ Self-Extend*
24.42
5.15
4.4
20.05
5.01
7.0
7.97
2.75
48.8
5.90
1.89
82.7
→ SWA(256,4)
36.68
5.35
3.5
40.24
5.75
3.6
56.79
6.06
3.4
153.55
5.56
2.8
Table 1: Length extrapolation on ELF-B/M/L at 1k–8k. PPL, entropy H and R4 (%) over 1000 samples per configuration; markers are defined in Appendix B .
Figure 3: Low Gen.PPL can mask severe text repetition. (a) Opening text of one sample per method on ELF-M at 8k. (b) Gen.PPL and R4 on ELF-B/M/L at 8k; hatched bars mark H<5 .
CoLa: 1k prompt
LangFlow: 1k
LangFlow: 2k
Method
PPL ↓
H
R4↓
PPL ↓
H
R4↓
PPL ↓
H
R4↓
Full
80.92
3.57
1.3
36.17
5.25
4.9
99.36
5.86
0.2
SWA
20.32
3.61
7.3
5.61
4.25
51.1
6.09
4.86
49.5
SumSWA
28.61
3.61
4.7
15.63
5.08
23.3
42.75
5.75
5.3
RangeSWA (Ours)
56.50
3.59
1.6
36.97
5.27
4.3
33.94
5.68
5.3
Table 3: Transfer to CoLa and LangFlow. PPL, H and R4 (%); CoLa is scored conditionally after a 1k prompt and LangFlow unconditionally (Appendix A.5 ).
Figure 4: Retaining body prefixes is not enough for sliding-window attention. (a) Dropping distant attention mass m scales control-token weights by 1/(1−m) . (b) Control-token mass under SWA(256,4) relative to Full, per layer at 1k. (c) With a 256-token window, LangFlow repeats heavily with or without prefixes.
2k
4k
8k
Method
s ↓
Speedup ↑
s ↓
Speedup ↑
s ↓
Speedup ↑
Full
2.08
–
4.48
–
10.84
–
RangeSWA (Ours)
2.36
0.88 ×
4.46
1.01 ×
8.98
1.21 ×
RangeSWA-Lite (Ours)
1.81
1.15 ×
3.41
1.32 ×
6.71
1.62 ×
Table 6: bfloat16 (BF16) sampling latency on ELF-B. Median time over five runs and paired speedup over Full on one H100 with batch size 16.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Fixed-state diagnostics at 1k. (a) Control attention mass ratio and (b) summary–readout cosine on ELF-B/M/L with a 256-token window and no retained body tokens. (c) LangFlow readout error with and without the first four body tokens.
Figure 7: Perplexity–repetition trajectories over 1k–8k. Each point is 1000 samples, and lines connect lengths. Full reaches the lowest PPL through heavy repetition at B4k/8k and M8k.
BF16
Strict FP32
Method
2k
4k
8k
2k
4k
8k
Full
2.08
4.48
10.84
7.74
20.17
58.30
RangeSWA Triton (Ours)
2.36 (0.88 × )
4.46 (1.01 × )
8.98 (1.21 × )
10.92 (0.71 × )
23.42 (0.86 × )
50.52 (1.15 × )
RangeSWA-Lite Triton (Ours)
1.81 (1.15 × )
3.41 (1.32 × )
6.71 (1.62 × )
6.63 (1.17 × )
13.41 (1.51 × )
26.74 (2.18 × )
Appendix
Table 15: Sampling time and speedup over Full. ELF-B on one H100 in BF16 and strict FP32; median of five runs.
Diffusion language models (DLMs) generate text by iteratively denoising masked sequences, but standard decoding either fixes the sequence length or relies on ad hoc stopping rules, often leading to unnecessary denoising steps. We recast length selection as a discrete-time survival problem over the end-of-sequence token and propose a plug-in, training-free length predictor that can be added to any existing DLM. Across reasoning and code-generation benchmarks, survival-guided length decoding speeds up inference by up to 7 times while preserving task accuracy. We further find that predicted lengths vary widely even within the same dataset, making model performance sensitive to the chosen length.
Ivan Kobyzev, Abbas Ghaddar, Yufei Cui
Huawei Noah’s Ark Lab, Montreal Research Center, Canada
Large language models have achieved remarkable success under the autoregressive paradigm, yet high-quality text generation need not be tied to a fixed left-to-right order. Existing alternatives still struggle to jointly achieve generation efficiency, scalable representation learning, and effective global semantic modeling. We propose Cola DLM, a hierarchical latent diffusion language model that frames text generation through hierarchical information decomposition. Cola DLM first learns a stable text-to-latent mapping with a Text VAE, then models a global semantic prior in continuous latent space with a block-causal DiT, and finally generates text through conditional decoding. From a unified Markov-path perspective, its diffusion process performs latent prior transport rather than token-level observation recovery, thereby separating global semantic organization from local textual realization. This design yields a more flexible non-autoregressive inductive bias, supports semantic compression and prior fitting in continuous space, and naturally extends to other continuous modalities. Through experiments spanning 4 research questions, 8 benchmarks, strictly matched ~2B-parameter autoregressive and LLaDA baselines, and scaling curves up to about 2000 EFLOPs, we identify an effective overall configuration of Cola DLM and verify its strong scaling behavior for text generation. Taken together, the results establish hierarchical continuous latent prior modeling as a principled alternative to strictly token-level language modeling, where generation quality and scaling behavior may better reflect model capability than likelihood, while also suggesting a concrete path toward unified modeling across discrete text and continuous modalities.
Hongcan Guo, Qinyu Zhao, Yian Zhao +8
ByteDance Seed · The Australian National University · Peking University +2
We introduce SERF (Self-Regularized Flow Language Models), a simple self-supervised framework for improving continuous diffusion language models by exploiting the denoising trajectory itself. While continuous flow-based language models offer a promising alternative to autoregressive generation, their training objectives typically rely on supervision from clean target tokens, leaving the denoising trajectory underexploited as a learning signal. SERF addresses this limitation through noisy-to-clean self-distillation: an exponential moving average (EMA) teacher receives a cleaner version of the student's noisy input along the same interpolation path and provides additional supervision. We investigate two variants: SERF-L, which aligns the student's intermediate representations with deeper teacher representations, and SERF-P, which aligns their output token distributions, either at the student's output or at an early student layer. All variants augment the standard training objective without requiring external pretrained models or modifying the inference procedure. Experiments across three continuous language-model paradigms (one-hot FLM, latent ELF, and hyper-spherical S-FLM) demonstrate substantial improvements in training efficiency, generation quality, and few-step generation. In particular, SERF achieves up to 7.7x wall-clock training acceleration on OpenWebText, improves MAUVE by 24.3% and GSM8K accuracy by up to 28.8%, and matches baseline performance with up to 64x fewer sampling steps. These results establish noisy-to-clean self-distillation as an effective approach to improving continuous diffusion language models.