Summarization with a fixed character length is important for tasks such as headline and article preview generation, where text must fit an exact space. Autoregressive language models (LMs) struggle with precise character-length control: training-based methods require task-specific fine-tuning, while zero-shot prompting strategies achieve only approximate compliance. We propose DELTA (Diffusion Expected Length Targeting Algorithm), a training-free inference-time method for masked diffusion LMs that estimates expected character length at each denoising step and dynamically inserts or removes masked positions to steer toward a target length. DELTA requires no fine-tuning and is designed for off-the-shelf masked diffusion LMs. Across three summarization benchmarks, it reduces character-length errors by an order of magnitude compared to baselines while maintaining competitive summary quality in both automatic and human evaluation.
Figures & tables
Figure 1: Overview of Delta . Top: Algorithm flowchart. Bottom: Example with target L∗=24 .
Length
Quality
G-Eval
Method
MAE
< 10
R-1
R-L
BF1
Rel
Con
Coh
Flu
XSum
Delta +LLaDA
2.3
99.6
0.286
0.217
0.882
3.41
4.21
4.61
4.66
Delta +Dream
2.3
99.7
0.333
0.255
0.890
3.17
4.14
4.76
4.82
LA-SF-AR
5.2
84.2
0.273
0.204
0.881
3.56
4.38
4.79
4.89
GPT-4o-mini
13.1
48.4
0.264
0.194
0.880
3.83
4.73
4.96
4.99
Table 1: Results on XSum, CNN/DailyMail, and arXiv, ordered by increasing reference summary length. MAE is mean absolute character-length error; < 10 is the percentage of outputs within 10 characters of the target. R-1/R-L = ROUGE-1/L; BF1 = BERTScore F1. G-Eval scores (1–5 Likert) use GPT-5-mini as judge ( Liu et al., 2023 ) . Best values per column within each dataset are bolded . BART-CNN is included only for CNN/DailyMail as it was fine-tuned on that dataset.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
XSum
CNN/DM
arXiv
Test examples
11,334
11,490
6,440
Avg src length (words)
376
684
5,906
Avg ref length (words)
21
55
163
Avg ref length (chars)
125
312
969
Appendix
Table 2: Dataset statistics for the test splits used in our evaluation.
Method
Rel
Con
Coh
Flu
Avg.
Delta +Dream
4.05
4.33
4.31
4.40
4.27
LA-SF-AR
4.18
4.36
4.32
4.42
4.32
GPT-4o-mini
4.26
4.32
4.26
4.42
4.31
Appendix
Table 3: Blind two-annotator human evaluation on 50 CNN/DailyMail examples. Within-one-point annotator agreement is 91.5% across dimensions.
Table 5: Comparison with DAEDAL. Time is wall-clock seconds per summary. DAEDAL on arXiv is evaluated on a 100-example diagnostic subset because of its high runtime cost; Delta +LLaDA uses the full test set. G-Eval scores use GPT-4o-mini.
Method
XSum
CNN/DM
arXiv
Delta +LLaDA
0.69±0.43
8.59±7.91
22.80±22.98
Delta +Dream
0.67±0.39
8.59±7.98
20.75±21.72
LA-SF-AR
1.43±0.33
2.19±0.64
13.22±8.04
DAEDAL (LLaDA)
2.50±7.54
5.94±4.22
306.56±233.80∗
BART-CNN
–
0.37±0.06
–
Lead-char-snap
<0.01
<0.01
<0.01
Appendix
Table 6: Wall-clock runtime in seconds per summary (mean ± standard deviation). ∗ DAEDAL on arXiv is measured on a 100-example diagnostic subset because of its high runtime cost. GPT-4o-mini is omitted because concurrent API calls make per-sample latency not directly comparable.
Method
Mean chars
MAE
< 10
R-1
R-L
BF1
Rel
Con
Coh
Flu
Lead-char-snap
252.5
59.4
21.4
0.363
0.230
0.868
2.85
4.92
4.22
4.40
Standard Lead-3
455.4
174.8
2.6
0.400
0.250
0.869
3.51
4.96
4.67
4.75
Delta +Dream
311.1
3.0
99.3
0.409
0.281
0.880
3.97
4.72
4.75
4.73
Appendix
Table 7: Standard Lead-3 diagnostic on CNN/DailyMail. Lead-3 achieves stronger extractive-summary quality than Lead-char-snap but substantially violates the target character budget. G-Eval scores use GPT-5-mini.
Dataset
Method
Judge
Rel
Con
Coh
Flu
XSum
Delta +LLaDA
GPT-4o-mini
3.33
3.98
3.38
4.54
GPT-5-mini
3.41
4.21
4.61
4.66
XSum
Delta +Dream
GPT-4o-mini
2.90
3.84
3.18
4.48
GPT-5-mini
3.17
4.14
4.76
4.82
XSum
LA-SF-AR
GPT-4o-mini
3.49
4.02
3.55
4.64
GPT-5-mini
3.56
4.38
4.79
4.89
Appendix
Table 8: Comparison of GPT-4o-mini and GPT-5-mini as G-Eval judges.
Diffusion language models (DLMs) generate text by iteratively denoising masked sequences, but standard decoding either fixes the sequence length or relies on ad hoc stopping rules, often leading to unnecessary denoising steps. We recast length selection as a discrete-time survival problem over the end-of-sequence token and propose a plug-in, training-free length predictor that can be added to any existing DLM. Across reasoning and code-generation benchmarks, survival-guided length decoding speeds up inference by up to 7 times while preserving task accuracy. We further find that predicted lengths vary widely even within the same dataset, making model performance sensitive to the chosen length.
Ivan Kobyzev, Abbas Ghaddar, Yufei Cui
Huawei Noah’s Ark Lab, Montreal Research Center, Canada
Unlike autoregressive language models, which terminate variable-length generation upon predicting an End-of-Sequence (EoS) token, Diffusion Language Models (DLMs) operate over a fixed maximum-length context window for a predetermined number of denoising steps. However, this process is independent of the required response length, resulting in computational waste for the majority of short responses common in reasoning and chat tasks. To address this problem, we conjecture that the latent prompt representation contains sufficient information to estimate the required output length. We provide empirical evidence for this phenomenon and propose a zero-shot mechanism to dynamically crop the context window before generation begins, leading to fewer diffusion steps and substantial computational savings. We evaluate our approach on four benchmarks with diverse tasks -- GSM8K (reasoning), HumanEval (code generation), IfEval (instruction following), and LongFormQA (question answering) -- revealing massive efficiency gains at minimal performance impact. We report significant reductions in FLOPs across all tasks, with no statistically significant performance degradation, and significant performance improvements in 2 out of 4 tasks.
Vittorio Rossi, Giacomo Cirò, Davide Beltrame +3
Department of Computing Sciences, Bocconi University, Milan, Italy.
We present a controlled empirical comparison between autoregressive (AR) and masked diffusion (MDLM) language models. Both models are trained on identical data (50M tokens from TinyStories), identical compute budget (20,000 steps, batch size 32, sequence length 512), and identical hardware (NVIDIA H100 80GB), isolating the generation paradigm as the sole variable. We report three findings. First, both paradigms achieve comparable training throughput (~50K tokens/second), with MDLM requiring only 4.7% more wall-clock time. Second, AR converges faster and begins overfitting by step 14,000, while MDLM converges more slowly and is still improving at step 20,000, suggesting different compute-optimal training regimes. Third, quantitative diversity analysis over 1,000 generated samples reveals a structural diversity-fluency trade-off: AR produces fluent but repetitive outputs (99.8% begin with the same word), while MDLM generates more diverse narratives (93.4% unique 5-word openings, higher Distinct-n, lower Self-BLEU), at the cost of occasional grammatical inconsistencies. All code, trained checkpoints, and data pipelines are released for reproducibility.