FastDiSS: Few-step Match Many-step Diffusion Language Model on Sequence-to-Sequence Generation--Full Version

Dat Nguyen-Cong; Tung Kieu; Hoang Thanh-Tung

cs.CLApr 7, 2026

FastDiSS: Few-step Match Many-step Diffusion Language Model on Sequence-to-Sequence Generation--Full Version

Authors: Dat Nguyen-Cong, Tung Kieu, Hoang Thanh-Tung

Organizations: FPT Software AI Center, FPT Corporation · Department of Computer Science, Aalborg University, Denmark · Quantum AI and Cyber Security Institute, FPT Corporation

Abstract

Self-conditioning has been central to the success of continuous diffusion language models, as it allows models to correct previous errors. Yet its ability degrades precisely in the regime where diffusion is most attractive for deployment: few-step sampling for fast inference. In this study, we show that when models only have a few denoising steps, inaccurate self-conditioning induces a substantial approximation gap; this mistake compounds across denoising steps and ultimately dominate the sample quality. To address this, we propose a novel training framework that handles these errors during learning by perturbing the self-conditioning signal to match inference noise, improving robustness to prior estimation errors. In addition, we introduce a token-level noise-awareness mechanism that prevents training from saturation, hence improving optimization. Extensive experiments across conditional generation benchmarks demonstrate that our framework surpasses standard continuous diffusion models while providing up to 400x faster inference speed, and remains competitive against other one-step diffusion frameworks.

Figures & tables

Figure 1: Overview of FastDiSS . The tokenized sequence is first encoded to

\boldsymbol{z}_{0}

, while concurrently sampling the initial timestep

t

. Both

\boldsymbol{z}_{0}

and

t

are passed into MANS to obtain the new timestep

t_{\theta}

. Subsequently, noise level at

t_{\theta}

is added to

\boldsymbol{z}_{0}

using SCP to obtain

\boldsymbol{z}_{t}^{\prime}

. The rest is the same as in the training objective in Eq. 5 .

Model	Number of denoising steps (NFEs)
	5	20	50	100	1,000
Original	27.85	29.83	29.97	30.10	30.12
Correct	29.70	30.21	30.34	30.20	30.23

Table 1: BLEU scores.

Figure 2: Validation loss and BLEU during training under fixed, double, and linear step noise scaling. Dashed lines denote BLEU scores, color-matched to the corresponding loss curves.

Method	MBR	NFE	IWSLT14		WMT14		WMT16
Method	MBR	NFE	DE $\to$ EN	EN $\to$ DE	DE $\to$ EN	EN $\to$ DE	RO $\to$ EN	EN $\to$ RO
Transformer	5	-	33.61	28.30	30.55	26.85	33.08	32.86
CMLM	5	-	29.41	24.34	28.71	23.22	31.13	31.26
DiffusionLM ‡	50	20	29.11	22.91	19.69	17.41	30.17	29.39
Diffomer ‡	20	20	28.01	23.31	25.30	23.80	29.37	29.20
DINOISER	50	20	31.61	25.70	29.05	24.26	31.22	31.08

Table 2: Main results on IWSLT14 , WMT14 , and WMT16 . The best NAR results are bold and the second-best results are underlined .

\ddagger

indicates results reported by Ye et al. (2023) . Other results are from their original papers.

Method	MBR	NFE	QQP			Wiki-Auto
Method	MBR	NFE	BLEU $\uparrow$	Rouge-L $\uparrow$	BertScore $\uparrow$	BLEU $\uparrow$	Rouge-L $\uparrow$	BertScore $\uparrow$
Transformer ‡	-	-	27.22	57.48	83.81	26.93	49.07	73.81
GPT2-large FT ‡	-	-	20.59	54.15	83.63	26.93	51.11	78.82
CMLM ×	-	10	21.78	56.12	-	35.26	58.46	81.83
LevT ‡	-	-	22.68	57.95	83.44	20.52	44.02	72.54
Difformer ∗	10	20	27.95	59.24	82.97	34.78	54.55	78.86

Table 3: Main results on QQP and Wiki-Auto . The best NAR results are bold and the second-best results are underlined .

\ddagger

\times

, and

\dagger

indicate results reported by Gong et al. (2023b) , Tang et al. (2023) , and Chuang et al. (2024) , respectively.

*

indicates reproduced results. Other results are from their original papers.

Figure 3: Generation speed and quality with different NFE. The speed is averaged over 3 runs.

Figure 4: Diversity and quality comparison.

SCP	MANS	$\text{NFE}=5$	$\text{NFE}=20$	$\Delta$ NFE
$\times$	$\times$	27.98	29.78	1.80
$\checkmark$	$\times$	29.64	30.36	0.72
$\times$	$\checkmark$	30.77	31.49	0.72
$\checkmark$	$\checkmark$	31.17	31.66	0.49

Table 4: Ablation study.

Figure 5: Effect of Number of Denoising Steps.

Steps	5	20	100	Fixed
$\lambda_{t}$	0.60 - 0.85	0.90 - 0.95	0.98 - 0.99	0.85
$\gamma_{t}$	0.25 - 0.90	0.15 - 0.35	0.05 - 0.15	0.50
BLEU	25.48	26.35	25.70	25.84
ROUGE-L	57.29	57.47	56.82	57.18

Table 5: Effect of

\gamma_{t}

and

\lambda_{t}

Noise Schedule		NFE=2	NFE=5	NFE=20
linear	Orig.	26.50	27.17	27.54
linear	Ours	26.84	27.52	27.54
cosine	Orig.	25.43	27.05	27.57
cosine	Ours	27.32	27.77	27.94

Table 6: Effect of Noise Schedulers.

NFE	DoT ( Plaid )		$\texttt{DoT}^{\text{mp}}$ ( CoT )
NFE	Normal	SCP	Normal	SCP
8	35.4 $\pm$ 0.5	$\mathbf{38.2\pm 0.7}$	30.9 $\pm$ 0.6	$\mathbf{35.1\pm 1.0}$
16	39.0 $\pm$ 0.5	$\mathbf{41.1\pm 0.7}$	36.1 $\pm$ 0.7	$\mathbf{39.8\pm 0.8}$
32	39.9 $\pm$ 0.3	$\mathbf{43.0\pm 1.1}$	37.2 $\pm$ 0.7	$\mathbf{40.4\pm 0.9}$
64	41.3 $\pm$ 0.2	$\mathbf{43.7\pm 0.2}$	36.6 $\pm$ 0.6	$\mathbf{40.8\pm 0.6}$

Table 7: Comparison of Accuracy on GSM8K benchmark. We experimented with two settings, DoT with Plaid inference and

\texttt{DoT}^{\text{mp}}

with CoT reasoning. The Accuracy is averaged over 5 runs.

NFE	Gen Perplexity ( $\downarrow$ )
NFE	None	SC	SCP
2	3178.74	3250.22	2873.67
5	1144.42	1433.79	1215.41
20	243.81	254.72	244.40
50	143.82	133.79	130.52
100	113.34	97.88	95.48

Table 8: Main results on MDLM . The best Generation Perplexity results are bold .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

NFEs	5	20	50	100
Sup $\mathbb{E}$	0.047	0.008	0.009	0.010

Appendix

Table 9: Estimation Error.

Panel of Figure 6 — Figure 6: Mean (top) and variance (bottom) ratio differences measured at different denoising steps. We manually select hyperparameters of the scaling terms to fit 20-step sampling.

Panel of Figure 7 — Figure 7: Inference mean $\hat{\lambda}_{t}$ (top) and variance $\hat{\gamma}_{t}$ (bottom) scaling factors of prediction mismatch in a pre-trained network, plotted across timestep $t$ for randomly selected embedding dimensions.

Panel of Figure 8 — Figure 8: The empirical distribution of $\epsilon_{t}^{i}$ with different random values of $t$ and $i$ .

Method	MBR	NFE	Rouge-1	Rouge-2	Rouge-L
Method	MBR	NFE	$(\uparrow)$	$(\uparrow)$	$(\uparrow)$
LSTM	5	-	34.2	16.0	31.8
CMLM	-	-	34.4	15.6	32.2
NAG-BERT	-	-	35.1	16.5	33.3
Difformer *	10	20	34.9	17.0	32.4
DiffuSeq	10	2000	31.2	12.2	29.2

Appendix

Table 10: Main results on Gigaword . The best NAR results are bold and the second-best results are underlined . Baseline results are from Difformer Gao et al. (2024) .

*

indicates reproduced results.

Param	Metric	Default	1	2	3	4	5	6	7	8	9
$\lambda_{\min}$	Value	–	0.85	0.86	0.87	0.88	0.89	0.90	0.91	0.92	0.93
	BLEU	26.94	28.67	28.58	28.73	28.65	28.68	28.67	28.34	28.11	28.44
$\lambda_{\max}$	Value	–	0.91	0.92	0.93	0.94	0.95	0.96	0.97	0.98	0.99
	BLEU	26.94	28.54	28.45	28.68	28.53	28.60	28.59	28.45	28.66	28.59
$\gamma_{\min}$	Value	–	0.11	0.12	0.13	0.14	0.15	0.16	0.17	0.18	0.19
	BLEU	26.94	28.60	28.37	28.00	28.46	28.53	28.48	28.40	28.79	28.52

Appendix

Table 11: BLEU on IWSLT14 baseline under different SCP schedule parameters. Default denotes

\lambda_{t}=1

and

\gamma_{t}=0

Configurations	WMT14	WMT16	IWSLT14	Gigaword	QQP	Wiki-Auto
Split
Training	4,500,966	608,319	160,215	3,803,957	144,715	677,751
Validation	3,000	1,999	7,282	189,651	2,048	2,048
Test	3,003	1,999	6,750	1,951	2,500	5,000
Preprocess
BPE	40,000	30,000	10,000	60,000	15,000	40,000

Appendix

Table 12: The dataset details, model architectures, and hyperparameters used in our experiments.

Figure 9: Word cloud of the easy and hard tokens during training.

Figure 10: SacreBLEU score on IWSLT14 De-En with various length beams and noise beams.

	Step	Example 1
Source	-	He also twice participated in the Summer Olympics, starting in 1996.
Target	-	He was also in the 1996 Summer Olympics.
Baseline	2	He also twice in the Summer Summer Olympics 1996 Summer 1996.
Baseline	1	He also twice in the Summer Summer in the in 1996.
FastDiSS	2	He also twice twice in in Summer Summer Olympics, starting in 1996.
FastDiSS	1	He also twice twice in the Summer Olympics, starting in 1996.

Appendix

Table 13: Examples of generation throughout the denoising process, with

\text{NFE}=2

	Example 1
Source	How do you think Federer, Nadal, Djokovic, Murray, Wawrinka, Delpo & Cilic rank in terms of potential
	when it comes to greatest spot in the tennis?
Target	How would you rank Federer, Nadal, Djokovic, Murray, Wawrinka, Delpo and Cilic in terms of greatness
	achieved and potential spot in the history of sport?
Baseline	How do you think about the Federer, rank in a good of Mury Wawrinka , Delgreatest rank and Cilic rank in
	terms of for tennis?

Appendix

Table 14: Examples of generation on Wiki-Auto dataset. The resulting outputs demonstrate higher coherence in the long-term context.

	Example 1
Source	This easy chicken curry recipe is made with just a handful of ingredients
Plaid	This easy chicken curry recipe is made with just a handful of ingredients make provide add that helps easy add help give add that helps welcome give help offer help for how you learn give help help help for act give help help help and perform give help help help for for how pay for help help help and even add help help help for give help that add give help up for the add provide help provide for help help help and add for how pay for help give help provide help and add give help help help and add help provide help to create help give help help help for for how doit add this give help help help that add help help give help help for and that give help add help give help give help for the run in the middle of the day add help help provide help help that gives simply give help help give help help provide help help provide help help for pay help help give help help pay give help give help give help help pay help the member on your family make help shop it …
Plaid + SCP	This easy chicken curry recipe is made with just a handful of ingredients at your disposal including a bow for glowing red ropes for strutter. It alone will be the trick since making anything else would normally be a lengthy method. Once that completes the dungeon is start laid out again towards the end where you learn the way to make naughty but once you find this little hidden dungeon point you will also need the basics of the tools to help you find your way to the recipe for cider seeds. $\hookleftarrow\hookleftarrow$ So many would say these simple caves and dungeons are like paster glue in a Oddband environment. It is not just the recipe that is perhaps also the highlight in that the have side rooms for dungeon for completion in small spells or just locked in a spooky cave somewhere. The lack of questline support will prevent really good and fun faciettes. The formula the ingredients are a lot worth then anything else but for an hour of stats capable of requiring three seemed some how effective when you have the basics …
	Example 2
Source	Generative models of text are very versatile: they can be used
Plaid	Negative models of text are very versatile: they can be used in the workplace and in the secondary market provide availability provides published versions varies and competitive paid provided available for tender available as the selection of availability choices were very common on a high … fair to read the initial sales framework very often … provide selection for the except immediately in commonly based on how pricing control edited ask that everyone creates as a matter of how much published provide is actually apply control on that immediately on offered provide on exchange for them provide paid write line of commissioned provide access made applied to more creative offered works cover different parts of the market agree that paid sharing is much more what use sharing work on the ability to in understanding in support of a contribution of a meaningful piece on say you can provide featured provide paid paid assignment paid submissions management provided applied arrangements with mixed paid paid shared fit easily perform create commissioned work in private … why is not offered applied availability on the subject or backing and reuse … to … more successful provide writing for working provide featured work submission creative exclusively created collection of icons is a symbol medium always learn how to make use of owns offering portfolio offering total free commissioned formats provided published licensing paid

Appendix

Table 15: Examples of 32-step long sequence generation on Prefix Completion with Plaid-1B . The output results indicate that SCP helps maintain the coherence and fluency of the paragraph, while ensuring consistency across sentences.

	Example
Source	Lexically Constraints: Donald
Plaid	Sample 1: … focused in comments on newly uncovered FBI collusion with President Donald Trump, Bill Nunes revelation that almost no trump prosecute matters, despite widespread evidence, making him the primary target of these very crimes …
	Sample 2: … It also includes some mud slinging with Obama FGA Chairman John Masters attacking Donald Trump for loose familiarity with the banking cartel. It’s no secret that Trump was beaten to the drapes by President Barack Obama on the actual exam.
	Sample 3: … March in London Members of Parliament looking at paintings of Prime Minister Donald trump during EU festival day Getty Images Brexit protest …
Plaid + SCP	Sample 1: … It is also an equal, far more debasive sellout than the academia; some may have reasonably attributed the whole spread to Donald Nhutbout not clicking into the comments I have written this time; it seems to be unfortunate because both the coverage and commentary involved that continued focus could just seem to call out bias and denial …
	Sample 2: … But at least on campuses, the so called feminist war on radical conservatives may actually be having a negative effect on Loomer attendance. When it comes to that Donald Trump presidency, we might actually find men who don’t have the viciousness, sorry love, of the feminist left …

Appendix

Table 16: Examples of 32-step long sequence generation on Lexically Constrained Generation with Plaid-1B . The Plaid-1B samples lack diversity while ours cover more word combinations.

	Example
Source	Negation: Donald but not Trump
Plaid	… [profile] Frederic Blanchfield Britain’s earliest basalt monument in Little Choyside, Chepole, Dorset. Source: MORI.mauipomwork-8x17-1.jpg… [profile] Marco Bori Special Projects Marco Bastoni Participants in discussion with Russian crionic residents in Venice. Artist/21-ponoco-8x17-1.jpg… [profile] Marco Baba.alexnet-8x17-1.jpg… [profile] Giananna Boone Interior Photography Histature from Rome Client – Social Architecture – Vanguard.photo/46-photograph-164x72-r.jpg… [profile] Donald Buckingham Industrial Art Izomage ProKnakon Call to exercise Tradarpones in Real Materials-stone-piping-8x17-608px.jpg… [slide] artmagazine-20x26x24.jpg… [slide] …
Plaid + SCP	… It was dress for contempt. The dishonesty of this whole development was clear, an astonishing and obvious, necessary to humanity, but seen in the remark Il Quartilver made which showed an almost impervolent fit of irony, necessary to discern truth from record. ‘ Our exile and our exile have been secured. The ocean has risen. Hell, Donald haath been a fool, before he came to save me, Taracharvi, and finally … not only in these troubled times it was taking a giant it-own for their charisma, but sol gallur phir bariet die Pope Donald . If they couldn’t say ‘wrong’, where did they get it? The slanders at Timon knew to just make radios, they could have solved …

Appendix

Table 17: Examples of 32-step long sequence generation on Negation with Plaid-1B . The results demonstrate that the Plaid-1B sample is likely memorized from the dataset when against an unusual pattern, while our method produces a normal paragraph that satisfies the constraint.

Diffusion Language Model Inference

Diffusion language model inference studies how to generate, guide, and serve text efficiently through iterative denoising and parallel token updates.

See all 165 papers

Few-Step Diffusion Language Models via Trajectory Self-Distillation
Feb 12, 2026Tunyu Zhang, Xinxi Zhang, Ligong Han +10Diffusion Model Distillation Few-Step Diffusion Sampling
Rutgers University · Red Hat AI Innovation · MIT-IBM Watson AI Lab
DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
May 22, 2026Jean-Marie Lemercier, Tomas Geffner, Karsten Kreis +3Diffusion Language Models Diffusion Language Model Distillation
NVIDIA
Consistent Diffusion Language Models
Apr 30, 2026Hasan Amin, Yuan Gao, Yaser Souri +4Few-Step Diffusion Sampling Diffusion Model Acceleration
Department of Computer Science, Purdue University · Microsoft

Feb 12, 2026J/K move · Enter open · S save

Few-Step Diffusion Language Models via Trajectory Self-Distillation

Tunyu Zhang, Xinxi Zhang, Ligong Han, Haizhou Shi+9

Rutgers University · Red Hat AI Innovation · MIT-IBM Watson AI Lab

Diffusion Model Distillation Few-Step Diffusion Sampling Diffusion Language Model Decoding Diffusion Language Models Diffusion Language Model Distillation

Open paper AskPDF arXiv

Diffusion large language models (DLLMs) have emerged as powerful generative models with the promise of fast text generation through parallel decoding. However, realizing this potential in practice remains challenging: reducing the number of decoding steps, typically causes a substantial degradation in output quality due to token factorization error. To alleviate this, we propose a self-distillation framework that trains a few-step student to match the generative trajectory of a full-step teacher. We theoretically and empirically show that trajectory-level supervision mitigates this factorization error, thereby enabling effective few-step decoding. We further incorporate Direct Discriminative Optimization (DDO), a reverse-KL objective that encourages mode-seeking toward the teacher's modes, yielding stronger performance on challenging reasoning tasks. Across reasoning and code-generation benchmarks, our method substantially narrows the gap between few-step and full-step decoding. The source code is available at https://github.com/Tyrion58/T3D.

Abstract

Diffusion Language Model Inference

Explore similar work

Few-Step Diffusion Language Models via Trajectory Self-Distillation

DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

Consistent Diffusion Language Models