Generalized Discrete Diffusion from Snapshots

Oussama Zekri; Théo Uscidda; Nicolas Boullé; Anna Korba

stat.MLMar 22, 2026

Generalized Discrete Diffusion from Snapshots

Authors: Oussama Zekri, Théo Uscidda, Nicolas Boullé, Anna Korba

Organizations: CREST, ENSAE, Institut Polytechnique de Paris, Paris, France · Department of Mathematics, Imperial College London, London, SW7 2AZ, UK

Abstract

We introduce Generalized Discrete Diffusion from Snapshots (GDDS), a unified framework for discrete diffusion modeling that supports arbitrary noising processes over large discrete state spaces. Our formulation encompasses all existing discrete diffusion approaches, while allowing significantly greater flexibility in the choice of corruption dynamics. The forward noising process relies on uniformization and enables fast arbitrary corruption. For the reverse process, we derive a simple evidence lower bound (ELBO) based on snapshot latents, instead of the entire noising path, that allows efficient training of standard generative modeling architectures with clear probabilistic interpretation. Our experiments on large-vocabulary discrete generation tasks suggest that the proposed framework outperforms existing discrete diffusion methods in terms of training efficiency and generation quality, and beats autoregressive models for the first time at this scale. We provide the code along with a blog post on the project page : https://oussamazekri.fr/gdds.

Figures & tables

Figure 1: Zero-shot transfer of OWT-trained models. Zero-shot perplexity (

\downarrow

) on three representative downstream validation sets from Table 3 : PTB, LM1B, and Wikitext. Across this high-to-low perplexity range, GDDS Gauss consistently achieves the lowest transfer perplexity, highlighting the stronger generalization capability induced by semantically structured noising processes.

Figure 2: Overview of GDDS. A clean sequence

\mathbf{x}_{0}

is first noised exactly by the forward CTMC at a sampled time

t\in[0,1]

, yielding a snapshot sequence

(\mathbf{x}_{t},t)

. The mean parametrization is then used as a denoiser: given the snapshot, the model predicts the clean-token posterior directly from

(\mathbf{x}_{t},t)

, so training is performed on snapshots rather than through a full path-wise objective.

Figure 3: Snapshot vs. path-wise training. The forward process corrupts the clean sequence “My name is David” . The blue path shows the beginning of the noising trajectory ${\color[rgb]{0.05,0,0.8}\omega}=\smash{\{({\color[rgb]{0.05,0,0.8}x_{t_{k}}^{(4)}},{\color[rgb]{0.05,0,0.8}t_{k}})\}_{k\geq 1}}$ of one tracked position ( $\ell=4$ ). Path-wise objectives condition on the entire trajectory ${\color[rgb]{0.05,0,0.8}\omega}$ , whereas our GDDS snapshot objective uses only one random-time observation ${\color[rgb]{1,0,0}s}=({\color[rgb]{1,0,0}x_{t^{\star}}},{\color[rgb]{1,0,0}t^{\star}})$ .

Method	BPC ( $\downarrow$ )
Continuous Diffusion
Plaid ( Gulrajani and Hashimoto, 2023 )	$\leq$ 1.48
BFN ( Graves et al., 2023 )	$\leq$ 1.41
Any-order Autoregressive
ARDM ( Hoogeboom et al., 2022 )	$\leq$ 1.43
MAC ( Shih et al., 2022 )	$\leq$ 1.40

Table 1: Bits Per Character (

\downarrow

) on Text8. Baseline results reported from ( Shi et al., 2024 ) . All models are trained for

1

M steps. Best results per model family are in bold , while best results among retrained models are underlined .

Method	Training token	PPL ( $\downarrow$ )
Autoregressive
GPT-2 †⋆ ( Radford et al., 2019 )	unknown	23.40
AR †	262B	16.11
AR (retrain)	262B	20.49
Uniform Discrete Diffusion
SEDD Uniform ‡ ( Lou et al., 2024 )	524B	$\leq$ 29.70

Table 2: OWT validation perplexity. Validation perplexity (

\downarrow

) on OWT. Best results are in bold . ⋆ Trained on the WebText dataset. † Result taken from ( Zhou et al., 2025 ) . ‡ Result taken from ( Sahoo et al., 2025 ) .

Figure 4: OWT training curves. Evolution of OWT validation perplexity during training for the retrained models reported in Table 2 . This complements the final numbers in Table 2 by showing the full optimization trajectory; both axes are shown on logarithmic scales.

	PTB	Wikitext103	LM1B	Lambada	AG News	Pubmed	Arxiv
Autoregressive
AR (retrain)	147.90	42.91	81.29	75.93	105.34	79.93	76.29
Masked Discrete Diffusion (upper bounds $\leq$ )
MDM (retrain)	181.36	45.42	92.58	58.89	123.51	66.46	56.82
GDDS Absorb	103.04	46.49	92.51	60.78	101.41	62.62	53.27
Uniform Discrete Diffusion (upper bounds $\leq$ )

Table 3: Zero-shot transfer perplexity. Zero-shot perplexity (

\downarrow

) of OWT-trained models. Models are evaluated on validation splits of

7

downstream datasets without additional fine-tuning. Best result per dataset is in bold ; second-best is underlined .

Figure 5: Generation quality-diversity tradeoff. Gen-PPL (

\downarrow

) vs Entropy tradeoff. For

K\in\{32,64,128,256,512,1024\}

decoding steps, we plot the generative perplexity of

N_{\text{gen}}=256

unconditional samples under a fixed evaluator (GPT2-large) against their sequence entropy (higher is better). Bubble radius increases with

K

. For reference, the AR baseline achieves Gen-PPL

56.82

at entropy

5.60

Model	Dist-1 ( $\uparrow$ )	Dist-2 ( $\uparrow$ )	Dist-3 ( $\uparrow$ )
AR (retrain)	0.10	0.57	0.88
MDM (retrain)	0.10	0.61	0.90
GDDS Absorb	0.10	0.60	0.89
UDLM (retrain)	0.08	0.53	0.85
GDDS Uniform	0.10	0.58	0.88

Table 4: Lexical diversity. Distinct-1/2/3 (

\uparrow

) computed on

N_{\text{gen}}=256

unconditional samples from OWT-trained models, measuring the fraction of unique

n

-grams among generated texts.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Setting
Sequence length	256 for Text8, 1024 for OWT
Hidden size / heads	768 / 12
MLP ratio / dropout	4 / 0.1
Time conditioning	AdaLN
Layers	12
Optimizer	AdamW ( $\beta_{1}=0.9,\beta_{2}=0.95,\epsilon=10^{-8}$ ), weight decay $=0.01$

Appendix

Table 5: GDDS Training configuration

Method	Mean $\pm$ Std (ms)
Absorbing	$0.09\pm 0.01$
Uniform	$0.07\pm 0.00$
SIK Gauss (KNN)	$8.97\pm 0.26$
SIK Gauss (KeOps)	$152.49\pm 1.86$
SIK Cosine (KNN)	$8.94\pm 0.27$
SIK Cosine (KeOps)	$162.45\pm 2.19$

Appendix

Table 6: Noising-time benchmark. Mean wall-clock latency in milliseconds for sampling

x_{t}\sim q_{t}(\cdot\mid x_{0})

on batches of size

512

and sequence length

1024

(

512\times 1024=524{,}288

positions). Reported values are mean

\pm

standard deviation over

5

random seeds; for each seed, latency is averaged over

10

timed runs after

3

warmup runs. Results use absorbing, uniform, and SIK-based forward processes with KNN and KeOps implementations.

[BOS]sessions studies professional present topics international literature university of global history of international shock artist as well as its global no actibilative clocks to ruin the strip individual martials in the image of information both side east[EOS]

[BOS]g strewn nearly confrentative and the bible tag strews which were very officialy dilgress in the usa japanepic first considered only parathislatic influences arabic christianity have played in their language slightly best known is the japanene term in ma[EOS]

Appendix

Table 7: Two unconditional samples for the GDDS Absorb model trained on Text8.

[BOS]heir ownest possession is the case despite cyclic subgraphs resulting in its first higher community area or less expensive [EOS]major national companies again below the company is a supporter of city trains or a record historically system is cut the large bu[EOS]

[BOS]ree the oxford university vol two six seven two two pp two seven three two x are not human definitions of oxford since others are seaset fara la conventions university of chicago based on bissaudi and known as the karlowe large content of the world elbia[EOS]

Appendix

Table 8: Two unconditional samples for the GDDS Uniform model trained on Text8.

<|endoftext|> to conform to routine training procedures for the handling of a crime or any mass deaths committed by police when it investigates crimes. Opponents of the entertainer’s departure say the department will offer guarantees in coverage for a crisis that in 2002 and 2009 frees law enforcement. The protesters, say police are taught to commit crime continuously with tear gas or bullets. The chaos comes amid sweeping changes in policing. Illinois law enforcement took last month’s a surprise given the growing appetite in some places for police from function as order in the state. “But more generally, it’s a safe haven,” said Lake-based Howard Hexano. Theneighbor Police Association of the United States said it replaced the training road used by law enforcement for initially international training exercises, rather opting to resemble standard “economic training” methods. Sarley opened questions about the leadership of Kenney Village, the same police department, that has the traditional candidate for the license for two men accused of shooting someone at the Colorado New Year’s Eve pageant annual competition, but it pays paid for the monthly pay of the victims. The agency won’t face any disciplinary action against officers in the scandal, and it has addressed the community about the past. Earley on Feb. 28 will officially investigate the incident, concluding that the men accused were not teaching acting well, or causing harm to others. The Tribune’s Matt Steward declined to comment. The department’s chief officer, J. Durkin, who was a local pastor following his Feb. 24 firing, did not respond to requests for earring for comment on the incidents. All grievances in the investigation came and there’s need for further technical inquiry. Even local law enforcement companies, including Illinois Public Affairs and other state law enforcement officials continue to have witness cooperation, such as W-Ferguson and Louisiana C-M., in the works. “We actively consult — municipal agencies and other social health and architectural agencies to deliver information services and information regarding current events,” Charlie Paley said in an announcement. “C-M., Under-Chief Shawn Leslie and Nick Stout, Executive Director are working closely with the person conducting the investigation for evaluation and possible update on future events. For more about what happened in Kidney Village, read on here.” The real highlights the six basic changes and guidelines recommended throughout the town’s history. For additional units, if completed, the Westlake Union Marriott Hotel will be located in the 650 block. Comprehensive plans include a spa spa. It is 45-foot by 40- feet wide and have a barbed roof. Regional — urban — will be 10,000 square feet and include RGB lighting, natural gas development, natural fire and swimming pool and golf courses. As the area grows diverse enough, police will become concerned about the potential threat to its security. Some cops want hotel gives new applicants for a recreational site to meet location hotel is home to the training headquarters of the U.S. Marines, home of about 700 horse and elderly confrontations with law enforcement officers. It’s normal for the U.S. government to increase its activist role — asserting that American police do slow to inflict damage on Americans during First World War. The U.S. Congress loosened constitutional restraints when it comes to police violations trial except that makes it a fine, similar to what is guilty under the federal Criminal Code. Also floating around is potential limited impacts — and public safety and safety have increased as a deterrent since when a school student was shot while running a road trip. Why is that? But tourists won’t allow it in. That common sense will prove little change for the town, either. And the impact will no longer homes. But any feeling of diversity will be gaining almost all in the surrounding institution, many of its core buildings and colleges. The Cenna-The Performing Arts exhibition is “Hop Art Thursday” by Sen. David D Ducey (D-D.). “To make everyday Americans realize that I think the museum stands out throughout much of American history,” he says when a 65-year-old drops an invitation from Harvard’s Dr. Richard B. Hoffman to enter the White House in 2006. And always will have been the center of the museum’s popularity. The visitors that the design be more limited in future exhibits, said Fucakis, are themes from past films, such as the arenas and cop offices, “brain longening” and “Hey, Captain. I have the power of a cat.” Fucakis, also a pollster, announced Thursday that the megamstitute is opening a doghouse and would be introducing video of historical footage <|endoftext|>

Appendix

Table 9: Unconditional sample for the GDDS Absorb model trained on OpenWebText.

<|endoftext|> the United Kingdom), in hopes that the U. president and his global leader will lodge a demand or promise to use its stated goal to sustain support to combat ISIL. (The United Arab Emirates responded for the first time.) On Saudi Arabia and its United States side on the matter, less than ever will affect the U.S. to longer series,” co-Nubal Hender explained. This kind of bullsrina zone is in the shape that Syria’s non-American interests may be in the West 2007. U.S recent political and economic quellings of the Egyptian mass-democratic regime, destabilizing Shiite and anti-Muslim militias will only escalate tensions between the core allies of Syria, Saudi Arabia, and Syria, independent of the central U.S., while maintaining a fair-minded, external foe whose leaders seek to assist them, as head of White House argued. Even if American and Pres. Kerry efforts succeed slack push, they are unlikely to continue to broker lobbying strategies. The best way to look at that is the case across the world that rather than accept the negative effects of Nazi method are much to completely out of reach for those who have demons. In fact, patients who are in progressanic – for their malign position limiting behavior may turn into a - or recommended - - for becausefama. Elimuting unsenate drugs as long as Western medical prohibition on marketing and other ministries have harm’s effect, with a perils annual mortality rate, despite often everywhere no-one makes patients plan for a continued suicide away. Indeed, suicide rates among the American population are abnormally low, and coronary myocardic degeneration remains at a preexisting rate the single highest.41. A third study conducted by the IBM School of Medicine also archives patient interpretation data for physicians and health researchers and encouraged the study. The study in July 2015 was published in Dec. 4 of the Proceedings of the National Academy of Sciences, which was produced this year. The authors responded with an order of published papers published in each major journal for the study of hypertension. The journal’s published papers were published online from Thursday, February 16 to Jan. 6, 2015 by Mawall and Sarah Leonard. “The 1VHP trials were the most selective Tract-based randomized trial to be administered under the ausp against the fact where they reported consistent that they were moderately popular in the internal health industry. But the motive of taking support for high-risk cardiovascular care tends to lead at even higher risk,” write Begojin Krishnan, who is the senior author of the New England Journal of Medicine. 41. B̈iopathological benefits of shoulder plans was again useful in triggering several potentially dangerous activity that could trigger strokes.Ḧe found that these outcomes cannot be altered because coronary artery consumption is harmful to the researchers, and his team among its patients to criticize the customary practice of nasation from shorter milliseconds of knee propelled toparse pain and flow from a higher-value knee plateau. In a study that took 36 weeks, Pro Medel’s 37 weeks of placebo caused s̈udden activities-̈-no orgasm for years of experience–and survive, and decreases were detected at

\leq

191 levels of cancer-seriousctal core actal. Backstroke-induced metabolic syndrome is a major shift in vascular function to dependence, which may cause time distances among Traitor intensive coronary artery consumption more associated with chronic stroke-staffing,n̈otes Schuning, the row center in the study. T̈his positively affected the vascular system almost all the time, and it represents his lower, older, rested region.4̈2 Despite by scientists its efforts to reduce coronary artery injury, concussion therapy impacted fatal deaths among documented patients with Naondigran, who suffered increased cardiovascular toxicity and significantly increased strokes among the patients investigated the November 1 to 12 June 2015. The efficacy of placebo The scope of the study for shoulder disease was completely triletified when the study showed moderate improvements in coronary artery activity. The main test of effectiveness was the use of alcohol-free root oils for impact and the intervention. free-toverorption properties also contained wide variety of complimentary placebo techniques. Only a run on the annual National Study of Clinical Study, significant clinical correlation was achieved by the study 1 trial that did not show the effects of treatment from their respective programmes. When fails all studies measuring menstrual side effects, another systematic review showed that one trial showed a linear scheme of the retrospective performance across knowledge units 1VHP. Modeling was predicted to be followed for each stroke due to a health insurance plan featuring RCT and Morse sequences over the day of visits. Thus the commensatory between risk of death and risk of strokes was reached, soon agreement with in the 321 1VHP trials involving practitioner using mathematical reasoning and the distribution provided by himself.44 Thus, septoristic treatments show a significant sum of the energy received in one dose of intervention; only a fraction of the energy is collected from the release of <|endoftext|>

Appendix

Table 10: Unconditional sample for the GDDS Uniform model trained on OpenWebText.

<|endoftext|> stories went in to TV in 2014 on Abboun on Zahora and became a busy town, rapidly knit and destabilized to men of almost 800 soldiers. Most frequently, Abcindi. the Aba Biba, boss of his lawyers, in Nahida, became a woman who enjoys his wife Helen‘s penchant for writing and writing to the LA Times. Abbindi, the wrinkled young woman, is sent fishing for the beach for one part in the incredibly horrific story find on eyewitness accounts and videos involving an immigrant family and local high school friends in AinsZi, a coastal enclave of Felton, N.C., Liberia and Dominican Republic. The Cairo story of The Los Angeles Times magazine tells the Egyptian State government to Abi ’Sadei was a huge hoax. The Prison cells of the LahaAidan of Cairo National University, a local university for the United States, built military antennas for the Egyptian army to cover the Ghouta bombing in Idlib. The Egyptian University builds anti-Islamic propaganda, challenged the group of soldier to get their flack on a private plane, leading to a photographers – Abno Ueh – having portrayed in photographs of Muslims. One explanation of what had caused the mayhem is for what’s worse happened on the covers and does not get to interest. That, they says, plenty of telefilmed spy community, of fear to beding in to the many stories and photos in the Abnot Ueh story. A little apart, however, the idea of the investigation was so near to this sites it became obvious that some that those individuals were presented for speculative speculation on that search result - and it is the way that has remained hidden for almost a year. It was not just a disturbing problem in trying to relate to personal data. Artificial intelligence is no confident in ability to pull a plot for it was as difficult to it was on a list the opponent has been to it. Research is to prone as it say to when we have requests in Google searches, they inevitably do asked to American espionage who care. Now they were looking for evidence pointing to action - to try to avoid that, and simply to try to get the first, or simply to get to the point for conjecture. Weaknesses could turn to brilliance on the very most dramatic weeks - power and intelligence are not mere real technologies, they are a incurable danger they have face maybe to a decade or we didn’t want to fully understand. The debate there seems interesting.Is it to several: Perhaps to be certain, the answer may sound much more than the midnight hour of 9/11. And in November 2015, Donald Trump’s first response on an outreach to try to infiltrate Twitter on Muslims was to open a hotline for U.S. for control of the World Trade Centre. And it is not a reference to what was happening in Saudi Arabia, as it is ISIS providing explicit terrorism was meant for attack that as detonate the target. It took much more room to laid the groundwork for it for many people looking to that - for anticipatory details that is not easier to find. there is an increased degree of concern when the group specifically targets terror-related activists and the National security adviser, Donald Trump tells audience gather in Saudi Arabia. to provide his account to Donald Trump as the story for put ubaida on Europe’s emerging refugee crises. – Kimberler Michael A- Doh6 Follow us on Twitter Google Whatsapp Tumblr Tumblr Pinterest Pinterest LinkedIn Google sharePARIS, New Jersey A building is deserted. It didn’t get cashing in for several years or it bombed to several.The biggest problems for that necessity of the huge building were dizors, patio cushions and wooden stitched.The journalist could hardly make up the case.A columnist in Ashraf Darrow, a reporter who is helped Donald Trump keep his watch as he is keen to become the non-American.This month, on Sunday a Faredan revealed the plan to embark in Montreal- doomed flights. “My plane had come up and I was actually only say to hell I’m have.” he told Time.“So I couldn’t do that It wasn’t anything for the metal gros and a communication tools I did,…It was sort of like an crazy …, ’cause, it got me on how I had to, [I Rediscovered that].I just wanted to prove I can have what to have on,” … French Transport Minister UEP S, Mariese Louse (C) and how Canada Led The War on fiscal Policy to a Future’S international Airport Centre. available Online at: 893/2313,349, online 915, (13,A), Part 2 The Economic Analysis Of Inflorating Railway Product Taxes (2013). French Transport Minister UEP S.Louse (C), Center to Transport Policy Group <|endoftext|>

Appendix

Table 11: Unconditional sample for the GDDS Gauss model trained on OpenWebText.

$K$	Entropy ( $\uparrow$ )	Gen-PPL ( $\downarrow$ )
8	5.34	402.25
16	5.45	230.64
32	5.59	207.48
64	5.59	176.68
128	5.66	189.06
256	5.67	254.15

Appendix

Table 12: GDDS-SIK sampling ablation on OpenWebText. We report decoding budget

K

, average sequence entropy, and Gen-PPL for unconditional samples. Natural OWT text typically lies around entropy

5.60

–

5.70

( Zheng et al., 2025 ) .

\bm{\mathcal{L}}(\theta)=\frac{1}{B}\sum_{b=1}^{B}\sum_{\ell=1}^{n}\sum_{k=1}^{N^{\ell,(b)}}-\log\mathsf{j}_{\theta}^{\ell}(\mathbf{x}^{(b)}_{T_{k}^{\ell,(b)}},\,T_{k}^{\ell,(b)})_{z_{k-1}^{\ell,(b)}},

Appendix

Algorithm 5 Training with Campbell estimator

Text8 BPC	OWT PPL
$\leq$ 1.75	$\leq$ 76.07

Appendix

Table 13: Validation perplexity of the two-stream architecture. We train the two-stream architecture on Text8 and OWT under the same experimental setup as in Appendix C , and report the BPC and validation perplexity.

Figure 6: Training loss stability on Text8. Train loss curves for AR, MDM, GDDS Absorb, and Campbell two-stream training, using the same setup as in Appendix C . While snapshot-based training exhibit noticeably higher short-term fluctuations, Campbell training yields a markedly smoother optimization trajectory, comparable to AR.

Diffusion Models

Diffusion models learn to reverse gradual data corruption, enabling flexible generation, editing, forecasting, and inference.

See all 1,169 papers

Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster
May 18, 2026Grigory Bartosh, Teodora Pandeva, Sushrut Karmalkar +1Few-Step Diffusion Sampling Diffusion Model Acceleration
University of Amsterdam · Microsoft Research, Cambridge
Discrete Diffusion Models: A Unified Framework from Tokenization to Generation
Jul 15, 2026Ye Yuan, Weien Li, Rui Song +19Masked Diffusion Models Diffusion Model Sampling
McGill University · University of Cambridge · Mila - Quebec AI Institute +6
Acceleration of Diffusion Language Model through Discrete Average Generator
Sep 29, 2026Yidong Ouyang, Zhengyan Wan, Themis Haris +8Few-Step Diffusion Sampling Diffusion Language Model Inference
University of California, Los Angeles · Google · Work done as a research intern at Google

May 18, 2026J/K move · Enter open · S save

Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster

Grigory Bartosh, Teodora Pandeva, Sushrut Karmalkar, Javier Zazo

University of Amsterdam · Microsoft Research, Cambridge

Few-Step Diffusion Sampling Diffusion Model Acceleration Diffusion Models Diffusion-Based Denoising

Open paper AskPDF arXiv

Discrete diffusion models are a powerful class of generative models with strong performance across many domains. For efficiency, however, discrete diffusion typically parameterizes the generative (reverse) process with factorized distributions, which makes it difficult for the model to learn the target process in a small number of steps and necessitates a long, computationally expensive sampling procedure. To reduce the gap between the target and model distributions and enable few-step generation, we propose Forward-Learned Discrete Diffusion (FLDD), which introduces discrete diffusion with a learnable forward (noising) process. Rather than fixing a Markovian forward chain, we adopt a non-Markovian formulation with learnable marginal and posterior distributions. This allows the generative process to remain factorized while matching the target defined by the noising process. We train all parameters end-to-end under the standard variational objective. Experiments on various benchmarks show that, for a given number of sampling steps, our approach produces a higher quality samples than conventional discrete diffusion models using the same reverse parameterization.

Abstract

Diffusion Models

Explore similar work

Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster

Discrete Diffusion Models: A Unified Framework from Tokenization to Generation

Acceleration of Diffusion Language Model through Discrete Average Generator