We introduce RefineMix, a framework for training discrete diffusion models under severe data scarcity, a common constraint in scientific applications. RefineMix uses out-of-distribution data at selected diffusion times to improve generalization without biasing the sampling distribution. Although this strategy has been explored in continuous diffusion, discrete diffusion presents a distinct challenge: unlike Gaussian noise, masking preserves domain information in surviving tokens, limiting the use of related data at high noise levels. At low noise levels, however, the domains effectively disjoint supports become an advantage, allowing the model to learn from both in-domain and out-of-distribution data without biasing the sampler. We formalize these intuitions and provide a theoretical analysis for the proposed method. Experimentally, across five domain-shift settings, RefineMix matches or outperforms in-domain finetuning and data mixing. For protein sequence generation, finetuning with just 197 in-domain examples nearly doubles the fraction of generated proteins that are simultaneously novel, foldable, and in-family compared to standard finetuning.
Figures & tables
Figure 1: Noise-selective mixing. (a) Noising widens the target (green) and source (orange) distributions until both reach the prior π at t=1 . They overlap (purple) only at high noise, so RefineMix admits source data only below the cutoff tmax (blue). (b) How well a classifier tells the domains apart, against the noise level: for text (solid), it stays sharp until almost every token is masked and for images under Gaussian noise (dotted), it goes blind early. (c) Why text stays separable: masked sequences at decreasing noise (top to bottom), with the classifier’s probability of the target. A single target-specific token (green) settles the domain, whereas shared tokens (white) add little.
Pair (src → tgt)
Tgt. tokens
Fine-tune
Data Mixing
Conditional
RefineMix ( tmax )
Math → Stat
12.96 M
19.92 ± 0.01
19.61 ± 0.01
19.10 ± 0.00
18.29 ± 0.01 (0.7)
Physics → Hep-Ex
4.71 M
7.49 ± 0.02
7.17 ± 0.01
7.09 ± 0.01
6.71 ± 0.02 (0.8)
cc_news → Hep-Ex
4.71 M
8.04 ± 0.05
8.05 ± 0.01
11.28 ± 0.01
7.39 ± 0.01 (0.8)
cc_news → Textiles
1.17 M
20.40 ± 0.04
19.73 ± 0.12
30.12 ± 1.19
19.70 ± 0.06 (0.9)
Patents-G → AG News
5.67 M
28.02 ± 0.15
31.55 ± 0.16
29.55 ± 0.04
27.47 ± 0.02 (0.2)
Table 1: RefineMix beats the baselines when adapting a source-pretrained model. Target validation perplexity (lower is better) after a source checkpoint (masked kernel, three seeds). Fine-tuning uses the target data alone, data mixing adds the source at every noise level, the conditional model adds it with a domain label, and RefineMix adds it only below tmax .
Figure 2: Testing the theory (masked kernel, three seeds). (a) CS → Math. A fully mixed model completes a noised sequence with n revealed tokens from CS, Math, or both. (b) cc_news → Hep-Ex, from the source checkpoint. Target validation perplexity against the cutoff, whose endpoints are fine-tuning and data mixing. (c) CS → Math, from scratch. Target validation perplexity against the fraction of the target corpus kept, for target-only training and RefineMix at its best cutoff.
Model
Target
Fine-tuning
Data mixing
LoRA
RefineMix ( tmax )
MDLM 130M
Lean/mathlib
3.289 ± 0.010
3.439
3.775 ± 0.007
3.234 ± 0.003 (0.2)
( Sahoo et al., 2024 )
Rare disease
15.703 ± 0.033
15.770
18.859 ± 0.117
14.933 ± 0.016 (0.4)
ChemRxiv
18.739 ± 0.002
20.260
25.248 ± 0.164
18.048 ± 0.017 (0.4)
SEDD 424M
Lean/mathlib
3.027 ± 0.004
3.147
3.314 ± 0.013
2.987 ± 0.005 (0.3)
( Lou et al., 2024 )
Rare disease
14.072 ± 0.002
13.720
15.473 ± 0.009
13.275 ± 0.055 (0.5)
ChemRxiv
16.588 ± 0.031
17.340
19.659 ± 0.126
15.915 ± 0.055 (0.4)
Table 2: RefineMix adapts pretrained diffusion language models to scarce domains. Target validation perplexity (lower is better). LoRA (rank 16 ) fine-tunes on the target alone. Mean ± std. over three seeds (data mixing: one). ∗ DiffuLLaMA uses LoRA (rank 64 ) in every column.
Family
Folds
Novel
All three
Real peroxiredoxins (ceiling)
94.0 ± 0.8
98.2 ± 0.9
78.1 ± 6.8
75.1 ± 5.7
Fine-tuning
51.8 ± 5.0
37.5 ± 4.6
99.0 ± 0.3
35.8 ± 4.6
Data mixing
36.2 ± 2.2
35.7 ± 2.2
98.7 ± 0.2
32.9 ± 2.2
RefineMix ( tmax=0.7 )
79.8 ± 2.4
74.3 ± 2.9
89.8 ± 1.8
63.3 ± 2.0
Generic UniRef50 (floor)
0.0
65.1
100.0
0.0
Table 3: RefineMix writes new, folding members of a protein family from 197 examples. Percentage of sequences generated by DPLM-650M that pass each test of Section 4 ; All three counts sequences that pass all tests at once. Real held-out peroxiredoxins and generic proteins mark the ceiling and the floor. Mean ± standard deviation over three train–test splits, each with three seeds.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Corpus
Tokenizer
Tokens
arXiv Math
bert-base-uncased
103.2 M
arXiv Stat
(our 139 M DiT)
14.4 M
arXiv Physics
46.5 M
arXiv Hep-Ex
5.3 M
arXiv CS
193.1 M
CC-News
373.5 M
Appendix
Table 4: Corpora , in tokens of the listed tokenizer. The upper block serves our 139 M model (Table 1 and Figure 2 ), the lower two the pretrained models of Table 2 . Of the two web corpora we use only one parquet file each of OpenWebText ( 100,173 documents) and of DKYoon/SlimPajama-6B . Figure 2(c) keeps 100% down to 1% of the Math training data.
Pair (src → tgt)
Tgt. tokens
Fine-tune
Data mixing
Conditional
RefineMix ( tmax )
cc_news → Hep-Ex
4.71 M
8.49 ± 0.03
8.76 ± 0.02
12.29 †
7.96 ± 0.01 (0.7)
Appendix
Table 5: Comparison to baselines under the uniform kernel. Target validation perplexity, same setting as Table 1 . Mean ± standard deviation over three seeds; † one seed.
Figure 3: The three predictions of the theory under the uniform kernel (three seeds). Same experiments as Figure 2 ; in (b), the source checkpoint is a DUO model of the same size, trained on same datasets.
Figure 4: Ablation of the source rate λ (CS → Math, 1% of the target, trained from scratch, masked kernel, one seed). (a) Target validation perplexity against λ . (b) On-target rate, the fraction of samples that the domain classifier assigns to the target, at sampling temperature 1.0 , against λ .
Figure 5: Samples generated by the models of Table 1 . For each of the five source–target pairs, all models start from the same source-pretrained checkpoint and are adapted to the target (masked kernel, mean over three seeds). We sample from pure noise over a sweep of sampling temperatures. (a) On-target rate: the fraction of samples that a domain classifier assigns to the target, against the sampling temperature, for fine-tuning, data mixing, the conditional model, and RefineMix. RefineMix stays on target at every temperature. (b) Generation frontier: generative perplexity of the samples, scored by OPT-2.7B, against their entropy, for fine-tuning and RefineMix. Perplexity is inverted, so up and to the right means more fluent and more diverse samples, and the dashed line marks real target text. The RefineMix frontier matches or lies above fine-tuning on every pair.
Figure 6: Domains stay separable almost to t=1 . Balanced accuracy of the domain classifier ct vs. noise level t , six source–target pairs of our text experiments; chance is 0.5 . Solid = far source, dashed = near. ct is a DDiT-style time-conditioned transformer ( 256 hidden, 4 blocks, 4 heads, attention-mean pooling) under the log-linear schedule, whose corrupted fraction is linear in t ; 4000 steps, batch 64 , lr 3×10−4 with cosine decay and 10% warmup, no label smoothing. The two panels differ only in the corruption kernel.
Figure 7: Where the source helps the target. For each pair of Table 1 , data mixing and fine-tuning start from the same source-pretrained checkpoint (masked kernel, mean over three seeds). We plot the loss gap g(t) , the per-level cross-entropy of data mixing minus that of fine-tuning, on held-out target text. Below zero, the source improves the target at that noise level; above zero, the price of mixing has taken over. On the arXiv pairs, the gap turns positive at the chosen cutoff; elsewhere, it is a rough guide.
Figure 8: RefineMix keeps what the source checkpoint knows. For each pair of Table 1 , fine-tuning and RefineMix start from the same source-pretrained checkpoint (masked kernel, mean over three seeds). We plot the change in per-level cross-entropy on held-out source text between the checkpoint and the end of training; positive values mean that source knowledge was lost. The dashed line marks the cutoff of RefineMix, and the shaded region is where it admits the source. Fine-tuning loses source knowledge at every level, whereas RefineMix keeps it below the cutoff and moves away only above it.
Figure 9: Generation frontiers of the adapted models of Table 2 . Each of the four pretrained models is adapted to each of the three targets, and we sample from pure noise over a sweep of sampling temperatures (mean over three seeds). We plot the generative perplexity of the samples, scored by OPT-2.7B, against their entropy, for fine-tuning and RefineMix. Perplexity is inverted, so up and to the right means more fluent and more diverse samples, and the dashed line marks real target text.
Figure 10: The adapted models of Table 2 stay on target. Each of the four pretrained models is adapted to each of the three targets, and we sample from pure noise over a sweep of sampling temperatures (mean over three seeds). We plot the on-target rate, the fraction of samples that a domain classifier assigns to the target rather than the source, against the sampling temperature, for all methods.
Figure 11: Testing Theorem 1 directly (CS → Math, three seeds). A fully mixed model ( λ=0.9 , 1% of the target) completes noised sequences in which n target-specific tokens are revealed and the rest is masked. We plot two estimates of how often a completion is source text, against n : the classifier’s source probability of the noised input, shifted to the prior λ equation 20 , and the classifier’s score of the completion. By Theorem 1 , the two agree for the optimal classifier; a trained classifier makes the input side an upper bound. Bands are bootstrap intervals over 256 samples per 3 seeds; the dashed line marks λ=0.9 .
We propose Ambient Dataloops, an iterative framework for refining datasets that makes it easier for diffusion models to learn the underlying data distribution. Modern datasets contain samples of highly varying quality, and training directly on such heterogeneous data often yields suboptimal models. We propose a dataset-model co-evolution process; at each iteration of our method, the dataset becomes progressively higher quality, and the model improves accordingly. To avoid destructive self-consuming loops, at each generation, we treat the synthetically improved samples as noisy, but at a slightly lower noisy level than the previous iteration, and we use Ambient Diffusion techniques for learning under corruption. Empirically, Ambient Dataloops achieve state-of-the-art performance in unconditional and text-conditional image generation and de novo protein design. We further provide a theoretical justification for the proposed framework that captures the benefits of the data looping procedure.
Adrián Rodríguez-Muñoz, William Daspit, Adam Klivans +3
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology · Department of Computer Science, University of Texas at Austin
Discrete diffusion models are a powerful class of generative models with strong performance across many domains. For efficiency, however, discrete diffusion typically parameterizes the generative (reverse) process with factorized distributions, which makes it difficult for the model to learn the target process in a small number of steps and necessitates a long, computationally expensive sampling procedure. To reduce the gap between the target and model distributions and enable few-step generation, we propose Forward-Learned Discrete Diffusion (FLDD), which introduces discrete diffusion with a learnable forward (noising) process. Rather than fixing a Markovian forward chain, we adopt a non-Markovian formulation with learnable marginal and posterior distributions. This allows the generative process to remain factorized while matching the target defined by the noising process. We train all parameters end-to-end under the standard variational objective. Experiments on various benchmarks show that, for a given number of sampling steps, our approach produces a higher quality samples than conventional discrete diffusion models using the same reverse parameterization.
Denoising Diffusion Probabilistic Models (DDPMs) generate samples by starting from noise and repeatedly denoising while keeping each update close to the current noisy state. This behavior is effective in many continuous domains, but its role is less clear for globally constrained discrete tasks, such as Sudoku, graph connectivity, Latin squares, and N-queens. In such settings, early discrete errors can be difficult to undo. As a result, standard diffusion sampling may preserve early mistakes, even when the model's clean predictions are informative. We compare standard samplers to sampling directly from the model's clean prediction. Without retraining, this single change improves Sudoku validity from 31% to 95%, with consistent gains across the other discrete tasks. We hypothesize that staying close to the current noisy state is harmful because the reverse trajectory can drift off the forward noising distribution the model was trained on. To reduce this train-test mismatch, we further introduce self-correction training, which exposes the model to its own predictions, improving robustness to errors that arise during inference. This substantially improves the performance of standard samplers. Our results suggest that continuous diffusion models can learn nontrivial global constraints, but discrete reasoning tasks require better alignment between training and inference: either through samplers that reduce commitment to early decisions, or through training that teaches the model to correct its own inference-time errors.
Mariia Drozdova, Stéphane Liem Nguyen, François Fleuret