Train the Sampler You Run: Few-Step Masked Diffusion Language Models via Mixture-Likelihood Fine-Tuning
Organizations: University of Zanjan
Abstract
When a masked diffusion language model is sampled in a few parallel steps in a random order, the model actually run is a mixture over reveal plans, while cross-entropy training fits token conditionals rather than this mixture. We introduce mixture-likelihood fine-tuning (Mix-FT), which fine-tunes a pretrained model on an importance-weighted bound of its own sampler's likelihood, with no teacher and no change to the architecture or the sampler. On LLaDA-8B, a LoRA fine-tune for 0.002% of the pretraining compute writes better OpenWebText continuations than continued cross-entropy and Alpha diffusion given the same compute, at every tested step count. With half the model evaluations, 16 rather than 32 steps per block, its continuations score better than the original model's in every seed under three judges, with confidence-based decoding as well as random order, and under confidence-based decoding two blinded LLM raters from different model families prefer them in 69 and 71% of pairs. On a public 110M-parameter block diffusion model, Mix-FT takes under four T4-hours and beats continued cross-entropy, Alpha diffusion and SDTT distillation at 4, 8 and 16 steps per block, under two judges in every seed. Exact likelihoods on text8, with a control that weights every plan equally, attribute about three quarters of the likelihood gain over continued cross-entropy to weighting each plan by how well it explains the sequence.
Figures & tables
| sampler bound | judge NLL, content tokens, at steps per block (time) | |||||||
| model | opt. steps | NELBO PPL | (55 s) | (93 s) | (162 s) | (305 s) | ||
| Start (BD3-LM) | – | 3.223 | 3.775 | 23.00 | 7.31 | 6.10 | 4.60 | 3.71 |
| first token fixed † | 7.46 | 6.17 | 4.54 | 3.63 | ||||
| Continued CE | 5,760 | 3.209 | 3.764 | 22.41 | – | 6.20 | 4.82 | 3.89 |
| SDTT-4 | 2 1,920 | 3.245 | 3.796 | 24.41 | 6.68 | 5.83 | 4.75 | 4.02 |
| SDTT-8 | 3,840 | 3.228 | 3.775 | 23.42 | – | – | 4.58 | 3.80 |
| random order | low confidence | |||||
| model | 16 | 32 | 16 | 32 | ||
| Start | 5.01 | 3.98 | 3.54 | 5.72 | 5.02 | 4.48 |
| Continued CE | 4.88 | 3.89 | – | 5.69 | 4.83 | – |
| Alpha ( ) | 4.67 | 3.74 | – | 5.55 | 4.76 | – |
| Mix-FT (ours) | 4.29 | 3.48 | – | 5.17 | 4.38 | – |
| start at minus Mix-FT at | ||||||
| arm | NLL | ||||
|---|---|---|---|---|---|
| Base 4 (start) | 1.804 | ||||
| Continued CE | 1.792 | 0 | 0 | 0 | 0 |
| SDTT | 1.802 | – | |||
| Per-path FT | 1.782 | – | – | ||
| Hybrid | 1.764 | – | |||
| Mix-FT | 1.751 |
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
| sampler bound | judge NLL at steps per block | |||||||
|---|---|---|---|---|---|---|---|---|
| training plans | opt. steps | NELBO PPL | ||||||
| , (main) | 1,152 | 3.181 | 3.723 | 24.51 | 7.07 | 5.54 | 4.10 | 3.37 |
| , (3 seeds) | 443 | 3.187 | 3.741 | 23.29 | 7.30 | 5.85 | 4.33 | 3.49 |
| , | 1,047 | 3.194 | 3.747 | 22.73 | – | 6.01 | 4.53 | – |
| , | 1,440 | 3.197 | 3.751 | 22.54 | – | 6.09 | 4.60 | – |
| , | 1,920 | 3.181 | 3.731 | 23.55 | – | 5.67 | 4.27 | – |
| model | ||||
|---|---|---|---|---|
| Start (BD3-LM) | 7.18 | 6.08 | 4.59 | 3.69 |
| Continued CE | – | 6.14 | 4.79 | 3.86 |
| SDTT-4 | 6.70 | 5.82 | 4.72 | 3.98 |
| SDTT-8 | – | – | 4.55 | 3.77 |
| Alpha diffusion ( ) | 6.99 | 5.75 | 4.52 | 3.75 |
| Mix-FT ( plans) | 6.96 | 5.50 | 4.07 | 3.33 |
| GPT-2-large | Pythia-1.4B | ||||||||
| Mix-FT against | seed 0 | seed 1 | seed 2 | mean | seed 0 | seed 1 | seed 2 | mean | |
| Start (BD3-LM) | 2 | 0.218 | 0.320 | 0.195 | 0.244 | 0.205 | 0.268 | 0.176 | 0.216 |
| 4 | 0.573 | 0.584 | 0.532 | 0.563 | 0.595 | 0.587 | 0.560 | 0.581 | |
| 8 | 0.521 | 0.497 | 0.469 | 0.496 | 0.544 | 0.518 | 0.492 | 0.518 | |
| 16 | 0.348 | 0.364 | 0.312 | 0.341 | 0.368 | 0.388 | 0.334 | 0.363 | |
| Continued CE | 4 | 0.569 | 0.735 | 0.673 | 0.659 | 0.573 | 0.709 | 0.660 | 0.647 |
| first arm | second arm | compute | updates | seeds | GPT-2 | Pythia | GPT-2 | Pythia | GPT-2 | Pythia |
| (a) plan coarseness and update count | ||||||||||
| two-step | four-step | 1 / 1 | 1,152 / 443 | 3 | ||||||
| two-step, 443 steps | four-step | 0.38 / 1 | 443 / 443 | 3 | – | – | ||||
| two-step, 443 steps | two-step | 0.38 / 1 | 443 / 1,152 | 3 | – | – | ||||
| two-step | four-step, 1,152 steps | 1 / 2.6 | 1,152 / 1,152 | 1 | – | – | ||||
| matched entropy | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 512 samples | 128 samples | 512 samples | 128 samples | |||||||||
| model | ||||||||||||
| Start (BD3-LM) | 8.0 | 5.4 | 4.2 | 10.9 | 8.7 | 5.2 | 4.3 | 3.5 | 3.3 | 6.3 | 4.4 | 3.3 |
| Continued CE | 9.4 | 6.1 | 4.7 | 12.5 | 9.6 | 6.1 | 4.9 | 3.8 | 3.5 | 6.5 | 5.0 | 3.6 |
| Alpha ( ) | 7.3 | 5.5 | 4.8 | 11.1 | 7.5 | 5.6 | 7.3 | 5.3 | 4.6 | 12.2 | 7.5 | 5.3 |
| SDTT-4 | 2.4 | 1.8 | 2.2 | 3.6 | 2.6 | 1.8 | 1.5 | 1.4 | 1.9 | 1.7 | 1.6 | 1.3 |
| model | Vendi | self-BLEU-4 | distinct-4 | rep-4 (%) | |
| Real held-out text | – | 172 | 0.341 | 0.898 | 3.6 |
| Start (BD3-LM) | 8 | 115 | 0.292 | 0.923 | 3.1 |
| Start (BD3-LM) | 16 | 137 | 0.353 | 0.891 | 3.8 |
| Continued CE | 16 | 129 | 0.358 | 0.892 | 3.6 |
| Alpha diffusion | 16 | 133 | 0.363 | 0.869 | 5.3 |
| SDTT-8 | 16 | 138 | 0.339 | 0.916 | 2.5 |
| sampler bound | vs continued CE | |||||
|---|---|---|---|---|---|---|
| model | NELBO PPL | GPT-2-L all | GPT-2-L content | Pythia content | ||
| Start (BD3-LM) | 3.223 | 3.775 | 23.00 | |||
| Continued CE | 3.209 | 3.764 | 22.41 | 0 | 0 | 0 |
| SDTT-4 | 3.245 | 3.796 | 24.41 | |||
| Hybrid | 3.200 | 3.752 | 22.69 | |||
| Alpha diffusion | 3.202 | 3.758 | 23.16 | |||
| IWAE-64 | IWAE-256 | ESS of 256 plans | tightening | ||||||
|---|---|---|---|---|---|---|---|---|---|
| model | NELBO | ||||||||
| Start (BD3-LM) | 3.027 | 2.966 | 3.008 | 2.955 | 10.8 | 18.0 | 0.002 | 0.001 | 3.135 |
| Continued CE | 3.010 | 2.943 | 2.993 | 2.933 | 11.7 | 19.6 | reference | 3.109 | |
| Continued CE 8 | 3.001 | 2.934 | 2.984 | 2.924 | 11.8 | 19.8 | 0.000 | 0.000 | 3.100 |
| Mix-FT 1 | 2.994 | 2.940 | 2.969 | 2.924 | 6.6 | 9.7 | 0.007 | 0.006 | 3.199 |
| Mix-FT 8 , seed 0 | 3.034 | 2.991 | 2.995 | 2.963 | 4.4 | 5.9 | 0.022 | 0.018 | 3.361 |
| MAUVE, random order | MAUVE, confidence planner | judge NLL gap | |||||||
| model | |||||||||
| Start (BD3-LM) | 0.04 | 0.55 | 0.95 | 0.01 | 0.11 | 0.72 | 0.04 | 0.56 | 0.50 |
| Continued CE | 0.08 | 0.48 | 0.90 | 0.04 | 0.22 | 0.73 | 0.45 | 0.42 | |
| Alpha ( ) | 0.17 | 0.68 | 0.92 | 0.06 | 0.29 | 0.73 | 0.14 | 0.47 | 0.41 |
| SDTT-8 | – | 0.52 | 0.92 | 0.01 | 0.06 | 0.51 | – | 0.60 | 0.52 |
| Mix-FT 1 , seed 0 | 0.18 | 0.89 | 0.96 | 0.06 | 0.54 | 0.91 | 0.20 | 0.51 | 0.45 |
| comparison | seed | |||
|---|---|---|---|---|
| Mix-FT vs continued CE | 0 | 0.258 / 0.243 | 0.407 / 0.409 | 0.412 / 0.413 |
| 1 | 0.310 / 0.290 | 0.455 / 0.475 | 0.430 / 0.436 | |
| 2 | 0.125 / 0.109 | 0.304 / 0.304 | 0.370 / 0.371 | |
| Mix-FT vs start | 0 | 0.265 / 0.264 | 0.267 / 0.296 | 0.255 / 0.278 |
| both fix the first token | 0 | 0.253 / 0.239 | 0.241 / 0.238 | 0.144 / 0.152 |
| continued CE vs start | 0 | 0.006 / 0.021 | 0.140 / 0.113 | 0.157 / 0.135 |
| gain from fixing the first token | G | P | G | P | G | P |
|---|---|---|---|---|---|---|
| Start (BD3-LM) | ||||||
| Continued CE, seed mean | ||||||
| Mix-FT 1 , seed mean | ||||||
| Mix-FT 8 , seed mean | – | – | ||||
| with per-step schedules | – | – | – | – | ||
| IWAE-32, | NELBO | |||
|---|---|---|---|---|
| model | ours | BD3-LM’s | ours | BD3-LM’s |
| Continued CE, seed 0 | ||||
| Mix-FT 1 , seed 0 | ||||
| Mix-FT 8 , seeds 0, 1, 2 | to | to | to | to |
| start | LLaDA-8B-Base ( Nie et al., 2025b ) , GSAI-ML/LLaDA-8B-Base , revision 0f2787f2 , six bf16 shards (sha256 checked); 8.0B parameters, pretrained on 2.3T tokens; vocabulary 126,464 |
|---|---|
| port | our implementation, checked against the Hub’s remote code (transformers 4.46.3) before any training: on 4,104 masked positions its fp32 logits agree within and its masked CE to nats; in fp16, the mode used for training and sampling, its masked CE is within nats of the fp32 reference; LLaDA’s own likelihood estimator gives the same mean on 16 prompt–continuation pairs to 0.002 nats (Monte Carlo SE 2.5) |
| data | OpenWebText, re-tokenized with LLaDA’s tokenizer, documents joined with EOS; 65,536 training windows of 256 tokens from shard 0; evaluation on 256 windows from the last 3,000 documents of shard 79; prompt: BOS and the window’s first 64 tokens; the model generates the other 192 |
| LoRA | rank 32, alpha 32, fp32 adapters on all attention and MLP linears (83,886,080 parameters); base frozen in fp16 with an fp32 residual stream, layers split at 16 over the two GPUs of a T4 2; per-layer gradient checkpointing |
| budget | FE: a 257-position pass counts 1, or 3 with a backward. Mix-FT: , , 8 examples (window, block) per update with 25 rows each, a plan-weight pass without gradient and a weighted gradient pass (4 FE per row), 600 updates, 480,000 FE; Mix-FT (seed 0): 4 examples per update, 309 updates, 479,568 FE; continued CE and Alpha ( ): 267 rows per update at the four-step random-order states, 600 updates, 480,600 FE. Each is 0.0018% of pretraining, counting a forward and a backward for each of the 2.3T pretraining tokens |
| optimizer | AdamW (0.9, 0.999), no weight decay, lr , 5% warmup, cosine to 0, clip 1, fp16 loss scaling from , micro-batch 8 rows; window order seed , LoRA init seed |
| temperatures | per model, planner and : a bracketing pass on 32 prompts, then four main temperatures where the entropy curve reaches real text’s per-sample entropy , , and nats, 256 continuations each, and further files until real text’s entropy and 0.3 nats below it are both bracketed; the start’s sweep is seeded from its sweep |
| (a) seed means | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| sampler bound | GPT-2-large, random order | GPT-2-large, low confidence | |||||||||
| model | 8 | 16 | 8 | 16 | 32 | 8 | 16 | 32 | |||
| Start (LLaDA-8B-Base) | 3.085 | 2.792 | 2.669 | 5.95 | 5.01 | 3.98 | 3.54 | 6.38 | 5.72 | 5.02 | 4.48 |
| Continued CE | 3.050 | 2.757 | 2.635 | 6.27 | 4.88 | 3.89 | – | 6.48 | 5.69 | 4.83 | – |
| Alpha ( ) | 3.045 | 2.754 | 2.632 | 6.18 | 4.67 | 3.74 | – | 6.44 | 5.55 | 4.76 | – |
| Mix-FT ( , ) | 3.048 | 2.761 | 2.643 | 5.85 | 4.29 | 3.48 | – | 6.19 | 5.17 | 4.38 | – |
| random order | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-2-large | Pythia-1.4B | Qwen2.5-7B | |||||||||||
| against | s0 | s1 | s2 | mean | s0 | s1 | s2 | mean | s0 | s1 | s2 | mean | |
| Continued CE | 4 | ||||||||||||
| 8 | |||||||||||||
| 16 | |||||||||||||
| Alpha | 4 | ||||||||||||
| planner | model | rep4 | distinct-4 | self-BLEU-4 | near-dup. | Vendi | |
|---|---|---|---|---|---|---|---|
| random order | Start | 32 | 0.052 | 0.930 | 0.160 | 0.000 | 96.87 |
| Mix-FT | 16 | 0.040 | 0.937 | 0.182 | 0.000 | 103.20 | |
| Start | 16 | 0.042 | 0.943 | 0.148 | 0.000 | 92.79 | |
| Start | 8 | 0.024 | 0.965 | 0.115 | 0.000 | 76.70 | |
| Mix-FT | 8 | 0.025 | 0.958 | 0.157 | 0.000 | 94.39 | |
| Continued CE | 8 | 0.018 | 0.967 | 0.149 | 0.000 | 81.06 |
| arm | seed 0 | seed 1 | seed 2 | vs per-path | ESS / gap (s0) |
|---|---|---|---|---|---|
| Base 5 (continued CE) | 1.7944 | 1.7946 | 1.7878 | to | 4.33 / 0.130 |
| Per-path FT ( , equal weights) | 1.7839 | 1.7867 | 1.7760 | 0 | 4.37 / 0.129 |
| Mix-FT ( ) | 1.7518 | 1.7552 | 1.7449 | , , | 3.57 / 0.241 |
| Mix-FT, | 1.7853 | 4.34 / 0.130 | |||
| Mix-FT, | 1.7645 | 4.01 / 0.168 | |||
| Mix-FT, | 1.7553 | 3.75 / 0.210 |
| arm | seed 0 | seed 1 | seed 2 |
|---|---|---|---|
| block 8, , exact | |||
| C0 (masked CE pretraining) | 1.9595 | 1.9540 | 1.9339 |
| Base 1 / Mix 1 (from C0) | 1.8988 / 1.9093 | 1.8862 / 1.9031 | 1.8768 / 1.8830 |
| Base 4 / Mix 4 (from C0) | 1.8076 / 1.8735 | 1.8054 / 1.8687 | 1.8005 / 1.8545 |
| Base 5 | 1.7944 | 1.7946 | 1.7878 |
| SDTT from Base 4 | 1.8041 | 1.8033 | 1.7974 |
| (a) width | |||||||
| continued | gain | vs continued CE | vs CE at states | ||||
| width | parameters | CE, | at | per step | gap | per step | gap |
| 256 | 4.8M | 1.794 | 0.042 | 0.024 | 0.018 | 0.026 | 0.016 |
| 512 | 19M | 1.683 | 0.036 | 0.018 | 0.018 | 0.019 | 0.017 |
| 768 | 43M | 1.666 | 0.032 | 0.013 | 0.019 | 0.015 | 0.017 |
| (b) training plans | |||||||
| sampler | true conditionals | best global temp. | per-step temp. | re-optimized table |
|---|---|---|---|---|
| , fixed plan | 1.2049 | – | – | 1.2049 (no change) |
| , random order | 0.3899 | 0.2141 | 0.1138 | 0.0299 |
| , fixed plan | 0.5158 | – | – | 0.5158 (no change) |
| , random order | 0.0556 | 0.0200 | 0.0111 | 0.0000 |
| sampler | true cond. | exact mixture | best temperature | |||
|---|---|---|---|---|---|---|
| , random order | 0.0440 | 0.0436 | 0.0103 | 0.0025 | 0.0000 | 0.0096 |
| , random order | 0.1481 | 0.1462 | 0.0380 | 0.0085 | 0.0008 | 0.0319 |
| target, | objective | TC | TC of | |
|---|---|---|---|---|
| HMM ( ), | true conditionals | 0.1375 | 0.240 | 0.806 |
| path reverse KL | 0.1197 | 0.268 | ||
| marginal reverse KL | 0.0008 | 0.769 | ||
| Ising , inv. temp. 0.9, | true conditionals | 1.144 | 1.720 | 4.546 |
| path reverse KL | 0.680 | 0.171 | ||
| marginal reverse KL | 0.573 | 1.419 |
| start | the public BD3-LM checkpoint of Appendix I (no earlier fine-tune) |
|---|---|
| plans | steps per block in training, plans per sequence, batch 8 sequences of 1,024 tokens, micro-batch 1 |
| budget | FE per sequence; 357 updates; 2,210,544 FE the 1 budget of 276,480 (under 0.13% of pretraining); the chunked backward executes 1,088 FE per sequence, 3,107,328 in all, 0.18% of pretraining ( Appendix I ) |
| sequences seen | 2,856, a prefix of the 1 run’s data order (no sequence repeats); same plan generator (RNG code 52, seed ) |
| optimizer | AdamW (0.9, 0.999), no weight decay, peak lr , warmup steps, cosine to 10% over the 357 steps, clip 1, fp16 autocast, gradient checkpointing |
| result | IWAE-32 at : 3.2141, 3.2128, 3.2145; at : 3.7293, 3.7289, 3.7300; NELBO PPL 28.82, 28.68, 28.85 (seeds 0, 1, 2) |
| setting | judge | seed 0 | seed 1 | seed 2 | seed mean | 95% interval |
|---|---|---|---|---|---|---|
| global | GPT-2-large | |||||
| Pythia-1.4B | ||||||
| Qwen2.5-1.5B | ||||||
| MAUVE at (start at : 0.948) | 0.940 | 0.934 | 0.926 | 0.933 | – | |
| schedule | GPT-2-large | |||||
| Pythia-1.4B |
| arm | recipe | steps | FE | warmup | sequences | gain | |
|---|---|---|---|---|---|---|---|
| continued CE | CE at the sampler’s states, lr cosine to 10% over all steps | 46,080 | 2,211,840 | 100 | 368,640 | / | / |
| seed 1 | as seed 0 | 46,080 | 2,211,840 | 100 | 368,640 | / | / |
| seed 2 | as seed 0 | 46,080 | 2,211,840 | 100 | 368,640 | / | / |
| Alpha | as CE, Alpha loss , centering constant from the start model ( ) | 46,080 | 2,211,840 | 100 | 368,640 | / | / |
| seed 1 | as seed 0, with its own constant ( ) | 46,080 | 2,211,840 | 100 | 368,640 | / | / |
| seed 2 | as seed 0, with its own constant ( ) | 46,080 | 2,211,840 | 100 | 368,640 | / | / |
| start | SMDM 1.1B base ( Nie et al., 2025a ) , nieshen/SMDM , file mdm_safetensors/mdm-1028M-3300e18-rsl-0.01-bs-1024.safetensors , revision f28f1dfe , sha256 1b7378eb ; 1,028M non-embedding parameters, pretrained on SlimPajama with FLOPs; TinyLlama (Llama-2) tokenizer |
|---|---|
| data | OpenWebText: 65,536 training windows of 256 tokens from shard 0 (documents packed with EOS); 2,048 held-out windows from the last 3,000 documents of shard 79; real reference: held-out windows 0 to 1,023 (per-sample entropy 4.7235 nats) |
| LoRA | rank 32, alpha 32, all attention and MLP linears (24,084,480 parameters); base frozen in fp16 |
| sampler | , block 16; FE: a 256-position pass counts 1, or 3 with a backward |
| 1 (stage 14) | Mix-FT: , , 22 examples (window, block) per update, 600 updates, 475,200 FE; continued CE and Alpha ( ): 267 rows per update at the states, 600 updates, 480,600 FE; under 0.01% of pretraining |
| 8 (stage 27) | Mix-FT: , , 11 examples per update, 339 updates, 3,833,412 FE (about 0.06% of pretraining) |
| optimizer | AdamW (0.9, 0.999), no weight decay, lr , 5% warmup (30 and 16 steps), cosine to 0, clip 1, fp16 loss scaling, micro-batch 16 rows; window order seed , LoRA init seed |
| (a) seed means | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sampler bound | GPT-2-large | Pythia-1.4B | ||||||||||
| model | opt. steps | NELBO PPL | 4 | 8 | 16 | 4 | 8 | 16 | ||||
| Start (SMDM base) | – | 3.279 | 3.868 | 23.47 | 7.34 | 6.68 | 5.42 | 4.37 | 7.23 | 6.58 | 5.34 | 4.31 |
| Continued CE | 600 | 2.997 | 3.605 | 17.88 | 7.71 | 6.63 | 5.24 | 4.29 | 7.36 | 6.43 | 5.07 | 4.13 |
| Alpha ( ) | 600 | 2.999 | 3.613 | 18.43 | 7.59 | 6.48 | 5.16 | 4.37 | 7.36 | 6.35 | 5.04 | 4.27 |
| Mix-FT 1 ( ) | 600 | 2.990 | 3.598 | 19.37 | 7.45 | 5.99 | 4.46 | 3.75 | 7.16 | 5.80 | 4.31 | 3.61 |
| GPT-2-large | Pythia-1.4B | ||||||||
| arm | seed 0 | seed 1 | seed 2 | mean | seed 0 | seed 1 | seed 2 | mean | |
| Mix-FT 1 ( ) against | |||||||||
| Start (SMDM base) | 2 | ||||||||
| 4 | |||||||||
| 8 | |||||||||
| 16 | |||||||||
| contrast | test side | reference side | temperatures | per-sample entropy (nats) | main pairs | control pairs | judgments |
|---|---|---|---|---|---|---|---|
| c1 | 8 model at | start at | each model’s own per-step schedule (the same shape for all four models), | test 5.431 to 5.444, reference 5.434 | 512, drawn at random from 1,536 (171, 171 and 170 per seed) | 192 on the 8 model at (64 per seed) | 1,408 |
| c0 | start at | start at | the same schedule for both, | test 5.403, reference 5.434 | 256, drawn at random from 512 | 64 on the start at | 640 |
| c3 | 8 model at | start at | one global temperature each: for the 8 model (every seed), for the start | test 5.385 to 5.414, reference 5.457 | 192 (64 per seed) | 96 on the start at | 576 |
| (a) all runs | |||||||
| rater | contrast | per seed | cons. | agr. | reading | ||
| first token fixed (27 to 31 October) | |||||||
| Claude Sonnet 5.5 helpers | c1 | 0.428 [0.385, 0.472] | 0.485, 0.415, 0.384 | 0.831 [0.783, 0.878] | 0.78 | 76% | worse; share 0.84 [0.74, 0.94] |
| c0 | 0.055 [0.032, 0.081] | – | 0.984 [0.953, 1.000] | 0.95 | 95% | ||
| c3 | 0.786 [0.710, 0.855] | 0.848, 0.781, 0.730 | 0.987 [0.971, 1.000] | 0.88 | 89% | preferred | |
| ChatGPT “6.1 Sol Ultra” | c1 | 0.468 [0.425, 0.513] | 0.515, 0.455, 0.435 | 0.850 [0.802, 0.896] | 0.80 | 76% | inconclusive; share 0.93 [0.82, 1.03] |
| model | run | budget | T4-hours |
|---|---|---|---|
| 110M | Mix-FT, , (1,152 steps) | 1 | 3.85, 3.38, 3.76 |
| 110M | continued CE (5,760 steps) | 1 | 3.10, 3.58, 3.37 |
| 110M | SDTT-8 (3,840 steps) | 1 | 3.35, 3.30, 3.28 |
| 110M | Mix-FT, , (357 steps) | 8 | 38.9, 37.7, 38.0 |
| 110M | Mix-FT, , (358 steps) | 16 | 49.7, 48.5, 49.2 |
| 110M | continued CE (46,080 steps) | 8 | 26.8, 26.0, 25.4 |
Diffusion Language Models
Diffusion language models generate and refine text through iterative denoising, enabling bidirectional context and parallel token updates.
See all 494 papers