The Lattice of Transition Laws

T. Y. Tsui; Jiatao Gu; Lingjie Liu

cs.LGOct 8, 2026

The Lattice of Transition Laws

Authors: T. Y. Tsui, Jiatao Gu, Lingjie Liu

Organizations: University of Pennsylvania

Abstract

Diffusion and autoregression (AR) have long been seen as different categories of generative models, with diffusion specialising in continuous fields and AR specialising in discrete tokens. Recent work seeks to combine the advantages of the two models, and each hybrid fixes its decoding schedule by design. In this paper, we ask whether the performance of decoding schedules of one model can be predicted before decoding at a fixed number of steps. We describe diffusion, AR, and models in between as paths on one corruption lattice, and define the cost of a schedule as the dependence its parallel steps discard. The cost shows that the fewest steps of a zero-cost schedule are set by the geometry of the data, in the same way for tokens and for continuous fields. In particular, for data that are Markov on a graph and dependent along its paths, the fewest steps equal the graph's treedepth, which is logarithmic in the length of a sequence and linear in the side length of a grid. With fewer steps than the treedepth, every schedule pays a positive cost, whose ranking we predict before decoding with a kernel of pairwise dependence estimated from pretrained weights. Across text generation, image generation, and video generation, we verify most of the predictions about the rankings of different schedules under different metrics and benchmarks. This work therefore provides a design principle for decoding for future AR models, diffusion models, and anything in between. Our code is available at https://github.com/TSUITUENYUE/The-Lattice-of-Transition-Laws.

Figures & tables

Figure 1: Decoder schedules on the lattice. Top: columns are coordinates, rows are successive states from $\top$ to $\bot$ , and shades run from white (clean) to blue (corrupted). Bottom: image-token schedules at 12.5%, 25% and 50% of the path; the last two show noise gradients in raster and spread order; Table 1 defines the rules.

Figure 2: Both corners from one weight set. Pixels: columns give schedules and step counts (Figure 1 ), rows give training distributions, and cells show 2 samples and CIFAR-10 FID-50K. Tokens: text8 bpc at 85M; AR-chain rows use the identity order and masked-ELBO rows the masked-diffusion bound (protocol in Appendix B.2 ).

Figure 3: (a) The contiguous, spread and separator rules for a width-4 step on a chain of 16 coordinates, the separator rule from a state with 3 revealed coordinates. Arcs connect consecutive advanced coordinates, with heavier arcs indicating shorter distances. (b) The midpoint rule on 15 coordinates, advancing the midpoint of every run of masked coordinates at each step.

Principle	Rule	Each step selects	Schedule	text8	MAR	LLaDA
released	contiguous	neighbours, in order (raster)		✓	✓
	random	random positions		✓	✓	✓
	confidence	the most confident		✓		✓
distance	spread	evenly spaced		✓	✓	✓
	dilated	a fixed interleaved order		✓
	low-discrepancy	an evenly covering order			✓

Table 1: Selection rules by principle, what each step selects, and the models they are tested on. Schematics as in Figure 1 : 16 coordinates from all masked (top) to all clean (bottom) under steps of widths 1, 2, 4 and 9, or one snapshot of an image-token grid. Contiguous, random and confidence selection follow Arriola et al. (2025) , Sahoo et al. (2024) and Nie et al. (2025) ; dilated and low-discrepancy selection follow Luxembourg et al. (2026) and Besnier et al. (2025) .

$d$	displacement	information	8 steps	displacement	information	FID
	kernel			floor
1	0.330	1.449	raster	77.1	338	139.80
2	0.119	0.209	spread, then raster	46.4	182	17.44
3	0.051	0.048	random	8.11	14.7	13.02
4	0.030	0.017	midpoint	7.56	12.5	10.63
5	0.021	0.009	low-discrepancy	3.87	1.57	8.32

Table 2: MAR-B kernels at revealed fraction 0.5 (left) and 8-step floors (right). The kernels give the relative displacement of a masked token’s prediction when a revealed neighbour at Chebyshev distance

d

is perturbed, and the Kullback–Leibler divergence in bits of its head law when such a neighbour is revealed. A floor sums the kernel at fraction 0.5 over consecutive tokens of each step and over the steps; spread-then-raster’s floors came after its decode. FID-50K as in Table 3 .

Figure 4: Kernels measured before decoding. (a) Entropy decrease in bits when a second text8 coordinate is revealed, at mask rate 0.5, by sequence distance: within-word and across-boundary pairs, mean and s.e.m. over 3 seeds, 512 pairs per cell. (b) Relative displacement of a masked-token prediction after a revealed-token perturbation on MAR-B, at revealed fraction 0.5, by grid distance, beside the across-boundary curve of (a); both are normalised at distance 1.

text8, bpc	8	16	32	64	MAR-B, FID	8	16	32	64	256
contiguous	3.14	2.44	1.77	1.32	raster	139.80	46.97	13.98	7.49	3.93
confidence	2.95	2.18	1.76	1.50	random, the model’s own	13.02	4.10	2.48	2.32	2.33
random	1.80	1.67	1.62	1.58	midpoint	10.63	3.63	2.48	2.36	2.31
dilated	1.66	1.60	1.59	1.59	spread	9.44	3.27	2.40	2.34	2.28
spread	1.65	1.56	1.49	1.32	low-discrepancy	8.32	3.13	2.36	2.27	2.29
separator	1.63	1.52	1.38	1.47	nested	13.73	4.26	2.37	—	—

Table 3: Measured quality by selection rule and number of decoding steps. Left: text8 bpc under the protocol of Figure 5 ; the contiguous rule uses one step per block. Right: FID-50K on ImageNet-256 from released MAR-B weights and MAR’s released evaluator, with only the reveal order changed; 1 seed per cell, except random and midpoint at 8 and 16 steps, averaged over 3 seeds within 0.25 FID. The 256-step random rule reproduces the published score.

Figure 5: Schedule quality versus decoding steps. (a) Excess bpc over the exact left-to-right chain on 64 held-out text8 characters after a revealed 192-character prefix, teacher forced on one weight set. Insets show the schedules. (b) Excess FID-50K over MAR-B’s 256-step random order; patches show the first 32 tokens of each order and the dashed blue curve is low-discrepancy. Both vertical axes are linear below 0.1 and logarithmic above.

Steps	8	16	32	64	128
accuracy, min-distance	$+9.7\pm 0.8$	$+7.6\pm 0.9$	$+5.3\pm 1.5$	$-0.9\pm 1.6$	$-0.2\pm 1.4$
accuracy, separator constraint	$+9.7\pm 0.8$	$+8.2\pm 0.9$	$+6.5\pm 1.5$	$-2.0\pm 1.6$	$-1.4\pm 1.3$
perplexity, min-distance	$-1.04$	$-0.90$	$-1.49$	$-1.56$	—
95% interval	[ $-1.25$ , $-0.83$ ]	[ $-1.16$ , $-0.65$ ]	[ $-1.82$ , $-1.15$ ]	[ $-1.88$ , $-1.25$ ]

Table 4: Changes relative to plain confidence on LLaDA-8B-Base. GSM8K: strict-match percentage points, 5-shot greedy decoding on 1,319 problems, paired standard errors. Perplexity: change in log generative perplexity (nats), bootstrap intervals (Table 25 ). One 256-position block; minimum distance 6. When too few positions satisfy a constraint, confidence fills the step: 1–21% of GSM8K steps for min-distance and 8–55% for the separator constraint.

estimate / measured	2 steps	8 steps	32 steps
contiguous	0.83 / 2.64	0.62 / 1.82	0.33 / 0.44
random	0.54 / 1.09	0.15 / 0.23	0.02 / 0.03
spread	0.32 / 0.78	0.06 / 0.09	0.00 / 0.01
separator	0.32 / 0.78	0.01 / 0.06	0.00 / 0.00

Table 5: Pairwise estimate and measured excess bpc over each schedule’s own sequential chain (protocol of Figure 5 ). The plot adds 4 and 16 steps; estimates filled, measurements open.

Steps	8	16	32	64
absorbing, spread	1.691	1.600	1.584	1.555
graded, class then character	1.790	1.736	1.716	1.703
graded, character at once	1.886	1.793	1.749	1.717

Table 6: Bpc of a complete decode, all stages scored, at equal steps on held-out text8; one 85M backbone per channel.

Appendix figures & tables23 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Intermediate states of the CIFAR-10 lattice model with 16 patches of 8 by 8 and 128 transitions. Each row shows one of five schedules at 6 points along its path; all rows use the same seed. The last column gives the level map at the fourth point, from white (clean) to blue (corrupted). On the grid of 128 transitions, the mean noise-level index in each patch column decreases from left to right by 19 at slope 8, by 24 at slope 32 and by 32 along the sequential path. It is constant along the synchronised path. In spread order, the 4 column means are 128, 47, 128 and 15.

	sync.	slope 1	raster	spread	raster	spread	seq.
			slope 4		slope 16
steps	128	143	188	188	368	368	2063
iid levels	15.8	15.5	15.2	14.0	13.1	12.0	13.0
mixed levels	9.8	10.1	12.2	11.0	11.6	12.1	11.9
at 120K steps	—	—	—	—	11.1	10.5	—
synchronised only	7.6	20.7	187.2	171.2	251.6	241.0	273.4

Appendix

Table 7: CIFAR-10 FID over 50K samples for the models in Figure 2 . Rows specify the training distribution and columns the decoding schedule; the two lattice distributions appear above the two single-schedule controls. On a sloped path each patch starts its 128 transitions a fixed number of steps after the patch before it, the slope

s

, and the path takes

128+15s

steps. Each column labelled spread uses the sloped path at the indicated slope, with patches selected by the spread rule in place of raster order. The first lattice distribution draws each coordinate’s level independently, reaching a corner with vanishing probability. The second assigns 25% of its mass to each corner; its row at 120K steps continues the same model on fresh data.

	synchronised	sequential
steps	249	256
lattice, mixed levels	43.3	98.7
synchronised only	39.9	205.5
lattice, synchronised at 16 steps	54.5	—

Appendix

Table 8: FID-50K on ImageNet-256 against the ADM reference for 2 SiT-B/2 models trained for 400K steps at batch size 256. The models differ only in their training distribution over levels and are sampled at guidance scale 1, using 1 seed. At the sequential corner, both follow the same path through 16 groups of 16 patches. The final row uses the synchronised path, updating each coordinate as often as at the sequential corner.

	prefix, width 1	interior, width 1	interior, width 8
causal	1.22 $\pm$ 0.11	—	—
bidirectional	1.41 $\pm$ 0.10	2.00 $\pm$ 1.10	1.01 $\pm$ 0.31
two-stream	1.26 $\pm$ 0.10	2.26 $\pm$ 1.13	1.31 $\pm$ 0.37

Appendix

Table 9: Advances from lattice states evaluated under each architecture at matched data, budget and size, 16 seeds with one shared conditioning pattern per seed. A cell reports the loss under the law expressed by its architecture: a sequential chain for the causal and two-stream rows and a product law for the bidirectional row. We report the mean and standard deviation over the 16 draws. An empty cell is an advance the architecture cannot express. Variation in the interior width-1 column reflects position-to-position differences in data entropy. The paired excess of the two-stream value over the bidirectional value in the interior width-1 column is 0.27

\pm

0.08 bits, positive in every draw.

Distance	1	2	4	8	16
inside one word	0.996 $\pm$ 0.050	0.694 $\pm$ 0.114	0.505 $\pm$ 0.030	0.187 $\pm$ 0.021	—
across a boundary	0.368 $\pm$ 0.086	0.269 $\pm$ 0.077	0.121 $\pm$ 0.058	0.023 $\pm$ 0.032	0.009 $\pm$ 0.005

Appendix

Table 10: The pair kernel

\kappa

: the decrease in entropy at one masked coordinate when a second coordinate at the given distance is revealed. Values are in bits at mask rate 0.5, mean

\pm

standard deviation over 3 seeds with 512 pairs per cell. A cell is left blank when too few pairs are available.

Rule	mask rate	width 8	width 16	width 32
random	0.3	26.4 $\pm$ 12.5	63.4 $\pm$ 9.8	119.7 $\pm$ 1.0
random	0.5	73.1 $\pm$ 7.7	144.6 $\pm$ 12.8	268.1 $\pm$ 18.6
random	0.7	102.7 $\pm$ 18.6	228.7 $\pm$ 9.2	420.6 $\pm$ 20.0
spread	0.3	2.7 $\pm$ 3.2	6.7 $\pm$ 2.5	25.0 $\pm$ 3.3
spread	0.5	4.4 $\pm$ 1.0	13.1 $\pm$ 1.1	68.6 $\pm$ 10.6
spread	0.7	6.1 $\pm$ 2.3	27.4 $\pm$ 8.6	130.7 $\pm$ 2.7

Appendix

Table 11: Total correlation of the advanced set at 3 mask rates, in

10^{-3}

bits per coordinate over 3 seeds.

Mask rate	$d=1$	$d=2$	$d=4$	$d=8$
0.1	0.216	0.115	0.020	0.003
0.2	0.298	0.162	0.051	0.000
0.3	0.425	0.289	0.093	0.013
0.5	0.785	0.480	0.214	0.030
0.7	0.936	0.646	0.331	0.118

Appendix

Table 12: The pair kernel in Table 10 at 5 mask rates, in bits, averaged over 3 seeds. At each distance, we weight pairs inside a word and pairs across a boundary by their proportions in the corpus. To predict the costs in Table 5 , we evaluate the kernel at each step’s mask rate, interpolating linearly between measured rates and using the nearest value outside their range.

predicted / measured	2 steps	4 steps	8 steps	16 steps	32 steps
contiguous	0.92 / 2.64	0.80 / 2.34	0.71 / 1.82	0.59 / 1.12	0.39 / 0.44
random	0.58 / 1.09	0.33 / 0.50	0.16 / 0.23	0.07 / 0.10	0.02 / 0.03
spread	0.31 / 0.78	0.11 / 0.27	0.07 / 0.09	0.04 / 0.03	0.01 / 0.01
separator	0.31 / 0.78	0.08 / 0.26	0.01 / 0.06	0.01 / 0.02	0.00 / 0.00

Appendix

Table 13: Pairwise contribution estimated from the 26M model’s kernel for the schedules and mask rates in Table 5 , compared with the 85M model’s measured excess over each schedule’s sequential chain. Values are in bits per character.

Steps	2	4	8	16	32
contiguous	1.32	1.32	1.32	1.32	1.32
confidence	1.40	1.38	1.35	1.44	1.48
random	1.51	1.55	1.57	1.57	1.58
spread	1.53	1.57	1.56	1.52	1.48
separator	1.53	1.57	1.56	1.50	1.38

Appendix

Table 14: Bits per character of each rule’s sequential chain on the 85M model, for the decodes in Table 5 . We reveal the coordinates in each selected set one at a time, following the schedule’s order. The entropy of the text does not depend on this order; differences between rows arise from the order-dependent error of the trained conditionals.

second-order estimate / measured	2 steps	4 steps	8 steps	16 steps	32 steps
contiguous	1.28 / 2.64	1.07 / 2.34	0.90 / 1.82	0.69 / 1.12	0.33 / 0.44
random	0.76 / 1.09	0.39 / 0.50	0.17 / 0.23	0.07 / 0.10	0.02 / 0.03
spread	0.48 / 0.78	0.12 / 0.27	0.06 / 0.09	0.03 / 0.03	0.00 / 0.01
separator	0.48 / 0.78	0.10 / 0.26	0.02 / 0.06	0.01 / 0.02	0.00 / 0.00
confidence, measured after the decode	1.12 / 2.13	0.94 / 1.96	0.81 / 1.60	0.42 / 0.74	0.16 / 0.28

Appendix

Table 15: The second-order estimate and measured excess, in bits per character under the convention of Table 5 . We add the pairwise estimate in that table to the triplet term

I(x_{c};\,x_{a}\mid x_{b},\ell)

of the proof of Lemma 2 , estimated on consecutive triples of the same schedule with no revealed coordinate between them, each step at its own mask rate, with 512 triples per cell over 3 seeds. For confidence selection, we compute the estimate after decoding from the sets selected during decoding. We compare it with the measured excess bpc over the sequential chain that reveals those same coordinates in the same order.

Distance, pair	$s=0$	$s=1$	$s=2$	$s=3$
2, within a word	0.83	0.51	—	—
2, across a boundary	0.34	0.17	—	—
3, within a word	0.81	0.60	0.36	—
3, across a boundary	0.32	0.18	0.08	—
4, within a word	0.69	0.66	0.43	0.21
4, across a boundary	0.25	0.16	0.12	0.04

Appendix

Table 16: Text8 kernel of the 85M model at mask rate 0.5, with

s

of the characters between the pair revealed and the rest of the window masked at that rate: entropy decrease in bits when the second coordinate is revealed, mean over 3 seeds with 512 pairs per cell and seed.

Rule		8 steps	16 steps	32 steps
separator	first-order reference	0.01	0.01	0.00
	with the state between	0.06	0.01	0.00
	measured	0.06	0.02	0.00
midpoint	first-order reference	0.01	0.00	0.00
	with the state between	0.18	0.14	0.09
	measured	0.26	0.19	0.09

Appendix

Table 17: Excess bpc over each schedule’s own sequential chain on the 85M model under the protocol of Table 5 , for three rules that advance at most one coordinate per masked run whenever the runs allow, beside 2 estimates from kernel reads like those of Table 16 at mask rates 0.1 to 0.7, each step at its own rate. The first-order reference charges a pair of advanced coordinates the kernel at its distance when they share a run and zero when a revealed character lies between them; the estimate with the state between charges every pair the kernel at its distance and its number of revealed characters between. The third rule advances the midpoints of the longest runs and spreads its choices among runs of equal length; its estimate was computed before its decode.

Steps	64	32	16	8	4	2
spread	1.32 $\pm$ 0.00	1.49 $\pm$ 0.00	1.56 $\pm$ 0.00	1.65 $\pm$ 0.00	1.84 $\pm$ 0.00	2.31 $\pm$ 0.00
random	1.58 $\pm$ 0.00	1.62 $\pm$ 0.01	1.67 $\pm$ 0.01	1.80 $\pm$ 0.00	2.05 $\pm$ 0.01	2.60 $\pm$ 0.01
contiguous	1.32 $\pm$ 0.00	1.77 $\pm$ 0.00	2.44 $\pm$ 0.00	3.14 $\pm$ 0.00	3.66 $\pm$ 0.00	3.96 $\pm$ 0.00

Appendix

Table 18: Bits per character on the last 64 characters of held-out windows, teacher forced after a revealed 192-character prefix, using one weight set over full sequences. Decoding is deterministic except for the random rule, whose selections vary across the 3 sampling seeds. The 64-step column evaluates the sequential chain.

Selection rule	bpc	separation	pairwise estimate
confidence	2.18	2.0	0.34
min-distance	1.66	11.7	0.02
random	1.67	13.0	0.06
spread	1.56	16.0	0.03

Appendix

Table 19: Selection rules at 16 steps over the 64 decoded characters, width 4, using the evaluation protocol of Table 18 . Separation is the mean gap between consecutive selected positions within a step, 16.0 for an even grid over the decoded region. The last column is the pairwise estimate of Table 5 on each rule’s own sets, in bits per character. Min-distance keeps the positions of a step at least 8 apart.

	contiguous 8	contiguous 16	spread 8	spread 16
uniform	3.96	3.70	1.74	1.64
wide first	3.94	3.69	1.92	1.81
wide late	3.97	3.75	1.79	1.69
geometric, wide first	3.94	3.84	1.95	1.91
geometric, wide late	3.99	3.89	1.86	1.85

Appendix

Table 20: Width profiles compared at matched step counts. We report bits per character for complete decodes using one weight set, deterministically evaluated on fixed windows. Every profile distributes the same total number of coordinate advances over the same number of steps. Wide-first and wide-late use

w_{t}\propto(t{+}1)^{\mp 1}

; the geometric profiles scale width by

2/3

3/2

per step.

Steps	8	16	32	64
chain, absorbing	1.586	1.564	1.545	1.542
chain, graded	1.628	1.644	1.683	1.688
excess, absorbing	0.105	0.036	0.039	0.012
excess, graded	0.162	0.091	0.033	0.015
estimate, absorbing	0.029	0.014	0.011	0.004
estimate, graded (class + character)	0.057	0.013	0.008	0.007

Appendix

Table 21: Sequential-chain bpc, excess bpc from parallel decoding, and the pairwise estimate for the two models on the schedules in Table 6 . For each model, the chain follows its own schedule, revealing the coordinates of each set one at a time. The excess is the difference between the parallel decode and this chain. The estimate is computed from the model’s own kernels (Table 22 ).

Distance	1	2	4	8
absorbing, character	1.172 / 0.797	0.900 / 0.446	0.609 / 0.315	0.374 / 0.112
graded, class	0.177 / 0.283	0.054 / 0.127	0.019 / 0.080	0.021 / 0.008
graded, character given classes	0.494 / 0.006	0.560 / 0.016	0.455 / 0.054	0.262 / 0.026

Appendix

Table 22: Pair kernels of the two models at mask rate 0.7, following the convention of Figure 4 . Entries give bits for a pair within one word / a pair across a boundary, with 512 pairs per cell. We report the absorbing model’s character kernel and the graded model’s class kernel and character kernel conditioned on the classes.

Rule	8	16	64	256	unigram KL
contiguous	6.35 $\pm$ .01	6.11 $\pm$ .01	5.38 $\pm$ .03	1.42 $\pm$ .02	.053 $\pm$ .001
confidence	6.01 $\pm$ .02	5.91 $\pm$ .01	4.69 $\pm$ .02	1.82 $\pm$ .01	.048 $\pm$ .001
random	3.81 $\pm$ .02	3.08 $\pm$ .02	2.41 $\pm$ .01	2.17 $\pm$ .01	.001 $\pm$ .000
spread	2.60 $\pm$ .01	2.16 $\pm$ .02	1.98 $\pm$ .02	1.46 $\pm$ .01	.002 $\pm$ .000
word-aware spread	2.70 $\pm$ .01	2.21 $\pm$ .04	2.03 $\pm$ .05	1.80 $\pm$ .01	.002 $\pm$ .000

Appendix

Table 23: Free generation at matched steps: bits per character of the sampled text under the dedicated autoregressive scorer over 3 sampling seeds. The last column is the unigram divergence against the corpus at 16 steps. Word-aware spread is the separator rule with words as the segments, at most one coordinate per word until every word has one, with word boundaries read from the spaces decoded so far and the model’s own space predictions.

Steps	8	16	32	64
pooled	$-1.04$	$-0.90$	$-1.49$	$-1.56$
controlled	$-0.90$	$-0.67$	$-1.20$	$-0.92$
95% interval	[ $-1.08$ , $-0.71$ ]	[ $-0.88$ , $-0.47$ ]	[ $-1.55$ , $-0.90$ ]	[ $-1.25$ , $-0.59$ ]
less repetitive half	$-0.83$	$-0.61$	$-1.16$	$-1.48$

Appendix

Table 24: Change in log generative perplexity (nats) of the min-distance rule against plain confidence, on the samples of Table 25 . Pooled: the difference in Table 4 . Controlled: the rule’s coefficient when each sample’s log perplexity is regressed on the rule, its unigram entropy and its share of repeated 4-token sequences, with a 95% bootstrap interval over samples. Less repetitive half: the pooled difference among the samples whose share of repeated 4-token sequences lies below the median of both rules’ samples.

unigram entropy, bits
Steps	8	16	32	64
random	283 [246,323]	147 [123,175]	54 [47,61]	27 [23,32]
confidence	375 [331,431]	212 [182,245]	180 [142,239]	97 [74,130]
min-distance	133 [114,157]	86 [71,105]	41 [33,52]	20 [17,24]
midpoint	163 [147,182]	94 [83,106]	50 [44,57]	31 [27,35]
spread	168 [149,190]	112 [98,130]	54 [47,64]	35 [31,41]

Appendix

Table 25: Generative perplexity on released LLaDA-8B-Base weights, with only the selection rule changed in the sampler. We generate 512-token samples with nucleus sampling at probability threshold 0.9; confidence ranks positions by the probability of the predicted token, and GPT-2-large scores each whole sample, re-tokenised, with a sliding window. Each cell pools 192 samples over 3 sampling seeds and reports perplexity with a 95% bootstrap interval over samples. Min-distance keeps the positions of a step at least 6 apart, the first distance at which the kernel on these weights falls to 10% of its value at distance 1; when it leaves too few positions, the most confident remaining positions fill the step. The lower blocks report the samples’ unigram entropy in bits and their share of repeated 4-token sequences in the scorer’s tokens, averaged over the 192 samples.

	LLaDA-8B-Base, 64 steps			text8 85M, 16 steps
Rule	forward, ms	select, ms	%	forward, ms	select, ms	%
contiguous	—	—	—	211.3	0.09	0.04
random	170.8	0.12	0.07	212.3	0.13	0.06
confidence	170.9	0.28	0.17	216.9	0.22	0.10
min-distance	170.3	1.24	0.73	217.6	1.01	0.46
separator constraint	171.6	0.78	0.46	—	—	—

Appendix

Table 26: Forward-pass and selection time per decoding step, reported as medians over the timed steps on one L40. The device is synchronised around each timed region. We evaluate LLaDA-8B-Base at 64 steps on the first 20 GSM8K problems in Table 4 , one prompt at a time, and the text8 85M decoder at 16 steps on 256 windows with a revealed 192-character prefix.

GSM8K, sampling, 64 steps
Steps	8	16	32	64	128
confidence	1.8 $\pm$ 1.0	2.4 $\pm$ 1.2	5.5 $\pm$ 1.8	11.0 $\pm$ 2.4	26.2 $\pm$ 3.4
min-distance	7.9 $\pm$ 2.1	9.1 $\pm$ 2.3	12.2 $\pm$ 2.6	17.7 $\pm$ 3.0	26.8 $\pm$ 3.5
separator constraint	7.9 $\pm$ 2.1	8.5 $\pm$ 2.2	13.4 $\pm$ 2.7	18.3 $\pm$ 3.0	25.6 $\pm$ 3.4
confidence	41.3 $\pm$ 1.4
min-distance	45.8 $\pm$ 1.4

Appendix

Table 27: HumanEval pass@1 in percent on released LLaDA-8B-Base weights, with only the selection rule changed in LLaDA’s sampler. We use greedy decoding on all 164 problems, one prompt at a time, score completions with the original code-execution harness in a sandbox and report binomial standard errors. Min-distance keeps the positions of a step at least 6 apart. The lower block reports GSM8K strict match at 64 steps on the 1,319 problems, using LLaDA’s sampling procedure in place of greedy decoding; the paired difference is

4.5\pm 1.6

points. Under plain confidence at 256 steps, the same harness gives 71.4

\pm

1.2 on GSM8K, against the 70.0 that LLaDA’s repository reports at this setting.

		subject	background	motion	imaging
Memory depth	steps	consistency	consistency	smoothness	quality
17 frames, as shipped	300	0.8998	0.9505	0.9937	65.70
5 frames	270	$-0.0087\pm 0.0206$	$-0.0053\pm 0.0047$	$-0.0015\pm 0.0013$	$-3.49\pm 2.44$
25 frames	330	$-0.0068\pm 0.0130$	$-0.0033\pm 0.0120$	$+0.0000\pm 0.0004$	$-3.50\pm 1.43$
33 frames	360	$-0.0109\pm 0.0128$	$-0.0007\pm 0.0062$	$+0.0001\pm 0.0006$	$-0.34\pm 1.41$
57 frames	540	$-0.0278\pm 0.0181$	$-0.0190\pm 0.0026$	$+0.0009\pm 0.0005$	$-5.15\pm 3.15$

Appendix

Table 28: Memory depth and decoding steps on the released SkyReels-V2 checkpoint ( Chen et al., 2025 ) at 8 times its training horizon, in the 4 VBench dimensions declared before decoding, measured over the entire 773-frame rollout with every frame denoised. The shipped row gives the mean over 3 prompts; the other rows give the paired difference from it over the same prompts, with its paired standard error.

Explore similar work

DiffusionGemma Technical Report
Jul 31, 2026DiffusionGemma Team, Adrien Ali Taïga, James Assiene +41Blockwise Diffusion Language Models Diffusion Model Fine-Tuning
Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models
Jun 22, 2026Jiawei Xu, Minghui Liu, Aakriti Agrawal +2Masked Diffusion Models Language Model Decoding
University of Maryland, College Park · University of California, Los Angeles.
Tensor-Train Joint Modeling for Few-Step Discrete Diffusion
Jul 4, 2026Byoungkwon Kim, Minhyuk SungMasked Diffusion Models Tensor Decomposition
KAIST

Jul 31, 2026, cs.CLJ/K move · Enter open · S save

DiffusionGemma Technical Report

DiffusionGemma Team, Adrien Ali Taïga, James Assiene, Daniele Calandriello+40

Blockwise Diffusion Language Models Diffusion Model Fine-Tuning Diffusion Language Model Decoding Diffusion Language Models Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.

Abstract

Explore similar work

DiffusionGemma Technical Report

Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Models

Tensor-Train Joint Modeling for Few-Step Discrete Diffusion