Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation

Hermann Kumbong; Ishan S. Khare; Iddah Mlauzi; Andrew Woen; Charles Ding; E. Kelly Buchanan; Simran Arora; Azalia Mirhoseini

cs.CLOct 9, 2026

Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation

Authors: Hermann Kumbong, Ishan S. Khare, Iddah Mlauzi, Andrew Woen, Charles Ding, E. Kelly Buchanan, Simran Arora, Azalia Mirhoseini,

Organizations: Department of Computer Science, Stanford University · Department of Electrical Engineering, Stanford University · Together AI

Abstract

Diffusion language models (DLMs) offer parallel sampling for faster generation than autoregressive models, an increasingly important advantage as inference becomes a dominant computational workload. However, DLMs degrade quickly with fewer sampling steps, process the full sequence at every step, and require many passes over the dataset to reach good quality. We introduce Next-Concept Modeling (NCM), a two-stage, coarse-to-fine framework: a multiscale residual VQ-VAE encodes text into a hierarchy of discrete codes, and a transformer generates one level at a time, predicting all codes within a level in parallel. At matched parameter scales, NCM achieves better generative perplexity than the strongest DLM baseline on LM1B while running $21\times$ faster. On OpenWebText, NCM is competitive with the strongest DLM while running $10\times$ faster; on XSum, it matches the strongest DLM's ROUGE-L while being $31\times$ faster. On the reasoning task of Sudoku, NCM outperforms autoregressive and diffusion baselines across all difficulty levels, using fewer function evaluations. On the reasoning task of Sudoku, NCM outperforms autoregressive and diffusion baselines across all difficulty levels, using fewer function evaluations.

Figures & tables

Panel of Figure 1 — Figure 1: Generation efficiency and reasoning quality. (a,b) NCM achieves better or on-par quality with $21\times$ and $31\times$ lower wall-clock time on LM1B and XSum, respectively (Section 4.6 ). $H$ denotes mean unigram entropy. (c) NCM maintains the highest Sudoku accuracy as difficulty increases, using fewer function evaluations (NFE) than the diffusion baselines.

Panel of Figure 2 — Figure 2: Comparing text generation paradigms. Next-token prediction generates one token per step, and masked diffusion (discrete diffusion) refines all $L$ positions at every step. NCM instead generates discrete codes $r_{k}^{i}$ , the $i$ -th code of level $k$ , from coarse to fine in $K\ll L$ steps: each step predicts the $\ell_{k}$ codes of one level in parallel, conditioned on the reconstruction $\hat{z}^{(k-1)}$ of the earlier levels, and after the final level, the cumulative reconstruction is decoded into tokens.

Panel of Figure 3 — Figure 3: NCM generation trajectory. Output decoded after selected sampling steps of one German–English translation (WMT14), shown verbatim; the last step (gold frame) is the final output. Words matching the reference are shaded green. Coarse levels fix the high-level idea; finer levels fill in and correct the content. Appendix J.2 shows all sampling steps with their scores.

Panel of Figure 4 — Figure 4: NCM’s two-stage training. (a) A multiscale residual VQ-VAE learns to encode text into a coarse-to-fine hierarchy of discrete codes $r_{1},\ldots,r_{K}$ and reconstruct the input. (b) The NCM transformer predicts all codes within level $k$ in parallel from the cumulative reconstruction $\hat{z}^{(k-1)}$ , resized to $\ell_{k}$ positions. In the unconditional setting, block-diagonal attention restricts each input block to itself. During training, all $K$ levels are predicted in one forward pass using a cross-entropy loss. During generation, levels are predicted sequentially from coarse to fine.

	Unconditional			Conditional		Reasoning (Sudoku)
	OWT	LM1B	TinyStories	XSum	WMT14	Easy	Medium	Hard
Recon. acc. (%) $\uparrow$	99.99	99.86	99.98	99.79	98.79	100.00	100.00	100.00
Utilization (%)	83.10	89.31	90.12	86.20	61.89	0 80.73	0 83.85	0 80.73

Table 1: Tokenizer reconstruction and codebook utilization. Reconstruction accuracy measures exact token recovery on held-out text using the full hierarchy. Utilization is the mean percentage of codebook entries used across levels.

Figure 5: Unconditional generation quality and sampling cost. Generative perplexity (Gen. PPL; lower is better) against the number of function evaluations on TinyStories and LM1B, and against the sampling time per

1{,}024

-token sequence (batch size

16

) on OpenWebText. NCM achieves competitive or better Gen. PPL compared to strong diffusion baselines with substantially fewer function evaluations and less time.

H

denotes mean per-sample unigram entropy.

		WMT14 De–En			XSum
Type	Method	Size	BLEU $\uparrow$	NFE $\downarrow$	Size	ROUGE-1 $\uparrow$	ROUGE-2 $\uparrow$	ROUGE-L $\uparrow$	NFE $\downarrow$
–	AR †	99M	25.2	64	99M	30.5	10.2	24.4	64
Without guidance
Diff.	MDLM †	99M	18.4	64	99M	33.4	11.6	25.8	64
	DUO †	205M	21.3	64	205M	31.4	10.1	25.0	64
	E2D2 †	99M	24.8	64	99M	28.4	8.3	22.0	64

Table 2: Conditional generation quality and sampling cost. BLEU on WMT14 De–En and ROUGE-1/2/L on XSum (higher is better). Size denotes task-specific generator parameters; NFE counts generator calls per output. Bold and underlined values mark the best and second-best results within each non-AR decoding setting, including ties. † : results from Hu et al. [13] .

Panel of Figure 6 — Figure 6: Model and test-time scaling. (a) Increasing generator size improves Gen. PPL while maintaining high sample entropy ( $H$ ) on LM1B at a fixed generator training-token budget; the shared tokenizer is excluded from this panel. (b) Generating more candidates and selecting with MBR decoding improves BLEU on WMT14 and ROUGE-L on XSum. $K$ denotes the number of candidates; wall-clock time measures sampling cost. On XSum, NCM outperforms diffusion baselines at matched inference time.

Panel of Figure 8 — Figure 8: Qualitative samples from NCM. Unconditional generation samples are on LM1B. Conditional generation samples use prompts from a held-out OWT validation set. Summarization and translation samples are on XSum and WMT14 De–En, respectively. Additional samples are provided in Appendix J .

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Abbreviation	Task	Text tokenizer	Context $c$	Target $L$
OpenWebText [ 16 ]	OWT	Unconditional	GPT-2	Null prefix	1,024
One Billion Word [ 17 ]	LM1B	Unconditional	bert-base-uncased	Null prefix	128
TinyStories [ 40 ]	–	Unconditional	Custom BPE	Null prefix	128
Extreme Summarization [ 44 ]	XSum	Summarization	GPT-2	Article (512)	48
WMT 2014 German–English [ 41 ]	WMT14	Translation	Qwen3-0.6B	German (96)	64
Sudoku [ 50 ]	–	Reasoning	FMLM+ [ 49 ]	Puzzle	89

Appendix

Table 4: Datasets. Target length

L

is the number of target tokens, which equals the length

\ell_{K}

of the finest level; context lengths are in tokens. “–” indicates that we use the full dataset name rather than a separate abbreviation.

Architecture
Setting	OWT	LM1B	TinyStories	XSum	WMT14	Sudoku
Text vocab. size	50,304	30,528	8,192	50,304	151,680	12
Target length $L$	1,024	128	128	48	64	89
Levels $K$	32	12	12	8	11	12
Codebook vocab. $V_{k}$	4,096	8,192	4,096	4,096	4,096	16
Latent width $d$	384	384	384	384	384	256

Appendix

Table 5: Tokenizers. Text vocabulary size includes unused padding rows. Nested dropout is the probability of dropping the finest levels; LM1B trains on the complete and a truncated hierarchy, weighted

0.7

and

0.3

Setting	OWT	LM1B	TinyStories	XSum	WMT14	Sudoku
Architecture
Parameters	94.65M	97.07M	88.61M	88.02M	119.96M	25.46M
Transformer blocks	12	12	12	12	12	8
Hidden width	768	768	768	768	768	512
Attention heads	12	12	12	12	12	8
Prefix length $P$	1	1	1	512	96	91

Appendix

Table 6: Generators. The prefix holds the source (XSum, WMT14), the puzzle (Sudoku), or a null token. Training settings for XSum and WMT14 describe conditional training. “–” means not applicable.

Model	Size	Tokens	Mistral-7B $\downarrow$	Llama-3.1-8B $\downarrow$	Gemma-2-9B $\downarrow$	Qwen-3-14B $\downarrow$	Avg. $\downarrow$
RDLM	108M	66B	179.54	315.40	1293.67	379.00	541.90
CANDI	139M	66B	67.75	99.96	208.59	125.87	125.54
MDLM	139M	66B	63.63	91.84	186.53	115.25	114.31
DUO	139M	66B	55.08	77.24	141.16	96.59	92.52
FLM	139M	66B	68.08	89.20	150.11	110.98	104.59
NCM	97M	13.36B	59.69	75.23	106.41	91.64	83.24

Appendix

Table 7: LM1B Gen. PPL (lower is better). Baselines use 128 steps; NCM uses 12. Size: generator parameters; Tokens: training budget (including tokenizer training for NCM ); Avg.: unweighted mean.

		TinyStories			LM1B			OpenWebText
Type	Method	NFE $\downarrow$	Gen. PPL $\downarrow$	$H$	NFE $\downarrow$	Gen. PPL $\downarrow$	$H$	NFE $\downarrow$	Gen. PPL $\downarrow$	$H$
–	Data	–	9.67	3.96	–	53.51	4.31	–	14.15	5.44
AR	Sample	128	8.89	4.01	–	–	–	1024	35.45	5.58
AR	Greedy	128	5.34	4.06	–	–	–	1024	1.07	1.65
Disc.	MDLM	128	17.89	4.03	128	115.45	4.33	1024	105.15	5.63
	Duo	128	19.63	4.01	128	97.30	4.30	1024	77.69	5.55

Appendix

Table 8: Unconditional generation on TinyStories, LM1B, and OpenWebText. Gen. PPL under GPT-2-large and mean per-sample unigram entropy

H

; the Data row scores ground-truth text; “–” means no result on that dataset. Lower Gen. PPL at an entropy below that of the data, as for FLM and FMLM+ on OpenWebText, partly reflects reduced diversity.

Figure 9: Ablation on the number of levels in the hierarchy. (a) Gen. PPL against training steps. (b) Gen. PPL at the final checkpoint. (c) Cumulative reconstruction accuracy per level.

Figure 10: Ablation on the level schedule. (a) Gen. PPL against training steps. (b) Gen. PPL of the final checkpoint at

10

k steps (bars). (c) Code lengths for six level schedules.

H

is the final per-sample unigram entropy.

Attention	Gen. PPL $\downarrow$	$H$	Time (s) $\downarrow$
Block-causal	19.63	4.01	7.90
Sliding window (2 levels)	17.84	3.99	6.83
Block-diagonal (ours)	15.41	3.98	5.65

Appendix

Table 9: Attention ablation on TinyStories. Quality uses the selected corruption recipe;

H

is mean unigram entropy. Times are for 1,024 TinyStories samples at 12 steps with batch size 16 (Section 4.6 ); block-causal and sliding-window times are

1.40\times

and

1.21\times

the block-diagonal time.

Figure 11: Attention over cumulative-reconstruction blocks. Allowed query–key pairs are colored by key level. (a) Causal (reference). (b) Block-causal. (c) Two-level sliding window. (d) Block-diagonal ( NCM ): attention stays within the current block, whose input already summarizes earlier levels.

Figure 12: Loss weighting on TinyStories. Left: loss share per level. Middle: Gen. PPL during training. Right: coarse-level validation loss, labelled with Gen. PPL after

10

k steps.

Figure 13: Effect of code corruption. Gen. PPL after

10

k steps vs. mean corruption rate. Nonzero rates use a shifted linear decreasing schedule;

H

is entropy.

Figure 14: Effect of classifier-free guidance. De–En BLEU and XSum ROUGE-1/2/L as a function of the guidance weight

w

. All three ROUGE metrics peak at

w\!=\!0.5

, the setting reported in Table 2 ; BLEU, which is length-corrected, peaks later at

w\!=\!1.0

Figure 15: Effect of MBR candidate count. De–En BLEU and XSum ROUGE-1/2/L against the number of sampled candidates

K

, on a log-spaced axis. Quality improves as more candidates are sampled. NCM ’s low sampling cost allows multiple candidates within the NFE budget that diffusion baselines use for a single sample, enabling extra inference compute to be used for sampling and verification through MBR selection.

Figure 16: Reconstruction and utilization by level. Left: exact token accuracy after decoding levels

0

through

k

at the final tokenizer checkpoint. Middle: percentage of the entries of each level’s codebook selected at least once. Right: effective codebook utilization, measured by the perplexity of each level’s code distribution, as a percentage of the codebook size.

Category	Original $\to$ reconstruction , in context
Named entities	…City Metro Sydney Melbourne Melbourne Brisbane Adelaide Adelaide Perth …
Diacritics	…ev sahibi takım takäm skoru … geriye düştü. düştüü …
Numerals	…1957 1956 1957 1955 1954 1955 1953 … 1949 1948 1947 1947 1946 1946 …
Usernames	…10 maxion 10 fishingcat 10 da_lan_xiong_mao da_op_xiong_mao …
Markup	…< head > < < > title > < #BlackoutUK …
Word boundaries	…Biological and Physical SciencesJournal Sciences Journal of Chemical …

Appendix

Table 11: Tokenizer reconstruction errors on OpenWebText, by category. Struck-through grey: original; red: its reconstruction.

Panel of Figure 17 — Figure 17: Shared structure, different realizations. Each column shows one source story and two samples generated with its coarse codes fixed. Text is shown verbatim to the 128-token limit, including unfinished endings. Yellow highlights added or replaced words relative to the corresponding source; deletions are not marked.

Panel of Figure 18 — Figure 18: Mixing coarse representations. Each column shows two sources and one output with newly generated finer levels. Green (source A) and yellow (source B) highlight selected shared wording and details. All text is verbatim, including unfinished endings.

Panel of Figure 19 — Figure 19: Replacing two early codes. Each example shows the original story, its level-1 code replacements, and paired clean and perturbed outputs with the same seed. Yellow marks word-level differences on both sides; punctuation-only changes are unmarked. Code positions index eight equal regions of the 128-token grid. Text is verbatim, including unfinished endings.

Panel of Table 12 — Table 12: German–English translation trajectory. Each row decodes the cumulative reconstruction after one level of a single NCM generation; text is shown as generated, and BLEU is computed per level against the reference. The final level (shaded) is the output.

Text Generation

Text generation studies methods for producing coherent, useful language from prompts, data, and dialogue context.

See all 423 papers

DiffusionGemma Technical Report
Jul 31, 2026DiffusionGemma Team, Adrien Ali Taïga, James Assiene +41Diffusion Language Model Decoding Blockwise Diffusion Language Models
CoBit: Language Modeling with Bitstream Diffusion
May 7, 2026Georgios Batzolis, Mark Girolami, Luca AmbrogioniDiffusion Language Model Decoding Diffusion Language Models
Department of Engineering, University of Cambridge. · Devotion AI Labs Ltd. · Donders Institute for Brain, Cognition, and Behaviour, Radboud University.
ALoDLM: Adaptively Looped Diffusion Language Models
Oct 3, 2026Liancheng Fang, Zhuowei Li, Youngeun Kim +10Adaptive Computation for Diffusion Models Diffusion Language Models
University of Illinois Chicago · Amazon AGI · Korea University

Jul 31, 2026J/K move · Enter open · S save

DiffusionGemma Technical Report

DiffusionGemma Team, Adrien Ali Taïga, James Assiene, Daniele Calandriello+40

Diffusion Language Model Decoding Blockwise Diffusion Language Models Diffusion Model Fine-Tuning Diffusion Language Models Diffusion Model Inference Acceleration Diffusion Language Model Training+1

Open paper AskPDF arXiv

We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.

Abstract

Text Generation

Explore similar work

DiffusionGemma Technical Report

CoBit: Language Modeling with Bitstream Diffusion

ALoDLM: Adaptively Looped Diffusion Language Models