Loopholing All the Way: Language Modeling with pure Self-Conditioning

Adrien Deschenaux; Nadezhda Ilieva; Justin Deschenaux; Manuel Madeira; Dorina Thanou; Aurélien Lucchi; Antonio Orvieto; Caglar Gulcehre

cs.LGOct 21, 2026

Loopholing All the Way: Language Modeling with pure Self-Conditioning

Authors: Adrien Deschenaux, Nadezhda Ilieva, Justin Deschenaux, Manuel Madeira, Dorina Thanou, Aurélien Lucchi, Antonio Orvieto, Caglar Gulcehre,

Organizations: EPFL, Lausanne, Switzerland · University of Basel, Switzerland · ELLIS Institute Tübingen, MPI for Intelligent Systems, Tübingen AI Center, Germany · Microsoft AI

Abstract

Recent Diffusion Language Models (DLMs) use Self-Conditioning (SC), or Loopholing, to improve their performance. While SC is traditionally seen as an auxiliary heuristic, its empirical success raises the question: can a continuous generative model be based purely on Self-Conditioning? We answer affirmatively with LAW (``Loopholing All the Way''), which propagates the Expected Embedding under the denoiser's prediction across steps, decoding the tokens once, at the end. On Sudoku, LAW outperforms previous Discrete and Continuous Diffusion Models, including Loopholing Discrete Diffusion. On Maze, LAW is the most accurate from 2 to 64 NFE. After training on TinyGSM and sampling at temperature 1 with 1024 NFE, LAW solves 42.2% of GSM8K problems against 34.5% for uniform Loopholing Discrete Diffusion. Additionally, fine-tuning a pre-trained Uniform Diffusion Model (Duo) with LAW increases its accuracy from 17.2% to 48.1%. Finally, on OpenWebText, LAW pushes the Pareto frontier and approaches Autoregressive Models (ARMs) with 64 NFE. Find our code at https://github.com/ilievanadezhda/law.

Figures & tables

Figure 1: GSM8K accuracy against the Number of Function Evaluations (NFE) at

T=0.1

(left) and

T=1

(right). LAW (Sec. ) trained from scratch with the training FLOPs of the baselines (solid), and fine-tuned from the Duo baseline for 50k steps (dashed). The dashed gray lines are an Autoregressive Model (ARM), sampled at

T=1

and with greedy decoding. Trained from scratch, LAW is the most accurate model from 32 NFE at

T=1

, and from 4 to 512 NFE at

T=0.1

. Fine-tuned from Duo, it comes closer to the ARM.

Figure 2: Overview. Sampling goes from left to right, applying the backbone

f_{\theta}

repeatedly. Boxed words are discrete tokens, colored cells SC states and white cells zero-valued states. Top: Loopholing (LDDM; Jo et al., 2026 ). At each step, discrete tokens are updated and forwarded alongside the SC state ( ). Bottom: LAW. At each step, we only propagate the SC state, computed by the state map

\Phi_{\theta}

( ). The initial tokens are drawn from the uniform prior, and we decode back to discrete tokens only at the end ( ).

Autoregressive
Model	Easy	Med.	Hard
Sample	13.9	5.1	0.6
Greedy	14.6	5.1	1.0
Discrete Diffusion
MDLM ( Sahoo et al., 2024 )	92.0	77.1	30.2
Duo ( Sahoo et al., 2025 )	96.3	84.7	58.4

Table 1: Exact-match accuracy (%) on Sudoku with 180 NFE. Baseline numbers are taken from prior work ( Deschenaux and Gulcehre, 2026 ) , except LDDMs, which we train with matched training FLOPs.

p

is the multi-pass probability, and reg. is temperature regularization with range

\rho

( ). Best per column in bold , second best underlined .

Figure 3: Maze valid-solution accuracy against NFE. Fraction (%) of the 1k evaluation mazes for which a valid solution is generated (Suppl. ). All methods are trained with matched training FLOPs (Suppl. ). LAW uses

M=2

p=0.75

, the NELBO, batch size 256 and 80k steps, with (

\rho=2

, solid) and without (dashed) temperature regularization. The dashed gray line is the ARM with greedy decoding (9.4%). Left: LAW against the DDM baselines. Right: LAW against the continuous baselines.

Figure 4: Ablations on GSM8K: how LAW uses the SC state (left,

T=0.1

) and fine-tuning from Duo (right,

T=1

). Left: the legend gives the SC state and the temperature regularization (

\rho=3

or none). Right: LAW fine-tuned from Duo for 15k and 50k steps, with

M=2

p=0.9

\rho=3

and batch size 512, against LAW trained from scratch.

Figure 5: Gen. PPL against unigram entropy on OpenWebText with 64 (left) and 1024 NFE (right). Each point is a sampling temperature

T\in\{0.70,0.75,\dots,1.20\}

. We sample the ARM with 1024 NFE, but show it in both panels for easier comparison with LAW. With 64 NFE, LAW dominates all diffusion and flow baselines and comes close to the ARM. With 1024 NFE, it dominates the ARM up to an entropy of 5.1–5.2.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Batch	Baseline steps	LDDM / LAW steps
Sudoku	256	20k	15.4k
TinyGSM	512	250k	192k
Maze	256	100k	76.9k
OpenWebText	512	400k	308k

Appendix

Table 2: Training budget per task. The baselines run standard updates (

3F

each). By default, LDDMs and LAW run

M=2

passes with

p=0.9

(

3.9F

per update) and train for fewer steps to match the training FLOPs of the baselines ( ).

Method	Difficulty	SC state	Loss	$M$	$p$	Stop-grad.	$\rho$	Mode emb.	Batch	Cost	Steps
Loss
LAW	all	E.E.	CE	2	0.9	yes	none	no	256	$3.9F$	15,385
LAW	all	E.E.	NELBO	2	0.9	yes	none	no	256	$3.9F$	15,385
Multi-pass probability (with the $p=0.9$ runs above)
LAW	all	E.E.	CE	2	0.75	yes	none	no	256	$3.75F$	16,000
LAW	hard	E.E.	NELBO	2	0.75	yes	none	no	256	$3.75F$	16,000

Appendix

Table 3: Training steps of the LAW runs on Sudoku, grouped by experiment. All runs are trained from scratch and match the training FLOPs of 20k baseline steps at batch size 256. Difficulty all stands for easy, medium, and hard. Runs marked

\star

appear in Table .

Method	SC state	Loss	$M$	$p$	Stop-grad.	$\rho$	Mode emb.	Batch	Cost	Steps
LDDMs (Fig. )
LDDM-U	Hid.	NELBO	2	0.9	yes	none	no	512	$3.9F$	192k
LDDM-M	Hid.	NELBO	2	0.9	yes	none	no	512	$3.9F$	192k
SC state and loss (Fig. )
LAW	E.E.	NELBO	2	0.9	yes	none	no	512	$3.9F$	192k
LAW	E.E.	CE	2	0.9	yes	none	no	512	$3.9F$	192k

Appendix

Table 4: Training steps of the LDDM and LAW runs on TinyGSM, grouped by the figures that show them. Runs trained from scratch match the training FLOPs of 250k baseline steps at batch size 512. Each fine-tuning configuration starts from the Duo baseline and is trained at batch sizes 64, 128, 256 and 512. The cost per update is given at batch size 512. The run marked

\star

is LAW from scratch in Figure , Figure and Table , and

\dagger

marks LAW without regularization. Outside Figure , the fine-tuned LAW is the run with

M=2

at batch size 512.

Method	SC state	Loss	$M$	$p$	Stop-grad.	$\rho$	Mode emb.	Batch	Cost	Steps
Loss
LAW	E.E.	CE	2	0.9	yes	none	no	256	$3.9F$	76,923
LAW	E.E.	NELBO	2	0.9	yes	none	no	256	$3.9F$	76,923
Multi-pass probability (with the $p=0.9$ runs above)
LAW	E.E.	NELBO	2	0.75	yes	none	no	256	$3.75F$	80,000
LAW	E.E.	CE	2	0.75	yes	none	no	256	$3.75F$	80,000

Appendix

Table 5: Training steps of the LAW runs on Maze-Hard, grouped by experiment. All runs are trained from scratch and match the training FLOPs of 100k baseline steps at batch size 256. The run marked

\star

is LAW in Figure , and

\dagger

marks the same run without regularization.

Method	SC state	Loss	$M$	$p$	Stop-grad.	$\rho$	Mode emb.	Batch	Cost	Steps
LDDMs (Figure and Figure )
LDDM-M	Hid.	NELBO	2	0.9	yes	none	no	512	$3.9F$	308k
LDDM-U	Hid.	NELBO	2	0.9	yes	none	no	512	$3.9F$	308k
LDDM-U	E.E.	NELBO	2	0.9	yes	none	no	512	$3.9F$	308k
LDDM-U	E.E.	NELBO	2	0.9	yes	3	no	512	$3.9F$	308k
LAW (Figure and Figure )

Appendix

Table 6: Training steps of the LDDM and LAW runs on OpenWebText. All runs are trained from scratch and match the training FLOPs of 400k baseline steps at batch size 512. The run marked

\star

is LAW in Figure .

Training	CE	NELBO
$p=0.9$	92.2	85.3
$p=0.75$	94.5	88.9
$p=0.5$	93.4	84.3
$p=0.25$	90.1	84.9
$p=0.1$	84.6	79.3

Appendix

Table 7: Multi-pass probability on hard Sudoku. LAW with

M=2

passes run with probability

p

, without regularization. Best per column in bold .

Training	CE	NELBO
$M=2$ , $p=0.9$	92.2	85.3
$M=3$ , $p=0.9$	94.5	75.4
$M=2$ , $p=0.75$	94.5	88.9
$M=3$ , $p=0.75$	94.2	79.6

Appendix

Table 8: Number of passes on hard Sudoku. LAW with

M

passes run with probability

p

, without regularization. Best per column in bold .

Training	none	$\rho=2$	$\rho=3$	$\rho=4$	$\rho=5$
CE, $M=2$ , $p=0.9$	92.2	96.5	94.3	95.2	94.5
NELBO, $M=2$ , $p=0.9$	85.3	84.5	85.2	87.1	86.7
CE, $M=2$ , $p=0.75$	94.5	95.4	96.2	94.6	94.5
NELBO, $M=2$ , $p=0.75$	88.9	88.5	87.8	86.0	85.9
CE, $M=3$ , $p=0.9$	94.5	94.5	94.9	95.1	93.3
NELBO, $M=3$ , $p=0.75$	79.6	86.5	83.1	83.5	81.1

Appendix

Table 9: Temperature regularization on hard Sudoku. Each row is one training configuration, without regularization (none) and with range

\rho

( ). Best per row in bold .

Figure 6: SC state and loss. LAW with the Expected Embedding or the hidden representation as SC state, trained with the NELBO or the CE, without regularization.

Figure 7: Temperature regularization and mode embedding. LAW with temperature regularization (

\rho\in\{3,4\}

), the mode embedding (Suppl. ), both, or neither.

Figure 8: Number of passes and multi-pass probability. LAW with

M\in\{2,3,4\}

passes, run with probability

p\in\{0.75,0.9\}

, and with

\rho=3

. Runs with more passes or a higher

p

train for fewer steps, to match the training FLOPs.

Figure 9: Gradient flow between passes. LAW with

M=2

and

\rho=3

, with the stop-gradient between passes (

p=0.9

) and without it (

p\in\{0.9,1\}

). Without the stop-gradient, each update is more expensive, so these runs train for 132k (

p=0.9

) and 125k (

p=1

) steps, against 192k with the stop-gradient.

Figure 10: Gradient flow between passes, over training. GSM8K accuracy with 256 NFE at

T=1

over training steps, for the runs of Figure . After 120k steps, the accuracy is 38.7% with the stop-gradient, against 20.9% (

p=0.9

) and 13.7% (

p=1

) without it. With the stop-gradient, the accuracy plateaus from about 70k steps. Without it, the run with

p=0.9

does not improve after 100k steps, and the run with

p=1

improves slowly.

Figure 11: LAW against LDDM-U with the Expected Embedding. The legend gives the SC state and the temperature regularization (

\rho=3

or none).

Figure 12: Fine-tuning from Duo. GSM8K accuracy over fine-tuning steps with 256 NFE at

T=1

, for

M=2

p=0.9

(solid) and

M=4

p=0.75

(dashed), both with

\rho=3

, at batch sizes 64 to 512. Step 0 is the Duo checkpoint, sampled with the sampler of LAW. The dashed gray line is the ARM sampled at

T=1

(53.9%, 512 NFE).

Figure 13: Early stopping. LAW fine-tuned from Duo with

M=2

p=0.9

\rho=3

and batch size 512, after 15k and 50k fine-tuning steps, against LAW trained from scratch (Figure ). The dashed gray line is the ARM, as in Figure .

Discrete Diffusion
Method	Decoding	1	2	4	8	16	32	64
MDLM (ancestral)	$T=0.1$	–	–	1.7	5.3	13.0	22.1	28.2
MDLM (confidence)	greedy	0.8	0.7	4.2	20.1	37.7	43.9	40.6
Duo (ancestral)	$T=0.1$	–	–	5.2	14.0	23.0	30.8	31.2
Discrete Diffusion, distilled
IDLM (MDLM)	$T=1$	–	–	0.4	1.8	6.5	12.8	14.9

Appendix

Table 10: GSM8K accuracy (%) with 1 to 64 NFE on TinyGSM. MDLM, Duo and IDLM are taken from Gushchin et al. (2026) , who re-evaluate the checkpoints of Deschenaux and Gulcehre (2026) and Li et al. (2026) (IDLM at 32 and 64 NFE is taken from Li et al. (2026) ). FMLM+ is taken from Agarwal et al. (2026) , DBTM from Tang and Wang (2026) , AlphaDLM from Gushchin et al. (2026) , and SDM from Deschenaux et al. (2026a) , whose sampler uses the stochasticity

\kappa=1

. SDM (distilled) starts from SDM (Expected Embedding), with a configuration selected on the GSM8K test set. FMLM+ (init. from MDLM) starts from an MDLM checkpoint, IDLM is distilled from MDLM or Duo, and the fine-tuned LAW starts from the Duo baseline. Greedy denotes argmax decoding, and for FMLM+ and DBTM, an NFE is one refinement round. MDLM (confidence) at 1 and 2 NFE is computed from the counts reported by Gushchin et al. (2026) . – marks values that are not reported. Best per column in bold .

Figure 14: A valid solution that fails exact match . On this Maze-Hard evaluation maze, LAW generates a shortest path that differs from the target. Left: the maze. Middle: the path generated by LAW. Right: cells on both paths (pink), only on the LAW path (vermilion), and only on the target path (blue). Both paths take 70 moves, so the generated path solves the maze, but exact match counts it as wrong.

Figure 15: Maze-Hard exact-match against valid-solution accuracy. Valid-solution (solid) and exact-match (dotted, hollow markers) accuracy against NFE of the baselines and of LAW with

\rho=2

(Figure ). Left: DDM baselines. Right: continuous baselines.

Figure 16: Multi-pass probability on Maze-Hard. Valid-solution accuracy against NFE at

T=1

of LAW with

M=2

passes run with probability

p

, trained with the CE (left) or the NELBO (right).

Figure 17: Number of passes on Maze-Hard. Valid-solution accuracy against NFE at

T=1

of LAW with

M=2

M=3

passes, run with probability

p=0.9

(left) or

p=0.75

(right), and trained with the CE (solid) or the NELBO (dashed).

Figure 18: Temperature regularization on Maze-Hard. Valid-solution accuracy against NFE at

T=1

. Each panel is one training configuration (x-axis label), without regularization (dashed black) and with

\rho\in\{2,3,4,5\}

Figure 19: Loss. Unigram entropy (left) and Gen. PPL (right) over training steps of LAW trained from scratch on OpenWebText with the NELBO or the CE ( ), without temperature regularization, and without the mode embedding.

Figure 20: Temperature regularization and mode embedding. Unigram entropy (top left), Gen. PPL (top right), hidden-state correlation (bottom left), and validation loss (bottom right) over training steps of LAW trained with the CE. No reg. is the CE run of Figure . The other runs add temperature regularization with

\rho\in\{3,4\}

( ), then the mode embedding (Suppl. ) on top of

\rho=3

Figure 21: LAW against LDDMs on OpenWebText. Gen. PPL against unigram entropy with 64 (left) and 1024 NFE (right), over the sampling temperatures of Figure . The legend gives the SC state and the regularization (Suppl. compares how LDDMs and LAW use the SC state). The highlighted run is the LAW model of Figure .

Panel of Figure 22 — Figure 22: Sample with 64 NFE. Gen. PPL 18.31, unigram entropy 5.60.

Panel of Figure 23 — Figure 23: Sample with 1024 NFE. Gen. PPL 22.09, unigram entropy 5.69.

Diffusion Language Models

Diffusion language models generate and refine text through iterative denoising, enabling bidirectional context and parallel token updates.

See all 466 papers

Oct 21, 2026

Looped Uniform Diffusion Language Models

Uniform diffusion language models (UDLMs) generate text by iteratively denoising sequences of ordinary vocabulary tokens, any of which may be corrupted and subsequently revised. During training, UDLMs must denoise from a context that may itself be unreliable, whereas masked diffusion language models (MDLMs) always condition on a clean context. This challenging training objective makes UDLMs more FLOP-intensive to train than MDLMs, but it yields stronger downstream reasoning performance. Recently, selective looping, which recursively applies a few early Transformer layers within each denoising step, has been shown to improve training efficiency in MDLMs under an iso-FLOP budget. We investigate looping in UDLMs while keeping the training objective and sampler unchanged. We find that looping significantly improves training efficiency: looped UDLMs reach the validation NLL of a non-looped UDLM trained on about 0.5 trillion tokens with up to 2.8x fewer training FLOPs on OpenWebText and 3.3x fewer on FineWeb-Edu. We also find that, at a matched NFE budget, looped UDLMs reach up to 9.8% lower generation perplexity than the non-looped UDLM, with most gains achieved at moderate loop counts. Finally, we study different loop-count distributions and find that sampling smaller loop counts more frequently during training improves inference efficiency: on downstream tasks, this strategy uses 12% fewer loops than uniform loop-count sampling without degrading accuracy.

Saiteja Utpala, Chieh-Hsin Lai, Yuki Mitsufuji +1

MBZUAI · Sony Group Corporation · MIT

Diffusion Language Models Diffusion Language Model Training Diffusion Language Model Inference

Ask PDF

Oct 9, 2026

On the Design Space of Continuous Diffusion Language Models

Continuous diffusion language models (CDLMs) promise to bring the benefits of continuous flow modelling to language, including accelerated sampling and controllable generation. The field has grown rapidly, and the capabilities of CDLMs have begun to rival those of discrete diffusion. Yet, unlike for continuous modalities such as images or videos, our grasp of the levers that matter when training CDLMs remains limited. In this work, we focus on two of these levers, namely self-conditioning and embedding geometry, and shed light on some of the mechanisms behind them, which in turn suggest cheaper and more effective designs. We begin by showing that self-conditioning corrects a train/inference mismatch by exposing the model to inference-like states that are otherwise rare during training. These gains can be recovered without an extra forward pass by a two-level interpolant, coined

\mathrm{DoubleCat}

, which improves TinyGSM accuracy more than threefold. Self-conditioning also aids in recognising tokens dominated by large-norm Gaussians

\unicode{x2013}

an issue known as token identifiability. To address it directly, we introduce discrete reveals to

\mathrm{DoubleCat}

, yielding

\mathrm{SemiCat}

, which achieves discrete diffusion-level performance on TinyGSM, reaching

49.94\%

with standard training hyperparameters and a tuned sampler. At the

1.7

B scale,

\mathrm{SemiCat}

outperforms discrete diffusion on four of six likelihood-based evaluations, and surpasses one-hot flows on unconditional generation. On embedded spaces, we find that cross-entropy drives embeddings towards a geometry whose decoding error rate closely matches the one-hot one. We briefly explore a range of further options, such as hybrid corruption processes and noise schedules, including asynchronous ones. We believe this exploration can help the community better understand and advance the design space of CDLMs.

Oscar Davis, Daan Roos, Alice Bizeul +5

Apple · University of Amsterdam · University of Oxford

Self-Conditioning Continuous Diffusion Language Models Representation Geometry in Language Models

Ask PDF

Apr 13, 2026

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow

Yuxin Chen, Chumeng Liang, Hangke Sui +4

UIUC

Self-Conditioning Diffusion Language Models Continuous Diffusion Language Models Diffusion Language Model Training Flow Matching

Ask PDF

Abstract

Diffusion Language Models

Explore similar work

Looped Uniform Diffusion Language Models

On the Design Space of Continuous Diffusion Language Models

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling