cs.LGNov 14, 2026

Empowering Masked Diffusion Models to Self-Correct with Leave-One-Out Transformers

Authors: Sofian Zalouk, Vincent Counathe, Paul Jünger, Daniel Cao, Daniel Zlotnick, Kilian Q. Weinberger, Christopher De Sa

Organizations: Department of Computer Science, Cornell University

Abstract

Masked diffusion models (MDMs) have emerged as a promising paradigm for language modeling, enabling parallel token generation with competitive performance. Yet standard MDMs suffer from two fundamental limitations: (1) inability to revise tokens, and (2) sparse training signal. We revisit hollow transformers, which address both limitations by predicting each token from its surrounding context, independently of the input at that position. We introduce the Leave-One-Out Transformer (LOOT), a more efficient and expressive architecture that advances hollow transformers through two complementary advances. First, we redesign the merging mechanism to enable frequent interaction between left and right contexts, substantially improving predictive performance over prior hollow transformers. Second, we derive a lower-variance estimator of the MDM NELBO that supervises all tokens and improves validation perplexity. Together, these advances yield a competitive standalone diffusion language model capable of effective self-correction during inference. On LM1B, LOOT matches the strongest dense-supervision MDM baseline in validation perplexity. On OpenWebText, LOOT achieves higher MAUVE and better generation quality-diversity tradeoffs than prior remasking methods, using 8×8\times fewer forward evaluations and 3.5×3.5\times fewer inference FLOPs.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Looped Diffusion Language Models

    May 25, 2026Sanghyun Lee, Chunsan Hong, Seungryong Kim +3Masked Diffusion ModelsDiffusion Language Model Inference

  2. Remask, Don't Replace: Token-to-Mask Refinement in Diffusion Large Language Models

    Apr 20, 2026Lin YaoLLM Self-CorrectionMulti-Token Prediction