cs.LGNov 12, 2026

On the Difficulty of Sequence-Level Supervised Learning for Autoregressive Language Models

Authors: Mathieu Blondel, Michaël E. Sander, Germain Vivier-Ardisson, Tianlin Liu, Vincent Roulet

Organizations: Google DeepMind

Abstract

During the supervised learning phase, autoregressive language models are typically trained by minimizing the negative log-likelihood of the training data, which decomposes as a sum of token-level cross-entropies via teacher forcing. This objective is simple and efficient but potentially suffers from so-called exposure bias: a mismatch between the way the model is trained and the way it is used at inference time. Can we instead design sequence-level loss functions that incorporate inference-time decoding procedures, such as greedy decoding or autoregressive sampling? We show that two natural attempts lead to loss functions that are fundamentally flawed: unbounded below. We further show that these two flawed loss functions are intimately connected: using autoregressive Gumbel tricks, the sampling based loss is the expected greedy loss under Gumbel perturbations. Since the pathology stems from the dependence of the decoded sequence on the model parameters, we analyze the update direction obtained by applying a stop gradient on the sampling procedure, leading to a moment matching direction. While it is not the gradient of any loss function, we prove that it defines a valid descent direction in function space for the autoregressive negative log-likelihood. Despite these theoretical insights, the moment matching direction suffers from high variance. We conclude that teacher forcing may well be ``all you need'' for supervised learning of autoregressive models.

Explore similar work

CardsList
  1. Sample Complexity of Autoregressive Reasoning: Chain-of-Thought vs. End-to-End

    Apr 13, 2026Steve Hanneke, Idan Mehalel, Shay MoranCoT SupervisionSample Complexity

  2. Hidden Biases in Conditioning Autoregressive Models

    Apr 9, 2026Francois Pachet, Pierre RoyConstrained Generative ModelingConstrained Decoding