Autoregressive Language Modeling
Overview
Autoregressive language models learn to predict each next token from the preceding context, a simple objective that supports both generation and tasks such as speech or protein modeling. Transformer attention remains a common approach, while recurrent and state-space models, sparse memory, and alternative tokenizations aim to reduce the cost of long contexts and large vocabularies. Current work also explores faster generation through multi-token prediction, latent lookahead, and combinations of causal and diffusion methods, evaluated on measures such as perplexity, retrieval, and downstream reasoning benchmarks. A central tension is how to gain speed and long-range capability without losing the quality, reliability, and generalization associated with left-to-right prediction.
Latest papers
132 papers15 papers in the last four weeks, up 25% on the four weeks before. 0.1% of all new papers.