Decoder-Only Language Models

Follow this topic Ask about this topic

Decoder-only language models, including GPT-style Transformers, predict each next token from the preceding context and now support text, code, speech, and other modalities. Research explores ways to improve their architecture and training, from recurrent or memory-augmented layers and alternative attention mechanisms to parameter-efficient fine-tuning, compression, and speculative decoding. Models are assessed on language-modeling benchmarks such as WikiText and C4, as well as reasoning, retrieval, classification, and multilingual tasks. A central tension is that changes that improve generation quality or inference efficiency do not always improve generalization or downstream accuracy, and causal models can lag bidirectional encoders on tasks such as fine-grained classification.

Latest papers

98 papers8 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Efficient Continual Learning in Language Models via Thalamically Routed Cortical Columns
Feb 25, 2026Afshin KhadangiContinual Learning for LLMs Decoder-Only Language Models
SnT, University of Luxembourg
Decoder-based Sense Knowledge Distillation
Feb 25, 2026Qitong Wang, Mohammed J. Zaki, Georgios Kollias +1Language Model Distillation Knowledge Distillation
Rensselaer Polytechnic Institute Troy, New York, USA · IBM Research Yorktown Heights, New York, USA
GATech at AbjadMed: Bidirectional Encoders vs. Causal Decoders: Insights from 82-Class Arabic Medical Classification
Feb 17, 2026Ahmed Khaled KhamisArabic NLP Clinical NLP
Georgia Institute of Technology
Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers
Feb 16, 2026Jonathan Lys, Vincent Gripon, Bastien Pasdeloup +4Transformer Residual Learning
IMT Atlantique, Lab-STICC, UMR CNRS 6285, Brest, France · Sony Europe Ltd. Stuttgart Technology Center, EUREC, Germany
On the Runway Cascade of Transformers for Language Modeling
Jan 20, 2026Hunjae Lee, Corey ClarkTransformer Attention Decoder-Only Language Models
Department of Computer Science, Southern Methodist University, Dallas TX USA.
Understanding the Staged Dynamics of Transformers in Learning Latent Structure
Nov 24, 2025Rohan Saha, Farzane Aminmansour, Alona FysheTransformer Training Dynamics Decoder-Only Language Models
Department of Computing Science, University of Alberta, Edmonton, Canada · Department of Computing Science and Psychology, University of Alberta, Edmonton, Canada
UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement
Oct 23, 2025Haoyin Yan, Chengwei Liu, Shaofei Xue +4Target Speaker Extraction Decoder-Only Language Models
Qwen Business Unit of Alibaba, China · Tongyi AI Lab of Alibaba, China
Any-Order GPT as Masked Diffusion Model: Decoupling Formulation and Architecture
Jun 24, 2025Shuchen Xue, Tianyu Xie, Tianyang Hu +5Masked Diffusion Models Decoder-Only Language Models
University of Chinese Academy of Sciences · Academy of Mathematics and Systems Science, Chinese Academy of Sciences · School of Mathematical Sciences, Peking University +2

Feb 25, 2026J/K move · Enter open · S save

Efficient Continual Learning in Language Models via Thalamically Routed Cortical Columns

Afshin Khadangi

SnT, University of Luxembourg

Continual Learning for LLMs Decoder-Only Language Models Memory-Augmented Language Models Replay-Based Continual Learning Modular Neural Networks

Open paper AskPDF arXiv

Large language models deployed in the wild must adapt to evolving data, user behavior, and task mixtures without erasing previously acquired capabilities. In practice, this remains difficult: sequential updates induce catastrophic forgetting, while many stabilization methods rely on external procedures that are costly, brittle, or difficult to scale. We present TRC

^{2}

(Thalamically Routed Cortical Columns), a decoder-only architecture that makes continual learning a property of the backbone itself. TRC

^{2}

combines stacked cortical columns with a thalamic modulatory pathway for selective inter-column communication and a hippocampal pathway for event selective retrieval, delayed surprise-based writing, and replay-driven consolidation. This design localizes fast plasticity while preserving a slower stable computation pathway. We further introduce a causal memory-update scheme and an online replay controller that adjusts consolidation strength from measured forgetting. Across a task-sequential language-modeling stream over C4, WikiText-103, and GSM8K, TRC

^{2}

consistently improves task-boundary modeling quality and substantially reduces cumulative forgetting relative to Transformer, Mamba, MoE, DeepSeek and continual learning baselines trained under the same pipeline. Ablations show that the thalamic and hippocampal components are central to the retention gains, while the full model remains competitive in throughput and training cost.