Decoder-Only Language Models

Follow this topic Ask about this topic

Decoder-only language models, including GPT-style Transformers, predict each next token from the preceding context and now support text, code, speech, and other modalities. Research explores ways to improve their architecture and training, from recurrent or memory-augmented layers and alternative attention mechanisms to parameter-efficient fine-tuning, compression, and speculative decoding. Models are assessed on language-modeling benchmarks such as WikiText and C4, as well as reasoning, retrieval, classification, and multilingual tasks. A central tension is that changes that improve generation quality or inference efficiency do not always improve generalization or downstream accuracy, and causal models can lag bidirectional encoders on tasks such as fine-grained classification.

Latest papers

98 papers8 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Stepwise Reasoning Verification: Don't Generate, Classify!
Oct 18, 2026Yifei Shi, Michalis Korakakis, Rami Aly +3Reasoning Verification Language Model Reasoning Evaluation
University of Cambridge · Alan Turing Institute · University of Edinburgh
Graph Metanetworks for Transformer Based Language Models
Oct 13, 2026Yonathan Wolanowsky, Guy Bar-Shalom, Fabrizio Frasca +1Weight-Space Learning Transformer
Technion – Israel Institute of Technology · NVIDIA Research
Project Greenhouse: Progress Toward Fully Open and Sovereign Agentic Search
Oct 8, 2026Jimmy Lin, Sahel Sharifymoghaddam, Lingwei Gu +1LLM Reranking Learning to Rank
David R. Cheriton School of Computer Science University of Waterloo
Do Language Models Need a Trainable Input Embedding Table? Fixed Minimal Token Codes at 1.7B-Class Scale
Oct 2, 2026A. BochkovLLM Training Word Embeddings
Sequential Functional Structured Tucker Compression for Large Language Model Attentions
Sep 30, 2026Jiangfeng Chen, Xinyu Wang, Tianshuo Yan +4Tucker Decomposition LLM Compression
University of Manitoba · McGill University · Simpleway +2
Is Weight Tying Still Beneficial for Decoder-Only LLMs in Private Settings Under DP-SGD?
Sep 30, 2026Razan El Mais, Ali Chehab, Ibrahim Issa +1Privacy-Preserving Language Models Differentially Private Stochastic Gradient Descent
Department of Electrical and Computer Engineering American University of Beirut Beirut, Lebanon
Predictive Geometry of Hidden Trajectories in Transformers
Sep 29, 2026Timur Mudarisov, Mikhail Burtsev, Tatiana Petrova +1Decoder-Only Language Models Representation Geometry in Language Models
University of Luxembourg, Luxembourg
SEED: Self-Speculative Decoding via Implicit Encoder-Decoder
Sep 29, 2026Hankun Lin, Patrick Pynadath, Ruqi ZhangSpeculative Decoding LLM Inference Acceleration
Department of Computer Science, Purdue University, USA
The Residual Stream's Effective Depth
Sep 25, 2026Barak Gahtan, Ido Galil, Alex M. BronsteinTransformer Interpretability Decoder-Only Language Models
Technion Israel Institute of Technology · Nvidia · Technion Israel Institute of Technology & ISTA Institute of Science and Technology Austria
Your Transformer Can Hold Two Thoughts at Once: Evidence of Linear Superposition in LLMs
Sep 24, 2026Pavel Tikhonov, Anton Korznikov, Matvey Mikhalchuk +6Language Model Decoding Decoder-Only Language Models
Repurposing Pre-trained LLMs as High Fidelity Continuous Text Autoencoders
Sep 23, 2026Arkanath Pathak, Unnat Jain, Alexander C. BergDecoder-Only Language Models Autoencoders
University of California, Irvine
Relational Attention for Data-Efficient Language Modeling
Sep 17, 2026Adrian Brasoveanu, Ece Takmaz, Jakub DotlačilLanguage Model Pretraining Decoder-Only Language Models
UC Santa Cruz · Utrecht University
Persistent Recurrent Memory Between Transformer Layers - Improves Language Model Generalization
Sep 15, 2026Eduardo Novaes HeringMemory-Augmented Language Models Decoder-Only Language Models
FITec Labs / Ericsson São Paulo
Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
Sep 11, 2026Kalyani Marathe, Artidoro Pagnoni, Tomasz Limisiewicz +4Decoder-Only Language Models Language Model Scaling Laws
University of Washington, Seattle · Work done at Meta FAIR · Meta FAIR
K/V-Cache Interventions Dissociate Representation Alignment from Persona Expression in Decoder-Only Language Models
Sep 10, 2026Yu Sun, Mengyin Lu, Cong Feng +2KV Caching Decoder-Only Language Models
LinkedIn
How Perturbations Propagate: A Multi-Level Analysis of Robustness in Large Language Models
Sep 3, 2026Dun Li Chan, Emily Liu, Niyathi Allu +1Decoder-Only Language Models Language Model Robustness
INTI International College Penang · Independent Researcher · FPT University
Generative vs. Encoder Models for Multilingual NER: A Comprehensive Empirical Study on Naamapadam
Aug 30, 2026Jakkala Mahesh, Jatavath Shravan Kumar, Komalla Shivani +1Named Entity Recognition Multilingual Language Model Evaluation
Rajiv Gandhi University of Knowledge Technologies, Basar, Telangana, India
DARTS: Decoder-Aware Representation Tuning via Surgery for Model Merging
Aug 28, 2026Aaryan Ajay Sharma, Sai Nishanth Padala, Seganrasan SubramanianLanguage Model Merging Decoder-Only Language Models
ServiceNow · University of Twente
An Exploratory Ablation of a Small MLA--SSM Hybrid Language Model
Aug 28, 2026Christos KoutsiarisHybrid Language Models Decoder-Only Language Models
Development Expert, SAP P&E, Cloud ERP, UX Foundation
Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study
Aug 26, 2026Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil +6ASR Evaluation Decoder-Only Language Models
Idiap Research Institute, Martigny, Switzerland · LIA, Avignon Universit´e, Avignon, France · LIUM, Le Mans Universit´e, Le Mans, France +1
XTC: Head-Aware Sampling by Excluding Top Choices
Aug 24, 2026Philipp Emanuel Weidmann, Allen Roush, Judah Goldfeder +2Language Model Decoding Diversity-Aware Sampling
Independent Researcher · Thoughtworks · New York University
Ask Self, Ask Others: Relation Is All You Need
Aug 20, 2026Yuting Ge, Pengju Yang, Mingkai NieSelf-Attention Decoder-Only Language Models
Department of Electrical Engineering City University of Hong Kong · Jilin University · National University of Singapore
Left-Branching Transformers Excel at Right-Branching Languages: Data Shapes Word Order Preferences in Language Models
Aug 15, 2026Varvara Arzt, Allan Hanbury, Terra BlevinsMultilingual Language Models Decoder-Only Language Models
Faculty of Informatics, TU Wien · Khoury College of Computer Sciences, Northeastern University
Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation
Aug 15, 2026Amr Hegazy, Amr Alanwar, Mostafa ElhoushiRecurrent Transformers Transformer Expressivity
The German University in Cairo · Technical University of Munich · Cerebras Systems Inc.
LoKiFormer: Locality-aware Attention with Decoupled Knowledge Memory for Efficient Large Language Model Pretraining
Aug 12, 2026Qiuwu Chen, Zimo Liu, Yuchen Li +8Language Model Pretraining Memory-Augmented Language Models
AIGCode · South China University of Technology · Pazhou Laboratory
Why Post-Norm Transformers Collapse: Attention Amplification and Gradient Repair Failure
Aug 10, 2026Xingjian Wang, Qingyu Han, Xiaodong Luo +1Transformer Self-Attention
The Chinese University of Hong Kong, Shenzhen · Shenzhen Research Institute of Big Data
Evaluating Dedicated Monolingual and Joint Multilingual Causal Models for Dravidian Languages
Aug 7, 2026Venkata Naga Sai Vishnu Rohit PulipakaMultilingual Language Models Multilingual Language Model Evaluation
Independent Researcher
Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLU
Aug 7, 2026Yuting Ge, Pengju Yang, Mingkai NieLanguage Model Pretraining Decoder-Only Language Models
Department of Electrical Engineering City University of Hong Kong · Beihang University · National University of Singapore
When Attention Goes Blind: Numerical Failure in ALiBi Positional Encodings
Aug 4, 2026Christopher Schröder, Lukas Gienapp, Ferdinand Schlatt +2Language Model Pretraining Long-Context Retrieval
Institute for Applied Informatics at Leipzig University (InfAI) · ScaDS.AI Dresden/Leipzig · Seltz +1
Separating quantum circuits from classical LLMs
Aug 4, 2026Srinivasan Arunachalam, Arkopal Dutt, Hari Krovi +1Decoder-Only Language Models Diffusion Language Models
IBM Research

Oct 18, 2026J/K move · Enter open · S save

Stepwise Reasoning Verification: Don't Generate, Classify!

Yifei Shi, Michalis Korakakis, Rami Aly, Eric Chamoun+2

University of Cambridge · Alan Turing Institute · University of Edinburgh

Reasoning Verification Language Model Reasoning Evaluation Decoder-Only Language Models

Open paper AskPDF arXiv

Verifying the reasoning of large language models is increasingly important as they are applied to complex mathematical and scientific problems. Existing work focuses on large decoder models and reports mixed results for generative and discriminative verifiers, leaving unclear which paradigm is more effective and whether large models are necessary. In this work, we cast stepwise verification as a natural language inference task: the problem and partial solution prefix form the premise, while the next reasoning step serves as the hypothesis to be classified as entailed or non-entailed. This formulation enables error localisation at individual reasoning steps and direct comparison of encoder classification, decoder classification, and decoder verdict generation across model scales and annotation budgets. Across five reasoning datasets, a 0.15B encoder classifier outperforms most generative verifiers with up to 4B parameters. On identical decoder backbones, classification outperforms generation from 0.5B to 4B parameters and matches it on the strongest 7B backbone---reaching comparable performance with roughly one tenth as many annotated problems. Our findings establish classification as a parameter- and data-efficient approach to stepwise verification and show that strong verification does not require large generative decoders.