Decoder-Only Language Models

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, up 80% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 67

All topics

The Wiola Architecture for Efficient Small Language Models
Jul 1, 2026Aryuemaan Kumar Chowdhury, Afreen Shaik, Yaparla Bhargavi +1Rotary Positional Embeddings Token Merging
Research and Development, Oscowl Ai IIT Hyderabad Hyderabad, India · Research and Development, Oscowl Ai Hyderabad, India
Recovering Input Text from Hidden States: Study of Gradient-Based Inversion of Decoder-Only Language Models
Jul 1, 2026Mikołaj Słowikowski, Maciej Witold MajewskiDecoder-Only Language Models Privacy Leakage in Language Models
AGH University of Krakow, Faculty of Physics and Applied Computer Science, al. Mickiewicza 30, 30-059 Krakow, Poland
Beyond Clean Text: Evaluating Encoder and Decoder Robustness for Bangla Event Detection in Noisy Text
Jun 29, 2026Tanvir Ahmed Sijan, S. M Golam Rifat, Nayeemul Islam +1Decoder-Only Language Models Low-Resource Language Processing
Jahangirnagar University, Dhaka, Bangladesh · Rajshahi University of Engineering & Technology, Rajshahi, Bangladesh · Bangladesh University of Engineering and Technology, Dhaka, Bangladesh
IG-Lens: Exact Additive Probability Attribution Across Transformer Layers via Telescoping Integrated Gradients
Jun 29, 2026Duc Anh NguyenGradient-Based Attribution Integrated Gradients
Hanoi University of Science and Technology
MultiHashFormer: Hash-based Generative Language Models
Jun 26, 2026Huiyin Xue, Atsuki Yamaguchi, Nikolaos AletrasLLM Compression Decoder-Only Language Models
School of Computer Science, University of Sheffield, United Kingdom
SamatNext v0.2-B: An Exploratory Study of RMS-Normalized Hybrid Decoders for Curriculum Retention in Small Code Models
Jun 20, 2026Samat ZharassovContinual Learning for LLMs Decoder-Only Language Models
Low-Rank Attention Residuals
Jun 19, 2026Jonathan SuLow-Rank Attention Decoder-Only Language Models
Independent Researcher
Dual Dimensionality for Local and Global Attention
Jun 17, 2026Zhiyuan Wang, Xuan Luo, Sirui Zeng +1LLM Compression Decoder-Only Language Models
UC Santa Barbara
Variable-Width Transformers
Jun 16, 2026Zhaofeng Wu, Oliver Sieberling, Shawn Tan +3Transformer Decoder-Only Language Models
ãMIT · èMIT-IBM Watson AI Lab
Surpassing Scale by Efficiency: A Compact 135M Parameter Foundational LLM Natively Adapted for the Bangla Language
Jun 15, 2026Rabindra Nath NandiDecoder-Only Language Models Small Language Models
Independent Researcher
SpikeDecoder: Realizing the GPT Architecture with Spiking Neural Networks
Jun 10, 2026Claas Beger, Florian Walter, Alois KnollEfficient Neural Network Inference Spiking Transformers
Chair of Robotics, Artificial Intelligence and Real-time Systems Technical University of Munich, Germany
DECSELFMASK: Leveraging Unlabeled Text via Self-Relevance-Guided Masking for Decoder-Only Classification
Jun 8, 2026Pietro Ferrazzi, Matteo Merler, Giovanni Bonetta +2Self-Supervised Pre-Training Decoder-Only Language Models
Fondazione Bruno Kessler, Trento, Italy · University of Padova, Italy
Where does Absolute Position come from in decoder-only Transformers?
Jun 4, 2026Valeria Ruscio, Umberto Nanni, Fabrizio SilvestriRotary Positional Embeddings Transformer Interpretability
Intuition Machines · Sapienza University of Rome
ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMs
Jun 4, 2026Ailiang Lin, Zhuoyun Li, Yusong Wang +3Text Embeddings Decoder-Only Language Models
Institute of Science Tokyo · Tencent
Pattern Selectivity is Not Task-Causal Structure: A Cross-Architecture Mechanistic Study of Composed-Task Circuits in 1B-Class Language Models
Jun 3, 2026Yongzhong XuAttention Head Analysis Decoder-Only Language Models
Depth-Attention: Cross-Layer Value Mixing for Language Models
Jun 3, 2026Boyi Zeng, Yiqin Hao, Zitong Wang +7Efficient Transformer Inference Decoder-Only Language Models
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Sun Yat-sen University · Shanghai Innovation Institute +2
AlignAtt4LLM: Fast AlignAtt for Decoder-Only LLMs at IWSLT 2026 Simultaneous Speech Translation Task
Jun 2, 2026Quentin Fuxa, Dominik MacháčekSpeech Translation Decoder-Only Language Models
Independent Researcher · Charles University, MFF, ÚFAL · & University of Edinburgh
DOA: Training-Free Decoder-Only Attention Policy for Long-Form Simultaneous Translation with SpeechLLMs
May 29, 2026Sara Papi, Luisa BentivogliSpeech Translation Decoder-Only Language Models
Fondazione Bruno Kessler, Italy
The Cognitive Categorical Transformer: Category-Theoretic Inductive Biases for Language Modeling
May 22, 2026Al KariDecoder-Only Language Models Message Passing Neural Networks
Manceps Inc.
Post-Hoc Understanding of Metaphor Processing in Decoder-Only Language Models via Conditional Scale Entropy
May 20, 2026Lawhori Chakrabarti, Jennifer Johnson-Leung, Bert Baumgaertner +3Figurative Language Understanding Transformer Interpretability
Department of Computer Science, University of Idaho, Moscow ID 83844, USA · Department of Mathematics and Statistical Science, University of Idaho, Moscow ID 83844, USA · Department of Politics and Philosophy, University of Idaho, Moscow ID 83844, USA +2
MiniGPT: Rebuilding GPT from First Principles
May 17, 2026Jibin JosephLanguage Model Pretraining Decoder-Only Language Models
Department of Computer Science The University of Texas at Austin Austin, TX 78712
Transformer-like Inference from Optimal Control
May 15, 2026Aditya Kudre, Heng-Sheng Chang, Prashant G. MehtaTransformer Inference Decoder-Only Language Models
Coordinated Science Laboratory · Electrical and Computer Engineering · University of Illinois Urbana-Champaign +1
Neural Activation Patterns Across Language Model Architectures: A Comprehensive Analysis of Cognitive Task Performance
May 14, 2026Mahdi Naser-Moghadasi, Faezeh GhaderiLLM Evaluation Decoder-Only Language Models
Research Division, BrightMind AI, Seattle, WA · Texas Tech University, Lubbock, TX · University of Texas at Arlington
A Study on Hidden Layer Distillation for Large Language Model Pre-Training
May 12, 2026Maxime Guigon, Lucas Dixon, Michaël E. SanderLanguage Model Pretraining Decoder-Only Language Models
Google DeepMind
Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining
May 11, 2026Jinchang Zhu, Jindong Li, Yuwen Hao +3Language Model Pretraining Self-Attention
The Hong Kong University of Science and Technology (Guangzhou)
Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization
May 7, 2026Andy Zeyi Liu, Elliot Paquette, John SousRepresentation Learning Decoder-Only Language Models
Department of Applied Physics, Yale University, New Haven, CT 06511, United States · Department of Mathematics and Statistics, McGill University · Department of Applied Physics, Yale University, New Haven, Connecticut 06511, USA +1
Escaping Mode Collapse in LLM Generation via Geometric Regulation
May 1, 2026Xin Du, Kumiko Tanaka-IshiiLanguage Model Steering Decoder-Only Language Models
Department of Communications and Computer Engineering, Waseda University, Tokyo, Japan · Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University, Shanghai, China · Department of Computer Science and Engineering, Waseda University, Tokyo, Japan
Graph Memory Transformer (GMT)
Apr 26, 2026Nicola Zanarini, Niccolò Ferrari, Evelina LammaMemory-Augmented Neural Networks Transformer FFNs
Bonfiglioli Engineering s.r.l. · Department of Engineering, University of Ferrara, Via Saragat 1, 44122 Ferrara, Italy · NAIS s.r.l., Via Maria Callas 4, 40131 Bologna, Italy
Transformer as an Euler Discretization of Score-based Variational Flow
Apr 26, 2026Huadong LiaoRepresentation Learning Transformer
Scaling Point-in-Time Language Models
Apr 24, 2026Bryan Kelly, Semyon Malamud, Johannes Schwab +1Decoder-Only Language Models Temporal Data Leakage
Yale School of Management · AQR Capital Management · NBER +3

Jul 1, 2026, cs.AIJ/K move · Enter open · S save

The Wiola Architecture for Efficient Small Language Models

Aryuemaan Kumar Chowdhury, Afreen Shaik, Yaparla Bhargavi, Brahma Kumar

Research and Development, Oscowl Ai IIT Hyderabad Hyderabad, India · Research and Development, Oscowl Ai Hyderabad, India

Rotary Positional Embeddings Token Merging Transformer Decoder-Only Language Models Small Language Models Transformer Attention

Open paper AskPDF arXiv

We present Wiola, a fully original Small Language Model (SLM) architecture built from first principles, sharing no structural lineage with any existing model family including GPT, LLaMA, Mistral, or Falcon. Wiola introduces five independently novel components: (i) Spiral Rotary Positional Encoding (SRPE), which embeds token positions on a three-dimensional helical manifold combining absolute, relative, and hierarchical positional signals; (ii) Gated Cross-Layer Attention (GCLA), providing each decoder layer with soft cross-attention access to compressed summaries of two preceding layers for inter-layer coherence; (iii) Adaptive Token Merging (ATM), which dynamically merges se mantically redundant adjacent tokens in middle network layers to reduce attention complexity without information loss; (iv) Dual Stream Feed-Forward (DSFF), replacing the conventional MLP with two parallel streams fused by a learned per-dimension gate; and (v) WiolaRMSNorm, a modified normalisation introducing a per-dimension learned offset vector that prevents representation collapse. We provide complete mathematical derivations, architectural block diagrams, complexity analyses, and systematic comparisons against GPT-2, LLaMA-2, and Mistral. Wiola is released in four sizes (120M, 360M, 700M, and 1.5B parameters) and is fully compatible with the HuggingFace Transformers ecosystem, with all 22 architectural unit tests passing.

Momentum

Related topics

Latest papers 67