Language Model Scaling

Follow this topic Ask about this topic

Language model scaling studies how changes in parameters, training tokens, compute, and inference effort affect language modeling and downstream abilities. Researchers use scaling laws and controlled size sweeps, and explore width or depth expansion, warm starts, sparse and hybrid architectures, external memory, and inference-time search. Larger models often improve language-model loss and performance on reasoning, question-answering, and coding tasks, but gains can diminish or vary with data, architecture, task, and evaluation format. A central challenge is allocating capacity and compute so improvements transfer reliably without excessive training or inference costs.

Latest papers

61 papers14 papers in the last four weeks, up 180% on the four weeks before. 0.1% of all new papers.

Transformer to GPT-5: Advancements, Limitations, and Future Directions in Large Language Model Design
Oct 25, 2026Harsidak Singh Banwait, Chander PrabhaLanguage Model Scaling LLM Training
Chitkara University of Engineering and Technology, Chitkara University, Punjab, India
When Does Pre-Trained Capacity Help Causal Learning? From Scalar Prediction to Counterfactual Post-Training
Oct 23, 2026Qi Xu, Zijie Lei, Zhifei Deng +2Causal Inference Causal Effect Estimation
Meta Platforms, Inc., Sunnyvale, CA, USA
Scaling Multilingual Pretraining of Pixel Language Models
Oct 21, 2026Ilker Kesen, Jenny Kunz, Kasra Malihi +1Multilingual Language Models Language Model Scaling
Humboldt-Universität zu Berlin · Linköping University · University of Copenhagen
BatchParScale: Sparse Batch-Mixing for Sequence Models
Oct 18, 2026Hans Peter Lyngsøe Raaschou-jensen, M. Onurcan Kaya, Desmond ElliottParallel Reasoning in Language Models Language Model Scaling
University of Copenhagen · Technical University of Denmark
Do Speech BCIs Need Larger Models? Rethinking Neural Decoding beyond Scaling
Oct 16, 2026Zehui Feng, Weichuan Wang, Xiaohan Chen +2Neural Decoding Brain-to-Speech Decoding
Shanghai Jiao Tong University · Nanyang Technological University · Zhejiang University
The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation
Oct 14, 2026Ian Magnusson, Noah A. Smith, Pang Wei KohLanguage Modeling Language Model Scaling
University of Washington · Allen Institute for AI
LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation
Oct 14, 2026Zefeng He, Yunkai Chai, Xiaoyang Chen +5Mixture-of-Experts Models Language Model Scaling
SJTU · Shanghai AI Lab · USTC +2
A Scaled-Up Empirical Study of Syntactic Language Models
Oct 12, 2026Pengcheng Wang, Zhexuan Zhang, Chao Lou +1Syntactic Generalization in Language Models Language Modeling
School of Information Science and Technology, ShanghaiTech University · Shanghai Engineering Research Center of Intelligent Vision and Imaging · Independent researcher
Do Small Language Models Learn to Negotiate? A Controlled Scaling Study of RL-Trained Sellers
Oct 5, 2026Pedro Tabacof, Sagar JoglekarRL for Language Models Multi-Agent Negotiation
Fin AI Research
Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation
Oct 5, 2026Jie Wang, Shiwei Luo, Qi Zhang +1Language Model Scaling Byte-Level Language Model
School of Computer Science, East China Normal University, Shanghai, China · School of Computer Science, Fudan University, Shanghai, China
ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control
Sep 30, 2026Peng Jin, Zihan Qiu, Zekun Wang +8Expert Load Balancing Mixture-of-Experts Language Models
Qwen Team, Alibaba Token Hub, Alibaba Group
Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories
Sep 29, 2026Zhehao Huang, Changxin Tian, Qingyuan Yang +5Language Model Scaling Model Merging
Ling Team, Ant Group · Shanghai Jiao Tong University
An Exact Generate - Transform Decomposition of Small-LLM Team Scaling Across Orchestration Architectures
Sep 28, 2026Blaz Bertalanic, Carolina FortunaMulti-Agent LLM Systems LLM Agent Orchestration
Jožef Stefan Institute, Ljubljana, Slovenia
When Harness Beats Scale, and When Reading Beats Both
Sep 28, 2026Ivan Bondarenko, Nikolay O. NikitinNumerical Reasoning in Language Models Language Model Scaling
Novosibirsk State University · ITMO University
Likelihood Ranking doesn't Scale Like Prompting in LLMs
Sep 24, 2026Alessandro Bondielli, Lucia Passaro, Davide Bacciu +1LLM Evaluation Language Model Generation Evaluation
CoLingLab, Department of Philology, Literature and Linguistics, University of Pisa · Department of Computer Science, University of Pisa
KITE: KV-Invariant Transformer Expansion for Efficient Agentic LLM Scaling
Sep 23, 2026Zhiheng Hu, Yixun Wei, Jian Zhou +8Language Model Scaling LLM Inference Acceleration
StepFun
Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M
Sep 13, 2026Dushyant Rajput, Nirdesh Chauhan, Siddharth KosarajuLanguage Model Pretraining Small Language Models
AltSlate Labs LLP
Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs
Sep 1, 2026Jingtan Wang, Arun Verma, Xiaoqiang Lin +4Supervised Fine-Tuning Language Model Scaling
Department of Computer Science, National University of Singapore, Republic of Singapore · Singapore-MIT Alliance for Research and Technology Centre, Republic of Singapore · Agency for Science, Technology, and Research (A*STAR), Republic of Singapore +1
When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning
Aug 31, 2026Hamed Babaei Giglou, Sören Auer, Jennifer D'SouzaLanguage Model Scaling Ontology Learning
TIB Leibniz Information Centre for Science and Technology, Hannover, Germany · L3S Research Center, Leibniz University of Hannover, Hannover, Germany
Asymmetric Capacity Allocation in Self-Refinement Pipelines
Aug 21, 2026Zhuoyi Yang, Ian G. Harris, Salar Hashemitaheri +7LLM Self-Refinement Language Model Scaling
University of California, Irvine · Drexel University
From BERT to Frontier Agents: Eight Years of Language-Model Progress, the Collapse of the Capability-Cost Curve, and the Rise of Task-Targeted Models
Aug 13, 2026Pranav Kumar KaliaperumalAI Coding Agents Language Model Scaling
Benchmarking LLM-Guided Control-Plane Policies for Backend Fault Isolation in HAProxy
Aug 11, 2026Aman Chauhan, Vishnu PendyalaLanguage Model-Based Control Language Model Scaling
Diminishing Returns of Intelligence: The Non-Linear Relationship Between LLM Scale and User Perception in Short-Duration Open-Ended Social Human-Robot Interactions
Aug 8, 2026Amanda Rasille Røn Volf, Morten Roed FrederiksenLLM Evaluation Language Model Scaling
Computer Science Department of The IT-University of Copenhagen
Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention
Aug 5, 2026George FountzoulasLanguage Model Generation Evaluation Autoregressive Language Modeling
Department of Computer Engineering & Informatics · Frederick University, Nicosia, Cyprus
Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
Jul 30, 2026Rubin Wei, Jiaqi Cao, Jiarui Wang +4Language Model Pretraining Memory-Augmented Language Models
LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory · Electronic Engineering, Tsinghua University
Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
Jul 9, 2026Aiwei Liu, Cheng Shi, Chuhan Wu +45Language Model Scaling Laws Transformer Attention
Understanding Layer Patching in Model Size Interpolation
Jul 9, 2026Sara Kangaslahti, Jonathan Geuter, Nihal V. Nayak +3Language Model Scaling
1Harvard University · 2Kempner Institute · 3IST Austria
Think Big, Search Small: Where Capacity Matters in Hierarchical Search Agents?
Jul 8, 2026Qinnan Cai, Yibo Zhao, Xiang LiMulti-Hop QA Multi-Agent LLM Systems
School of Data Science and Engineering East China Normal University
Fixed RAG Compression Collapses Measured Reader Scaling
Jun 20, 2026Sugam Panthi, Rabab AbdelfattahRetrieval-Augmented Generation LLM Evaluation
The University of Southern Mississippi
Variable-Width Transformers
Jun 16, 2026Zhaofeng Wu, Oliver Sieberling, Shawn Tan +3Transformer Decoder-Only Language Models
ãMIT · èMIT-IBM Watson AI Lab

Oct 25, 2026J/K move · Enter open · S save

Transformer to GPT-5: Advancements, Limitations, and Future Directions in Large Language Model Design

Harsidak Singh Banwait, Chander Prabha

Chitkara University of Engineering and Technology, Chitkara University, Punjab, India

Language Model Scaling LLM Training

Open paper AskPDF arXiv

Large Language Models (LLMs) have revolutionized Artificial Intelligence (AI) by their extraordinary capabilities, like contextual understanding, few-shot/zero-shot learning, code generation, ability to reason and problem-solving. Recent studies have shown immense work on LLMs. Yet, there is limited work on the thorough taxonomy of LLMs. This study would showcase a detailed classification of LLMs, starting with the earlier models like BERT and leading to LLMs like GPT, Gemini, Claude and Grok. We would delve into topics like datasets used to train LLMs, computational power used during training, their hardware specifications, architecture and parameters. Further, we would also look at the limitations, computational challenges and open problems as well. This study would primarily focus on the timeline of LLMs, signifying their growth over the period. Findings reveal a clear pattern: models that optimize the ratio between architectural complexity and data quality exhibit stronger reasoning stability and better alignment, without proportionally increasing the computational load. This desired balance between scaling and sustainability is the defining characteristic of the current generation of AI.