Overview

Language model scaling studies how changes in parameters, training tokens, compute, and inference effort affect language modeling and downstream abilities. Researchers use scaling laws and controlled size sweeps, and explore width or depth expansion, warm starts, sparse and hybrid architectures, external memory, and inference-time search. Larger models often improve language-model loss and performance on reasoning, question-answering, and coding tasks, but gains can diminish or vary with data, architecture, task, and evaluation format. A central challenge is allocating capacity and compute so improvements transfer reliably without excessive training or inference costs.

Latest papers

CardsList

67 papers8 papers in the last four weeks, down 38% on the four weeks before. 0.1% of all new papers.

  1. Multi-Branch Transformer

    Nov 15, 2026Gaotang Li, Lingjie Chen, Chunyuan Deng +6TransformerLanguage Model Scaling

  2. When Bigger Becomes Brittle: Why Compression Hurts More as Language Models Scale

    Nov 5, 2026Gautam Gupta, Dikshant Kukreja, Unique Kumar +7LLM CompressionLanguage Model Scaling

  3. From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language Models

    Oct 30, 2026Navapat Nananukul, Mayank KejriwalMedical LLMsMedical QA

  4. Transformer to GPT-5: Advancements, Limitations, and Future Directions in Large Language Model Design

    Oct 25, 2026Harsidak Singh Banwait, Chander PrabhaLanguage Model ScalingLLM Training

  5. Scaling Multilingual Pretraining of Pixel Language Models

    Oct 21, 2026Ilker Kesen, Jenny Kunz, Kasra Malihi +1Multilingual Language ModelsLanguage Model Scaling

  6. BatchParScale: Sparse Batch-Mixing for Sequence Models

    Oct 18, 2026Hans Peter Lyngsøe Raaschou-jensen, M. Onurcan Kaya, Desmond ElliottParallel Reasoning in Language ModelsLanguage Model Scaling

  7. Do Speech BCIs Need Larger Models? Rethinking Neural Decoding beyond Scaling

    Oct 16, 2026Zehui Feng, Weichuan Wang, Xiaohan Chen +2Neural DecodingBrain-to-Speech Decoding

  8. The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation

    Oct 14, 2026Ian Magnusson, Noah A. Smith, Pang Wei KohLanguage ModelingLanguage Model Scaling

  9. LatentLoop: Scaling Looped MoE Models with Asymmetric Compute Allocation

    Oct 14, 2026Zefeng He, Yunkai Chai, Xiaoyang Chen +5Mixture-of-Experts ModelsLanguage Model Scaling

  10. A Scaled-Up Empirical Study of Syntactic Language Models

    Oct 12, 2026Pengcheng Wang, Zhexuan Zhang, Chao Lou +1Syntactic Generalization in Language ModelsLanguage Modeling

  11. Byte Language Models: Scaling, Emergent Abstractions, and Information Allocation

    Oct 5, 2026Jie Wang, Shiwei Luo, Qi Zhang +1Language Model ScalingByte-Level Language Model

  12. ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control

    Sep 30, 2026Peng Jin, Zihan Qiu, Zekun Wang +8Expert Load BalancingMixture-of-Experts Language Models

  13. Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories

    Sep 29, 2026Zhehao Huang, Changxin Tian, Qingyuan Yang +5Language Model ScalingModel Merging

  14. When Harness Beats Scale, and When Reading Beats Both

    Sep 28, 2026Ivan Bondarenko, Nikolay O. NikitinNumerical Reasoning in Language ModelsLanguage Model Scaling

  15. Likelihood Ranking doesn't Scale Like Prompting in LLMs

    Sep 24, 2026Alessandro Bondielli, Lucia Passaro, Davide Bacciu +1LLM EvaluationLanguage Model Generation Evaluation

  16. Depth and Scale in the Sub-150M Regime: JugnuLM-53M vs JugnuLM-110M

    Sep 13, 2026Dushyant Rajput, Nirdesh Chauhan, Siddharth KosarajuLanguage Model PretrainingSmall Language Models

  17. Scaling Near-Optimal SFT-RL Annotation Budget Allocation from Small to Large LLMs

    Sep 1, 2026Jingtan Wang, Arun Verma, Xiaoqiang Lin +4Supervised Fine-TuningLanguage Model Scaling

  18. When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning

    Aug 31, 2026Hamed Babaei Giglou, Sören Auer, Jennifer D'SouzaLanguage Model ScalingOntology Learning

  19. Asymmetric Capacity Allocation in Self-Refinement Pipelines

    Aug 21, 2026Zhuoyi Yang, Ian G. Harris, Salar Hashemitaheri +7LLM Self-RefinementLanguage Model Scaling

  20. Kathleen Writes: Autoregressive Generation and Data Scaling Without Attention

    Aug 5, 2026George FountzoulasLanguage Model Generation EvaluationAutoregressive Language Modeling

  21. Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

    Jul 30, 2026Rubin Wei, Jiaqi Cao, Jiarui Wang +4Language Model PretrainingMemory-Augmented Language Models