Efficient Language Model Training

Ask about this topic

Overview

This field asks how to train language models with fewer resources while preserving their quality, making model development more accessible and large-scale training more practical. Approaches include selecting and scheduling data, changing tokenization or model architecture, improving optimizers and gradient computation, and coordinating parallelism, memory, and hardware; related work also targets fine-tuning and reasoning-model training. Researchers evaluate these methods with measures such as training loss, downstream benchmarks, throughput, memory use, and energy across different model sizes and hardware settings. A central tension is that gains depend on the task and system, and lower training cost may come with losses in generalization, robustness, or reproducibility, so efficiency must be assessed alongside model quality.

Latest papers

CardsList

162 papers27 papers in the last four weeks, up 108% on the four weeks before. 0.2% of all new papers.

  1. Auto-Research Arena: Benchmarking and Advancing Auto-Research in Language Modeling

    Oct 26, 2026Xi Chen, Chongjian Ge, Yiran Xu +5Automated Machine LearningEfficient Language Model Training

    University of California San Diego · Adobe Research

  2. HobbyLM: A Sparse Mixture-of-Experts Model Family on a Hobby Budget

    Oct 25, 2026Harish Santhanalakshmi Ganesan, Hariharan M. S., Prabhurajhan RMixture-of-Experts Language ModelsEfficient Language Model Training

  3. Error Diffusion Replaces Latent Weights: Training Binary-Weight Language Models on a Few Bits of Dithered State

    Oct 25, 2026Pierre-Olivier BoninBinary Neural NetworksEfficient Language Model Training

    Independent Researcher

  4. HyScale: Scaling Long-Context Training for Learned Sparse Attention

    Oct 23, 2026Yangshen Deng, Xinjing Huang, Yulong Wang +8Efficient Language Model TrainingSparse Attention

    University of Edinburgh · Tencent Hunyuan

  5. AQL-Lite: Enabling Efficient Knowledge Acquisition for Arabic LLMs via Data-Centric Continual Pretraining

    Oct 21, 2026Houssam Eddine Boukhalfa, Mourad Bouache, Mourad DahmaneArabic Language ModelsLanguage Model Pretraining

    National School of Artificial Intelligence (ENSIA), Algiers, Algeria · Nojoom AI

  6. Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation

    Oct 19, 2026Xuyuan Xiong, Siyu Chen, Zihan Shao +5Truncated Backpropagation Through TimeEfficient Language Model Training

    MMLab, The University of Hong Kong · Yale University

  7. PRECOCIAL: Pretraining with Code Curriculums via Iterative Search for Accelerated Learning

    Oct 18, 2026Edward Gu, Ali Payani, Kevin EllisSynthetic Data PretrainingSynthetic Data Generation for Language Models

    Cornell University · Cisco Research

  8. AdamPM: Memory-Efficient LLM Optimization via Moment Pooling

    Oct 17, 2026Xiaowei Ji, Lishan Xu, Jianing An +3Efficient Language Model TrainingMemory-Efficient Fine-Tuning

    SKLCCSE, School of Artificial Intelligence, Beihang University, Beijing, China · Hangzhou International Innovation Institute, Beihang University, Hangzhou, China

  9. The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation

    Oct 14, 2026Ian Magnusson, Noah A. Smith, Pang Wei KohLanguage ModelingLanguage Model Scaling

    University of Washington · Allen Institute for AI

  10. TurboMamba: Faster Inference, Faster Training, and Better Prediction with Lexical Memory

    Oct 14, 2026Hanxin ZhangHybrid Transformer-State Space Model ArchitecturesAutoregressive Language Modeling

    Department of Neurology, University of Pittsburgh

  11. ProtModernBERT: Compute-Efficient Protein Language Modeling

    Oct 14, 2026Alberto Martínez, Natalia Flechas Manrique, Llorenç Espinosa-Portalés +1Protein Language ModelsEfficient Language Model Training

    Multiverse Computing, Parque Científico y Tecnológico de Gipuzkoa, Paseo de Miramón 170, 20014 Donostia-San Sebastián, Spain. · Multiverse Computing, 7 rue de la Croix Martre, 91120 Palaiseau, Paris, France

  12. Copy-Paste Language Models

    Oct 13, 2026Nathan Godey, Yoav ArtziLanguage Model PretrainingEfficient Language Model Training

    Cornell University New York City, USA

  13. A Refresh Interval Does Not Fully Specify a Refresh Schedule: Learning-Rate-Dependent Placement in Kronecker-Preconditioned Language Model Training

    Oct 13, 2026Xinyu Wang, Zhiwei Zhai, Yu Gu +6Deep Learning OptimizationEfficient Language Model Training

    McGill University · The Chinese University of Hong Kong · University of Manitoba +1

  14. ATTUNE: Agentic Throughput Tuning for LLM Training

    Oct 13, 2026Leo Fan, Hongyang Wang, Wenbo Chen +10Efficient Language Model TrainingDeep Learning Optimization

    Amazon

  15. Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs

    Oct 11, 2026Sewon MinLLM TrainingData Repetition in Language Model Training

    UC Berkeley & Allen Institute for AI

  16. Adaptive-Depth Is a Finishing Stage

    Oct 9, 2026Andrei Piterbarg, Sean McLeish, Dhruv Pai +1Efficient Language Model TrainingAdaptive Learning

    University of Pennsylvania · University of Maryland

  17. Learning More by Training Less: Uncertainty-Guided Progressive Contraction for Efficient Reasoning Post-Training

    Oct 9, 2026Suorong Yang, Jiaxin Wang, Jiaming Su +4Efficient Language Model ReasoningEfficient Language Model Training

    National University of Singapore · Nanjing University · Fudan University +3

  18. Towards Looped Models Done Right, Part II: Rethinking at Fixed Points

    Oct 5, 2026Benhao Huang, Chufan Shi, Junlin Chen +4Efficient Language Model TrainingRecurrent Transformers

    Institute of Foundation Models · CMU · USC

  19. Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models

    Oct 5, 2026Joe DwyerLanguage Model Scaling LawsEfficient Language Model Training

    ECPI University

  20. More Than Words: Compositional Tokenization for Efficient Language Models

    Oct 4, 2026Yuval Reif, Guy Kaplan, Roy SchwartzLanguage Model PretrainingEfficient Language Model Training

    The Hebrew University of Jerusalem

  21. Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining

    Oct 4, 2026Ziyue WANG, T. KanamoriLanguage Model PretrainingLanguage Model Scaling Laws

    Institute of Science Tokyo

  22. Constraint-Aware Training

    Oct 2, 2026Jinwoo KimConstrained Generative ModelingEfficient Language Model Training

    Department of Computer Science and Engineering University of California-San Diego La Jolla, CA

  23. Beyond State-of-the-Art: Standardising Environmental Impact Metrics for AI Research

    Oct 1, 2026Lachlan McGinness, Dan Pagendam, Robert OffnerLLM Inference EfficiencyEnergy-Efficient ML

    School of Computing, Australian National University · Commonwealth Scientific and Industrial Research Organisation

  24. Closing the Loop: Practical Training Recipes for Looped Language Models

    Sep 30, 2026Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5Efficient Language Model TrainingLLM Training

    RND NLP, DAIMLD, Russian Federation

  25. It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

    Sep 29, 2026Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois +7Language Model PretrainingSynthetic Data Pretraining

    PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab +6

  26. MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining

    Sep 28, 2026Chang-Wei Shi, Xu Wang, Wu-Jun LiLanguage Model PretrainingEfficient Language Model Training

    National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China

  27. SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning

    Sep 28, 2026Bojian Yin, Shurong Wang, Yuqi Pan +1Language Model PretrainingEfficient Language Model Training

    Institute of Automation, Chinese Academy of Sciences

  28. LionMuon: Alternating Spectral and Sign Descent for Efficient Training

    Sep 28, 2026Arman Bolatov, Artem Riabinin, Nikita Kornilov +4Stochastic OptimizationLanguage Model Pretraining

    Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Applied Artificial Intelligence Institute +1

  29. No Free Efficiency: Revisiting the Trade-off Between Training Efficiency and Model Vulnerability

    Sep 27, 2026Yiyong Liu, Jun Sakuma, Michael Backes +1Adversarial RobustnessEfficient Language Model Training

    CISPA Helmholtz Center for Information Security · Institute of Science Tokyo

  30. Elastic Selective Spectral Hybrids for Train-Once, Export-Many Budgeted Inference

    Sep 26, 2026Dachuan Song, Chuchu Chen, Xuan WangSelective SSMsLLM Inference Acceleration

    George Mason University · George Washington University