Efficient Language Model Training

Overview

This field asks how to train language models with fewer resources while preserving their quality, making model development more accessible and large-scale training more practical. Approaches include selecting and scheduling data, changing tokenization or model architecture, improving optimizers and gradient computation, and coordinating parallelism, memory, and hardware; related work also targets fine-tuning and reasoning-model training. Researchers evaluate these methods with measures such as training loss, downstream benchmarks, throughput, memory use, and energy across different model sizes and hardware settings. A central tension is that gains depend on the task and system, and lower training cost may come with losses in generalization, robustness, or reproducibility, so efficiency must be assessed alongside model quality.

Latest papers

CardsList

172 papers15 papers in the last four weeks, down 37% on the four weeks before. 0.1% of all new papers.

  1. Pruning DNA Sequence

    Nov 7, 2026Faias Satter, Shaker Islam, Aminul IslamDataset PruningEfficient Language Model Training

  2. On the Design of Hyper-Connections: Simplification, Stability, and Scalability

    Nov 4, 2026Xiangdong Zhang, Xiaohan Qin, Sunan Zou +8Hyper-ConnectionsEfficient Language Model Training

  3. Language Models Can Benefit from Domain-Specific Inductive Biases

    Nov 2, 2026Damien Teney, Liangze Jiang, Zachary Shinnick +2Language Model PretrainingEfficient Language Model Training

  4. UpLoop: Training Looped Transformers via Upcycling at No Extra Pretraining Compute

    Oct 30, 2026Cheng Liang, Tengfei Zhang, Ya Zhang +2Efficient Language Model TrainingRecurrent Transformers

  5. Auto-Research Arena: Benchmarking and Advancing Auto-Research in Language Modeling

    Oct 26, 2026Xi Chen, Chongjian Ge, Yiran Xu +5Automated Machine LearningEfficient Language Model Training

  6. HyScale: Scaling Long-Context Training for Learned Sparse Attention

    Oct 23, 2026Yangshen Deng, Xinjing Huang, Yulong Wang +8Efficient Language Model TrainingSparse Attention

  7. AQL-Lite: Enabling Efficient Knowledge Acquisition for Arabic LLMs via Data-Centric Continual Pretraining

    Oct 21, 2026Houssam Eddine Boukhalfa, Mourad Bouache, Mourad DahmaneArabic Language ModelsLanguage Model Pretraining

  8. Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation

    Oct 19, 2026Xuyuan Xiong, Siyu Chen, Zihan Shao +5Truncated Backpropagation Through TimeEfficient Language Model Training

  9. AdamPM: Memory-Efficient LLM Optimization via Moment Pooling

    Oct 17, 2026Xiaowei Ji, Lishan Xu, Jianing An +3Optimizer State CompressionEfficient Language Model Training

  10. The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation

    Oct 14, 2026Ian Magnusson, Noah A. Smith, Pang Wei KohLanguage ModelingLanguage Model Scaling

  11. ProtModernBERT: Compute-Efficient Protein Language Modeling

    Oct 14, 2026Alberto Martínez, Natalia Flechas Manrique, Llorenç Espinosa-Portalés +1Protein Language ModelsEfficient Language Model Training

  12. Copy-Paste Language Models

    Oct 13, 2026Nathan Godey, Yoav ArtziLanguage Model PretrainingEfficient Language Model Training

  13. Adaptive-Depth Is a Finishing Stage

    Oct 9, 2026Andrei Piterbarg, Sean McLeish, Dhruv Pai +1Efficient Language Model TrainingAdaptive Learning

  14. Learning More by Training Less: Uncertainty-Guided Progressive Contraction for Efficient Reasoning Post-Training

    Oct 9, 2026Suorong Yang, Jiaxin Wang, Jiaming Su +4Efficient Language Model ReasoningEfficient Language Model Training

  15. Towards Looped Models Done Right, Part II: Rethinking at Fixed Points

    Oct 5, 2026Benhao Huang, Chufan Shi, Junlin Chen +4Efficient Language Model TrainingRecurrent Transformers