Efficient Language Model Training

Follow this topic Ask about this topic

This field asks how to train language models with fewer resources while preserving their quality, making model development more accessible and large-scale training more practical. Approaches include selecting and scheduling data, changing tokenization or model architecture, improving optimizers and gradient computation, and coordinating parallelism, memory, and hardware; related work also targets fine-tuning and reasoning-model training. Researchers evaluate these methods with measures such as training loss, downstream benchmarks, throughput, memory use, and energy across different model sizes and hardware settings. A central tension is that gains depend on the task and system, and lower training cost may come with losses in generalization, robustness, or reproducibility, so efficiency must be assessed alongside model quality.

Latest papers

162 papers27 papers in the last four weeks, up 108% on the four weeks before. 0.2% of all new papers.

Rethinking Token Prediction: Tree-Structured Diffusion Language Model
Apr 4, 2026Zihao Wu, Haoming Yang, Juncheng Dong +1Diffusion Language Models Efficient Language Model Training
Department of Electrical & Computer Engineering, Duke University
Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer
Apr 1, 2026Dharma Teja Vooturi, Dhiraj Kalamkar, Dipankar Das +1Language Model Pretraining LLM Training
Parallel Computing Lab (India) Intel Corporation
Training-Free Dynamic Upcycling of Expert Language Models
Mar 31, 2026Eros Fanì, Oğuzhan ErsoyMixture-of-Experts Language Models Efficient Language Model Training
Gensyn
Time is Not Compute: Scaling Laws for Wall-Clock Constrained Training on Consumer GPUs
Mar 29, 2026Yi LiuScaling Laws Language Model Scaling Laws
Independent Researcher
Throughput Optimization as a Strategic Lever in Large-Scale AI Systems: Evidence from Dataloader and Memory Profiling Innovations
Mar 27, 2026Mayank JhaEfficient Language Model Training LLM Training
Amazon Robotics
MAGNET: Autonomous Expert Model Generation via Decentralized Autoresearch and BitNet Training
Mar 26, 2026Yongwan Kim, Sungchul ParkAutomated Machine Learning LLM Training
Holo Studio Co., Ltd.
SozKZ: Training Efficient Small Language Models for Kazakh from Scratch
Mar 21, 2026Saken TukenovLanguage Model Pretraining Small Language Models
Independent Researcher
SAGE: Sustainable Agent-Guided Expert-tuning for Culturally Attuned Translation in Low-Resource Southeast Asia
Mar 20, 2026Zhixiang Lu, Chong Zhang, Yulong Li +5Machine Translation Low-Resource MT
Xi’an Jiaotong-Liverpool University Suzhou, China · University of Liverpool Liverpool, United Kingdom · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates
Optimal Splitting of Language Models from Mixtures to Specialized Domains
Mar 19, 2026Skyler Seto, Pierre Ablin, Anastasiia Filippova +4Efficient Language Model Training Domain-Adaptive Pretraining
Apple · National University of Singapore
Anatomical Heterogeneity in Transformer Language Models
Mar 19, 2026Tomasz WietrzykowskiTransformer Efficient Language Model Training
Independent Researcher, Wroclaw, Poland
SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
Mar 19, 2026Shenggui Li, Chao Wang, Yikai Zhu +14Speculative Decoding LLM Inference Acceleration
Nanyang Technological University · SpecForge Team · Meituan +2
Learning to Reason with Curriculum I: Provable Benefits of Autocurriculum
Mar 18, 2026Nived Rajaraman, Audrey Huang, Miro Dudik +3Efficient Language Model Training Curriculum RL
Microsoft Research · University of Illinois Urbana Champaign
EngGPT2: Sovereign, Efficient and Open Intelligence
Mar 17, 2026G. Ciarfaglia, A. Rosanova, S. Cipolla +17Language Modeling Open-Weight Language Models
HO-SFL: Hybrid-Order Split Federated Learning with Backprop-Free Clients and Dimension-Free Aggregation
Mar 16, 2026Qiyuan Chen, Xian Wu, Yi Wang +1Communication-Efficient Distributed Training Zeroth-Order Optimization
Department of Electrical and Computer Engineering, The University of Hong Kong, Hong Kong SAR, China.
Towards Next-Generation LLM Training: From the Data-Centric Perspective
Mar 16, 2026Hao Liang, Zhengyang Zhao, Zhaoyang Han +8LLM Training Efficient Language Model Training
Peking University, Beijing, China · Hong Kong University of Science and Technology, Hong Kong SAR, China · Institute for Advanced Algorithms Research, Shanghai, China +2
Lost in Backpropagation: The LM Head is a Gradient Bottleneck
Mar 10, 2026Nathan Godey, Yoav ArtziLLM Training Efficient Language Model Training
Cornell University, New York, USA
Progressive Residual Warmup for Language Model Pretraining
Mar 5, 2026Tianhao Chen, Xin Xu, Lu Yin +4Language Model Pretraining LLM Training
The Hong Kong University of Science and Technology, Hong Kong, China · University of Surrey, Guildford, UK · The University of Hong Kong, Hong Kong, China +1
Bidirectional Curriculum Generation: A Multi-Agent Framework for Data-Efficient Mathematical Reasoning
Mar 5, 2026Boren Hu, Xiao Liu, Boci Peng +4Curriculum Learning LLM Mathematical Reasoning
Zhejiang University · University of Macau · Peking University +3
Distribution-Aware Companding Quantization of Large Language Models
Feb 27, 2026Athul Radhakrishnan, Siddhant Mohan, Mahima SachdevaLLM Training Multi-Token Prediction
Preference Packing: Efficient Preference Optimization for Large Language Models
Feb 27, 2026Jaekyung ChoEfficient Language Model Training Preference Optimization
AWS GenAI Innovation Center
RLHFless: Serverless Computing for Efficient RLHF
Feb 26, 2026Rui Wei, Hanfei Yu, Shubham Jain +5Efficient Language Model Training RL from Human Feedback
Stevens Institute of Technology · Northeastern University · Stony Brook University +1
Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA
Feb 26, 2026Hai Huang, Yann LeCun, Randall BalestrieroEfficient Language Model Training LLM Training
Atlassian · NYU · Brown
A Diversity Diet for a Healthier Model: A Case Study of French ModernBERT
Feb 25, 2026Louis Estève, Christophe Servan, Thomas Lavergne +1Language Model Pretraining Diversity-Aware Sampling
Université Paris-Saclay, CNRS, LISN
Efficient Scaling of LLM Training with Flexible Context Parallelism
Feb 25, 2026Yifan Niu, Han Xiao, Dongyi Liu +2Context Parallelism Efficient Language Model Training
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co., Ltd. · The Hong Kong University of Science and Technology
Entropy-Based Data Selection for Language Models
Feb 19, 2026Hongming Li, Yang Liu, Chao HuangData Selection for Language Model Fine-Tuning Training Data Selection
School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China · National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence, Beijing 100086, China · Shunde Innovation School, University of Science and Technology Beijing, Foshan 528399, China
Predictive Batch Scheduling: Accelerating Language Model Training Through Loss-Aware Sample Prioritization
Feb 19, 2026Sumedh RasalEfficient Language Model Training Adaptive Sampling
Georgia Institute of Technology
ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset
Feb 16, 2026DatologyAI, :, Aldo Gael Carranza +32Multilingual Language Models Training Data Curation
DatologyAI
Stabilizing Native Low-Rank LLM Pretraining
Feb 12, 2026Paul Janson, Edouard Oyallon, Eugene BelilovskyLanguage Model Pretraining Low-Rank Matrix Decomposition
Concordia University, Montreal, Canada · Mila Quebec AI Institute, Montreal, Canada · Sorbonne University, CNRS, Paris, France
Revisiting the Shape Convention of Transformer Language Models
Feb 6, 2026Feng-Ting Liao, Guan-Ting Yi, Tzu-Quan Lin +2Efficient Transformer Inference Transformer FFNs
MediaTek Research · National Taiwan University
Variance-Adaptive Muon: Pre-Orthogonalization Variance Modulation for Efficient Language Model Pretraining
Jan 21, 2026Jingru Li, Yibo Fan, Huan LiStochastic Optimization Language Model Pretraining
College of Artificial Intelligence, Nankai University

Apr 4, 2026J/K move · Enter open · S save

Rethinking Token Prediction: Tree-Structured Diffusion Language Model

Zihao Wu, Haoming Yang, Juncheng Dong, Vahid Tarokh

Department of Electrical & Computer Engineering, Duke University

Diffusion Language Models Efficient Language Model Training

Open paper AskPDF arXiv

Discrete diffusion language models have emerged as a competitive alternative to auto-regressive language models, but training them efficiently under limited parameter and memory budgets remains challenging. Modern architectures are predominantly based on a full-vocabulary token prediction layer, which accounts for a substantial fraction of model parameters (e.g., more than 20% in small scale DiT-style designs) and often dominates peak GPU memory usage. This leads to inefficient use of both parameters and memory under constrained training resources. To address this issue, we revisit the necessity of explicit full-vocabulary prediction, and instead exploit the inherent structure among tokens to build a tree-structured diffusion language model. Specifically, we model the diffusion process with intermediate latent states corresponding to a token's ancestor nodes in a pre-constructed vocabulary tree. This tree-structured factorization exponentially reduces the classification dimensionality, makes the prediction head negligible in size, and enables reallocation of parameters to deepen the attention blocks. Empirically, under the same parameter budget, our method reduces peak GPU memory usage by half while matching the perplexity performance of state-of-the-art discrete diffusion language models.