LLM Training

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

31 papers in the last four weeks, up 72% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 226

All topics

Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
May 30, 2026Qiao Xiao, Boqian Wu, Patrik Okanovic +6Language Model Pretraining Efficient Language Model Training
Eindhoven University of Technology · University of Luxembourg · University of Twente +1
GNMR: Runtime Stability Control for Low-Precision Large Language Model Training
May 30, 2026Boao Kong, Weichen Jia, Engao Zhang +6LLM Training
Peking University · Huawei Technologies Ltd.
Mellum2 Technical Report
May 29, 2026Marko Kojic, Ivan Bondyrev, Aral de Moor +6Mixture-of-Experts Language Models Open-Weight Language Models
1JetBrains · 2Constructor University, Bremen, Germany
D $^3$ : Dynamic Directional Graph-Constrained Data Scheduling for LLM Training
May 29, 2026Yuanjian Xu, Jianing Hao, Guang Zhang +1Efficient Language Model Training LLM Training
1HKUST-GZ · Microsoft Research Asia · 2Microsoft Research
Unlocking the Working Memory of Large Language Models for Latent Reasoning
May 28, 2026Lukas Aichberger, Sepp HochreiterMemory-Augmented Language Models Efficient Language Model Reasoning
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria · NXAI GmbH, Linz, Austria
Demystifying Data Organization for Enhanced LLM Training
May 28, 2026Yalun Dai, Yangyu Huang, Tongshen Yang +8Language Model Pretraining Training Data Curation
Nanyang Technological University · Microsoft Research · The Hong Kong University of Science and Technology
MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection
May 28, 2026Haowen Wang, Yaxin Du, Jian Yang +9Data Selection Efficient Language Model Training
1Beihang University · 2IQuest Research · 3Shanghai Jiao Tong University +2
AMDP: Asynchronous Multi-Directional Pipeline Parallelism for Large-Scale Models Training
May 28, 2026Ling Chen, Houming Wu, Wenjie YuPipeline Parallelism LLM Training
State Key Laboratory of Blockchain and Data Security, Zhejiang University, Hangzhou, China · College of Computer Science and Technology, Zhejiang University, Hangzhou, China
Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
May 28, 2026Jing Huang, Daniel Wurgaft, Rachit Bansal +6Long-Tail Learning Language Model Scaling Laws
Stanford University · Kempner Institute at Harvard University · MIT +1
Training Stratigraphy: Persistent Behavioral Artifacts in Large Language Models Observed Through Longitudinal AI-Human Interaction
May 27, 2026Chen Ying Claude, Zhihan LuoLLM Auditing Human-in-the-Loop AI
*Anthropic · †Independent Researcher
Heterogeneous Parallelism for Multimodal Large Language Model Training
May 26, 2026Yashaswi Karnati, Kamran Jafari, Akash Mehra +10Communication-Efficient Distributed Training Multimodal Large Language Models
NVIDIA
Laguna M.1/XS.2 Technical Report
May 26, 2026Julien Abadji, Marah Abdin, Connor Adams +93Mixture-of-Experts Language Models Software Engineering Agents
Negligible in Size, Significant in Effect: On Scale Vectors in Large Language Models
May 26, 2026Mingze Wang, Shuchen Zhu, Yuxin Fang +3Language Model Pretraining Weight Decay
ByteDance Seed · Peking University
Elias in the Lighthouse, Again? Diagnosing Low Diversity in LLM Stories
May 26, 2026Sil Hamilton, David MimnoLLM Alignment Diverse Text Generation
Department of Information Science Cornell University
MuCon: Clipped Muon Updates for LLM Training
May 26, 2026Albert YiMuon Optimizer Spectral Methods
Merge-Bench: Resolve Merge Conflicts with Large Language Models
May 25, 2026Benedikt Schesch, Michael D. ErnstSoftware Engineering Code Language Models
Amazon · University of Washington, USA
PowLU: An Activation Function for Stable Pre-Training of LLMs
May 25, 2026Peijie Jiang, Yuqi Feng, Cunyin Peng +5Language Model Pretraining Neural Network Activation Functions
Ling Team, Ant Group
How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws
May 25, 2026Zhitao Zhu, Xili Wang, Shizhe Wu +2Language Model Scaling Laws LLM Training
Peking University · Meituan
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
May 25, 2026Zili Zhang, Chengxu Yang, Shenglong Zhang +8Deep Learning Optimization Multimodal Large Language Models
1Peking University · †Work done during internships at Xiaohongshu, Inc. · 2Independent Researcher +2
When Mean CE Fails: Median CE Can Better Track Language Model Quality
May 23, 2026Hao Guo, Simon Dennis, Rivaan Patil +1LLM Evaluation LLM Training
1i14 · University of Melbourne · University of California, Santa Cruz
Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap
May 23, 2026Tianlang Chen, Shirley Wu, Jure LeskovecLanguage Model Distillation Teacher-Student Learning
Stanford
ScaleAcross Explorer: Exploring Communication Optimization for Scale-Across AI Model Training
May 23, 2026Minghao Li, Alicia Golden, Samuel Hsia +14Communication-Efficient Distributed Training Distributed Training
1Harvard University · 2Meta Platforms, Inc.
Unified Data Selection for LLM Reasoning
May 21, 2026Xiaoyuan Li, Yubo Ma, Chengpeng Li +6Training Data Selection LLM Training
University of Science and Technology of China · Alibaba Group · National University of Singapore
One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
May 21, 2026Di He, Songjun Tu, Keyu Wang +2Learning Rate Scheduling Efficient Language Model Training
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +5
Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate
May 20, 2026Dayal Singh Kalra, Maissam BarkeshliLanguage Model Scaling Laws Hyperparameter Transfer
Department of Computer Science, University of Maryland, College Park · Department of Physics, University of Maryland, College Park · Joint Quantum Institute, University of Maryland, College Park +1
DEL: Digit Entropy Loss for Numerical Learning of Large Language Models
May 19, 2026Zhaohui Zheng, Chenhang He, Shihao Wang +3Numerical Reasoning in Language Models Neural Network Optimization
The Hong Kong Polytechnic University · VCIP, College of Computer Science, Nankai University
A Tabular Schedule Abstraction for Communication-Aware Evaluation of Pipeline-Parallel LLM Training
May 19, 2026Daniel Barley, Jonathan Leis, Benjamin Klenk +1Communication-Efficient Distributed Training Pipeline Parallelism
Hardware and Artificial Intelligence (HAWAII) Lab, Heidelberg University, Heidelberg, Germany · NVIDIA Corporation, Santa Clara, CA, USA
A Bitter Lesson for Data Filtering
May 19, 2026Christopher Mohri, John Duchi, Tatsunori HashimotoLanguage Model Pretraining Training Data Curation
Department of Computer Science Stanford University · Departments of Statistics and Electrical Engineering Stanford University
Language models struggle with compartmentalization
May 19, 2026Thomas Vincent Howe, David WingateMultilingual Language Models LLM Interpretability
Department of Computer Science · Brigham Young University · Provo, UT 84602
ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
May 18, 2026Aaron DefazioLanguage Model Pretraining Learning Rate Scheduling
FAIR at Meta Super-Intelligence Labs

May 30, 2026, cs.LGJ/K move · Enter open · S save

Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling

Qiao Xiao, Boqian Wu, Patrik Okanovic, Tomasz Sternal+5

Eindhoven University of Technology · University of Luxembourg · University of Twente+1

Language Model Pretraining Efficient Language Model Training LLM Training Memory-Efficient Training

Open paper AskPDF arXiv

Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates. In this work, we show that the naive use of standard Adam-based optimizers leads to a cold-start issue for newly regrown parameters, resulting in excessively large updates and disrupted training dynamics. To address this issue, we propose Sparse Memory-Efficient Training (SMET), which stabilizes DST with optimizer warm-up and improves training progress through density-aware learning-rate scaling. SMET further reduces memory consumption by storing gradients and optimizer states only for active parameters. We provide a theoretical analysis of the update behaviors under SMET, showing improved optimization stability. Extensive experiments demonstrate that SMET enables stable, scalable, and memory-efficient sparse pre-training of LLMs, paving the way for sparse training as a practical alternative to dense training. Our code is publicly available at: https://github.com/QiaoXiao7282/SMET.

Momentum

Related topics

Latest papers 226