LLM Training

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

31 papers in the last four weeks, up 72% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 226

All topics

Harnessing the Potential of Optimizing Data Mixtures via Bayesian Domain Reweighting
Jul 30, 2026Xiang Yuan, Kaiqing Lei, Zhenyu Jin +3Data Mixture Optimization LLM Training
School of Mathematics and Statistics, Xi’an Jiaotong University · Northwest University, Shaanxi, China
Towards joint scaling laws with optimal batch size schedules
Jul 30, 2026Jiaxiang Li, Zhiqi Bu, Shiyun XuDeep Learning Optimization Language Model Scaling Laws
Meta · Independent researcher
The Kinetics of Training: A Driven-Nucleation Rate Law for Emergence, Plasticity Loss, and Circuit Control in Language Models
Jul 29, 2026Lei DongLoss of Plasticity Phase Transitions
Independent Researcher
From Found to Designed: Concepts as a Design Axis for Large Language Models
Jul 29, 2026Chen ShaniLLM Grounding LLM Interpretability
Tel Aviv University
Cutting AI Datacenter Energy with Reinforcement Learning: Measured Power Control of LLM Training from One GPU to the Fleet
Jul 27, 2026Eliseo CurcioRL Control Energy-Efficient ML
Advanced Department of Artificial Intelligence and Energy – New York
Stable FP4 Training via Transposition-Invariant Block Quantization
Jul 27, 2026Mehdi Rahimifar, Amin Darabi, Mehran Taghian Jazi +6Mixed-Precision Quantization 4-Bit Quantization
Hierarchical Grading in Large Language Models
Jul 23, 2026T. ShaskaTransformer LLM Training
SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD
Jul 22, 2026Dongfang Li, Xiaodong Luo, Ruoyu Sun +62Communication-Efficient Distributed Training LLM Post-Training
Solar Open 2 Technical Report
Jul 22, 2026Sungrae Park, Sanghoon Kim, Gyoungjin Gim +50Mixture-of-Experts Language Models Long-Context Language Modeling
Training Continuous Chain of Thought Models: A Tale of Two Regimes
Jul 18, 2026Varun Yerram, He He, Eunsol ChoiCoT Reasoning Continuous Latent Reasoning
Center for Data Science, New York University
The Information Shadow: Measuring Structural Limits on What Language Models Can Learn
Jul 17, 2026Priyansh Srivastava, Romit ChatterjeeLLM Auditing LLM Training
Sirena Ai · Independent Researcher
The Cost and Network Limits of Space-Based AI Compute
Jul 15, 2026Kees van BerkelLLM Inference Cost-Aware Inference
Technical University Eindhoven, Dept. of Math. & Computer Science P.O. Box 513, 5600 MB Eindhoven, The Netherlands
Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models
Jul 14, 2026Gil Avraham, Violetta Shevchenko, Hadi Mohaghegh Dolatabadi +9Communication-Efficient Distributed Training Fault-Tolerant Distributed Training
Pluralis Research
Function-Aware Fill-in-the-Middle as Mid-Training for Coding Agent Foundation Models
Jul 14, 2026Yubo Wang, Jiarong Liang, Yuxuan Zhang +5Masked Language Modeling Coding Agents
University of Waterloo · 5Vector Institute · University of British Columbia +2
Hidden Decoding at Scale: Latent Computation Scaling for Large Language Models
Jul 9, 2026Aiwei Liu, Cheng Shi, Chuhan Wu +45Language Model Scaling Laws Transformer Attention
No Subspace to Track: Non-Identifiability and Optimizer State in Low-Rank Training
Jul 7, 2026Noel ThomasDeep Learning Optimization Parameter Identifiability
Mohamed bin Zayed University of Artificial Intelligence
Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training
Jul 6, 2026Jingwei Zuo, Cong Zeng, Ilyas Chahed +6Memorization in Language Models Neural Network Memorization
Technology Innovation Institute Abu Dhabi, UAE
Improving LLMs via Validator-to-Generator Alignment
Jul 2, 2026Juan Diego Rodriguez, Jocelyn Zhang, Katrin Erk +1LLM Alignment LLM Reliability
Department of Computer Science, The University of Texas at Austin · Departments of Linguistics and Computer Science, University of Massachusetts Amherst · Department of Computer Science & Center for Data Science, New York University
SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communication
Jul 2, 2026Mingkai Zheng, Junlin Chen, Haotian Xie +1Language Model Pretraining Communication-Efficient Distributed Training
Rutgers University
PHOENIX: Resilient LLM Training with Hot-Swapping via Zero-Overhead Checkpoint
Jul 2, 2026Haotian Xie, Junlin Chen, Mingkai Zheng +2Fault-Tolerant Distributed Training Distributed Training
Rutgers University · George Mason University
CausalMix: Data Mixture as Causal Inference for Language Model Training
Jul 1, 2026Zinan Tang, Yukun Zhang, Shaomian Zheng +6Data Mixture Optimization LLM Training
Tsinghua University · Ant Group · Renmin University of China
Prototype Language Models
Jul 1, 2026Dan Ley, Giang Nguyen, Himabindu Lakkaraju +1LLM Interpretability Language Modeling
1Harvard University · 2Guide Labs Inc. *Work initiated during an internship at Guide Labs. · 2Guide Labs Inc.
One-Step Gradient Delay is Not a Barrier for Large-Scale Asynchronous Pipeline Parallel LLM Pretraining
Jun 29, 2026Philip Zmushko, Egor Petrov, Nursultan Abdullaev +2Muon Optimizer Pipeline Parallelism
Institute of Science and Technology Austria (ISTA), Austria · Yandex, Russia · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab), Russia +2
HSAP: A Hierarchical Sequence-aware Parallelism for Hybrid-Context Generative Models
Jun 29, 2026Songxin Zhang, Zejian Xie, Zhuoyang Song +4Communication-Efficient Distributed Training LLM Training
Department of Statistics and Data Science, Southern University of Science and Technology · International Digital Economy Academy
Smooth Scaling Laws Hide Stepwise Token Learning
Jun 29, 2026Pingjie Wang, Zechen Hu, Peiru Yang +2Language Model Pretraining Language Model Scaling Laws
Dots Studio, Xiaohongshu Inc. · Shanghai Jiao Tong University · Tsinghua university
How much of an LLM-generated clinical corpus is actually new? A production-scale measurement of content redundancy for provenance classification
Jun 28, 2026Ali H. Lazem, William J. TeahanData Provenance Clinical NLP
School of Computer Science and Engineering, Bangor University, Bangor, LL57 2DG, Gwynedd, United Kingdom. · University of Thi-Qar, Nasiriyah, 64001, Iraq.
On the Nonlinearity of Learning Rate Scaling for LLM Training
Jun 28, 2026Zaiwen Yang, Huaqing Zhang, Jing Xu +1Language Model Scaling Laws Hyperparameter Transfer
Tsinghua University · Tsinghua University, Shanghai Qi Zhi Institute
Developmental Trajectories of Situation Modeling and Mentalizing in Transformer Language Models
Jun 26, 2026Pamela D. Rivière, Cameron Jones, Sean TrottLLM Evaluation Theory of Mind
Rutgers University - Newark · Stony Brook University
Mechanism-Driven Monitors for Preemptive Detection of LLM Training Instability
Jun 26, 2026Ruixuan Huang, Yipei Wang, Wenyi Fang +7Fault Detection LLM Training
1HKUST · 2Huawei · 3Independent Researcher
Enhancing Numerical Prediction in LLMs via Smooth MMD Alignment
Jun 26, 2026Zhuo Zuo, Li Yue, Wenhao Zheng +2Numerical Reasoning in Language Models Maximum Mean Discrepancy
College of Computer Science, Sichuan University, Chengdu, China · Dongfang Electric (Chengdu) Academy of Science and Technology Co., Ltd., Chengdu, China · Institute of Medical Information & Library, CAMS & PUMC, Beijing, China

Jul 30, 2026, cs.LGJ/K move · Enter open · S save

Harnessing the Potential of Optimizing Data Mixtures via Bayesian Domain Reweighting

Xiang Yuan, Kaiqing Lei, Zhenyu Jin, Jun Shu+2

School of Mathematics and Statistics, Xi’an Jiaotong University · Northwest University, Shaanxi, China

Data Mixture Optimization LLM Training

Open paper AskPDF arXiv

The performance of Large Language Models (LLMs) is fundamentally influenced by the distributional composition of multi-domain pre-training data. While manual heuristics were prevalent in early models, they increasingly fail to capture the intricate synergies between domains as data complexity grows. To overcome the issue, a dominant approach seeks to fit a proxy function mapping between domain weights and their corresponding validation losses, and then find the optimal domain weights to minimize validation losses. These methods rely on strong structural assumptions, such as rank invariance or scaling laws, which are often violated, resulting in non-negligible estimation bias. A promising approach is to directly optimize the weighting scheme from data. However, it suffers from unstable optimization trajectory and prohibitive computational overhead, limiting its potential to search better domain weights configurations. This paper presents a Bayesian domain weighting method to infer the weights from a Dirichlet distribution via introducing Gamma prior information learned from observations. Experimental results demonstrate that proposed method could achieve stable and efficient domain weights learning, and identifies optimal mixtures while consuming substantially less data than search-based function-fitting methods, revitalizing optimization-based domain weighting for large-scale applications.

Momentum

Related topics

Latest papers 226