LLM Training

LLM: Large Language Model

Ask about this topic

Momentum

31 papers in the last four weeks, up 72% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28
View on map

Latest papers 226

All topics
CardsList

  1. Fault-tolerant foundation models

    Oct 7, 2026Trevor McCourt, Ila R. Fiete, Isaac L. ChuangLLM TrainingLanguage Model Scaling Laws

    Department of Electrical Engineering and Computer Science, MIT · Department of Brain and Cognitive Sciences, McGovern Institute, MIT · Department of Physics, MIT

  2. Algorithmic Scratchpads and Curriculum Staging for Arithmetic Reasoning in Tiny Transformers

    Oct 6, 2026Sourabh KasliwalNumerical Reasoning in Language ModelsNeural Algorithmic Reasoning

  3. Dynamic Minimax Regret Optimization for Robust LLM Post-Training

    Oct 5, 2026Chengbo Zang, Haoyu Dong, Mehmet Kerem Turkcan +3Dynamic Regret MinimizationDistributionally Robust Optimization

    Department of Electrical Engineering, Columbia University · Department of Computer Science & Engineering, University of California, Santa Cruz · Department of Civil Engineering, Columbia University

  4. ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training

    Oct 5, 2026Yuanshi Liu, Boyuan Jiang, Liang Hou +4Spectral RegularizationDeep Learning Optimization

    School of Intelligence Science and Technology, Peking University · Kling Team

  5. CIPHER-MoE: Balancing Efficiency and Routing Fidelity in Trillion-Scale MoE Training

    Oct 5, 2026Jing Li, Jian Meng, Yingmeng Gao +8Expert Load BalancingExpert Routing

    Tongji University · Cornell University · Harbin Institute of Technology, Shenzhen +1

  6. Invent a Dataset: Measuring dataset generation abilities with zero seed

    Oct 1, 2026Shivalika Singh, Andrija Djurisic, Gbemileke Onilude +2Synthetic Data GenerationSynthetic Data Generation for Language Models

    Adaption

  7. AF-Muon: An AdamW-Free Muon Optimizer for Tied-Embedding Models

    Oct 1, 2026Arash Lagzian, Paniz Halvachi, Junming Zhang +2Memory-Efficient OptimizationMuon Optimizer

    National University of Singapore, Singapore · Independent Researcher · LUMIA Lab, School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China

  8. Persistent Depth Ordering amid Shifting Block-Bypass Responses in Language Model Pretraining

    Oct 1, 2026Shengye Tao, Yinzhu Cheng, Haihua XieLLM InterpretabilityCausal Interventions in Language Models

    Beijing University of Civil Engineering and Architecture · Beijing Institute of Mathematical Sciences and Applications (BIMSA) · Institute of Statistics and Big Data, Renmin University of China

  9. Leto: Fast In-Place Recovery for LLM Training on Surviving Hardware

    Sep 30, 2026Geon-Woo Kim, Joon Ha Kim, Daehyeok KimFault-Tolerant Distributed TrainingLLM Training

    The University of Texas at Austin Austin, Texas, USA

  10. Closing the Loop: Practical Training Recipes for Looped Language Models

    Sep 30, 2026Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5Efficient Language Model TrainingLLM Training

    RND NLP, DAIMLD, Russian Federation

  11. Forking: Sudden Overfitting Under Replay

    Sep 30, 2026Shanbin Yu, Shaoyang Guo, Haoran Zhao +2Neural Network GeneralizationLLM Training

    MetaCircle (元环智能) · Tsinghua University · Peking University +1

  12. MASCRDM: Multi-Agent System for Compliance Risk Detection and Mitigation in Training Process of Large Language Models

    Sep 30, 2026Yan Zhang, Chuming Wei, Ruien Li +3LLM SafetyLanguage Model Bias Evaluation

    Department of Computer Science and Technology, Tsinghua University · Academy of Artificial Intelligence and Advanced Technology, Xi’an Jiaotong-Liverpool University · Department of Computer Sciences, University of Wisconsin–Madison +1

  13. Multi-LLM Collaborative Alignment via Stackelberg Games

    Sep 30, 2026Christina Hahn, Shangbin Feng, Dean Light +3RL for Language ModelsLLM Alignment

    University of Washington · Amazon · University of British Columbia

  14. Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectories

    Sep 29, 2026Zhehao Huang, Changxin Tian, Qingyuan Yang +5Language Model ScalingModel Merging

    Ling Team, Ant Group · Shanghai Jiao Tong University

  15. ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models

    Sep 29, 2026Jingnan Pu, Zi-En Fan, Feng LianMemory-Augmented Language ModelsJoint-Embedding Predictive Architecture

    School of Automation Science and Technology, Xi’an Jiaotong University No. 28, West Xianning Road, Xi’an, Shaanxi 710049, China

  16. Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training

    Sep 29, 2026Zixuan Gong, Zeyu Gan, Jiaye Teng +1Language Model PretrainingMatrix Optimization

    Gaoling School of Artificial Intelligence Renmin University of China · School of Statistics and Management Shanghai University of Finance and Economics

  17. AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization

    Sep 29, 2026Pengyu He, Yan Zhang, Ruien Li +1Communication-Efficient Distributed TrainingDistributed Training

    Department of Electrical Engineering, California Institute of Technology · Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison

  18. Principled Thoughts for Latent Recursive LLM Systems

    Sep 28, 2026Fahd Seddik, Fatemeh FardContinuous Latent ReasoningMulti-Agent LLMs

    FARD Lab, University of British Columbia, Okanagan, Canada

  19. Telescopic Language Models

    Sep 28, 2026Zhilin Guo, Boqiao Zhang, Hakan Aktas +14Language ModelingEfficient Language Model Inference

    University of Cambridge · University of British Columbia · Google

  20. MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining

    Sep 28, 2026Chang-Wei Shi, Xu Wang, Wu-Jun LiLanguage Model PretrainingEfficient Language Model Training

    National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China

  21. Nereus: Adaptive Parallelism for LLM Post-Training

    Sep 28, 2026Songlin Jiang, Tuo Shi, Sitong Zhang +3LLM Inference SchedulingRL Post-Training

    Aalto University · Shenzhen University of Advanced Technology · Zhejiang University

  22. Commutator Memory: Sparse, Path-Local Reading and Steering in Language Models

    Sep 28, 2026John SweeneyLLM Training

    Sideplane AI

  23. Direct Self-Evolving Optimization: Evolving LLMs without Challenger Training

    Sep 28, 2026Yuyang Deng, Yu Wang, Jiayun WangSelf-Training for Language ModelsLarge Language Model-Guided Optimization

    Accenture, Center for Advanced AI · Georgia Institute of Technology

  24. Training and Inference Dynamics of PLDR-LLMs: Row-Map Collapse, Renormalization, and Predictive Reduction

    Sep 28, 2026Burc GokdenLLM InferenceLanguage Model Scaling Laws

    Fromthesky Research Labs LLC, Oregon, USA

  25. Technical Manual for Toolkit for Confidence-Corpus Consistency, Corpus Absorption and Rule Learning via Fine-Tuning on a Fabricated Corpus

    Sep 23, 2026José Luciano Verçosa Marques, Frederico Jorge Heitmann, Daniel Omar Perez +3LLM Training

    Center for Electric Mobility Research (CEMOBE) / Power Electronics Laboratories (LEPO), University of Campinas (Unicamp) · Institute of Computing (IC), University of Campinas (Unicamp) · Center for Logic, Epistemology and History of Science (CLE), University of Campinas (Unicamp) +1

  26. Direct Optimization of Generators for Search in Automated Theorem Proving

    Sep 22, 2026Adam Ousherovitch, Ambuj TewariInference-Time SearchAutomated Theorem Proving

    Department of Statistics University of Michigan

  27. The Life of a Token: from Words to Bits on the Wire

    Sep 17, 2026Davide Avesani, Pengwenlong Gu, Sotiris Skaperas +1Communication-Efficient Distributed TrainingHigh-Performance Computing

    CEDRIC - ROC · cnam · Cnam

  28. Learn Your Own Thoughts: Abstract Token Curriculum

    Sep 17, 2026Khashayar Gatmiry, Avrajit Ghosh, Parsa Mirtaheri +4CoT ReasoningCurriculum Learning

    UC San Diego · Google DeepMind and UC Berkeley

  29. Preventing Model Collapse: A Fisher-Rao Perspective on the Dynamics of Training with Synthetic Data

    Sep 16, 2026Matteo Marchi, João Pedro Silvestre, Bahman Gharesifard +1Synthetic Data PretrainingModel Collapse

    Electrical and Computer Engineering Department, University of California at Los Angeles, Los Angeles, CA 90095 USA · Department of Mathematics and Statistics at Queens University, Kingston, ON, Canada

  30. MiST: Mid-Training LLMs for Cybersecurity

    Sep 16, 2026Oded Ovadia, Elad Ben Zaken, Elad Guttman +1Synthetic Data PretrainingDomain-Adaptive Pretraining

    Dream

  31. GraphEcho: Structural Redundancy and Evidence Provenance in LLM Graph Agents

    Sep 15, 2026Sikun Wang, Yixi Zhou, Lei Fan +1LLM Reasoning with GraphsLLM Agent Evaluation

    Tokyo University of Science · Hong Kong Baptist University · University of Michigan +1

  32. MoARa: Module-Aware Rank Allocation and Structure-Preserving Decomposition for Low-Rank LLM Pre-training

    Sep 14, 2026Keunyoung Kim, Nojun KwakLanguage Model PretrainingLow-Rank Approximation

    Seoul National University

  33. Characterizing Job Power Elasticity for Power-Flexible AI Training

    Sep 12, 2026Philip Colangelo, Charles Dawson, Shayan Sengupta +2Energy-Efficient MLEfficient Language Model Training

    Emerald AI

  34. Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data

    Sep 10, 2026Atindra Jha, Margaret Li, Jure Leskovec +2Mixture of ExpertsSparse Mixture-of-Experts

    Stanford University · Paul G. Allen School of Computer Science, University of Washington

  35. Rebalancing Token Importance in Language Models with TF-IDF Weighted Cross-Entropy Loss

    Sep 10, 2026Zhijian Li, Stefan Larson, Kevin LeachMemorization in Language ModelsLLM Fine-Tuning

    University of Southern California, Los Angeles · Vanderbilt University, Nashville

  36. Parallelism Strategy Chaining for Fast Training Convergence

    Sep 7, 2026Minchul Kang, Changyong Shin, Younghun Go +4Efficient Language Model TrainingLLM Training

    Department of Computer Science and Engineering, Korea University, Seoul, South Korea

  37. Language, Language Models, and What We're Talking About

    Sep 3, 2026Malvina NissimLanguage ModelingLLM Training

    CLCG, University of Groningen, The Netherlands

  38. BASP: Communication-Efficient Batch-Aware Sequence Parallelism for LLM Training

    Sep 2, 2026Bigyan Ghimire, Jon C. CalhounCommunication-Efficient Distributed TrainingLong-Context Language Modeling

    Holcombe Department of Electrical and Computer Engineering Clemson University Clemson, USA

  39. Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall

    Sep 1, 2026Jacqueline He, Howard Yen, Shuyue Stella Li +9Language Model DistillationFactual Knowledge in Language Models

    Meta AI · University of Washington · Princeton University

  40. Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search

    Sep 1, 2026Zhiliang Chen, Sebastian Ament, David Eriksson +4Bayesian OptimizationLanguage Model Scaling Laws

    Meta · National University of Singapore · Atomic Machines; work done at Meta

  41. When Tokenization is Secretly Output Supervision

    Sep 1, 2026Tanja Baeumel, Josef van Genabith, Simon OstermannLLM Training

    1German Research Center for AI (DFKI) · Saarland University · Center for European Research in Trusted AI (CERTAIN)

  42. Lagged Coupling: Internal Representations Become Readable Before They Become Causal

    Sep 1, 2026Xining XunCausal InterpretabilityCausal Interventions in Language Models

    Tsingjiao Information Science (Beijing) Co., Ltd.

  43. Instella-MoE Technical Report

    Sep 1, 2026Jiang Liu, Sudhanshu Ranjan, Prakamya Mishra +10Mixture-of-Experts Language ModelsLanguage Model Post-Training

  44. TrainSDC: Characterizing and Mitigating Silent Data Corruption in Large Language Model Training

    Aug 31, 2026Zhipeng Xia, Haotian Xu, Siyu Yun +4Fault-Tolerant Distributed TrainingLanguage Model Robustness

    Zhejiang University · Huawei

  45. On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

    Aug 31, 2026Zihan Qiu, Zekun Wang, Xiao Li +33Efficient Language Model TrainingSparse Attention

    Qwen Team

  46. Context Staircase: Signature-Aligned Dynamics of Token Embeddings under Small Initialization

    Aug 31, 2026Junjie Yao, Liangkai Hang, Zhi-Qin John XuLanguage Model PretrainingWord Embeddings

    School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China · Institute of Natural Sciences, School of Mathematical Sciences Shanghai Jiao Tong University Shanghai 200240, China

  47. CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents

    Aug 31, 2026Amir Saeidi, Zehua Zhang, Rishitosh Singh +6LLM AgentsLLM Agent Training

    Arizona State University · Cisco Research

  48. DataFoundry: Evolving Data Preparators via Recursive Self-Improvement

    Aug 30, 2026Cehao Yang, Xiaojun Wu, Xueyuan Lin +4Training Data CurationDomain Adaptation for LLMs

    1IDEA Research, International Digital Economy Academy · 2Hong Kong University of Science and Technology (Guangzhou) · 3DataArc Tech Ltd.

  49. DFM Mimir v1: An Open HRM Delivering Frontier Performance at 1B Parameters Using Only Permissible Post-Training Data

    Aug 13, 2026Peter Schneider-Kamp, Jacob Nielsen, Gianluca Barmina +2Language ModelingLLM Training

    University of Southern Denmark · Ordbogen A/S · Aarhus University

  50. Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model

    Aug 13, 2026Mohammed Sabry, Sean Augenstein, Keith Rush +1Language Model PretrainingEfficient Language Model Training

    Google, Mountain View, CA, USA · School of Computing, Dublin City University, Dublin, Ireland.

  51. Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge

    Aug 12, 2026Arda Uzunoglu, Benjamin Van Durme, Daniel KhashabiLong-Context Language ModelingFactual Knowledge in Language Models

    Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA

  52. LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training

    Aug 12, 2026Xiaojun Wu, Cehao Yang, Honghao Liu +5Efficient Language Model TrainingLLM Training

    1IDEA Research · 2The Hong Kong University of Science and Technology (Guangzhou) · 3DataArcTech Ltd.

  53. Decodable But Not Detachable: Training Data Granularity Determines Parametric Modularity in Large Language Models

    Aug 10, 2026Marcus Armstrong, Navid Ayoobi, Arjun MukherjeeLLM InterpretabilityCircuits in Language Models

    Department of Computer Science University of Houston Houston, TX 77204

  54. Fusion Training for Mathematical Generalization in Large Language Models

    Aug 10, 2026Congfeng Cao, Pengyu Zhang, Jelke BloemLLM Training

    Institute for Logic, Language and Computation, University of Amsterdam · INDE Lab, University of Amsterdam

  55. Instability of LLM Pre-Pretraining: It Doesn't Always Help. An Investigation on Multiple Languages

    Aug 9, 2026Sofiia Riazhskykh, Nam Luu, Ondřej BojarLanguage Model PretrainingLinguistic Typology

    Charles University, Faculty of Mathematics and Physics

  56. Scaling Inherently Interpretable Language Models

    Aug 6, 2026Guide Labs Team, Andreas Madsen, Aya Abdelsalam Ismail +7Language Model SteeringLLM Interpretability

    Guide Labs Team

  57. OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

    Aug 5, 2026Indraneil Paul, Falko Helm, Goran Glavaš +1Software EngineeringLong-Context Language Modeling

    UKP Lab & Center for Applied Cybersecurity ATHENE, TU Darmstadt · CAIDAS, University of Würzburg

  58. Training nGPT

    Aug 2, 2026Ilya Loshchilov, Boris GinsburgHyperspherical Representation LearningEfficient Language Model Training

    NVIDIA

  59. AdaMTP: An Adaptive Training Paradigm for Multi-Token Prediction

    Aug 1, 2026Ziqiang Cui, Han Shi, Bowei He +8Multi-Token PredictionLLM Inference Acceleration

    City University of Hong Kong · Huawei Technologies · Mohamed bin Zayed University of Artificial Intelligence +2

  60. Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training

    Jul 30, 2026Jiawen Tao, Miao Peng, Yaoming Li +7Language Model PretrainingSynthetic Data Pretraining

    Hunyuan Team, Tencent · Peking University · The Hong Kong University of Science and Technology (Guangzhou)