LLM Training

LLM: Large Language Model

Overview

Large language model training studies how models acquire language and task abilities, from large-scale next-token pretraining to supervised fine-tuning and preference or reinforcement learning. Common approaches combine data curation, filtering and mixture design with curricula, scaling laws, and optimization methods such as AdamW and matrix-based optimizers. Systems research makes training more practical through low-precision arithmetic, memory-efficient methods, and distributed parallelism, while domain adaptation and synthetic reasoning data target specific capabilities. A central tension is how to improve generalization and reliability while controlling compute and data costs, avoiding memorization and instability, and measuring gains beyond benchmark scores.

Latest papers

CardsList

377 papers38 papers in the last four weeks, down 30% on the four weeks before. 0.3% of all new papers.

  1. OceanCoder: A Verification-Guided Large Language Model for Automated Simulation Modelling in Ocean Engineering

    Nov 19, 2026Can Xue, Changhong He, Ziyi Sun +3Large Language Model-Based SimulationLLM Code Generation

  2. General progress substitutes for specific training data in language models

    Nov 18, 2026Mark Marosi, Peter AntalLLM Training

  3. A Survey on RLOps for Large Language Models

    Nov 17, 2026Lingzhe Zhang, Tong Jia, Chiming Duan +7Reinforcement LearningLLM Training

  4. On the Difficulty of Sequence-Level Supervised Learning for Autoregressive Language Models

    Nov 12, 2026Mathieu Blondel, Michaël E. Sander, Germain Vivier-Ardisson +2Moment MatchingAutoregressive Language Modeling

  5. daVinci-Prover: Agent-Native Mid-Training for Formal Theorem Proving

    Nov 12, 2026Wenhong Zhu, Xuefeng Li, Jiashuo Zhang +4Mid-Training for Language ModelsLarge Language Model-Guided Theorem Proving

  6. Starscream: Reimagining LLM Sharding on AMD Instinct GPUs with GPU Partitioning

    Nov 11, 2026Sumanth Gudaparthi, Smeet Chheda, Ganesh DasikaLLM Inference AccelerationGPU Acceleration

  7. LM-SimBench: A Unified Framework, Dataset, and Benchmark for Large Language Model Training Simulation

    Nov 10, 2026Yu Dong, Hong Li, Qiang Sun +4LLM Training

  8. The WoW: A Large-Scale World Knowledge Corpus for Full-Lifecycle LLM Training

    Nov 10, 2026Tianze Sun, Yongyao Duan, Sam ChenLLM TrainingKnowledge Augmentation for Language Models

  9. RPEG: Training Reasoning Language Models Scalably and Cost-Effectively

    Nov 6, 2026Maciej Besta, Leonard Schmidt, Pierre Pang +3LLM TrainingDistributed Training

  10. Mass Producing Training Data with Surprising Side Effects

    Nov 4, 2026Yaowen Ye, Jacob Steinhardt, Sarah Schwettmann +1LLM TrainingEmergent Misalignment in Language Models

  11. On the Design of Hyper-Connections: Simplification, Stability, and Scalability

    Nov 4, 2026Xiangdong Zhang, Xiaohan Qin, Sunan Zou +8Hyper-ConnectionsEfficient Language Model Training

  12. Scheduling Data to Compute: Hierarchical Data Pipelining for LLM Training on AMD GPUs

    Nov 3, 2026Gengyu Rao, Ghadeer Ahmed H Alabandi, Su Ann Chong +28LLM TrainingGPU Kernel Optimization

  13. Training Specialist LLMs on Your Phone: Forward-Only Optimization via Block-Spectral Coordinates

    Nov 3, 2026Liyan Tan, Yequan Zhao, Ziyue Liu +3On-Device TrainingLLM Training

  14. A Scaling Theory of Multi-Epoch Pretraining: Late-Time Overfitting and Ensembling

    Nov 2, 2026Zhen Yang, Yidi Miao, Blake BordelonLanguage Model Scaling LawsLLM Training

  15. FARO: Function-Space Adaptive Return-to-Risk Optimization via Cross-Block Equal-Risk-Contribution Budget Allocation

    Oct 29, 2026Xiaojun Wu, Cehao Yang, Xuhui Jiang +3Deep Learning OptimizationLLM Training

  16. Training Foundation Models with a Single Algorithm

    Oct 29, 2026Yuda Song, Alvin Zhang, Fahim Tajwar +7LLM TrainingRL for Language Models

  17. The Role of Quantization in Loss Spikes

    Oct 28, 2026Zaiqian Chen, Aditya Somasundaram, Sophie Li +4LLM TrainingLanguage Model Pretraining

  18. Locating Sycophancy Across Training Stages in Open Language Models

    Oct 27, 2026Jiarui Yao, Shweta Parihar, Nghia Tuan Bui +2LLM SycophancyLLM Training

  19. Advancement or Acceleration? How Data Curation Improves LLM Training

    Oct 27, 2026Jiaxing Wang, Zicheng Zhang, Deping Xiang +5Training Data CurationLLM Training