LLM Training

LLM: Large Language Model

Overview

Large language model training studies how models acquire language and task abilities, from large-scale next-token pretraining to supervised fine-tuning and preference or reinforcement learning. Common approaches combine data curation, filtering and mixture design with curricula, scaling laws, and optimization methods such as AdamW and matrix-based optimizers. Systems research makes training more practical through low-precision arithmetic, memory-efficient methods, and distributed parallelism, while domain adaptation and synthetic reasoning data target specific capabilities. A central tension is how to improve generalization and reliability while controlling compute and data costs, avoiding memorization and instability, and measuring gains beyond benchmark scores.

Latest papers

CardsList

377 papers38 papers in the last four weeks, down 30% on the four weeks before. 0.3% of all new papers.

  1. HetCCL: Efficient LLM Training on Heterogeneous Vendor GPUs

    Jan 30, 2026Heehoon Kim, Chris Jaehwan Lee, Taejeoung Kim +6Heterogeneous ComputingDistributed Training

  2. Self-Improving Training & Thinking

    Jan 29, 2026Ellen Xiaoqing Tan, Jack Lanchantin, Zichun Yu +11Mid-Training for Language ModelsLLM Training

  3. Quantifying the Effect of Test Set Contamination on Generative Evaluations

    Jan 7, 2026Rylan Schaeffer, Joshua Kazdan, Baber Abbasi +8Language Model Generation EvaluationBenchmark Contamination

  4. Distilling the Essence: Efficient Reasoning Distillation via Sequence Truncation

    Dec 24, 2025Wei-Rui Chen, Vignesh Kothapalli, Ata Fatahibaarzi +5CoT DistillationSelective Knowledge Distillation

  5. A Causal Perspective on the Role of Training Dynamics for Interpreting Privacy Risks in Code Models

    Dec 8, 2025Hua Yang, Alejandro Velasco, Sen Fang +2Privacy Leakage in Language ModelsLLM Training

  6. EDGC: Entropy-driven Dynamic Gradient Compression for Efficient LLM Training

    Nov 13, 2025Qingao Yi, Jiaang Duan, Jun Zhang +5Communication-Efficient Distributed TrainingGradient Compression

  7. LLMs Can Get "Brain Rot": A Pilot Study on Twitter/X

    Oct 15, 2025Shuo Xing, Junyuan Hong, Yifan Wang +5LLM TrainingLLM Reliability

  8. The Environmental Impacts of Language Model Training Keep Rising Now is the Time to Catch Impacts on the Rebound

    Oct 10, 2025Clément Morand, Anne-Laure Ligozat, Aurélie NévéolEnergy-Efficient MLLife Cycle Assessment

  9. Geometrically Principled Randomized Optimization for Efficient LLM Training

    Oct 2, 2025Sahar Rajabi, Nayeema Nonta, Sirisha RambhatlaLanguage Model PretrainingDeep Learning Optimization

  10. Chronoberg: A 250-Year Diachronic Corpus with Temporal Affective Lexicons for Language Model Analysis

    Sep 26, 2025Niharika Hegde, Subarnaduti Paul, Lars-Joel Frey +4Language ModelingSemantic Change Detection

  11. Open-sci-ref-0.02: open reference baselines for reproducible language model and dataset comparison

    Sep 10, 2025Marianna Nezhurina, Harsh Raj, Victor May +7LLM TrainingLLM Evaluation

  12. Understanding the Interplay between Memorization and Learning in Large Language Models

    Jul 20, 2025Bishwamittra Ghosh, Soumi Das, Qinyuan Wu +4Memorization in Language ModelsLLM Training

  13. Learning Composable Chains-of-Thought

    May 28, 2025Fangcong Yin, Zeyu Leo Liu, Liu Leqi +2LLM Fine-TuningCoT Reasoning

  14. ZenFlow: Enabling Stall-Free Offloading for LLM Training

    May 18, 2025Tingfeng Lan, Yusen Wu, Bin Ma +7LLM TrainingLLM Fine-Tuning

  15. Learning by Surprise: Adaptive Mitigation of Model Collapse in Large Language Models

    Oct 16, 2024Daniele Gambetta, Gizem Gezici, Fosca Giannotti +3Model CollapseTraining Data Selection

  16. Thought-Like-Pro: Enhancing Reasoning of Large Language Models through Self-Bootstrapped Prolog-based Chain-of-Thought

    Jul 18, 2024Xihe Qiu, Yongxin Deng, Xiaoyu Tan +2Symbolic Reasoning in Language ModelsLLM Training