LLM Training

LLM: Large Language Model

Large language model training studies how models acquire language and task abilities, from large-scale next-token pretraining to supervised fine-tuning and preference or reinforcement learning. Common approaches combine data curation, filtering and mixture design with curricula, scaling laws, and optimization methods such as AdamW and matrix-based optimizers. Systems research makes training more practical through low-precision arithmetic, memory-efficient methods, and distributed parallelism, while domain adaptation and synthetic reasoning data target specific capabilities. A central tension is how to improve generalization and reliability while controlling compute and data costs, avoiding memorization and instability, and measuring gains beyond benchmark scores.

Latest papers

353 papers62 papers in the last four weeks, up 94% on the four weeks before. 0.4% of all new papers.

Alignment midtraining for animals
Mar 21, 2026Jasmine Brazilek, Miles TidmarshLLM Alignment LLM Training
Compassion Aligned Machine Learning
Mi:dm K 2.5 Pro
Mar 19, 2026KT Tech innovation GroupLLM Training LLM Tool Use
Tech. Innovation Group, KT
Humans and transformer LMs: Abstraction drives language learning
Mar 18, 2026Jasper Jian, Christopher D. ManningLLM Training Language Model Pretraining
Stanford University
Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Training
Mar 17, 2026Elnaz Rahmati, Nona Ghazizadeh, Zhivar Sourati +2LLM Training Continual Learning for LLMs
University of Southern California
PRISM: Demystifying Retention and Interaction in Mid-Training
Mar 17, 2026Bharat Runwal, Ashish Agrawal, Anurag Roy +1LLM Training Scientific Reasoning in Language Models
IBM Research, MIT-IBM Watson AI Lab
TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities
Mar 17, 2026Victoria Graf, Valentina Pyatkin, Nouha Dziri +2Multi-Turn Dialogue Evaluation Multi-Turn Dialogue Management
University of Washington · Allen Institute for AI
Towards Next-Generation LLM Training: From the Data-Centric Perspective
Mar 16, 2026Hao Liang, Zhengyang Zhao, Zhaoyang Han +8LLM Training Efficient Language Model Training
Peking University, Beijing, China · Hong Kong University of Science and Technology, Hong Kong SAR, China · Institute for Advanced Algorithms Research, Shanghai, China +2
Drift and selection in LLM text ecosystems
Mar 15, 2026Søren RiisModel Collapse LLM Training
Queen Mary University of London, London, United Kingdom
FLUX: Data Worth Training On
Mar 14, 2026Gowtham, Sai Rupesh, Sanjay Kumar +2Training Data Curation LLM Training
Language Generation with Replay: A Learning-Theoretic View of Model Collapse
Mar 12, 2026Giorgio Racca, Michal Valko, Amartya SanyalModel Collapse Data Contamination in Language Models
University of Copenhagen · Isara Labs
Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Data
Mar 10, 2026Hyunji Nam, Haoran Li, Natasha JaquesLLM Personalization Mutual Information Maximization
Stanford University · Character AI · University of Washington
Mashup Learning: Faster Finetuning by Remixing Past Checkpoints
Mar 10, 2026Sofia Maria Lo Cicero Vaina, Artem Chumachenko, Max RyabininLLM Fine-Tuning LLM Training
Together AI.
Lost in Backpropagation: The LM Head is a Gradient Bottleneck
Mar 10, 2026Nathan Godey, Yoav ArtziLLM Training Efficient Language Model Training
Cornell University, New York, USA
You Didn't Have to Say It like That: Subliminal Learning from Faithful Paraphrases
Mar 10, 2026Isaia Gisler, Zhonghao He, Tianyi QiuSubliminal Learning LLM Training
ETH Zürich · University of Cambridge · Peking University
HTMuon: Improving Muon via Heavy-Tailed Spectral Correction
Mar 10, 2026Tianyu Pang, Yujie Fang, Zihang Liu +4Muon Optimizer LLM Training
Dartmouth College · Microsoft · International Computer Science Institute +2
Capacity-Aware Mixture Law Enables Efficient LLM Data Optimization
Mar 9, 2026Jingwei Li, Xinran Gu, Jingzhao ZhangData Mixture Optimization LLM Training
Institute for Interdisciplinary Information Sciences, Tsinghua University · Shanghai Qizhi Institute
Unlocking Data Value in Finance: A Study on Distillation and Difficulty-Aware Training
Mar 7, 2026Chuxue Cao, Honglin Lin, Zhanping Zhong +5Financial Language Model Evaluation LLM Training
Shanghai Artificial Intelligence Laboratory, OpenDataLab, OpenDataArena · Hong Kong University of Science and Technology
Addressing the Ecological Fallacy in Larger LMs with Human Context
Mar 6, 2026Nikita Soni, Dhruv Vijay Kunjadiya, Pratham Piyush Shah +3Language Modeling LLM Training
Department of Computer Science, Stony Brook University · College of Connected Computing, Vanderbilt University, USA
Orion: Characterizing and Programming Apple's Neural Engine for LLM Training and Inference
Mar 6, 2026Ramchand KumaresanNeural Network Hardware Acceleration On-Device Training
POET-X: Memory-efficient LLM Training by Scaling Orthogonal Transformation
Mar 5, 2026Zeju Qiu, Lixin Liu, Adrian Weller +2LLM Training Memory-Efficient Training
The Chinese University of Hong Kong · MPI for Intelligent Systems · University of Cambridge +1
Progressive Residual Warmup for Language Model Pretraining
Mar 5, 2026Tianhao Chen, Xin Xu, Lu Yin +4Language Model Pretraining LLM Training
The Hong Kong University of Science and Technology, Hong Kong, China · University of Surrey, Guildford, UK · The University of Hong Kong, Hong Kong, China +1
.tmu: A Low-Entropy Tree-Structured Representation for LLM-Assisted Scientific Writing
Mar 3, 2026Tianyou Liu, Ziqiang Li, Xurui Liu +2AI-Assisted Scientific Research LLM Training
Southern University of Science and Technology · Alibaba · Tsinghua University +2
Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs
Mar 3, 2026Wuyue Zhang, Chongdong Huang, Chunbo You +3Low-Precision Training LLM Training
Zhejiang Lab, Hangzhou, China
Generative AI & Fictionality: How Novels Power Large Language Models
Mar 1, 2026Edwin Roland, Richard Jean SoText Generation LLM Training
School of Information Science University of Illinois Urbana-Champaign · Department of English Duke University
Thoth: Mid-Training Bridges LLMs to Time Series Understanding
Mar 1, 2026Jiafeng Lin, Yuxuan Wang, Jialong Wu +3Temporal Reasoning in Language Models LLM Training
School of Software, BNRist, Tsinghua University.
Improving Automatic Summarization of Radiology Reports through Mid-Training of Large Language Models
Feb 28, 2026Mengxian Lyu, Cheng Peng, Ziyi Chen +3Text Summarization Medical LLMs
Department of Health Outcomes and Biomedical Informatics, University of Florida, Gainesville, FL, USA · Department of Engineering Education, University of Florida, Gainesville, FL, USA · Department of Health Outcomes and Biomedical Informatics, Preston A. Wells, Jr. Center for Brain Tumor Therapy, University of Florida, Gainesville, FL, USA
Distribution-Aware Companding Quantization of Large Language Models
Feb 27, 2026Athul Radhakrishnan, Siddhant Mohan, Mahima SachdevaLLM Training Multi-Token Prediction
How Large Language Models Get Stuck: Early structure with persistent errors
Feb 27, 2026Alokesh Manna, William Snyder, Whitney TaborLanguage Modeling LLM Training
Department of Statistics University of Connecticut, Storrs, USA · Department of Linguistics University of Connecticut, Storrs, USA · Department of Psychological Sciences University of Connecticut, Storrs, USA
Semantic Tube Prediction: Beating LLM Data Efficiency with JEPA
Feb 26, 2026Hai Huang, Yann LeCun, Randall BalestrieroEfficient Language Model Training LLM Training
Atlassian · NYU · Brown
Efficient Scaling of LLM Training with Flexible Context Parallelism
Feb 25, 2026Yifan Niu, Han Xiao, Dongyi Liu +2Context Parallelism Efficient Language Model Training
The Hong Kong University of Science and Technology (Guangzhou) · Huawei Technologies Co., Ltd. · The Hong Kong University of Science and Technology

Mar 21, 2026J/K move · Enter open · S save

Alignment midtraining for animals

Jasmine Brazilek, Miles Tidmarsh

Compassion Aligned Machine Learning

LLM Alignment LLM Training Moral Reasoning in Language Models

Open paper AskPDF arXiv

We investigate the robustness of value alignment via midtraining with synthetic documents, using animal compassion as a value that is both important in its own right and orthogonal to existing alignment efforts. To evaluate compassionate reasoning, we develop and publicly release Animal Norms In Moral Assessment (ANIMA), a 26-question evaluation spanning 13 ethical dimensions, publicly available as a dataset and Inspect evaluation. On ANIMA, training with 3000 documents achieves 77% compared to 40% for instruction-tuning approaches, with generalization to human compassion and no degradation in standard safety benchmarks or capabilities. However, subsequent unrelated instruction-tuning degrades the intervention, with the advantage disappearing after 5000 samples. Our exploratory results suggest document-based value interventions may require explicit preservation strategies to remain effective through typical training pipelines.