LLM Training

LLM: Large Language Model

Large language model training studies how models acquire language and task abilities, from large-scale next-token pretraining to supervised fine-tuning and preference or reinforcement learning. Common approaches combine data curation, filtering and mixture design with curricula, scaling laws, and optimization methods such as AdamW and matrix-based optimizers. Systems research makes training more practical through low-precision arithmetic, memory-efficient methods, and distributed parallelism, while domain adaptation and synthetic reasoning data target specific capabilities. A central tension is how to improve generalization and reliability while controlling compute and data costs, avoiding memorization and instability, and measuring gains beyond benchmark scores.

Latest papers

353 papers62 papers in the last four weeks, up 94% on the four weeks before. 0.4% of all new papers.

Normalized Architectures are Natively 4-Bit
May 7, 2026Maxim Fishman, Brian Chmiel, Ron Banner +2LLM Quantization Neural Representation Geometry
∧NVIDIA · ◦Department of Electrical and Computer Engineering - Technion, Haifa, Israel
Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization
May 7, 2026Andy Zeyi Liu, Elliot Paquette, John SousRepresentation Learning Decoder-Only Language Models
Department of Applied Physics, Yale University, New Haven, CT 06511, United States · Department of Mathematics and Statistics, McGill University · Department of Applied Physics, Yale University, New Haven, Connecticut 06511, USA +1
When2Speak: A Dataset for Temporal Participation and Turn-Taking in Multi-Party Conversations for Large Language Models
May 7, 2026Vihaan Nama, Shreya Mendi, Zian Ye +1Synthetic Data Generation Synthetic Data Generation for Language Models
Pratt School of Engineering Duke University
FinRAG-12B: A Production-Validated Recipe for Grounded Question Answering in Banking
May 6, 2026Denys Katerenchuk, Pablo Duboue, Keelan Evanini +6Financial Services LLM Grounding
Kasisto, New York, NY · Textualization, Vancouver, Canada · NBME, Philadelphia, PA
Perturbation is All You Need for Extrapolating Language Models
May 5, 2026Zetai Cen, Jin Zhu, Xinwei Shen +1OOD Generalization Autoregressive Language Modeling
School of Mathematics, University of Bristol · School of Mathematics, University of Birmingham · Department of Statistics, University of Washington +1
Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer
May 5, 2026Jinghui Yuan, Jiaxuan Zou, Shuo Wang +2Neural Network Optimization Matrix Optimization
1. School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University. · 2. School of Mathematics and Statistics, Xi’an Jiaotong University. · 4. Gaoling School of Artificial Intelligence, Renmin University of China. +1
Model Spec Midtraining: Improving How Alignment Training Generalizes
May 3, 2026Chloe Li, Nevan Wichers, Sara Price +2LLM Alignment LLM Safety Alignment
Anthropic Fellows Program · Anthropic
Activation Compression in LLMs: Theoretical Analysis and Efficient Algorithm
May 2, 2026Wen-Da Wei, Han-Bin Fang, Yang-Di Liu +3LLM Compression Gradient Compression
Nanjing University, Nanjing, China · Tsinghua University, Beijing, China · Huazhong University of Science and Technology, Wuhan, China +1
AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs
May 1, 2026Wenxiang Lin, Juntao Huang, Luhan Zhang +5LLM Quantization Communication-Efficient Distributed Training
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Huawei Technologies Ltd.
Assert, don't describe: Linguistic features that shift LLM reasoning about animal welfare
Apr 30, 2026Jasmine Brazilek, Harper DunnMoral Reasoning in Language Models Language Model Post-Training
Compassion Aligned Machine Learning (CaML) · Independent researcher
Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling
Apr 30, 2026Ansar Aynetdinov, Patrick Haller, Alan AkbikLanguage Model Pretraining Efficient Language Model Training
Humboldt-Universität zu Berlin
ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
Apr 30, 2026Wenxiang Lin, Xinglin Pan, Ruibo Fan +2Communication-Efficient Distributed Training Lossless Data Compression
Harbin Institute of Technology, Shenzhen, China · The Hong Kong University of Science and Technology (Guangzhou), China · The Hong Kong University of Science and Technology, Hong Kong SAR
XekRung Technical Report
Apr 30, 2026Jiutian Zeng, Junjie Li, Chengwei Dai +13LLM Fine-Tuning Cybersecurity
Alibaba Security AGI Lab
ChipLingo: A Systematic Training Framework for Large Language Models in EDA
Apr 30, 2026Lei Li, Xingwen Yu, Jianguo Ni +4Knowledge Augmentation for Language Models LLM Fine-Tuning
Ickylin AI Team
AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
Apr 29, 2026Ahan Gupta, Zhihao Wang, Neel Dani +3Efficient Language Model Training LLM Training
SSAIL Lab, University of Illinois Urbana-Champaign Champaign, IL, USA · Anyscale San Francisco, CA, USA · Snowflake San Mateo, CA, USA
What Kind of Language is Easy to Language-Model Under Curriculum Learning?
Apr 29, 2026Nadine El-Naggar, Tatsuki Kuribayashi, Ted BriscoeInductive Bias Linguistic Bias in Language Models
Mohamed bin Zayed University of Artificial Intelligence
From Architecture to Output: Structural Origins of Hallucination in Large Language Models and the Amplifying Role of Data
Apr 29, 2026Md. Rejaul Korim Sadi, Toufiqur Rahman Tasin, Golam Mostofa NaeemHallucination in Language Models LLM Reliability
*Dept. of Computer Science and Engineering, Metropolitan University, Sylhet, Bangladesh
DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
Apr 29, 2026Tianhao Hu, Xiangcheng Liu, Youshao Xiao +15RL for Language Models Asynchronous RL
Meituan, China
LegalMidm: Use-Case-Driven Legal Domain Specialization for Korean Large Language Model
Apr 28, 2026Youngjoon Jang, Chanhee Park, Hyeonseok Moon +5Legal NLP LLM Fine-Tuning
Department of Computer Science and Engineering, Korea University · KT (Korea Telecom)
Programming with Data: Test-Driven Data Engineering for Self-Improving LLMs from Raw Corpora
Apr 27, 2026Chenkai Pan, Xinglong Xu, Yuhang Xu +6LLM Evaluation Knowledge Augmentation for Language Models
Zhejiang University · University of Chinese Academy of Sciences · Shanghai Artificial Intelligence Laboratory
Intrinsic Mutual Information as a Modulator for Preference Optimization
Apr 27, 2026Peng Liao, Peijia Zheng, Lingbo Li +2LLM Alignment Offline Preference Optimization
1Sun Yat-sen University · University of Warwick · 3Macao Polytechnic University
CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training
Apr 27, 2026Rezaul Karim, Austin Wen, Wang Zongzuo +3Communication-Efficient Distributed Training Distributed Optimization
Toronto Ascend Team
DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
Apr 26, 2026DeepSeek-AI, Anyi Xu, Bangcai Lin +316Mixture-of-Experts Language Models Long-Context Language Model Inference
Scaling Point-in-Time Language Models: Enabling Economic Evaluation of Embeddings
Apr 24, 2026Bryan Kelly, Semyon Malamud, Johannes Schwab +1Decoder-Only Language Models Temporal Data Leakage
Yale School of Management · AQR Capital Management · NBER +3
Utility-Aware Data Pricing: Token-Level Quality and Empirical Training Gain for LLMs
Apr 24, 2026Minghui Xu, Qi Luo, Kun Li +1Data Valuation LLM Training
Shandong University, China
Commonsense Knowledge with Negation: A Resource to Enhance Negation Understanding
Apr 21, 2026Zijie Wang, MohammadHossein Rezaei, Farzana Rashid +1Knowledge Augmentation for Language Models LLM Training
University of Arizona · University of North Carolina Asheville
Characterizing Model-Native Skills
Apr 19, 2026Feiyang Kang, Mahavir Dabas, Myeongseob Ko +1LLM Alignment LLM Training
Virginia Tech Blacksburg, VA 24060, USA
PoliLegalLM: A Technical Report on a Large Language Model for Political and Legal Affairs
Apr 19, 2026Yuting Huang, Yinghao Hu, Qian Xiao +7LLM Grounding Language Model Post-Training
Zhejiang University, Hangzhou, China · Tongyi Lab, Alibaba Group, Hangzhou, China · Zhejiang Zhengfa Xinxi Guanli Zhongxin, Hangzhou, China
Where does output diversity collapse in post-training?
Apr 17, 2026Constantinos Karouzos, Xingwei Tan, Nikolaos AletrasOutput Diversity in Language Models Language Model Post-Training
School of Computer Science University of Sheffield, UK
GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimization
Apr 17, 2026Jixuan Leng, Si Si, Hsiang-Fu Yu +2LLM Alignment Direct Preference Optimization
Carnegie Mellon University · Google DeepMind · Google

May 7, 2026J/K move · Enter open · S save

Normalized Architectures are Natively 4-Bit

Maxim Fishman, Brian Chmiel, Ron Banner, Daniel Soudry+1

∧NVIDIA · ◦Department of Electrical and Computer Engineering - Technion, Haifa, Israel

LLM Quantization Neural Representation Geometry 4-Bit Quantization Neural Network Robustness Low-Precision Training LLM Training

Open paper AskPDF arXiv

Training large language models at 4-bit precision is critical for efficiency. We show that nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to low-precision arithmetic. This removes the need for interventions-such as applying random Hadamard transforms and performing per-tensor scaling calculations-to preserve model quality, and it enables stable end-to-end NVFP4 training. We validate this approach on both a 1.2B dense model and hybrid (Mamba-Transformer) MoE models of up to 3B/30B parameters. We trace this robustness to the dot product: while quantization noise remains largely uncorrelated in both standard and normalized architectures, the signal behaves differently. In nGPT, the hypersphere constraint enhances weak positive correlations among the element-wise products, leading to a constructive accumulation of the signal across the hidden dimension while the noise continues to average out. This yields a higher effective signal-to-noise ratio and a flatter loss landscape, with the effect strengthening as the hidden dimension grows, suggesting increasing advantages at scale. A reference implementation is available at https://github.com/anonymous452026/ngpt-nvfp4