LLM Training

LLM: Large Language Model

Follow this topic Ask about this topic

Momentum

31 papers in the last four weeks, up 72% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 226

All topics

CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention
Jun 25, 2026Sayak DuttaLinear Attention Delta-Rule Linear Attention
Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models
Jun 23, 2026Kwok Chun Au, Adam BlockDeep Learning Optimization LLM Training
Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University
Internal Data Repetition Destroys Language Models
Jun 23, 2026Jessica Chudnovsky, Joshua Kazdan, Noam Levi +6Memorization in Language Models Language Model Scaling Laws
1Stanford Computer Science. · 2Stanford Statistics. · 3Tel Aviv University. +1
FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training
Jun 22, 2026Dikshant Kukreja, Kritarth Prasad, Avinash Anand +6Automatic Differentiation Backpropagation
Puch AI · Singapore Institute of Technology · Nanyang Technological University +2
Words as Difference Makers: How Large Language Models Determine Causal Structure in Text
Jun 21, 2026Wolfgang PietschCausal World Models LLM Interpretability
tum
Towards Understanding the Power and Limits of the Muon Optimizer: A River-Valley Perspective
Jun 19, 2026Tianqi Shen, Jinji Yang, Runze Shi +3Deep Learning Optimization Muon Optimizer
Department of Computer Science City University of Hong Kong (CityUHK) · Department of Statistics and Data Science University of Pennsylvania · School of Statistics and Data Science Shanghai University of Finance and Economics
Online Dynamic Batching with Formal Guarantees for LLM Training
Jun 18, 2026Dian Li, Zekun Wang, Yaoru Wang +1LLM Fine-Tuning Distributed Training
Tencent
RegMix-D: Dynamic Data Mixing via Proxy Training Trajectories
Jun 17, 2026Kaiyan Zhao, Zhongtao Miao, Akiko Aizawa +1Language Model Pretraining Training Data Selection
The University of Tokyo · National Institute of Informatics
BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training
Jun 17, 2026Jiaxing Wang, Deping Xiang, Jin Xu +9Training Data Selection Bilevel Optimization
The Stanford EDGAR Filings Dataset: Reconstructing U.S. Corporate and Financial Disclosures into Layout-Faithful and Token-Efficient Pretraining Data
Jun 16, 2026Nick Bettencourt, Xiaowei Ding, Kay GieseckeDocument Understanding Long-Context Language Modeling
Department of Applied Statistics and Data Science University of California, Los Angeles · School of Information Management Nanjing University · Advanced Financial Technologies Lab Stanford University
Small Initialization Matters for Large Language Models
Jun 16, 2026Liangkai Hang, Junjie Yao, Zhiyu Li +3Language Model Pretraining Neural Network Initialization
School of Mathematical Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China. · MemTensor (Shanghai) Technology Co., Ltd.. for Advanced Algorithms Research, Shanghai, China. · Institute of Natural Sciences, Shanghai Jiao Tong University, Shanghai, 200240, China.
Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
Jun 15, 2026Martin JaggiMixture-of-Experts Language Models LLM Compression
EPFL
From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Models
Jun 12, 2026Zixuan Gu, Xiaojun Ye, Yang LiuPrivacy-Preserving Language Models Privacy Leakage in Language Models
School of Software, Tsinghua University, Beijing, China · the Hong Kong Polytechnic University, Hongkong, China
Small LLMs: Pruning vs. Training from Scratch
Jun 12, 2026Yufeng Xu, Taiming Lu, Kunjun Li +3Language Model Pretraining LLM Pruning
Princeton University · New York University · Carnegie Mellon University
Natively Unlearnable Large Language Models
Jun 11, 2026Gaurav R. Ghosal, Pratyush Maini, Aditi RaghunathanActivation Sparsity Machine Unlearning
Carnegie Mellon University · DatologyAI
A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget
Jun 11, 2026Joe DwyerLLM Evaluation Efficient Language Model Training
Department of Computer Information Science ECPI University Virginia Beach, VA 23462
Ouroboros-Spatial: Closing the Data-Model Loop for Spatial Reasoning
Jun 10, 2026Enhan Zhao, Wei Wu, Yuanrui Zhang +23D Spatial Reasoning Synthetic Data Generation
1Peking University · 2Ant International · 3The University of Hong Kong
Improving Cross-Format Robustness in Language Models with Multi-Format Training
Jun 10, 2026June M. Liu, Shaomian Zheng, He Cao +3Language Model Robustness LLM Training
Ant Group · International Digital Economy Academy (IDEA)
Life Cycle Assessment of Pre-training the Lucie 7B Open-Source Large Language Model on the Jean Zay Supercomputer
Jun 9, 2026Marc Léobet, Pierre-François Lavallée, Jean-Pierre LorréEnergy-Efficient ML LLM Training
Mens Data · CNRS / IDRIS · LINAGORA
Unifying Data, Memory, and Compute Efficiency in LLM training: A Survey
Jun 9, 2026Vanessa Schmidt, Huy Hoang Nguyen, Cédric Jung +2Efficient Language Model Training LLM Training
Chair of Information Theory and Data Analytics (INDA), RWTH Aachen University, 52074 Aachen, Germany · AIT Austrian Institute of Technology GmbH, 1210 Vienna, Austria · Automation and Control Institute, Technische Universität Wien (TUW), 1040 Vienna, Austria
IS-CoT: Breaking the Long-form Generation Collapse via Interleaved Structural Thinking
Jun 8, 2026Zechen Sun, Yuyang Sun, Zecheng Tang +6Open-Ended Generation Long-Form Document Generation
Institute of Computer Science and Technology, Soochow University · 2Information Research Center of Military Science, PLA Academy of Military Science
FlashCP: Load-Balanced Communication-Efficient Context Parallelism for LLM Training
Jun 7, 2026Zheng Wang, Eric Liu, Linan Jiang +5Communication-Efficient Distributed Training Long-Context Language Modeling
University of California San Diego, La Jolla, USA · University of Southern California, Los Angeles, USA · Rice University, Houston, USA.
Minibatch Selection for Language Models via Partition Matroid Constrained Gradient Matching
Jun 6, 2026Prayas Agrawal, Prateek Chanda, Ishita Khatri +3Submodular Optimization LLM Fine-Tuning
Department of Computer Science and Engineering, Indian Institute of Technology Bombay · Microsoft Research India · Centre for Machine Intelligence and Data Science, Indian Institute of Technology Bombay +1
Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling
Jun 5, 2026Rohan ShravanMixture-of-Experts Language Models Efficient Language Model Training
The School of AI Bengaluru, India
DataEvolver: Automatic Data Preparation for Large Language Models through Multi-Level Self-Evolving
Jun 5, 2026Chao Deng, Shaolei Zhang, Ju Fan +1Training Data Curation LLM Training
Renmin University of China, Beijing, China
Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search
Jun 4, 2026Shing Yin Wong, Shaocheng Liu, Linqi Song +2Automated Theorem Proving LLM Training
Department of Computer Science, City University of Hong Kong · Department of Information Engineering, The Chinese University of Hong Kong
Spectral Scaling Laws of Muon
Jun 2, 2026Gagik Magakyan, Pablo Parrilo, Asuman OzdaglarNewton-Schulz Iteration Momentum Methods
MIT
Unlocking Feature Learning in Gated Delta Networks at Scale
Jun 2, 2026Yifeng Liu, Quanquan GuNeural Network Optimization Hyperparameter Transfer
University of California, Los Angeles
Learning from Saturated Data: Signals Beyond Correctness for LLM Training
May 31, 2026Hanno Hiss, Jasper Dekoninck, Martin VechevLanguage Model Self-Improvement LLM Training
ETH Zurich
MiCU: End-to-End Smart Home Command Understanding with Large Language Model
May 31, 2026Haowei Han, Kexin Hu, Weiwei Cai +6Internet of Things LLM Training
Xiaomi Corporation · School of Computer Science Wuhan University Wuhan, China · Institute for Math & AI Wuhan University Wuhan, China

Jun 25, 2026, cs.CLJ/K move · Enter open · S save

CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention

Sayak Dutta

Linear Attention Delta-Rule Linear Attention Attention Mechanisms LLM Training

Open paper AskPDF arXiv

Recurrent delta-rule models keep a fixed-size state matrix S (d_v x d_k) that compresses all past context. The state of the art (GDN-2) gates this update with element-wise matrix erase/write masks. This is powerful but has two defects. First, both gates are computed from the incoming token alone, making the model memory-blind: it decides what to erase without seeing what it has stored. Second, value-axis coupling in the erase gate blocks the WY-form triangular chunk solver that drives efficient training -- the intra-chunk system splits into d_v independent solves, collapsing throughput to serial-recurrence cost. We introduce CARVE (Content-Aware Recurrent with Value Efficiency), which fixes both and, via a single-launch "megakernel" scheduling of the same WY-form math, trains faster than the matrix-gated baseline it replaces. The key idea is architectural: restricting all gating to the key axis makes the intra-chunk coupling independent of the value index, restoring one unmodified WY-form solve. Within this constraint, CARVE conditions both gates on a content signal read once per chunk from the chunk-boundary state and folded algebraically into each gate's low-rank projection (by associativity, U(Sq)=(US)q), giving memory-aware gating at negligible extra traffic. At init the content projections are zero, so CARVE is bit-identical to the baseline; we prove the one-chunk staleness perturbs gates by only O(1/sqrt(L)), matching a measured 0.18% deviation flat up to L=128. At 1.3B parameters / 100B FineWeb-Edu tokens on H100 (three seeds), CARVE improves every axis: WikiText perplexity 15.72 vs 15.90 (hybrid 15.41 vs 15.62), +0.63 pp average common-sense accuracy, and state-of-the-art RULER and real-world recall -- while training +1.4% faster at matched depth and +19.3% at iso-quality depth, at +13% peak memory. Backed by six formal guarantees.

Momentum

Related topics

Latest papers 226