Efficient Language Model Training

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 109

All topics

Towards Looped Models Done Right, Part II: Rethinking at Fixed Points
Oct 5, 2026Benhao Huang, Chufan Shi, Junlin Chen +4Efficient Language Model Training Recurrent Transformers
Institute of Foundation Models · CMU · USC
Scaling Down the Scaling Laws: Parameter Efficiency and Compute-Optimal Training in Resource-Constrained Large Language Models
Oct 5, 2026Joe DwyerLanguage Model Scaling Laws Efficient Language Model Training
ECPI University
More Than Words: Compositional Tokenization for Efficient Language Models
Oct 4, 2026Yuval Reif, Guy Kaplan, Roy SchwartzLanguage Model Pretraining Efficient Language Model Training
The Hebrew University of Jerusalem
Selecting Repetition Counts Across Model Scales in Data-Constrained Pretraining
Oct 4, 2026Ziyue WANG, T. KanamoriLanguage Model Pretraining Language Model Scaling Laws
Institute of Science Tokyo
Beyond State-of-the-Art: Standardising Environmental Impact Metrics for AI Research
Oct 1, 2026Lachlan McGinness, Dan Pagendam, Robert OffnerLLM Inference Efficiency Energy-Efficient ML
School of Computing, Australian National University · Commonwealth Scientific and Industrial Research Organisation
Closing the Loop: Practical Training Recipes for Looped Language Models
Sep 30, 2026Andrei Marchenko, Viacheslav Bezrukov, Oleg Kashurin +5Efficient Language Model Training LLM Training
RND NLP, DAIMLD, Russian Federation
It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
Sep 29, 2026Pierre-Carl Langlais, Pieter Delobelle, Yannick Detrois +7Language Model Pretraining Synthetic Data Pretraining
PleIAs · Sorbonne Center for Artificial Intelligence · Sciences Po Médialab +6
MeqMuon: Matrix-Equilibrating Muon for LLM Pretraining
Sep 28, 2026Chang-Wei Shi, Xu Wang, Wu-Jun LiLanguage Model Pretraining Efficient Language Model Training
National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, P. R. China
SOLO: Pretraining Billion-Parameter Language Models with Shared-Output Local Learning
Sep 28, 2026Bojian Yin, Shurong Wang, Yuqi Pan +1Language Model Pretraining Efficient Language Model Training
Institute of Automation, Chinese Academy of Sciences
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
Sep 28, 2026Arman Bolatov, Artem Riabinin, Nikita Kornilov +4Stochastic Optimization Language Model Pretraining
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI) · Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Applied Artificial Intelligence Institute +1
No Free Efficiency: Revisiting the Trade-off Between Training Efficiency and Model Vulnerability
Sep 27, 2026Yiyong Liu, Jun Sakuma, Michael Backes +1Adversarial Robustness Efficient Language Model Training
CISPA Helmholtz Center for Information Security · Institute of Science Tokyo
The Life of a Token: from Words to Bits on the Wire
Sep 17, 2026Davide Avesani, Pengwenlong Gu, Sotiris Skaperas +1Communication-Efficient Distributed Training High-Performance Computing
CEDRIC - ROC · cnam · Cnam
Size Matters: Foundation Model for Czech HTML documents
Sep 16, 2026Martin Dvořák, Vít Tlustoš, Artyom Voronin +4Language Model Pretraining Efficient Language Model Training
Seznam.cz, Prague, Czech Republic
Structural priors for data-efficient language learning
Sep 12, 2026Yana Veitsman, Jonas Mayer Martins, Jonathan Lautenschlager +1Language Model Pretraining Neural Network Initialization
University of Göttingen, Germany
Characterizing Job Power Elasticity for Power-Flexible AI Training
Sep 12, 2026Philip Colangelo, Charles Dawson, Shayan Sengupta +2Energy-Efficient ML Efficient Language Model Training
Emerald AI
Data-Efficient Language Modeling: From Frontier Advancement to Principle-Guided Model Improvement
Sep 12, 2026Shuxing Yang, Kaihao Zhu, Junjie Yang +13Efficient Language Model Training Language Modeling
College of Information Science and Electronic Engineering, Zhejiang University · Qiushi Engine Team
Parallelism Strategy Chaining for Fast Training Convergence
Sep 7, 2026Minchul Kang, Changyong Shin, Younghun Go +4Efficient Language Model Training LLM Training
Department of Computer Science and Engineering, Korea University, Seoul, South Korea
Subword Segmental BabyLMs: Learning to Tokenise for Sample-Efficient Pretraining
Sep 1, 2026Francois MeyerLanguage Model Pretraining Autoregressive Language Modeling
Department of Computer Science University of Cape Town
Toppling the Hierarchy in Byte-level Language Modeling
Aug 31, 2026Lukas Edman, Alexander FraserEfficient Language Model Training Byte-Level Language Model
School of Computation, Information and Technology, TU Munich · Munich Center for Machine Learning · Munich Data Science Institute
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Aug 31, 2026Zihan Qiu, Zekun Wang, Xiao Li +33Efficient Language Model Training Sparse Attention
Qwen Team
Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090
Aug 27, 2026Kairong Luo, Jiarui Cui, Yaorui Yin +8Language Model Pretraining Language Model Scaling Laws
Tsinghua University · Pengcheng Laboratory
Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model
Aug 13, 2026Mohammed Sabry, Sean Augenstein, Keith Rush +1Language Model Pretraining Efficient Language Model Training
Google, Mountain View, CA, USA · School of Computing, Dublin City University, Dublin, Ireland.
LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training
Aug 12, 2026Xiaojun Wu, Cehao Yang, Honghao Liu +5Efficient Language Model Training LLM Training
1IDEA Research · 2The Hong Kong University of Science and Technology (Guangzhou) · 3DataArcTech Ltd.
TELLME: Test-Enhanced Learning for Language Model Enrichment
Aug 12, 2026Minjun Kim, Inho Won, Hyeonseok Lim +6Domain-Adaptive Pretraining Efficient Language Model Training
Korea Advanced Institute of Science and Technology · Seoul National University of Science and Technology · National Security Research Institute
Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
Aug 11, 2026Soumajyoti Sarkar, Yuxin Tang, Sheng ZhaMixture-of-Experts Language Models Language Model Scaling Laws
Amazon AGI Foundations
Matryoshka Language Model Suites
Aug 10, 2026Nathan Godey, Yoav ArtziEfficient Language Model Training LLM Inference Acceleration
Cornell University
Gradient Under Microscope: Benchmarking Resource Utilization of Memory-Efficient Gradient Computation Methods
Aug 9, 2026Sarthak Mahapatra, Zihan Zhou, Khatoon Khedri +2Efficient Language Model Training Adaptive Gradient Methods
Efficient Knowledge Distillation for LLMs: Offline Top-K Logits and a Fused Chunked KL Loss
Aug 4, 2026Bakbergen Ryskulov, Iker García-Ferrero, David Montero +5Efficient Language Model Training Language Model Distillation
Multiverse Computing
Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension
Aug 4, 2026Raviraj Joshi, Utkarsh Vaidya, Sanjay Singh Chauhan +1Efficient Language Model Training Continued Pretraining
NVIDIA
Training nGPT
Aug 2, 2026Ilya Loshchilov, Boris GinsburgHyperspherical Representation Learning Efficient Language Model Training
NVIDIA
PARALLEL: A Prefrontal-Aligned Reinforcement inspired Approach for Language-Model Learning under Explicit Limits
Jul 31, 2026Namkyung Yoon, Sanghong Kim, Hwangnam KimContinual Learning for LLMs LLM Fine-Tuning
School of Electrical Engineering, Korea University Seoul 02841, Republic of Korea
SDO: Structure-Aware Data Organization for Efficient LLM Post-Training
Jul 29, 2026Jinliang Gao, Ning Yang, Hai Wang +2Training Data Curation Efficient Language Model Training
Institute of Automation, Chinese Academy of Sciences · National University of Defense Technology
From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference
Jul 27, 2026Darina Gold, Alexander Schwirjow, Viktor Haag +4On-Device Language Model Inference Training Data Curation
IIS Fraunhofer
DomainPilot: Domain-Level Loss-Guided Two-Stage Data Mixture Optimization for Efficient Language Model Fine-Tuning
Jul 24, 2026He ZhangSupervised Fine-Tuning Large Language Model-Guided Optimization
Tsinghua University
Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
Jul 21, 2026Nuemaan MalikDeep Learning Optimization Mixture-of-Experts Language Models
Independent Researcher
PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning
Jul 20, 2026Hang Zhang, Warren J. GrossLLM Fine-Tuning Efficient Language Model Training
Department of Electrical and Computer Engineering, McGill University Montreal, QC, Canada
OpenLanguageModel: Readable and Composable Small-Language-Model Pretraining for Education and Research
Jul 18, 2026Tavish Mankash, Vardhaman Kalloli, Keshava Prasad +1Language Model Pretraining Small Language Models
FAIRC · Plaksha University
Long-Context Fine-Tuning with Limited VRAM
Jul 16, 2026Vladimir Fedosov, Aleksandr Sazhin, Artemiy Grinenko +1Fine-Tuning KV-Cache Offloading
BMW Group
LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget
Jul 16, 2026Changhai Zhou, Kieran Liu, Yuhua Zhou +17Efficient Language Model Training Long-Context Modeling
MindLab · Fudan University
Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training
Jul 6, 2026Jingwei Zuo, Cong Zeng, Ilyas Chahed +6Memorization in Language Models Neural Network Memorization
Technology Innovation Institute Abu Dhabi, UAE
Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
Jul 2, 2026Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao +3Communication-Efficient Distributed Training High-Performance Computing
Salesforce AI Research
Parameter Golf: What Really Works?
Jul 1, 2026Prashanna Mani Paudel, Shivanand Venkanna SheshappanavarLLM Evaluation LLM Compression
Geometric Intelligence Research Lab., Department of Electrical Engineering and Computer Science University of Wyoming
Token Geometry
Jul 1, 2026Kathan ShahLanguage Model Pretraining LLM Fine-Tuning
Preprint.
CHERRY: Compressed Hierarchical Experts with Recurrent Representational Yield
Jun 30, 2026Dohyeon Kwon, Youngjin ParkMixture-of-Experts Language Models LLM Compression
AXOps Team, TeamSparta Inc., Seoul, South Korea
BluTrain: A C++/CUDA Framework for AI Systems
Jun 23, 2026Adhitya Charan, Adwaid Suresh, Anuj Kumar +19GPU Acceleration High-Performance Computing
Blubridge AI
FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training
Jun 22, 2026Dikshant Kukreja, Kritarth Prasad, Avinash Anand +6Automatic Differentiation Backpropagation
Puch AI · Singapore Institute of Technology · Nanyang Technological University +2
L20-Edu-135M: An Auditable Single-GPU Study of Data-Efficient Small Language Modeling
Jun 20, 2026Yin LiLanguage Model Pretraining Small Language Models
University of Birmingham
Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale
Jun 13, 2026Ang Li, Ben Liu, Bin Han +215Efficient Language Model Training Linear Attention
A Quantitative Experimental Repeated Measures Study of Training Dynamics in a Small Llama Style Language Model Under a Compute-Aware Token Budget
Jun 11, 2026Joe DwyerLLM Evaluation Efficient Language Model Training
Department of Computer Information Science ECPI University Virginia Beach, VA 23462
Pythagoras-Prover: Advancing Efficient Formal Proving via Augmented Lean Formalisation
Jun 10, 2026Joshua Ong Jun Leang, Zheng Zhao, Mihaela Cătălina Stoian +5Automated Theorem Proving Efficient Language Model Training
Imperial College London · University of Edinburgh · Nanyang Technological University +1
Unifying Data, Memory, and Compute Efficiency in LLM training: A Survey
Jun 9, 2026Vanessa Schmidt, Huy Hoang Nguyen, Cédric Jung +2Efficient Language Model Training LLM Training
Chair of Information Theory and Data Analytics (INDA), RWTH Aachen University, 52074 Aachen, Germany · AIT Austrian Institute of Technology GmbH, 1210 Vienna, Austria · Automation and Control Institute, Technische Universität Wien (TUW), 1040 Vienna, Austria
Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling
Jun 5, 2026Rohan ShravanMixture-of-Experts Language Models Efficient Language Model Training
The School of AI Bengaluru, India
q0: Primitives for Hyper-Epoch Pretraining
Jun 2, 2026Bishwas Mandal, Shmuel Berman, Akshay Vegesna +1Language Model Pretraining Efficient Language Model Training
Q Labs · Princeton University
PortBERT: Navigating the Depths of Portuguese Language Models
Jun 1, 2026Raphael Scheible-Schmitt, Henry He, Armando B. MendesLanguage Model Pretraining Efficient Language Model Training
School of Computation, Information and Technology, Technical University of Munich, Munich, Germany · IS2E - Intelligent Systems, Science and Engineering, LIACC polo on Azores University, Ponta Delgada, Portugal
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
May 31, 2026Boqian Wu, Qiao Xiao, Patrik Okanovic +6Language Model Scaling Laws Efficient Language Model Training
University of Luxembourg · University of Twente · Eindhoven University of Technology +1
Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
May 30, 2026Qiao Xiao, Boqian Wu, Patrik Okanovic +6Language Model Pretraining Efficient Language Model Training
Eindhoven University of Technology · University of Luxembourg · University of Twente +1
PithTrain: A Compact and Agent-Native MoE Training System
May 29, 2026Ruihang Lai, Hao Kang, Haozhan Tang +6Mixture-of-Experts Language Models Efficient Language Model Training
1Carnegie Mellon University · 3NVIDIA · 2Xlue
Towards Efficient LLMs Annealing with Principled Sample Selection
May 29, 2026Yuanjian Xu, Jianing Hao, Wanbo Zhang +2Language Model Pretraining Efficient Language Model Training
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · Microsoft Research Asia, Beijing, China · Fudan University, Shanghai, China
D $^3$ : Dynamic Directional Graph-Constrained Data Scheduling for LLM Training
May 29, 2026Yuanjian Xu, Jianing Hao, Guang Zhang +1Efficient Language Model Training LLM Training
1HKUST-GZ · Microsoft Research Asia · 2Microsoft Research
MIRA: Mid-training Rubric Anchoring for Source-Aware Data Selection
May 28, 2026Haowen Wang, Yaxin Du, Jian Yang +9Data Selection Efficient Language Model Training
1Beihang University · 2IQuest Research · 3Shanghai Jiao Tong University +2

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

Towards Looped Models Done Right, Part II: Rethinking at Fixed Points

Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen+3

Institute of Foundation Models · CMU · USC

Efficient Language Model Training Recurrent Transformers LLM Inference Acceleration Recursive Language Models

Open paper AskPDF arXiv

Every recurrence of a looped language model adds cost in training, decoding, prefill, and reinforcement learning (RL). The closer recurrent states get to fixed points, the less the path to them matters. This enables truncated backpropagation in training; terminal key-value (KV) sharing for decoding with almost no loss in accuracy; a distilled student that prefills up to 1.79x faster; and RL updates that compute gradients from saved rollout states, 2x faster than backpropagating through the replayed trajectory. We therefore improve the two components of training that shape these fixed points: the depth prior and input injection. Fixed-depth training breaks KV sharing, and Huginn's broad depth prior supports sharing but dilutes supervision at the target depth more than sharing requires; we learn the prior from prediction feedback, with an entropy term that keeps it broad. Existing injection schemes let the state's component along the input amplify or cancel the injection; we remove this component with orthogonal injection. From 100M to 1.6B parameters, the learned prior and orthogonal injection lower perplexity at every scale relative to Huginn's prior and existing injection schemes, respectively. At 1.6B, the learned prior with a 3x smaller KV cache matches the downstream average of fixed-depth training with the full cache.

Momentum

Related topics

Latest papers 109