Efficient Language Model Training

Follow this topic Ask about this topic

Momentum

15 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 109

All topics

Inference-Native Zeroth-Order Optimization for LLMs
May 27, 2026Zelin Li, Caiwen DingLLM Serving Zeroth-Order Optimization
University of Minnesota
BPPO: Binary Prefix Policy Optimization for Efficient GRPO-Style Reasoning RL with Concise Responses
May 27, 2026Qingfei Zhao, Huan Song, Shuyu Tian +2RL for Language Model Reasoning Group Relative Policy Optimization
1TeleAI · 2Shanghai Jiao Tong University
Hierarchical Prompt-Domain Control and Learning for Resource-Constrained Agentic Language Models
May 26, 2026Joan Vendrell Gallart, Russell Bent, Michael GrosskopfContinual Learning for LLM Agents Language Model-Based Control
Department of Mechanical and Aerospace University of California Irvine Irvine, CA 92617-4322, USA · T-5 Los Alamos National Laboratory Los Alamos, NM 88220, USA · CAI-4 Los Alamos National Laboratory Los Alamos, NM 88220, USA
BigMac: Breaking the Pareto Frontier of Compute and Memory in Multimodal LLM Training
May 25, 2026Zili Zhang, Chengxu Yang, Shenglong Zhang +8Deep Learning Optimization Multimodal Large Language Models
1Peking University · †Work done during internships at Xiaohongshu, Inc. · 2Independent Researcher +2
Convex Optimization for Alignment and Preference Learning on a Single GPU
May 22, 2026Miria Feng, Mert PilanciPairwise Preference Learning LLM Alignment
Department of Electrical Engineering, Stanford University, California, United States.
One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
May 21, 2026Di He, Songjun Tu, Keyu Wang +2Learning Rate Scheduling Efficient Language Model Training
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences · Peng Cheng Laboratory · University of Chinese Academy of Sciences +5
torchtune: PyTorch native post-training library
May 20, 2026Mark Obozov, Maxime Griot, Joseph Cummings +8LLM Fine-Tuning Efficient Language Model Training
Work done at PyTorch, Meta · Stanford · Meta-FAIR
OlmoEarth v1.2: A more efficient family of OlmoEarth models
May 20, 2026Gabriel Tseng, Yawen Zhang, Favyen Bastani +8Efficient Neural Network Inference Geospatial Foundation Models
Allen Institute for AI
Correcting Stochastic Update Bias in Preconditioned Language Model Optimizers
May 20, 2026Nikhil Nayak, Julia White, Urchade Zaratiana +7Efficient Language Model Training Adaptive Gradient Methods
Fastino Labs
HRM-Text: Efficient Pretraining Beyond Scaling
May 20, 2026Guan Wang, Changling Liu, Chenyu Wang +6Language Model Pretraining Recurrent Neural Networks
1Sapient Intelligence · 2MIT
Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stages
May 19, 2026Brandon Cui, Ximing Lu, Jaehun Jung +7Language Model Pretraining Efficient Language Model Training
1NVIDIA · University of Washington · 3Carnegie Mellon University +1
Heterogeneity-Aware Dataset Scheduling for Efficient Audio Large Language Model Training
May 18, 2026Yanru Wu, Jianning Wang, Chongxin Gan +1Audio QA Efficient Language Model Training
1Shenzhen International Graduate School, Tsinghua University · 2Independent Researcher · 3The Hong Kong Polytechnic University
Learn-by-Wire Training Control Governance: Bounded Autonomous Training Under Stress for Stability and Efficiency
May 18, 2026Anis RadianisEfficient Language Model Training LLM Training
Research Scientist at Qluon Inc. Delaware, United States
DynMuon: A Dynamic Spectral Shaping View of Muon
May 16, 2026Fangzhou Wu, Rikhav Shah, Sandeep Silwal +1Neural Network Optimization Muon Optimizer
University of Wisconsin–Madison · MIT · Elorian AI
DualKV: Shared-Prompt Flash Attention for Efficient RL Training with Large Rollouts and Long Contexts
May 14, 2026Jiading Gai, Shuai Zhang, Xiang Song +2GPU Acceleration Self-Attention
1Amazon Web Services · 2Google · University of Minnesota
EMO: Frustratingly Easy Progressive Training of Extendable MoE
May 13, 2026Linghao Jin, Chufan Shi, Huijuan Wang +4Sparse Mixture-of-Experts Efficient Language Model Training
USC-ISI · MBZUAI-IFM
Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time
May 13, 2026Michael Y. Hu, Apurva Gandhi, Kyunghyun Cho +2Language Model Pretraining Continual Learning for LLMs
New York University · Carnegie Mellon University · Microsoft
DynaTrain: Fast Online Parallelism Switching for Elastic LLM Training
May 12, 2026Yuanqing Wang, Yuchen Zhang, Hao Lin +9Efficient Language Model Training Distributed Training
Peking University · Infinigence AI · Institute of Computing Technology, CAS +2
Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation
May 12, 2026Yuchen Cai, Ding Cao, Liang Lin +9Efficient Language Model Training On-Policy Distillation
1USTC · 2Tencent · 4HKUST(GZ) +3
Predicting Large Model Test Losses with a Noisy Quadratic System
May 9, 2026Chuning Li, Chris J. MaddisonLanguage Model Scaling Laws Efficient Language Model Training
University of Toronto · Vector Institute
Structured Recurrent Mixers for Massively Parallelized Sequence Generation
May 9, 2026Benjamin L. BadgerRecurrent Neural Networks Autoregressive Language Modeling
IBM
Efficient Pre-Training with Token Superposition
May 7, 2026Bowen Peng, Théo Gigant, Jeffrey QuesnelleLanguage Model Pretraining Efficient Language Model Training
Nous Research
Rethinking Local Learning: A Cheaper and Faster Recipe for LLM Post-Training
May 6, 2026Hengyu Shi, Tianyang Han, Peizhe Wang +3Efficient Language Model Training LLM Post-Training
1Independent Researcher · 2D4 Lab · 3Southeast University
Budget-aware Auto Optimizer Configurator
May 6, 2026Kang Liu, Wei Peng, Jianchen HuDeep Learning Optimization Memory-Efficient Optimization
School of Future Technology Xi’an Jiaotong University Xi’an, China · School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China
ELAS: Efficient Pre-Training of Low-Rank Large Language Models via 2:4 Activation Sparsity
May 5, 2026Jiaxi Li, Lu Yin, Li Shen +5Language Model Pretraining Activation Sparsity
University of Surrey · Sun Yat-sen University · Bytedance +2
Compute Optimal Tokenization
May 2, 2026Tomasz Limisiewicz, Artidoro Pagnoni, Srini Iyer +6Language Model Scaling Laws Efficient Language Model Training
1FAIR at Meta · University of Washington
Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient German Language Modeling
Apr 30, 2026Ansar Aynetdinov, Patrick Haller, Alan AkbikLanguage Model Pretraining Efficient Language Model Training
Humboldt-Universität zu Berlin
Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
Apr 29, 2026Théo Gigant, Bowen Peng, Jeffrey QuesnelleLanguage Model Pretraining Efficient Language Model Training
Nous Research
AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
Apr 29, 2026Ahan Gupta, Zhihao Wang, Neel Dani +3Efficient Language Model Training LLM Training
SSAIL Lab, University of Illinois Urbana-Champaign Champaign, IL, USA · Anyscale San Francisco, CA, USA · Snowflake San Mateo, CA, USA
Efficient Training on Multiple Consumer GPUs with RoundPipe
Apr 29, 2026Yibin Luo, Shiwei Gao, Huichuan Zheng +2GPU Acceleration LLM Fine-Tuning
Tsinghua University Beijing, China

May 27, 2026, cs.LGJ/K move · Enter open · S save

Inference-Native Zeroth-Order Optimization for LLMs

Zelin Li, Caiwen Ding

University of Minnesota

LLM Serving Zeroth-Order Optimization Efficient Language Model Training LLM Inference Acceleration

Open paper AskPDF arXiv

Zeroth-order (ZO) methods train large language models using only forward passes, yet common ZO execution paths perform substantial work beyond what the algorithm itself requires. To remove this extra execution overhead, we present Infer-ZO, which separates the evaluations required by the algorithm from how they are executed, allowing them to reuse existing inference-engine optimizations. After removing inherited execution work, a complete Infer-ZO step on Qwen3-14B adds only 1.16% wall-clock time over matched inference, bringing ZO execution close to its fundamental inference workload. With Infer-ZO, ZO evaluations run at near-inference cost with frozen base weights and can share GPU batches with ordinary inference requests. In co-serving experiments, foreground and background throughput remain within 1-3% of the corresponding background-request baseline. Across 15 Qwen3, Llama, and OPT models, Infer-ZO achieves 2.10x-9.94x end-to-end step speedups over released LoZO. Code is available at https://github.com/playeriv65/zo-vllm.

Momentum

Related topics

Latest papers 109