Optimizer Design

Follow this topic Ask about this topic

Momentum

2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 24

All topics

AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization
Sep 29, 2026Pengyu He, Yan Zhang, Ruien Li +1Large Language Model Training Batch
Department of Electrical Engineering, California Institute of Technology · Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison
Muon-C: Operator-Aligned Muon for Convolutional Kernels
Sep 9, 2026Jiaxin Qing, Lexin LiMuon Deep Unfolding Networks
University of California, Berkeley Berkeley, CA 94720-1776, USA
LazyTrain: Limited-resource Allocation toward Zero-waste Yield Optimization in Large Language Model Training
Aug 12, 2026Xiaojun Wu, Cehao Yang, Honghao Liu +5Large Language Model Training Graphics Processing Unit Memory
1IDEA Research · 2The Hong Kong University of Science and Technology (Guangzhou) · 3DataArcTech Ltd.
A Hybrid Nested Harness for Decoupling Structure and Parameters in LLM-Driven Optimization
Aug 8, 2026Víctor GallegoEvolutionary Search Optimization Modeling
Komorebi AI Technologies
Muon Meets Mamba: Spectral Optimization for State Space Models
Aug 4, 2026Arslan Battalov, Karim Kramin, Alexander Markotenko +1Muon State Space Models
HSE University · Moscow, Russia
Draining the Energy Commons: Self-Defeating Over-Appropriation as a Coordination Failure in Agentic LLM Collectives
Jul 24, 2026Marcantonio Bracale Syrnicov, Federico Pierucci, Matteo Prandi +4Optimizer Design
Icaro Lab · VU Amsterdam · M. Bracale Syrnikov1,4 +8
Sampling on Random Subspaces under Limited Data in the Context of Exploratory Landscape Analysis
Jul 8, 2026Iván Olarte Rodríguez, Anja Jankovic, Thomas Bäck +1Optimization Landscape Optimal Design
Leiden Institute of Advanced Computer Science (LIACS), Leiden University, The Netherlands · RWTH Aachen University, Germany
The Anatomy of Implicit Bias: Information Allocation in Neural Network Training
Jul 8, 2026Zhang Gongyue, Wang Zhiyong, Liu Donghan +3Neural Network Optimization Optimizer Design
1State Key Laboratory of Robotics and Systems, Harbin Institute of Technology Shenzhen, Shenzhen, 518055, China.
Bridging Spherical Black-Box Optimizers
Jun 24, 2026Johannes Ackermann, Stefano PeluchettiBlack-Box Optimization Optimizer Design
The University of Tokyo · Sakana AI
FORGE: Fused On-Register Gradient Elimination for Memory-Efficient LLM Training
Jun 22, 2026Dikshant Kukreja, Kritarth Prasad, Avinash Anand +6Matched Fp16 Intermediate Optimizer Design
Puch AI · Singapore Institute of Technology · Nanyang Technological University +2
MSC-CMA-ES: Structure-Aware Restarts for CMA-ES via Cyclic Nearest-Better Basin Discovery
Jun 14, 2026Dimitar Nedanovski, Svetoslav Nenov, Dimitar PilevMetaheuristics Basin
Faculty of Mathematics and Informatics Sofia University St. Kliment Ohridski James Baucher Blvd., 1164 Sofia, Bulgaria · Department of Mathematics University of Chemical Technology and Metallurgy Sofia, Bulgaria · Department of Informatics University of Chemical Technology and Metallurgy Sofia, Bulgaria
Structure from Reasoning, Numbers from Search: On-Premise Open LLMs as Structural Priors for Coupled MIMO Controller Tuning
Jun 9, 2026Jiaxuan Chen, Haonan Li, Yang ShuTuning Robust Control
Jinling Institute of Technology, Nanjing, China · College of Water Resources and Civil Engineering, China Agricultural University, Beijing, China · Zhejiang University, Hangzhou, China
On the Optimizer Dependence of Neural Scaling Laws
May 28, 2026Vansh Ramani, Shourya Vir JainScaling Laws Optimizer Design
Department of Computer Science and Engineering, Indian Institute of Technology Delhi · Department of Electrical Engineering, Indian Institute of Technology Delhi
From SGD to Muon: Adaptive Optimization via Schatten-p Norms
May 19, 2026Thomas Massena, Corentin Friedrich, Mathieu SerrurierOptimizer Design Muon
IRIT, DTIPG - SNCF, UT3 · IRIT · IRT Saint Exupery
Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers
May 18, 2026Tim Tsz-Kit Lau, Weijie SuAdaptive Optimizers Optimizer Design
University of Pennsylvania, Philadelphia, PA 19104, USA.
Context Training with Active Information Seeking
May 13, 2026Zeyu Huang, Adhiguna Kuncoro, Qixuan Feng +4Optimizer Design Physical Activities
The University of Edinburgh · Google DeepMind
Muown: Row-Norm Control for Muon Optimization
May 11, 2026Kai Lion, Florian Hübler, Bingcong Li +2Muon Optimizer Design
Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, School of Computation, Information and Technology; Technical University of Munich, Germany · ELLIS Institute Tübingen, MPI-IS, Tübingen AI Center, Germany
Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
May 9, 2026Aditya RanganathOptimizer Design Large Language Model Training
Center for Applied Scientific Computing Lawrence Livermore National Laboratory Livermore, CA, USA
Budget-aware Auto Optimizer Configurator
May 6, 2026Kang Liu, Wei Peng, Jianchen HuOptimizer Design Graphics Processing Unit Memory
School of Future Technology Xi’an Jiaotong University Xi’an, China · School of Automation Science and Engineering Xi’an Jiaotong University Xi’an, China
FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correction
May 5, 2026Duc Dm, Thao Do, Minh Son Hoang +3Optimizer Design Privacy-Utility Trade-Offs
School of Computing, Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea · Center for Environmental Intelligence (CEI), VinUniversity, Hanoi, Vietnam · Khoury College of Computer Sciences, Northeastern University, USA
Benchmarking Optimizers for MLPs in Tabular Deep Learning
Apr 16, 2026Yury Gorishniy, Ivan Rubachev, Dmitrii Feoktistov +1Tabular Learning Optimizer Design
α Yandex · β HSE University
Prism: Symbolic Superoptimization of Tensor Programs
Apr 16, 2026Mengdi Wu, Xiaoyu Jiang, Oded Padon +1Optimizer Design Tensor Networks
Carnegie Mellon University USA · Tsinghua University China · Weizmann Institute of Science Israel
To Use or not to Use Muon: How Simplicity Bias in Optimizers Matters
Feb 28, 2026Sara Dragutinović, Yedi Zhang, Rajesh RanganathMuon Adam
Courant Institute School of Mathematics, Computing, and Data Science, New York University · Gatsby Computational Neuroscience Unit, University College London
On the Gradient Complexity of Private Optimization with Private Oracles
Nov 17, 2025Michael Menart, Aleksandar NikolovFirst Order Oracle Complexity Convex Loss
Department of Computer Science, University of Toronto, Vector Institute · Department of Computer Science, University of Toronto

Sep 29, 2026, cs.LGJ/K move · Enter open · S save

AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronization

Pengyu He, Yan Zhang, Ruien Li, Guangwen Yang

Department of Electrical Engineering, California Institute of Technology · Department of Computer Science and Technology, Tsinghua University · Department of Computer Sciences, University of Wisconsin - Madison

Large Language Model Training Batch Optimizer Design Mechanics

Open paper AskPDF arXiv

The pre-training of Large Language Models (LLMs) is increasingly conducted across multiple data centers. As training scales to a larger number of accelerators, the fraction of time spent on computation decreases, while the fraction spent on communication increases. Therefore, frequent synchronization becomes a growing bottleneck. Local update methods reduce this cost by allowing workers to perform several optimizer steps between synchronizations. Most local update methods set the number of local optimizer steps between synchronizations before training and keep this interval fixed throughout the run. However, the best interval can change during the entire train process. If the interval and optimizer are adapted to the current training state, the communication frequency is reduced while maintaining the training performance. In this work, we introduce AutoLoCo, an adaptive training framework to reduce communication in LLM training. It adapts the local interval using scalar training statistics and corrects each outer update. Our method is motivated by two observations: 1) the appropriate local interval varies across training stages, and 2) changing the number of inner steps per interval creates a mismatch with an unchanged outer optimizer, requiring a correction to the outer update. We optimize this mismatch by correction of the outer optimizer for the momentum and the learning rate using the accumulated inner learning rate. Our experiments under communication constraints demonstrate that AutoLoCo reduces communication frequency by 27% relative to DiLoCo while maintaining training performance.

Momentum

Related topics

Latest papers 24