cs.LGNov 12, 2026

Factored One-sided Adam-Moment for Practical and Scalable SOAP

Authors: Jaemyung Yu, Byeongho Heo, Sangdoo Yun, Dongyoon Han

Organizations: NAVER AI Lab

Abstract

Second-order optimizers (e.g., Shampoo, SOAP) have emerged as compelling alternatives to Adam/AdamW and promise faster per-step convergence than AdamW in transformer training. However, they still remain costly in practice, and even SOAP still pays substantial optimizer-state memory and eigendecomposition overhead. We ask whether the benefits of SOAP can be retained at nearly the efficiency of AdamW. In this paper, we answer yes with a lightweight SOAP variant, Factored One-sided Adam-Moment (FOAM). The path to practical impact is twofold: first, FOAM employs input-only preconditioning, dropping the output-side Kronecker factor to reduce memory and remove one eigendecomposition. The Kronecker structure of the Hessian shows that this is exact for a diagonal output-side curvature and nearly lossless for a near-diagonal one, as we observe empirically. Second, FOAM replaces SOAP's dense rotated second moment with a factored Adam-moment estimator based on row-by-column statistics. Under the K-FAC Fisher assumption with a diagonal output curvature, this estimator exactly recovers SOAP's dense second moment, and we measure only a 15-20% K-FAC violation, with negligible effect on the update. Our empirical results support the claim that FOAM matches strong SOAP and Muon baselines in validation loss across scales of large language model (LLM) pre-training, uses half of SOAP's optimizer state, runs at Muon-class wall-clock time, and retains robustness across a three-fold learning-rate range. The results hold from 124M to 760M parameters, demonstrating that SOAP's gains need not require SOAP's cost. Code: https://github.com/naver-ai/foam

Figures & tables

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. AdamPM: Memory-Efficient LLM Optimization via Moment Pooling

    Oct 17, 2026Xiaowei Ji, Lishan Xu, Jianing An +3Optimizer State CompressionEfficient Language Model Training

  2. Gefen: Optimized Stochastic Optimizer

    Jun 11, 2026Nadav Benedek, Tomer Koren, Ohad FriedDeep Learning OptimizationMemory-Efficient Optimization

  3. Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

    Aug 26, 2026Xiaodong Wu, Wenyi Yu, Chao Zhang +1Deep Learning OptimizationMuon Optimizer