cs.AIMay 11, 2026

Optimizer-Induced Mode Connectivity: From AdamW to Muon

Authors: Fangzhao ZhangSungyoon KimErica ZhangYiqi JiangMert Pilanci

Organizations: Stanford University

Abstract

Mode connectivity has been widely studied, yet the role of the optimizer remains underexplored. We revisit it through optimizer-induced implicit regularization, asking how connectivity behaves when restricted to solutions constrained by a given optimizer. For two-layer ReLU networks, we show that solutions from a single optimizer -- AdamW, Muon, or others in the Lion-K\mathcal{K} family -- form a connected set at sufficiently large width, a result not implied by prior work. We then characterize how optimizer-induced regions interact: at large width two different regions can be disjoint or overlap depending on regularization, while in our small-width example AdamW and Muon converge to disconnected zero-loss components separated by a provable loss barrier. Empirically, in GPT-2 pretraining, we observe same-optimizer paths preserve each model's spectrum while cross-optimizer paths traverse a smooth transition. Our results reveal optimizer-dependent structure beyond classical mode connectivity literature.

Explore similar work

CardsList
  1. Why Muon Outperforms Adam: A Curvature Perspective

    Jun 3, 2026Shuche Wang, Fengzhuo Zhang, Jiaxiang Li +2AdamMuon