cs.LGNov 6, 2026

BiMuon: A Geometry-Aware Stochastic Bilevel Optimization Approach

Authors: Zhen Qin, Zhiyao Zhang, Yingbin Liang, Jia Liu

Organizations: Department of Electrical and Computer Engineering, The Ohio State University

Abstract

Stochastic bilevel optimization provides a general framework for many machine learning applications, such as hyperparameter learning, data reweighting, language model alignment, and actor-critic reinforcement learning. However, most existing bilevel optimization methods treat matrix-valued gradients as flattened vectors and perform optimization in the resulting vectorized space, ignoring the matrix structures common in modern neural networks. Moreover, the lower-level (LL) problem curvature can induce significant spectral imbalance in the upper-level (UL) geometry. Such limitations of existing bilevel optimization methods motivate us to develop BiMuon, a geometry-aware stochastic bilevel optimization method that integrates implicit hypergradient estimation in bilevel optimization with Muon-style orthogonalized updates at the UL. To our knowledge, BiMuon is the first stochastic BLO method with Muon-style orthogonalized matrix updates and an O(ε−4)O(ε^{-4}) iteration-complexity guarantee, matching the best-known rates achieved by Muon-type methods based on matrix parameterizations. To improve computational and parallel efficiency for models with multiple matrix parameters, we further study BiMuon for blockwise diagonal matrix approximation and derive a relative stepsize allocation rule based on UL progress and blockwise LL sensitivity. We evaluate BiMuon on representation learning and blockwise bilevel optimization, providing empirical support for our methods.

Figures & tables

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Lower-Level Agnostic Bilevel Optimization

    Nov 6, 2026Peiwen Qiu, Prashant Khanduri, Jia LiuBilevel OptimizationHypergradient Optimization

  2. Efficient Hessian-Free Methods for Multi-Objective Bilevel Optimization with Nonconvex Lower Level

    Aug 13, 2026Yicong Jiang, Feihu HuangBilevel Optimization

  3. To Solve Bilevel Optimization with Nonconvex Lower Levels, We Need Second-Order Stationarity

    Sep 24, 2026Zhiyao Zhang, Menglu Yu, Alvaro Velasquez +2Bilevel OptimizationSecond-Order Optimization