cs.LGSep 7, 2026

HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball

Authors: Jinghui Yuan, Hongtao Zhang, Jade Zou, Tianyu Li, Wenjie Zhou, Tianyu He, Wei Chen

Organizations: StepFun · University of Chinese Academy of Sciences, StepFun · University of Chinese Academy of Sciences · Tsinghua University, StepFun · 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Hyperball optimizers constrain parameter norms and update only their directions, establishing a distinct paradigm for neural network optimization. Although this geometry appears fundamentally different from that of conventional Base Optimizers, which update both parameter norms and directions, we show that the two paradigms are dynamically equivalent for scale-invariant networks. Building on this equivalence, we propose HyperTransfer, which constructs a Hyperball optimizer that reproduces the dynamics of a target Base Optimizer using only its initialization and learning-rate schedule, without running the target optimizer itself. We further derive the inverse mapping and extend the framework to non-scale-invariant networks. Experiments show that both HyperTransfer and the inverse mapping produce loss trajectories nearly identical to those of their targets, suggesting that Hyperball dynamics are governed primarily by the induced effective learning-rate schedule and optimizer state.

Figures & tables

Figure 1: Geometric interpretation of HyperTransfer.
Figure 2: Bidirectional transfer between independently tuned Muon and MuonH baselines on the scale-invariant model M1\mathcal{M}_{1} under a linear learning-rate schedule decaying to zero. The two baselines exhibit distinct training dynamics: Muon reduces the validation loss substantially faster during the early stage of training, whereas MuonH eventually overtakes Muon and achieves a lower final loss. In panel (a), HyperTransfer enables MuonH to reproduce the target Muon trajectory. In panel (b), Inverse HyperTransfer enables Muon to reproduce the target MuonH trajectory.
Figure 3: Parameter-wise dynamical alignment under HyperTransfer on the strictly scale-invariant network M1\mathcal{M}_{1} with a linear learning-rate schedule decaying to zero. Left: the proxy norm sts_{t} versus the parameter norm ∥wt∥\lVert w_{t}\rVert along the target Muon run. Right: the effective learning rate estimated by HyperTransfer versus that measured from Muon. In both panels, the curves closely overlap across representative hidden matrices, showing that HyperTransfer aligns the norm and effective-learning-rate dynamics of the target optimizer.
Figure 4: Bidirectional trajectory transfer on the non-scale-invariant network M2\mathcal{M}_{2} under nominal learning-rate schedules ending at zero and at 0.1×0.1\times the peak learning rate. (a) HyperTransfer+: MuonH follows the target Muon trajectory. (b) Inverse HyperTransfer+: Muon follows the target MuonH trajectory. (c) Effective-learning-rate trajectories of the Muon and MuonH baselines. The transferred trajectories closely match their respective targets under both schedules, whereas the nominally matched schedules induce different effective-learning-rate decay profiles.
Figure 5: Parameter-wise final-to-peak effective-learning-rate ratios of Muon on the non-scale-invariant network M2\mathcal{M}_{2} under a nominal final-to-peak ratio of 0.10.1 . Rows correspond to parameter types and columns correspond to Transformer blocks, with each cell reporting the ratio for one hidden matrix. The average across all hidden matrices is approximately 4.7×10−24.7\times 10^{-2} , showing that the nominal decay ratio does not equal the effective-learning-rate decay ratio for Muon.
Figure 6: Transfer under matched final-to-peak effective-learning-rate ratios on the non-scale-invariant network M2\mathcal{M}_{2} . The peak learning rates are searched separately, while the effective final-to-peak ratio is matched across the two optimizers. The left panel compares the Muon baseline, the MuonH baseline, and the Muon trajectory obtained through Inverse HyperTransfer+. The right panel compares the corresponding Muon and MuonH effective-learning-rate schedules. The effective-learning-rate decay ratio of MuonH is matched to the average ratio measured for Muon, approximately 4.7×10−24.7\times 10^{-2} . The transferred Muon trajectory remains close to the target MuonH trajectory under effective-learning-rate alignment.
Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Architecture of the strictly scale-invariant variant. RMSNorm (denoted by LN) is applied before each attention and FFN sublayer and after the selected linear transformations within the attention and FFN modules, ensuring invariance of the corresponding weight matrices to positive rescaling. The non-scale-invariant variant follows the original NanoGPT configuration and retains only the block-level pre-norm and Q/K normalization, removing the post-linear normalization.
Figure 8: Parameter-wise proxy norm tracking under HyperTransfer on the scale-invariant network with a learning-rate schedule decaying to zero. Each panel corresponds to one hidden matrix. The red dashed curve shows the proxy norm sts_{t} maintained by the MuonH→Muon\mathrm{MuonH}\rightarrow\mathrm{Muon} transfer run, while the blue solid curve shows the actual parameter norm ∥wt∥\lVert w_{t}\rVert measured along the target Muon trajectory. Their close agreement demonstrates that sts_{t} accurately recovers the parameter-norm evolution of the target optimizer.
Figure 9: Parameter-wise effective-learning-rate tracking for the scale-invariant network under a schedule decaying to zero. Each small panel corresponds to one hidden matrix. The red dashed curves show the HyperTransfer estimates, while the blue curves show the corresponding Muon references.
Figure 10: Parameter-wise nominal learning-rate schedules of Muon induced by Inverse HyperTransfer on the scale-invariant network, using a reference Hyperball schedule that decays to zero. Each panel corresponds to one hidden matrix. The red dashed curve shows the nominal learning rate ηt\eta_{t} assigned to Muon, while the blue solid curve shows the nominal learning-rate schedule of the independently tuned Muon reference.
Figure 11: Parameter-wise proxy norm tracking under HyperTransfer+ on the non-scale-invariant network with a learning-rate schedule decaying to zero. Each panel corresponds to one hidden matrix. The red dashed curve shows the proxy norm sts_{t} maintained by the MuonH→Muon\mathrm{MuonH}\rightarrow\mathrm{Muon} transfer run, while the blue solid curve shows the actual parameter norm ∥wt∥\lVert w_{t}\rVert measured along the target Muon trajectory. Their close agreement demonstrates that sts_{t} accurately recovers the parameter-norm evolution of the target optimizer.
Figure 12: Parameter-wise effective-learning-rate tracking for the non-scale-invariant network under a schedule decaying to zero. Each small panel corresponds to one hidden matrix. The red dashed curves show the HyperTransfer+ estimates, while the blue curves show the corresponding Muon references.
Figure 13: Parameter-wise nominal learning-rate schedules of Muon induced by Inverse HyperTransfer+ on the non-scale-invariant network, using a reference Hyperball schedule that decays to zero. Each panel corresponds to one hidden matrix. The red dashed curve shows the nominal learning rate ηt\eta_{t} assigned to Muon, while the blue solid curve shows the nominal learning-rate schedule of the independently tuned Muon reference.
Figure 14: Parameter-wise proxy norm tracking under HyperTransfer+ on the non-scale-invariant network with a learning-rate schedule decaying to 0.1×0.1\times the peak learning rate. Each panel corresponds to one hidden matrix. The red dashed curve shows the proxy norm sts_{t} maintained by the MuonH→Muon\mathrm{MuonH}\rightarrow\mathrm{Muon} transfer run, while the blue solid curve shows the actual parameter norm ∥wt∥\lVert w_{t}\rVert measured along the target Muon trajectory.
Figure 15: Parameter-wise effective-learning-rate tracking for the non-scale-invariant network under a schedule decaying to 0.1×0.1\times the peak learning rate. Each small panel corresponds to one hidden matrix. The red dashed curves show the HyperTransfer+ estimates, while the blue curves show the corresponding Muon references.
Figure 16: Parameter-wise nominal learning-rate schedules of Muon induced by Inverse HyperTransfer+ on the non-scale-invariant network, using a reference Hyperball schedule that decays to 0.1×0.1\times peak learning rate. Each panel corresponds to one hidden matrix. The red dashed curve shows the nominal learning rate ηt\eta_{t} assigned to Muon, while the blue solid curve shows the nominal learning-rate schedule of the independently tuned Muon reference.
Figure 17: Bidirectional transfer between AdamW and AdamH on the scale-invariant network M1\mathcal{M}_{1} under a linear learning-rate schedule decaying to zero. The left panel shows HyperTransfer, where AdamH →\rightarrow AdamW closely follows the target AdamW loss trajectory. The right panel shows Inverse HyperTransfer, where AdamW →\rightarrow AdamH closely follows the target AdamH trajectory.
Figure 18: HyperTransfer on the 500M-parameter dense language model trained on 70B tokens. The upper panel compares the training-loss trajectories of the target Muon Optimizer and the transferred MuonH. The lower panel shows the difference between their training losses. The results demonstrate that HyperTransfer maintains its transfer capability even in long-duration training.
Figure 19: Inverse HyperTransfer on the 500M-parameter dense language model trained on 70B tokens. The upper panel compares the training-loss trajectories of the target MuonH optimizer and the Base Muon Optimizer obtained through Inverse HyperTransfer. The lower panel shows the difference between their training losses. The results demonstrate that Inverse HyperTransfer maintains its transfer capability even in long-duration training.
More in this topic

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 331 papers

Explore similar work

CardsList
Jul 24, 2026

Hyperball May Not Be a Free Lunch

For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates. However, the source of their advantage remains unclear. Starting from the angular displacement between consecutive parameter states, we derive an angular effective learning rate that accounts for the parameter-update angle, parameter norm, and update norm. We also show that the conventional norm-based measure is a special case under parameter-update orthogonality. We then decompose optimizer updates into radial and tangential components and analyze how radial updates affect one-step angular displacement. Under the training configurations considered, numerical results show that the radial component has only a limited direct effect on the angular effective learning rate. It therefore cannot explain why MuonH converges more slowly than MuonWD early in training but overtakes it later. To further isolate the underlying mechanism, we devise a heuristic experiment that modifies only the learning-rate schedule so that the dynamics of each optimizer reproduce those of the other. The results suggest that their main difference stems from the evolution of the effective step size rather than an intrinsically superior update direction induced by Hyperball. Our pretraining experiments further show that more aggressive learning-rate decay can accelerate MuonH early in training but may impair its later performance. Thus, maintaining a constant angular velocity does not eliminate the learning-rate-scheduling problem; careful scheduling remains essential to realizing the potential of Hyperball-style optimizers. Our code is publicly available at https://github.com/mangocrazz/hyperball-may-not-be-a-free-lunch.

Yihao Xiao, Jialong Sun, Zitian Gao +5

1IQuest Research · 4Shenzhen University of Advanced Technology · 2Peking University +2

Learning Rate SchedulingNeural Network OptimizationMuon OptimizerNeural Network Training Dynamics
AskPDF
Mar 30, 2026

Rethinking Language Model Scaling under Transferable Hypersphere Optimization

Scaling laws for large language models depend critically on the optimizer and parameterization. Existing hyperparameter transfer laws are mainly developed for first-order optimizers, and they do not structurally prevent training instability at scale. Recent hypersphere optimization methods constrain weight matrices to a fixed-norm hypersphere, offering a promising alternative for more stable scaling. We introduce HyperP (Hypersphere Parameterization), the first framework for transferring optimal learning rates across model width, depth, training tokens, and Mixture-of-Experts (MoE) granularity under the Frobenius-sphere constraint with the Muon optimizer. We prove that weight decay is a first-order no-op on the Frobenius sphere, show that Depth-μμP remains necessary, and find that the optimal learning rate follows the same data-scaling power law with the "magic exponent" 0.32 previously observed for AdamW. A single base learning rate tuned at the smallest scale transfers across all compute budgets under HyperP, yielding 1.58×1.58\times compute efficiency over a strong Muon baseline at 6×10216\times10^{21} FLOPs. Moreover, HyperP delivers transferable stability: all monitored instability indicators, including ZZ-values, output RMS, and activation outliers, remain bounded and non-increasing under training FLOPs scaling. We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities for improved granularity scaling, and show that hypersphere optimization enables substantially larger auxiliary load-balancing weights, yielding both strong performance and good expert balance. We release our training codebase at https://github.com/microsoft/ArchScale.

Liliang Ren, Yang Liu, Yelong Shen +1

Microsoft

Language Model Scaling LawsHyperparameter TransferConstrained OptimizationMixture-of-Experts Language Models
AskPDF
Jun 15, 2026

Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

Matrix based optimizers such as Muon can substantially speed up language model pretraining, but their gains over AdamW are observed to shrink as model size and data scale grow when using standard constant decoupled weight decay. We propose Hyperball, a simple optimizer wrapper that addresses this issue. Given a base optimizer such as Adam or Muon, Hyperball sets the Frobenius norms of weight matrices and their corresponding optimizer updates to fixed constants. On Qwen3 style models up to 1.2B parameters, Muon Hyperball achieves 20--30% token equivalent speedup over weight decay baselines. Hyperball also improves learning rate transfer across widths and depths compared to decoupled weight decay. This method is motivated by prior theory showing that training with weight decay leads to an equilibrium weight norm that only depends on the training hyperparameters. Through this mechanism, the weight decay then decides the angular learning rate, i.e. how fast the direction of the weight matrix changes.

Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2

†Stanford University. · ‡Princeton University. · §Tsinghua University.

Language Model PretrainingWeight DecayNeural Network OptimizationHyperparameter TransferConstrained Optimization
AskPDF