HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball

Jinghui Yuan; Hongtao Zhang; Jade Zou; Tianyu Li; Wenjie Zhou; Tianyu He; Wei Chen

cs.LGSep 7, 2026

HyperTransfer: Understanding the Equivalence between Base Optimizer and Hyperball

Authors: Jinghui Yuan, Hongtao Zhang, Jade Zou, Tianyu Li, Wenjie Zhou, Tianyu He, Wei Chen

Organizations: StepFun · University of Chinese Academy of Sciences, StepFun · University of Chinese Academy of Sciences · Tsinghua University, StepFun ·

Abstract

Hyperball optimizers constrain parameter norms and update only their directions, establishing a distinct paradigm for neural network optimization. Although this geometry appears fundamentally different from that of conventional Base Optimizers, which update both parameter norms and directions, we show that the two paradigms are dynamically equivalent for scale-invariant networks. Building on this equivalence, we propose HyperTransfer, which constructs a Hyperball optimizer that reproduces the dynamics of a target Base Optimizer using only its initialization and learning-rate schedule, without running the target optimizer itself. We further derive the inverse mapping and extend the framework to non-scale-invariant networks. Experiments show that both HyperTransfer and the inverse mapping produce loss trajectories nearly identical to those of their targets, suggesting that Hyperball dynamics are governed primarily by the induced effective learning-rate schedule and optimizer state.

Figures & tables

Figure 1: Geometric interpretation of HyperTransfer.

Figure 2: Bidirectional transfer between independently tuned Muon and MuonH baselines on the scale-invariant model

\mathcal{M}_{1}

under a linear learning-rate schedule decaying to zero. The two baselines exhibit distinct training dynamics: Muon reduces the validation loss substantially faster during the early stage of training, whereas MuonH eventually overtakes Muon and achieves a lower final loss. In panel (a), HyperTransfer enables MuonH to reproduce the target Muon trajectory. In panel (b), Inverse HyperTransfer enables Muon to reproduce the target MuonH trajectory.

Figure 3: Parameter-wise dynamical alignment under HyperTransfer on the strictly scale-invariant network

\mathcal{M}_{1}

with a linear learning-rate schedule decaying to zero. Left: the proxy norm

s_{t}

versus the parameter norm

\lVert w_{t}\rVert

along the target Muon run. Right: the effective learning rate estimated by HyperTransfer versus that measured from Muon. In both panels, the curves closely overlap across representative hidden matrices, showing that HyperTransfer aligns the norm and effective-learning-rate dynamics of the target optimizer.

Figure 4: Bidirectional trajectory transfer on the non-scale-invariant network

\mathcal{M}_{2}

under nominal learning-rate schedules ending at zero and at

0.1\times

the peak learning rate. (a) HyperTransfer+: MuonH follows the target Muon trajectory. (b) Inverse HyperTransfer+: Muon follows the target MuonH trajectory. (c) Effective-learning-rate trajectories of the Muon and MuonH baselines. The transferred trajectories closely match their respective targets under both schedules, whereas the nominally matched schedules induce different effective-learning-rate decay profiles.

Figure 5: Parameter-wise final-to-peak effective-learning-rate ratios of Muon on the non-scale-invariant network $\mathcal{M}_{2}$ under a nominal final-to-peak ratio of $0.1$ . Rows correspond to parameter types and columns correspond to Transformer blocks, with each cell reporting the ratio for one hidden matrix. The average across all hidden matrices is approximately $4.7\times 10^{-2}$ , showing that the nominal decay ratio does not equal the effective-learning-rate decay ratio for Muon.

Figure 6: Transfer under matched final-to-peak effective-learning-rate ratios on the non-scale-invariant network

\mathcal{M}_{2}

. The peak learning rates are searched separately, while the effective final-to-peak ratio is matched across the two optimizers. The left panel compares the Muon baseline, the MuonH baseline, and the Muon trajectory obtained through Inverse HyperTransfer+. The right panel compares the corresponding Muon and MuonH effective-learning-rate schedules. The effective-learning-rate decay ratio of MuonH is matched to the average ratio measured for Muon, approximately

4.7\times 10^{-2}

. The transferred Muon trajectory remains close to the target MuonH trajectory under effective-learning-rate alignment.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Architecture of the strictly scale-invariant variant. RMSNorm (denoted by LN) is applied before each attention and FFN sublayer and after the selected linear transformations within the attention and FFN modules, ensuring invariance of the corresponding weight matrices to positive rescaling. The non-scale-invariant variant follows the original NanoGPT configuration and retains only the block-level pre-norm and Q/K normalization, removing the post-linear normalization.

Figure 8: Parameter-wise proxy norm tracking under HyperTransfer on the scale-invariant network with a learning-rate schedule decaying to zero. Each panel corresponds to one hidden matrix. The red dashed curve shows the proxy norm

s_{t}

maintained by the

\mathrm{MuonH}\rightarrow\mathrm{Muon}

transfer run, while the blue solid curve shows the actual parameter norm

\lVert w_{t}\rVert

measured along the target Muon trajectory. Their close agreement demonstrates that

s_{t}

accurately recovers the parameter-norm evolution of the target optimizer.

Figure 9: Parameter-wise effective-learning-rate tracking for the scale-invariant network under a schedule decaying to zero. Each small panel corresponds to one hidden matrix. The red dashed curves show the HyperTransfer estimates, while the blue curves show the corresponding Muon references.

Figure 10: Parameter-wise nominal learning-rate schedules of Muon induced by Inverse HyperTransfer on the scale-invariant network, using a reference Hyperball schedule that decays to zero. Each panel corresponds to one hidden matrix. The red dashed curve shows the nominal learning rate

\eta_{t}

assigned to Muon, while the blue solid curve shows the nominal learning-rate schedule of the independently tuned Muon reference.

Figure 11: Parameter-wise proxy norm tracking under HyperTransfer+ on the non-scale-invariant network with a learning-rate schedule decaying to zero. Each panel corresponds to one hidden matrix. The red dashed curve shows the proxy norm

s_{t}

maintained by the

\mathrm{MuonH}\rightarrow\mathrm{Muon}

transfer run, while the blue solid curve shows the actual parameter norm

\lVert w_{t}\rVert

measured along the target Muon trajectory. Their close agreement demonstrates that

s_{t}

accurately recovers the parameter-norm evolution of the target optimizer.

Figure 12: Parameter-wise effective-learning-rate tracking for the non-scale-invariant network under a schedule decaying to zero. Each small panel corresponds to one hidden matrix. The red dashed curves show the HyperTransfer+ estimates, while the blue curves show the corresponding Muon references.

Figure 13: Parameter-wise nominal learning-rate schedules of Muon induced by Inverse HyperTransfer+ on the non-scale-invariant network, using a reference Hyperball schedule that decays to zero. Each panel corresponds to one hidden matrix. The red dashed curve shows the nominal learning rate

\eta_{t}

assigned to Muon, while the blue solid curve shows the nominal learning-rate schedule of the independently tuned Muon reference.

Figure 14: Parameter-wise proxy norm tracking under HyperTransfer+ on the non-scale-invariant network with a learning-rate schedule decaying to

0.1\times

the peak learning rate. Each panel corresponds to one hidden matrix. The red dashed curve shows the proxy norm

s_{t}

maintained by the

\mathrm{MuonH}\rightarrow\mathrm{Muon}

transfer run, while the blue solid curve shows the actual parameter norm

\lVert w_{t}\rVert

measured along the target Muon trajectory.

Figure 15: Parameter-wise effective-learning-rate tracking for the non-scale-invariant network under a schedule decaying to

0.1\times

the peak learning rate. Each small panel corresponds to one hidden matrix. The red dashed curves show the HyperTransfer+ estimates, while the blue curves show the corresponding Muon references.

Figure 16: Parameter-wise nominal learning-rate schedules of Muon induced by Inverse HyperTransfer+ on the non-scale-invariant network, using a reference Hyperball schedule that decays to

0.1\times

peak learning rate. Each panel corresponds to one hidden matrix. The red dashed curve shows the nominal learning rate

\eta_{t}

assigned to Muon, while the blue solid curve shows the nominal learning-rate schedule of the independently tuned Muon reference.

Figure 17: Bidirectional transfer between AdamW and AdamH on the scale-invariant network

\mathcal{M}_{1}

under a linear learning-rate schedule decaying to zero. The left panel shows HyperTransfer, where AdamH

\rightarrow

AdamW closely follows the target AdamW loss trajectory. The right panel shows Inverse HyperTransfer, where AdamW

\rightarrow

AdamH closely follows the target AdamH trajectory.

Figure 18: HyperTransfer on the 500M-parameter dense language model trained on 70B tokens. The upper panel compares the training-loss trajectories of the target Muon Optimizer and the transferred MuonH. The lower panel shows the difference between their training losses. The results demonstrate that HyperTransfer maintains its transfer capability even in long-duration training.

Figure 19: Inverse HyperTransfer on the 500M-parameter dense language model trained on 70B tokens. The upper panel compares the training-loss trajectories of the target MuonH optimizer and the Base Muon Optimizer obtained through Inverse HyperTransfer. The lower panel shows the difference between their training losses. The results demonstrate that Inverse HyperTransfer maintains its transfer capability even in long-duration training.

Neural Network Optimization

Neural network optimization studies algorithms and training dynamics that help models learn efficiently, reliably, and generalize well.

See all 331 papers

Hyperball May Not Be a Free Lunch
Jul 24, 2026Yihao Xiao, Jialong Sun, Zitian Gao +5Learning Rate Scheduling Neural Network Optimization
1IQuest Research · 4Shenzhen University of Advanced Technology · 2Peking University +2
Rethinking Language Model Scaling under Transferable Hypersphere Optimization
Mar 30, 2026Liliang Ren, Yang Liu, Yelong Shen +1Language Model Scaling Laws Hyperparameter Transfer
Microsoft
Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization
Jun 15, 2026Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2Language Model Pretraining Weight Decay
†Stanford University. · ‡Princeton University. · §Tsinghua University.

Jul 24, 2026J/K move · Enter open · S save

Hyperball May Not Be a Free Lunch

Yihao Xiao, Jialong Sun, Zitian Gao, Zeming Wei+4

1IQuest Research · 4Shenzhen University of Advanced Technology · 2Peking University+2

Learning Rate Scheduling Neural Network Optimization Muon Optimizer Neural Network Training Dynamics

Open paper AskPDF arXiv

For scale-invariant deep networks, Hyperball-style optimizers have shown strong performance in large-scale training by fixing the norms of matrix-valued parameters and normalizing updates. However, the source of their advantage remains unclear. Starting from the angular displacement between consecutive parameter states, we derive an angular effective learning rate that accounts for the parameter-update angle, parameter norm, and update norm. We also show that the conventional norm-based measure is a special case under parameter-update orthogonality. We then decompose optimizer updates into radial and tangential components and analyze how radial updates affect one-step angular displacement. Under the training configurations considered, numerical results show that the radial component has only a limited direct effect on the angular effective learning rate. It therefore cannot explain why MuonH converges more slowly than MuonWD early in training but overtakes it later. To further isolate the underlying mechanism, we devise a heuristic experiment that modifies only the learning-rate schedule so that the dynamics of each optimizer reproduce those of the other. The results suggest that their main difference stems from the evolution of the effective step size rather than an intrinsically superior update direction induced by Hyperball. Our pretraining experiments further show that more aggressive learning-rate decay can accelerate MuonH early in training but may impair its later performance. Thus, maintaining a constant angular velocity does not eliminate the learning-rate-scheduling problem; careful scheduling remains essential to realizing the potential of Hyperball-style optimizers. Our code is publicly available at https://github.com/mangocrazz/hyperball-may-not-be-a-free-lunch.

Abstract

Neural Network Optimization

Explore similar work

Hyperball May Not Be a Free Lunch

Rethinking Language Model Scaling under Transferable Hypersphere Optimization

Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization