cs.LGAug 2, 2026

Training nGPT

Authors: Ilya LoshchilovBoris Ginsburg

Organizations: NVIDIA

Abstract

The normalized Transformer (nGPT) realizes hyperspherical representation learning by constraining model parameter vectors and activation vectors to the unit hypersphere. In this paper, we describe a practical training recipe for nGPT and evaluate it on modern hybrid Mamba-2--Transformer Mixture-of-Experts (MoE) models. The recipe introduces Logit Gradient Preconditioning, Logarithmic Learning Rate Decay, GatedAdamW, angular update control, and optional exploration mechanisms. Compared with an unnormalized model of the same hybrid MoE architecture trained with AdamW, the 30B-total-parameter nGPT model reaches the same validation loss using approximately half as many training tokens. The recipe scales across the models considered, which contain up to 30B total parameters.

Explore similar work

CardsList
  1. Learning Rate Transfer in Normalized Transformers

    Apr 29, 2026Boris Shigida, Boris Hanin, Andrey GromovLearning RatesTransfer Learning

  2. Muown Implicitly Performs Angular Step-size Decay

    Jun 22, 2026Florian Hübler, Kai Lion, Antonio Orvieto +1Muon OptimizerMuon