Hyperparameter Transfer

Ask about this topic

Overview

Hyperparameter transfer asks when settings such as learning rate, batch size, and weight decay tuned in one model or training run will work in another. Common approaches use maximal-update parameterizations such as μP, spectral or smoothness analyses, and empirical scaling laws fitted on smaller proxy models. Researchers test these methods across language models, mixture-of-experts, graph networks, and other architectures, aiming to reduce costly tuning at large scale. Transfer can be reliable under suitable scaling rules, but its success depends on architecture, optimizer, batch size, and training horizon, and scaling laws may need recalibration across regimes.

Latest papers

CardsList

42 papers11 papers in the last four weeks, up 120% on the four weeks before. 0.1% of all new papers.

  1. Toto 2.0: Time Series Forecasting Enters the Scaling Era

    May 19, 2026Emaad Khwaja, Chris Lettieri, Gerald Woo +10Time Series ForecastingHyperparameter Transfer

    Datadog AI Research · Carnegie Mellon University

  2. GQA-μP: The maximal parameterization update for grouped query attention

    May 14, 2026Kyle R. Chickering, Huijuan Wang, Mengxi Wu +7Grouped-Query AttentionRepresentation Learning

    UC Davis & MBZUAI IFM · USC & MBZUAI IFM · USC +4

  3. Hyperparameter Transfer for Dense Associative Memories

    May 11, 2026Roi Holtzman, Dmitry Krotov, Boris HaninHyperparameter TransferAssociative Memory

    Rudolf Peierls Centre for Theoretical Physics, University of Oxford, Oxford OX1 3PU, UK · Dynamical Mind, IBM Research · Princeton

  4. Learning Rate Transfer in Normalized Transformers

    Apr 29, 2026Boris Shigida, Boris Hanin, Andrey GromovTransformerHyperparameter Transfer

    Meta Superintelligence Labs · Princeton University

  5. Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer

    Apr 28, 2026Penghao Kuang, Haoyi Wu, Kewei TuTransformerNeural Network Optimization

    School of Information Science and Technology, ShanghaiTech University

  6. Rethinking Language Model Scaling under Transferable Hypersphere Optimization

    Mar 30, 2026Liliang Ren, Yang Liu, Yelong Shen +1Language Model Scaling LawsHyperparameter Transfer

    Microsoft

  7. Deriving Hyperparameter Scaling Laws via Modern Optimization Theory

    Mar 16, 2026Egor Shulgin, Dimitri von Rütte, Tianyue H. Zhang +3LMO MethodsScaling Laws

    King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi Arabia · ETH Zürich, Zürich, Switzerland · Mila, Quebec AI Institute & Université de Montréal, Quebec, Canada +1

  8. On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer

    Mar 10, 2026Ruihan Xu, Jiajin Li, Yiping LuNeural Network OptimizationHyperparameter Transfer

    University of Chicago · Sauder School of Business, University of British Columbia · Industrial Engineering & Management Science, Northwestern University

  9. Meta-Learning Hyperparameters for Parameter Efficient Fine-Tuning

    Mar 2, 2026Zichen Tian, Yaoyao Liu, Qianru SunParameter-Efficient Fine-TuningVision Foundation Model Adaptation

    Singapore Management University · University of Illinois Urbana-Champaign

  10. Spectral Condition for μμP under Width-Depth Scaling

    Feb 28, 2026Chenyu Zheng, Rongzhen Wang, Xinyu Zhang +1Deep Learning OptimizationHyperparameter Transfer

    Gaoling School of AI, Renmin University of China · ByteDance Seed

  11. Extending μμP: Spectral Conditions for Feature Learning Across Optimizers

    Feb 24, 2026Akshita Gupta, Marieme Ngom, Sam Foreman +1Hyperparameter TransferAdaptive Gradient Methods

    Purdue University · Argonne National Laboratory

  12. μμpscaling small models: Principled warm starts and hyperparameter transfer

    Feb 11, 2026Yuxin Ma, Nan Chen, Mateo Díaz +3Neural Network OptimizationHyperparameter Transfer

    Department of Applied Mathematics and Statistics Johns Hopkins University