Hyperparameter Transfer

Ask about this topic

Overview

Hyperparameter transfer asks when settings such as learning rate, batch size, and weight decay tuned in one model or training run will work in another. Common approaches use maximal-update parameterizations such as μP, spectral or smoothness analyses, and empirical scaling laws fitted on smaller proxy models. Researchers test these methods across language models, mixture-of-experts, graph networks, and other architectures, aiming to reduce costly tuning at large scale. Transfer can be reliable under suitable scaling rules, but its success depends on architecture, optimizer, batch size, and training horizon, and scaling laws may need recalibration across regimes.

Latest papers

CardsList

42 papers11 papers in the last four weeks, up 120% on the four weeks before. 0.1% of all new papers.

  1. Improving Data-Constrained Scaling with Hyperparameter Transfer and Ensembling

    Oct 27, 2026Ethan Baron, Shmuel Berman, Anirudh Satheesh +3Language Model Scaling LawsHyperparameter Transfer

    New York University · Princeton University · Q Labs

  2. MetaARIMA: Configuration-Level Transfer Learning for ARIMA Models

    Oct 25, 2026Vitor Cerqueira, Ricardo Inácio, Carlos SoaresTime Series ModelingTime Series Forecasting

    University of Coimbra; CISUC/LASI – Centre for Informatics and Systems of the University of Coimbra; Coimbra, Portugal · Faculty of Engineering, University of Porto; Porto, Portugal · Faculty of Engineering, University of Porto; Laboratory for Artificial Intelligence and Computer Science (LIACC), Portugal; Fraunhofer Portugal AICOS, Portugal; Porto, Portugal

  3. The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It

    Oct 20, 2026Hannan Mahadik, David Salinas, Jacek Golebiowski +1Direct Preference OptimizationHyperparameter Optimization

    ELLIS Institute Tübingen Germany · ELLIS Institute Tübingen & Prior Labs Germany · distil labs Germany

  4. Towards Hyperparameter Transfer for Differentially Private Optimization

    Oct 16, 2026Tianze Wang, Zhiqi Bu, Linjun ZhangNeural Network OptimizationHyperparameter Transfer

    Department of Statistics, Rutgers University · Independent Researcher

  5. Measuring the local shape of learning-rate transfer curves

    Oct 15, 2026Satoki Ishikawa, Rio Yokota, Ryo KarakidaNeural Network OptimizationNeural Network Training Dynamics

    Institute of Science Tokyo · AIST

  6. ShellElastic: Scalable Elastic Pretraining with Simultaneous Maximal Updates and Hyperparameter Transfer

    Oct 10, 2026Can Jin, Yan Kang, Yifan Gong +5Elastic Neural NetworksTransformer

    Rutgers University · Adobe Research

  7. On the Depth Transfer of Neural Optimizers via L-Smoothness

    Oct 9, 2026Maximilian Wong, Chenyu Xu, Jiajin Li +1Deep Learning OptimizationHyperparameter Transfer

    University of Chicago · Peking University · Sauder School of Business University of British Columbia +1

  8. σσTransfer: Uncertainty Transfer from Small to Large Networks under μPμ\mathrm{P}

    Oct 8, 2026Richard Bergna, Fernando Ruiz Mazo, Nicolò Felicioni +2Uncertainty QuantificationBayesian Inference

    University of Cambridge · Spotify

  9. The Best Optimizer Depends on Batch Size

    Oct 6, 2026Xingyu Dang, Kaiyue Wen, Sadhika MalladiDeep Learning OptimizationHyperparameter Transfer

    Princeton University · Stanford University · University of California, San Diego

  10. ααTransfer: Coefficient Transfer for Efficient Model Merging

    Oct 6, 2026Shih-Cheng Huang, Zhi Rui Tam, Chieh-Yen Lin +3Hyperparameter TransferLanguage Model Merging

    Appier AI Research · National Taiwan University

  11. Learning Rate Transfer for Hybrid Transformer-SSM Architectures

    Oct 1, 2026Jimin Seo, Gyubok Lee, Yeonsik Jo +11Hyperparameter TransferHybrid Transformer-State Space Model Architectures

    Seoul National University · SB Intuitions · LG AI Research +1

  12. Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons

    Sep 28, 2026Mana Sakai, Masaaki ImaizumiHyperparameter TransferNeural Network Training Dynamics

    The University of Tokyo · RIKEN Center for Advanced Intelligence Project · Kyoto University

  13. Balancing Early Performance Sacrifices with Long-Term Gains: Scaling Learning-Rate Warmup Duration Across Training Horizons

    Sep 27, 2026Kristi Topollai, Anna ChoromanskaLearning Rate SchedulingNeural Network Training Dynamics

    New York University

  14. Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters

    Sep 23, 2026Egor Romanyukov, Timofey Novikov, Timur Shokarov +3Language Model Scaling LawsHyperparameter Transfer

    Faculty of Computer Science HSE University

  15. The Drift Contract: Spectral Updates for Depth-Robust Local Learning

    Sep 16, 2026Fabien PollySpectral OptimizationLocal Learning

    Independent researcher

  16. Hyperparameter Scaling Laws Across MoE Sparsity

    Sep 8, 2026Changxin Tian, Kunlong Chen, Jia Liu +3Language Model Scaling LawsSparse Mixture-of-Experts

    Ling Team, Ant Group

  17. Investigating Hyperparameter Optimization and Transferability for ES-HyperNEAT: A TPE Approach

    Aug 31, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1Evolutionary OptimizationHyperparameter Transfer

    University of Neuchâtel Neuchâtel, Switzerland · University College Dublin Dublin, Ireland

  18. Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts

    Aug 20, 2026Nayeon Kim, Hojin Lee, Yunju Bak +2Hyperparameter TransferHyperparameter Optimization

    Kakao Corp. · Upstage AI

  19. Pitfalls and Remedies for Multi-Task Bayesian Optimization

    Jul 10, 2026Carl Hvarfner, Sam Daulton, Max Balandat +1Bayesian OptimizationHyperparameter Transfer

    Meta

  20. LLM-Driven Neural Network Generation with Same-Family Architecture Guidance: Disentangling Transfer and Adaptation

    Jul 6, 2026Kabir Dev Paul Baghel, Radu Timofte, Dmitry IgnatovLarge Language Model-Guided OptimizationHyperparameter Transfer

    Computer Vision Lab, CAIDAS, University of Würzburg, Germany

  21. Hyperparameter Transfer in Graph Neural Networks

    Jul 6, 2026Gage DeZoort, Boris HaninGraph Neural NetworksNeural Network Optimization

    Department of Operations Research and Financial Engineering · School of Engineering and Applied Science · Princeton University +1

  22. On the Nonlinearity of Learning Rate Scaling for LLM Training

    Jun 28, 2026Zaiwen Yang, Huaqing Zhang, Jing Xu +1Language Model Scaling LawsHyperparameter Transfer

    Tsinghua University · Tsinghua University, Shanghai Qi Zhi Institute

  23. Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation

    Jun 24, 2026Luyang Fang, Haoran Lu, Yongkai Chen +2Hyperparameter TransferKnowledge Distillation

    Department of Statistics University of Georgia, Athens, GA, 30602, USA

  24. Loop-μμP: Residual Scaling and Hyperparameter Transfer in Looped Transformers

    Jun 16, 2026Shaowen Wang, Bingrui Li, Tansheng Zhu +4Residual LearningHyperparameter Transfer

    Tsinghua University · ByteDance Seed · M-A-P

  25. Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

    Jun 15, 2026Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2Language Model PretrainingWeight Decay

    Stanford University. · Princeton University. · Tsinghua University.

  26. Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training

    Jun 4, 2026Yongwei Zhou, Juncheng Diao, Junlin Shang +2Language Model Scaling LawsHyperparameter Transfer

    MeiTuan · University of Chinese Academy of Sciences · Harbin Institute of Technology

  27. Unlocking Feature Learning in Gated Delta Networks at Scale

    Jun 2, 2026Yifeng Liu, Quanquan GuNeural Network OptimizationHyperparameter Transfer

    University of California, Los Angeles

  28. Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models

    May 22, 2026Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong +3TransformerNeural Network Optimization

    Adobe Research

  29. Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate

    May 20, 2026Dayal Singh Kalra, Maissam BarkeshliLanguage Model Scaling LawsHyperparameter Transfer

    Department of Computer Science, University of Maryland, College Park · Department of Physics, University of Maryland, College Park · Joint Quantum Institute, University of Maryland, College Park +1

  30. Oracle Supervision Transfers for Hyperparameter Prediction in Model-Based Image Denoising

    May 19, 2026Jianmin Liao, Lixin Shen, Yuesheng XuHyperparameter TransferTransfer Learning

    Department of Mathematics Syracuse University · Department of Mathematics & Statistics Old Dominion University