Hyperparameter Transfer

Latest papers 26

All topics
CardsList
  1. σσTransfer: Uncertainty Transfer from Small to Large Networks under μPμ\mathrm{P}

    Oct 8, 2026Richard Bergna, Fernando Ruiz Mazo, Nicolò Felicioni +2Uncertainty QuantificationBayesian Inference

  2. The Best Optimizer Depends on Batch Size

    Oct 6, 2026Xingyu Dang, Kaiyue Wen, Sadhika MalladiDeep Learning OptimizationHyperparameter Transfer

  3. ααTransfer: Coefficient Transfer for Efficient Model Merging

    Oct 6, 2026Shih-Cheng Huang, Zhi Rui Tam, Chieh-Yen Lin +3Hyperparameter TransferLanguage Model Merging

  4. Learning Rate Transfer for Hybrid Transformer-SSM Architectures

    Oct 1, 2026Jimin Seo, Gyubok Lee, Yeonsik Jo +11Hyperparameter Transfer

  5. Fast Learning Rate Transfer in Shallow Linear Networks at Growing Training Horizons

    Sep 28, 2026Mana Sakai, Masaaki ImaizumiHyperparameter TransferNeural Network Training Dynamics

  6. Does Step Law Transfer to Small-Scale Language Models? An Empirical Recalibration Below 59M Parameters

    Sep 23, 2026Egor Romanyukov, Timofey Novikov, Timur Shokarov +3Language Model Scaling LawsHyperparameter Transfer

  7. Hyperparameter Scaling Laws Across MoE Sparsity

    Sep 8, 2026Changxin Tian, Kunlong Chen, Jia Liu +3Language Model Scaling LawsSparse Mixture-of-Experts

  8. LLM-Driven Neural Network Generation with Same-Family Architecture Guidance: Disentangling Transfer and Adaptation

    Jul 6, 2026Kabir Dev Paul Baghel, Radu Timofte, Dmitry IgnatovLarge Language Model-Guided OptimizationHyperparameter Transfer

  9. Hyperparameter Transfer in Graph Neural Networks

    Jul 6, 2026Gage DeZoort, Boris HaninGraph Neural NetworksNeural Network Optimization

  10. On the Nonlinearity of Learning Rate Scaling for LLM Training

    Jun 28, 2026Zaiwen Yang, Huaqing Zhang, Jing Xu +1Language Model Scaling LawsHyperparameter Transfer

  11. Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation

    Jun 24, 2026Luyang Fang, Haoran Lu, Yongkai Chen +2Hyperparameter TransferKnowledge Distillation

  12. Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

    Jun 15, 2026Kaiyue Wen, Xingyu Dang, Kaifeng Lyu +2Language Model PretrainingWeight Decay

  13. Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training

    Jun 4, 2026Yongwei Zhou, Juncheng Diao, Junlin Shang +2Language Model Scaling LawsHyperparameter Transfer

  14. Unlocking Feature Learning in Gated Delta Networks at Scale

    Jun 2, 2026Yifeng Liu, Quanquan GuNeural Network OptimizationHyperparameter Transfer

  15. Quantifying Hyperparameter Transfer and the Importance of Embedding Layer Learning Rate

    May 20, 2026Dayal Singh Kalra, Maissam BarkeshliLanguage Model Scaling LawsHyperparameter Transfer

  16. Oracle Supervision Transfers for Hyperparameter Prediction in Model-Based Image Denoising

    May 19, 2026Jianmin Liao, Lixin Shen, Yuesheng XuHyperparameter TransferTransfer Learning

  17. Toto 2.0: Time Series Forecasting Enters the Scaling Era

    May 19, 2026Emaad Khwaja, Chris Lettieri, Gerald Woo +10Time Series ForecastingHyperparameter Transfer

  18. GQA-μP: The maximal parameterization update for grouped query attention

    May 14, 2026Kyle R. Chickering, Huijuan Wang, Mengxi Wu +7Grouped-Query AttentionRepresentation Learning

  19. Hyperparameter Transfer for Dense Associative Memories

    May 11, 2026Roi Holtzman, Dmitry Krotov, Boris HaninHyperparameter TransferAssociative Memory

  20. Learning Rate Transfer in Normalized Transformers

    Apr 29, 2026Boris Shigida, Boris Hanin, Andrey GromovTransformerHyperparameter Transfer

  21. Scaling Probabilistic Transformer via Efficient Cross-Scale Hyperparameter Transfer

    Apr 28, 2026Penghao Kuang, Haoyi Wu, Kewei TuTransformerNeural Network Optimization

  22. μμpscaling small models: Principled warm starts and hyperparameter transfer

    Feb 11, 2026Yuxin Ma, Nan Chen, Mateo Díaz +3Neural Network OptimizationHyperparameter Transfer

  23. On the Residual Scaling of Looped Transformers: Stability and Transferability

    Date pendingShaowen Wang, Bingrui Li, Ge Zhang +3Residual LearningHyperparameter Transfer

  24. Investigating Hyperparameter Optimization and Transferability for ES-HyperNEAT: A TPE Approach

    Date pendingRomain Claret, Michael O'Neill, Paul Cotofrei +1Evolutionary OptimizationHyperparameter Transfer