Hyperparameter Optimization

Also known as HPO

Overview

Hyperparameter optimization studies how to choose settings such as learning rates, regularization, architecture, batch size, and algorithm-specific controls using finite validation or compute budgets. Common approaches range from grid and random search to Bayesian optimization with Gaussian processes or tree-structured Parzen estimators, bandit and multi-fidelity methods that stop weak runs early, and evolutionary or gradient-based bilevel methods. Meta-learning, scaling laws, transferred priors, and language-model agents can reuse prior experiments or guide proposals, including for costly language-model training, reinforcement learning, and scientific applications. A central challenge is that useful settings and method rankings vary across datasets, seeds, scales, and hardware, so sample efficiency must be balanced against reliable validation, transferability, and tuning cost.

Latest papers

CardsList

178 papers20 papers in the last four weeks, up 54% on the four weeks before. 0.1% of all new papers.

  1. GRIMIP: A General Framework for Instance-Specific Configuration of MIP Solvers Using LLMs

    Jun 22, 2026Yidong Luo, Chenguang Wang, Fangzhou Zhu +2Surrogate-Assisted OptimizationMixed-Integer Linear Programming

  2. Leveraging AutoML for Sustainable Deep Learning: A Multi-Objective HPO Approach on Deep Shift Neural Networks

    Jun 22, 2026Leona Hennig, Marius LindauerMulti-Fidelity OptimizationEnergy-Efficient ML

  3. Gradient-Descent Steps to Success over Mean Accuracy: A Paradigm Shift for ML

    Jun 20, 2026Riccardo Poli, Ahmet YilmazAutomated Machine LearningHyperparameter Optimization

  4. Towards Robust Training in NNGPT AutoML Pipeline: A Loss-Optimizer Pairing Selection Study

    Jun 18, 2026Anton Abramochkin, Radu Timofte, Dmitry IgnatovDeep Learning OptimizationNeural Network Optimization

  5. Evolutionary Two-Stage Hyperparameter Optimization Strategies for Physics-Informed Neural Networks

    Jun 18, 2026Fedor Buzaev, Dmitry Efremenko, Egor Bugaev +4Multi-Fidelity OptimizationEvolutionary Optimization

  6. The Hidden Power of Scaling Factor in LoRA Optimization

    Jun 11, 2026Zicheng Zhang, Haoran Li, Jiaxing Wang +10Low-Rank AdaptationHyperparameter Optimization

  7. Importance-Aware Scheduling for High-Dimensional Hyperparameter Optimization

    Jun 8, 2026Ruinan Wang, Ian Nabney, Mohammad GolbabaeeBayesian OptimizationHyperparameter Optimization

  8. Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training

    Jun 4, 2026Yongwei Zhou, Juncheng Diao, Junlin Shang +2Language Model Scaling LawsHyperparameter Transfer

  9. LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval

    Jun 3, 2026Shahrzad Esmat, Chaunte W. Lacewell, Sameh Gobriel +2Large Language Model-Guided OptimizationHyperparameter Optimization

  10. BBOmix: A Tabular Benchmark for Hyperparameter Optimization of Unsupervised Biological Representation Learning

    Jun 3, 2026Luca Thale-Bombien, Jan Ewald, Ralf König +1Representation LearningAutoencoders

  11. Provably Reduced Sample Cost in Prior-Guided Hyperparameter Optimization

    Jun 3, 2026Leona Hennig, Jasmin Brandt, Lukas Fehring +3Multi-Fidelity OptimizationMulti-Armed Bandits

  12. Hyperparameter Learning for Latent Factorization of Tensors for Representation Learning to Large-scale Dynamic Weighted Directed Network

    Jun 3, 2026Yaqian Zhan, Jialan He, Tianzhu ChenDifferential EvolutionGraph Representation Learning

  13. How Many Trees in a Random Forest? A Revisited Approach with Plateau Search and Optuna Integration

    Jun 2, 2026Vadim Porvatov, Andrey Dukhovny, Andrey LangeEnsemble LearningRandom Forest

  14. Efficient Hyperparameter Optimization for LLM Reinforcement Learning

    Jun 2, 2026Minping Chen, Bowen Xiao, Du Liang +2RL for Language ModelsRL Post-Training

  15. Self-Tuning Regularization for Image Scanning Microscopy

    May 29, 2026Sofia Agostoni, Lisa Cuneo, Christian Daniele +5Image ReconstructionImage Restoration

  16. RAISE: RAG Design as an Architecture Search Problem

    May 28, 2026Zhen Chen, Yibing Liu, Weihao Xie +3RAG EvaluationNeural Architecture Search

  17. Selection Hyper-heuristics Can Automatically Adjust the Learning Period to Optimally Solve Pseudo-Boolean Problems

    May 28, 2026Benjamin Doerr, Pietro S. Oliveto, John Alasdair WarwickerMetaheuristic OptimizationCombinatorial Optimization

  18. Optimal ridge regularization revisited

    May 27, 2026Jack Timmermans, Sergio A. AlvarezLinear RegressionStatistical Learning Theory

  19. Oracle Supervision Transfers for Hyperparameter Prediction in Model-Based Image Denoising

    May 19, 2026Jianmin Liao, Lixin Shen, Yuesheng XuHyperparameter TransferTransfer Learning

  20. Training Neural Networks with Optimal Double-Bayesian Learning

    May 19, 2026Vy Bui, Hang Yu, Karthik Kantipudi +2Neural Network OptimizationBayesian Decision Theory

  21. Evolutionary Algorithm for Reservoir Learning and Yielding

    May 19, 2026Julien Testu, Pierrick Legrand, Xavier HinautReservoir ComputingEvolutionary NAS

  22. A Feature-Driven Framework for Software Fault Prediction

    May 17, 2026Ahmad Nauman Ghazi, Nagajyothi Devarapalli, Ashir Javeed +3Feature SelectionHyperparameter Optimization

  23. BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks

    May 16, 2026Ruth Wan Theng Chew, Zhiliang Chen, Apivich Hemachandra +1Bayesian OptimizationBenchmark Design

  24. WASHH: An Anchor-Aware Whale-Guided Selection Hyper-Heuristic for Continuous Optimization and SVC Configuration

    May 13, 2026Yifu Zhao, Xiaofan Zou, Junhao Wei +9Black-Box OptimizationMetaheuristic Optimization