Hyperparameter Optimization

Also known as HPO

Overview

Hyperparameter optimization studies how to choose settings such as learning rates, regularization, architecture, batch size, and algorithm-specific controls using finite validation or compute budgets. Common approaches range from grid and random search to Bayesian optimization with Gaussian processes or tree-structured Parzen estimators, bandit and multi-fidelity methods that stop weak runs early, and evolutionary or gradient-based bilevel methods. Meta-learning, scaling laws, transferred priors, and language-model agents can reuse prior experiments or guide proposals, including for costly language-model training, reinforcement learning, and scientific applications. A central challenge is that useful settings and method rankings vary across datasets, seeds, scales, and hardware, so sample efficiency must be balanced against reliable validation, transferability, and tuning cost.

Latest papers

CardsList

178 papers20 papers in the last four weeks, up 54% on the four weeks before. 0.1% of all new papers.

  1. Large Language Models as a Universal Searcher for Open-World Automated Machine Learning

    Nov 9, 2026Yuxuan Hu, Jihao Liu, Weikang Shi +6Automated Machine LearningLarge Language Model-Guided Optimization

  2. Simple Meta-Learning for HPO and CASH Using Performance Representations

    Nov 5, 2026Emiel Witting, Cheng Yan, Tom Julian VieringHyperparameter OptimizationMeta-Learning

  3. Beyond Foundation Models: PFN Priors Improve Tabular Hyperparameter Optimization

    Nov 3, 2026Artem Chistyakov, Renat Sergazinov, Artem BabenkoHyperparameter OptimizationTabular ML

  4. ExaTune: Exhaustive Hyperparameter Landscape Exploration at Scale

    Nov 2, 2026Kevin Dick, Katie LonswayHyperparameter OptimizationFitness Landscape Analysis

  5. Online Autotuning of Algebraic Multigrid via Contextual Bandits and Reinforcement Learning

    Oct 31, 2026Xudong He, Ruiyu Wang, Yilang Ding +2Hyperparameter OptimizationReinforcement Learning

  6. Log-Driven Semantic Tuning for Mixed Integer Programming Solvers

    Oct 31, 2026Zhengzhong Ricky You, Kaizhao Sun, Zeping Min +2Hyperparameter OptimizationLarge Language Model-Guided Optimization

  7. Cost-Aware Stopping for Freeze-Thaw Bayesian Optimization: When Does One-Step Lookahead Suffice?

    Oct 30, 2026Long Vu, Jongseong Brad Choi, Binoy Ravindran +1Bayesian OptimizationOptimal Stopping

  8. Distributed Grid Search for Hyperparameter Tuning

    Oct 30, 2026Constança De Herdeiro Cachim, Andrea Martin, Nicola BastianelloHyperparameter OptimizationDistributed Optimization

  9. Enhancing Atrial Fibrillation Detection Through Hyperparameter Optimization

    Oct 29, 2026Sara Gjorgjieva, Ilinka Ivanoska, Marjan GusevAtrial Fibrillation DetectionElectrocardiogram Classification

  10. Meta Models: Models that Model Learning

    Oct 27, 2026Peihang Li, Ziming LiuLearning Curve PredictionLanguage Model Performance Prediction

  11. Inference Latency-Aware Tsetlin Machine Training

    Oct 26, 2026Yefan Zeng, Shengyu Duan, Rishad Shafik +1Efficient InferenceHyperparameter Optimization

  12. REDAR: Reinforcement-Driven Adaptive Regularisation for Class Incremental Learning

    Oct 22, 2026Akila Ayanthi, Darshana Priyasad, Tharindu Fernando +3RL HPOClass-Incremental Learning

  13. The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It

    Oct 20, 2026Hannan Mahadik, David Salinas, Jacek Golebiowski +1Direct Preference OptimizationHyperparameter Optimization

  14. IRAH: Hierarchical Agentic Tree Search for Retrieval Pipeline Optimization

    Oct 16, 2026Maria Ana Cardei, Aniket Singh, Amir Moghadam +2Information RetrievalHyperparameter Optimization

  15. FGSS-BO: Feedback-Guided Hypothesis Management for LLM-Based Bayesian Optimization

    Oct 15, 2026Hao Chen, Ziyang Ling, Ronald X. Xu +1Large Language Model-Guided OptimizationBayesian Optimization

  16. Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training

    Oct 14, 2026Anshu Raina, Peyman Razaghi, Yuankai Chen +7Distributed TrainingLLM Training

  17. Revisited Support Vector Machines: A Unified Joint Optimization Framework with Nyström Approximation for Scalable Adaptive Learning

    Oct 11, 2026Nassou Gongon Benjamin, Noulapeu Ngaffo, Patrick Njionou SadjangNyström MethodSupport Vector Machines

  18. Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf

    Oct 9, 2026Priya Kasimbeg, Ahmed Khaled, Aahlad Chandrabhatta +9Deep Learning OptimizationBenchmark Design

  19. Pre-training of Bayesian Optimization Algorithm through Bayesian Optimization

    Oct 7, 2026Satoshi Katayama, Shoyo Hunt, Shintaro Masuda +1Bayesian OptimizationBlack-Box Optimization

  20. Early Memory Selection for Balanced Adam

    Oct 6, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Adaptive Gradient MethodsHyperparameter Optimization

  21. On Hyperparameter Tuning on the Test Set

    Oct 5, 2026Matteo Fregonara, Tom Viering, Jan van GemertModel SelectionBenchmark Validity

  22. Hyperparameter selection for equation learning with biologically-informed neural networks

    Oct 2, 2026William Lavery, Jodie A. Cochrane, John T. Nardini +1Hyperparameter OptimizationPhysics-Informed ML

  23. Calibrating Prediction Timeliness Through Multi-Objective Hyperparameter Optimization for Remaining Useful Life Prediction

    Oct 1, 2026Tugrul Cabir Hakyemez, Ener Uras GokhanRUL EstimationMulti-Objective Evolutionary Optimization

  24. UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?

    Sep 29, 2026Puning Yang, Qizhou Wang, Junchi Yu +2LLM UnlearningHyperparameter Optimization