Hyperparameter Optimization

Also known as HPO

Overview

Hyperparameter optimization studies how to choose settings such as learning rates, regularization, architecture, batch size, and algorithm-specific controls using finite validation or compute budgets. Common approaches range from grid and random search to Bayesian optimization with Gaussian processes or tree-structured Parzen estimators, bandit and multi-fidelity methods that stop weak runs early, and evolutionary or gradient-based bilevel methods. Meta-learning, scaling laws, transferred priors, and language-model agents can reuse prior experiments or guide proposals, including for costly language-model training, reinforcement learning, and scientific applications. A central challenge is that useful settings and method rankings vary across datasets, seeds, scales, and hardware, so sample efficiency must be balanced against reliable validation, transferability, and tuning cost.

Latest papers

CardsList

178 papers20 papers in the last four weeks, up 54% on the four weeks before. 0.1% of all new papers.

  1. Batch Size or Negatives? A Selection Rule for Memory-Constrained Recommender Training

    Aug 11, 2026Artyom Sabitov, Daniil Volkov, Alexey ZaytsevMemory-Efficient OptimizationRecommender Systems

  2. On Cross-Validation for Hyperparameter Optimization of Deep Learning Image Classifiers

    Aug 10, 2026Ljubomir ButurovicHyperparameter OptimizationCross-Validation

  3. Out-Of-The-Loop Multi-Fidelity Bayesian Optimization

    Aug 4, 2026Gustavo Sutter, Hao Wang, Luis Ricardez-Sandoval +2Multi-Fidelity OptimizationBayesian Optimization

  4. Predicting Deep Neural Network Training Outcomes from Early Training Telemetry

    Aug 4, 2026Ranjita Naik, Anh D. Nguyen, Pankaj Kumar SinghLearning Curve PredictionNeural Network Training Dynamics

  5. A Simple Approximation to the Distribution of the Ridge Regression Estimator

    Aug 3, 2026José Luis Montiel Olea, Ryan Strong, Amilcar Velez +2Ridge RegressionHyperparameter Optimization

  6. AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers

    Jul 31, 2026Tianyu Huai, Tingshuo Fan, Xinchi Chen +5LLM Agent EvaluationAI Agent Benchmarks

  7. Efficient Heteroscedastic Bayesian Optimization for Risk-Aware AutoRL

    Jul 29, 2026Mingxuan Che, Tsung-Yuan Tseng, Theresa Eimer +2Bayesian OptimizationRisk-Sensitive RL

  8. Using Data-Derived Priors to Guide CNN Architecture Design for NIR Chemometrics

    Jul 28, 2026Dário PassosConvolutional Neural NetworksNeural Architecture Search

  9. Formulation-Level Auto-Tuning for QUBO-Based Machine Learning: A Case Study Across Multiple Quantum-Inspired Annealers

    Jul 21, 2026Naoya Mizuki, Takahiro Katagiri, Daichi Mukunoki +1Black-Box OptimizationQuantum-Inspired ML

  10. Automated Discovery Has No Universally Superior Harness

    Jul 20, 2026Akshat Gupta, Jermaine Lei, Alexander Lu +2Automated Algorithm DiscoveryAutomated Heuristic Design

  11. Which Hyperparameters Matter? A Game-Theoretic Framework for Interpretable Hyperparameter Sensitivity Analysis

    Jul 17, 2026Nyi Nyi Aung, Heepeom Shin, Abigail Lawlor +1Hyperparameter OptimizationSensitivity Analysis

  12. Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures

    Jul 9, 2026Hafsa Mateen, Radu Timofte, Dmitry IgnatovLearning Rate SchedulingNeural Network Optimization

  13. Bayesian Optimization of Genetic Algorithm Hyperparameters in a Multi-Fidelity Framework for Efficient Lattice Material Design

    Jul 8, 2026Sergei Zorkaltsev, Maciej Haranczyk, Christina SchenkMulti-Fidelity OptimizationBayesian Optimization

  14. The Regularization Parameter: Sparse Precision Matrix Estimation

    Jul 7, 2026Aryan Eftekhari, Daniel Sergio Vega, Ernst-Jan Camiel Wit +1Graph Structure LearningParameter Estimation

  15. Beyond Heuristic Tuning: Power-Calibrated LLM Watermarking

    Jul 6, 2026Xiaopu Wang, Zelin He, Chengyuan Liu +1Statistical Power AnalysisLLM Watermarking

  16. TabPack: Efficient Hyperparameter Ensembles for Tabular Deep Learning

    Jul 6, 2026Yury Gorishniy, Akim Kotelnikov, Ivan Rubachev +1Multilayer PerceptronsEnsemble Learning

  17. MMAO-Cls: Metabolic Multi-Agent Optimization for Joint Feature Selection and Classifier Tuning

    Jul 1, 2026Jinliang Xu, Liping MaModel SelectionMulti-Agent System Optimization

  18. How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size

    Jul 1, 2026Fabian SchaippLanguage Model Scaling LawsScaling Laws

  19. Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learning

    Jul 1, 2026Haemin Park, Diego Klabjan, Martin W. Braun +2Class-Imbalanced LearningDeep Learning Optimization

  20. SemiScope: Disentangling Classifier Tuning and Joint Optimization in Semi-Supervised Security Classification

    Jun 30, 2026Rui Shu, Tianpei Xia, Jingzhu HeSelf-TrainingCybersecurity

  21. A Stationary-Distribution Theory for Triplet-Based Plateau Search in Random Forest Ensemble-Size Selection

    Jun 29, 2026Andrey A. Dukhovny, Andrey M. LangeRandom ForestHyperparameter Optimization

  22. Evolutionary Hyperparameter Optimization to Find Lightweight CNN Models for Autonomous Steering

    Jun 29, 2026Devson Butani, Ryan Kaddis, Chan-Jin ChungEfficient Neural Network InferenceAutonomous Driving

  23. Bilevel Optimization for Neural Architecture Search

    Jun 28, 2026Abhishek Shukla, Ankur Sinha, Faiz HamidBilevel OptimizationNeural Architecture Search

  24. How Good Can Linear Models Be for Time-Series Forecasting?

    Jun 25, 2026Lang Huang, Jinglue Xu, Luke DarlowTime Series ForecastingHyperparameter Optimization

  25. Statistically Valid Post-Training Hyperparameter Selection: From Tuning to Guarantees

    Jun 24, 2026Amirmohammad Farzaneh, Osvaldo SimeoneHyperparameter Optimization

  26. Solve for the Hyperparameter, Skip the Search: Kolmogorov-Optimal Scaling Laws for Spline Regression

    Jun 22, 2026Yong Yi Bay, Kathleen A. YearickModel SelectionScaling Laws