Hyperparameter Optimization

Also known as HPO

Overview

Hyperparameter optimization studies how to choose settings such as learning rates, regularization, architecture, batch size, and algorithm-specific controls using finite validation or compute budgets. Common approaches range from grid and random search to Bayesian optimization with Gaussian processes or tree-structured Parzen estimators, bandit and multi-fidelity methods that stop weak runs early, and evolutionary or gradient-based bilevel methods. Meta-learning, scaling laws, transferred priors, and language-model agents can reuse prior experiments or guide proposals, including for costly language-model training, reinforcement learning, and scientific applications. A central challenge is that useful settings and method rankings vary across datasets, seeds, scales, and hardware, so sample efficiency must be balanced against reliable validation, transferability, and tuning cost.

Latest papers

CardsList

179 papers18 papers in the last four weeks, up 20% on the four weeks before. 0.1% of all new papers.

  1. SPSA Hyperparameter Tuning for Variational Quantum Natural Language Inference

    Aug 11, 2026Nayan D'Souza, Christopher J. AgostinoHyperparameter OptimizationQuantum NLP

  2. Batch Size or Negatives? A Selection Rule for Memory-Constrained Recommender Training

    Aug 11, 2026Artyom Sabitov, Daniil Volkov, Alexey ZaytsevMemory-Efficient OptimizationRecommender Systems

  3. On Cross-Validation for Hyperparameter Optimization of Deep Learning Image Classifiers

    Aug 10, 2026Ljubomir ButurovicHyperparameter OptimizationCross-Validation

  4. Out-Of-The-Loop Multi-Fidelity Bayesian Optimization

    Aug 4, 2026Gustavo Sutter, Hao Wang, Luis Ricardez-Sandoval +2Multi-Fidelity OptimizationBayesian Optimization

  5. Predicting Deep Neural Network Training Outcomes from Early Training Telemetry

    Aug 4, 2026Ranjita Naik, Anh D. Nguyen, Pankaj Kumar SinghLearning Curve PredictionNeural Network Training Dynamics

  6. A Simple Approximation to the Distribution of the Ridge Regression Estimator

    Aug 3, 2026José Luis Montiel Olea, Ryan Strong, Amilcar Velez +2Ridge RegressionHyperparameter Optimization

  7. AgentHPOBench: A Benchmark For Evaluating LLM Agents as Sequential Hyperparameter Optimizers

    Jul 31, 2026Tianyu Huai, Tingshuo Fan, Xinchi Chen +5LLM Agent EvaluationAI Agent Benchmarks

  8. Efficient Heteroscedastic Bayesian Optimization for Risk-Aware AutoRL

    Jul 29, 2026Mingxuan Che, Tsung-Yuan Tseng, Theresa Eimer +2Bayesian OptimizationRisk-Sensitive RL

  9. Using Data-Derived Priors to Guide CNN Architecture Design for NIR Chemometrics

    Jul 28, 2026Dário PassosConvolutional Neural NetworksNeural Architecture Search

  10. Formulation-Level Auto-Tuning for QUBO-Based Machine Learning: A Case Study Across Multiple Quantum-Inspired Annealers

    Jul 21, 2026Naoya Mizuki, Takahiro Katagiri, Daichi Mukunoki +1Black-Box OptimizationQuantum-Inspired ML

  11. Automated Discovery Has No Universally Superior Harness

    Jul 20, 2026Akshat Gupta, Jermaine Lei, Alexander Lu +2Automated Algorithm DiscoveryAutomated Heuristic Design

  12. Which Hyperparameters Matter? A Game-Theoretic Framework for Interpretable Hyperparameter Sensitivity Analysis

    Jul 17, 2026Nyi Nyi Aung, Heepeom Shin, Abigail Lawlor +1Hyperparameter OptimizationSensitivity Analysis

  13. Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures

    Jul 9, 2026Hafsa Mateen, Radu Timofte, Dmitry IgnatovLearning Rate SchedulingNeural Network Optimization

  14. Bayesian Optimization of Genetic Algorithm Hyperparameters in a Multi-Fidelity Framework for Efficient Lattice Material Design

    Jul 8, 2026Sergei Zorkaltsev, Maciej Haranczyk, Christina SchenkMulti-Fidelity OptimizationBayesian Optimization

  15. The Regularization Parameter: Sparse Precision Matrix Estimation

    Jul 7, 2026Aryan Eftekhari, Daniel Sergio Vega, Ernst-Jan Camiel Wit +1Graph Structure LearningParameter Estimation

  16. Beyond Heuristic Tuning: Power-Calibrated LLM Watermarking

    Jul 6, 2026Xiaopu Wang, Zelin He, Chengyuan Liu +1Statistical Power AnalysisLLM Watermarking

  17. TabPack: Efficient Hyperparameter Ensembles for Tabular Deep Learning

    Jul 6, 2026Yury Gorishniy, Akim Kotelnikov, Ivan Rubachev +1Multilayer PerceptronsEnsemble Learning

  18. MMAO-Cls: Metabolic Multi-Agent Optimization for Joint Feature Selection and Classifier Tuning

    Jul 1, 2026Jinliang Xu, Liping MaModel SelectionMulti-Agent System Optimization

  19. How to Allocate Your Tokens? Scaling Laws with Training Steps and Batch Size

    Jul 1, 2026Fabian SchaippLanguage Model Scaling LawsScaling Laws

  20. Class-Grouped Normalized Momentum and Faster Hyperparameter Exploration to Tackle Class Imbalance in Federated Learning

    Jul 1, 2026Haemin Park, Diego Klabjan, Martin W. Braun +2Class-Imbalanced LearningDeep Learning Optimization

  21. SemiScope: Disentangling Classifier Tuning and Joint Optimization in Semi-Supervised Security Classification

    Jun 30, 2026Rui Shu, Tianpei Xia, Jingzhu HeSelf-TrainingCybersecurity

  22. A Stationary-Distribution Theory for Triplet-Based Plateau Search in Random Forest Ensemble-Size Selection

    Jun 29, 2026Andrey A. Dukhovny, Andrey M. LangeRandom ForestHyperparameter Optimization

  23. Evolutionary Hyperparameter Optimization to Find Lightweight CNN Models for Autonomous Steering

    Jun 29, 2026Devson Butani, Ryan Kaddis, Chan-Jin ChungEfficient Neural Network InferenceAutonomous Driving

  24. Bilevel Optimization for Neural Architecture Search

    Jun 28, 2026Abhishek Shukla, Ankur Sinha, Faiz HamidBilevel OptimizationNeural Architecture Search

  25. How Good Can Linear Models Be for Time-Series Forecasting?

    Jun 25, 2026Lang Huang, Jinglue Xu, Luke DarlowTime Series ForecastingHyperparameter Optimization

  26. Statistically Valid Post-Training Hyperparameter Selection: From Tuning to Guarantees

    Jun 24, 2026Amirmohammad Farzaneh, Osvaldo SimeoneHyperparameter Optimization