Hyperparameter Optimization

Also known as HPO

Overview

Hyperparameter optimization studies how to choose settings such as learning rates, regularization, architecture, batch size, and algorithm-specific controls using finite validation or compute budgets. Common approaches range from grid and random search to Bayesian optimization with Gaussian processes or tree-structured Parzen estimators, bandit and multi-fidelity methods that stop weak runs early, and evolutionary or gradient-based bilevel methods. Meta-learning, scaling laws, transferred priors, and language-model agents can reuse prior experiments or guide proposals, including for costly language-model training, reinforcement learning, and scientific applications. A central challenge is that useful settings and method rankings vary across datasets, seeds, scales, and hardware, so sample efficiency must be balanced against reliable validation, transferability, and tuning cost.

Latest papers

CardsList

178 papers20 papers in the last four weeks, up 54% on the four weeks before. 0.1% of all new papers.

  1. Anomaly-Free Self-Optimization via AUC Bounds

    Sep 23, 2026Kevin Wilkinghoff, Zheng-Hua TanUnsupervised Anomaly DetectionHyperparameter Optimization

  2. Bilevel Optimization of Topology and Hyperparameters (BOTH)

    Sep 18, 2026Suryanarayanan Manoj Sanu, Miguel Anibal Bessa, Alejandro Marcos AragónHypergradient OptimizationTopology Optimization

  3. Agentic Search Spaces for Tabular Machine Learning

    Sep 14, 2026Renat Sergazinov, Artem Chistyakov, Sergey Pankevich +1Large Language Model-Guided OptimizationTabular ML

  4. JumpStart Your Policy Learning with Lessons from 160,000 Training Runs

    Sep 12, 2026Nabil Omi, Eric Bae, Chung Yik Edward Yeung +2Offline RLAlgorithm Selection

  5. Early-Stopping Thresholds for ES-HyperNEAT: A Data-Driven Approach from Fitness Dynamics

    Sep 11, 2026Romain Claret, Arthur Gygax, Michael O'Neill +2Early StoppingEvolutionary Optimization

  6. Parameter Sensitivity Analysis for Aerial LiDAR-Inertial Odometries in low-altitude flights

    Sep 11, 2026Robert Milijas, Jose Ramiro Martinez-de Dios, Stjepan BogdanSimultaneous Localization and MappingUAV Navigation

  7. Revisiting Multi-Object Tracking Baselines: Hyperparameter Optimization with Multi-Fidelity Greedy Coordinate Search

    Sep 10, 2026Momir AdžemovićMulti-Fidelity OptimizationHyperparameter Optimization

  8. TASTE: Throughput-Aware Batch Size Tuning for On-Device Edge Learning

    Sep 7, 2026Avik Bhatnagar, Federico Nicolas Peccia, Oliver BringmannOn-Device TrainingEdge ML

  9. Constrained Bayesian Optimization for Hierarchical Federated Learning in IoT Networks for Plant Disease Classification

    Sep 6, 2026Athanasios Papanikolaou, Athanasios Tziouvaras, Apostolos Xenakis +6Hierarchical Federated LearningInternet of Things

  10. Amortizing Scaling Law Construction Costs

    Sep 4, 2026Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6Language Model Scaling LawsBayesian Optimization

  11. HyperMC: Multi-Fidelity Hyperparameter Tuning for Stochastic Gradient MCMC

    Sep 2, 2026Ming Tan, Xiyun JiaoHyperparameter OptimizationMarkov Chain Monte Carlo

  12. Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search

    Sep 1, 2026Zhiliang Chen, Sebastian Ament, David Eriksson +4Bayesian OptimizationLanguage Model Scaling Laws

  13. Bandits in Prod: Hyperparameter Optimization at Inference Time

    Sep 1, 2026Louis Abraham, Tuan-Anh Nguyen, Nicolas DevatineMulti-Armed BanditsInference-Time Optimization

  14. Post-Training Science for Supervised Fine-Tuning

    Sep 1, 2026Charles O'Neill, Mudith Jayasekara, Harry PartridgeSupervised Fine-TuningHyperparameter Optimization

  15. Investigating Hyperparameter Optimization and Transferability for ES-HyperNEAT: A TPE Approach

    Aug 31, 2026Romain Claret, Michael O'Neill, Paul Cotofrei +1Evolutionary OptimizationHyperparameter Transfer

  16. Constrained Hyperparameter Optimization for Streaming Data

    Aug 25, 2026Bruno Veloso, João GamaHyperparameter OptimizationConstrained Optimization

  17. Free-Probability Kernels for Zero-Rollout Hyperparameter Selection in Reservoir Computing

    Aug 21, 2026Sara Malacarne, Andrea Ceni, Claudio GallicchioReservoir ComputingHyperparameter Optimization

  18. Regularised Iterative Generalised Least Squares with Optimal Selection of the Hyper-Parameter for Identifying Nonlinear Phenomenological Models

    Aug 19, 2026Mark Cary, Charles BokorNonlinear System IdentificationRidge Regression

  19. Tight Bounds for Data-driven Multiple Hyper-parameter Tuning with Structured Loss Function

    Aug 18, 2026Anh Tuan Nguyen, Viet Anh NguyenHyperparameter OptimizationGeneralization Bounds

  20. TRACE-CASH: Trial-History-Conditioned Reinforcement Learning for Adaptive Configuration Exploration in Time-Series CASH

    Aug 17, 2026Yu-Han Huang, Yujia Wu, Vincent S. TsengRL HPOHyperparameter Optimization

  21. Small-Scale Experiments: Are We There Yet?

    Aug 12, 2026Nicholas Lourie, Kyunghyun Cho, Karen Ullrich +1Language Model Scaling LawsHyperparameter Optimization

  22. Dynamics Models for Offline Hyperparameter Selection in Real-World RL

    Aug 11, 2026Jordan Coblin, Han Wang, Martha White +1RL ControlOffline RL

  23. Exploring new directions in enhancing the ACTS parameter optimization suite

    Aug 11, 2026Chance LaVoie, Qi Bin Lei, Rocky Bala Garg +1Hyperparameter OptimizationBayesian Optimization

  24. SPSA Hyperparameter Tuning for Variational Quantum Natural Language Inference

    Aug 11, 2026Nayan D'Souza, Christopher J. AgostinoHyperparameter OptimizationQuantum NLP