Hyperparameter Optimization

Also known as HPO

Hyperparameter optimization studies how to choose settings such as learning rates, regularization, architecture, batch size, and algorithm-specific controls using finite validation or compute budgets. Common approaches range from grid and random search to Bayesian optimization with Gaussian processes or tree-structured Parzen estimators, bandit and multi-fidelity methods that stop weak runs early, and evolutionary or gradient-based bilevel methods. Meta-learning, scaling laws, transferred priors, and language-model agents can reuse prior experiments or guide proposals, including for costly language-model training, reinforcement learning, and scientific applications. A central challenge is that useful settings and method rankings vary across datasets, seeds, scales, and hardware, so sample efficiency must be balanced against reliable validation, transferability, and tuning cost.

Latest papers

161 papers16 papers in the last four weeks, down 11% on the four weeks before. 0.1% of all new papers.

Jan 18, 2026

Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH

To lower the expertise barrier in machine learning, the AutoML community has focused on the CASH problem, which jointly automates algorithm selection and hyperparameter tuning. While traditional methods like Bayesian Optimization (BO) struggle with cold-start issues, Large Language Models (LLMs) can mitigate these through semantic priors. However, existing LLM-based optimizers generalize poorly to high-dimensional, structured CASH spaces. CASH requires coordinated exploration and exploitation both across candidate algorithms and within their algorithm-specific hyperparameter spaces. Yet existing approaches often rely on flat optimization histories that mix observations across algorithms and obscure configuration-refinement trajectories. In this paper, we propose LB-MCTS, a trajectory-structured optimization framework that uses a Monte Carlo Tree Search tree as a shared state for algorithm selection, hyperparameter refinement, and BO-LLM proposer synergy. Within this shared state, BO provides algorithm-specific surrogate modeling for quantitative search, while the LLM exploits path-aware selective memory to generate semantic proposals and reflections. As the surrogate model improves, a reliability-aware proposer policy adaptively shifts from LLM-driven to BO-driven proposals within a unified search trajectory. Experiments on 104 AMLB datasets demonstrate that LB-MCTS consistently outperforms BO-based, LLM-based, and hybrid baselines. Code: https://github.com/PKU-DAIR/open-box.

Beicheng Xu, Weitong Qian, Lingching Tung +2

School of Computer Science, Peking University, Beijing, China

Automated Machine Learning Hyperparameter Optimization Large Language Model-Guided Optimization Bayesian Optimization Monte Carlo Tree Search

Ask PDF

Oct 22, 2025

Survival of the fittest Cox model: Pivotal variable selection for time-to-event data

We revisit Cox's proportional hazards model to improve variable selection in survival analysis. A square-root transformation of the partial likelihood renders the selection of the regularization parameter pivotal, free of the unknown baseline hazard and censoring mechanism. The resulting criterion borrows from information criteria such as BIC and from penalized regression methods such as the lasso, taking the best of both. On simulated and real data, our method substantially improves upon state-of-the-art approaches used daily in support recovery.

Maxime van Cutsem, Sylvain Sardy

Department of Mathematics, University of Geneva

Model Selection Survival Analysis Feature Selection Hyperparameter Optimization

Ask PDF

Oct 7, 2025

Bayesian Optimization under Uncertainty for Training a Scale Parameter in Stochastic Models

Hyperparameter tuning is a challenging problem especially when the system itself involves uncertainty. Due to noisy function evaluations, optimization under uncertainty can be computationally expensive. In this paper, we present a novel Bayesian optimization framework tailored for hyperparameter tuning under uncertainty, with a focus on optimizing a scale- or precision-type parameter in stochastic models. The proposed method employs a statistical surrogate for the underlying random variable, enabling analytical evaluation of the expectation operator. Moreover, we derive a closed-form expression for the optimizer of the random acquisition function, which significantly reduces computational cost per iteration. Numerical experiments demonstrate substantial gains in optimization efficiency: about 40-fold reduction in data usage relative to a Monte Carlo-based one-dimensional optimization scheme, and an order of magnitude improvement relative to a Gaussian process-based Bayesian optimization baseline.

Akash Yadav, Ruda Zhang

University of Houston

Bayesian Optimization Hyperparameter Optimization Stochastic Optimization

Ask PDF

Aug 22, 2025

Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders

Sparse Autoencoders (SAEs) extract features from LLM internal activations, meant to correspond to interpretable concepts. A core SAE training hyperparameter is L0: how many SAE features should fire per token on average. Existing work compares SAE algorithms using sparsity-reconstruction tradeoff plots, implying L0 is a free parameter with no inherently correct value aside from its effect on reconstruction. In this work we study the effect of L0 on SAEs, and show that if L0 is not set correctly, the SAE fails to disentangle the underlying features of the LLM. If L0 is too low, the SAE will mix correlated features to improve reconstruction. If L0 is too high, the SAE finds degenerate solutions that also mix features. Further, we present a proxy metric that can help guide the search for the correct L0 for an SAE on a given training distribution. We show that our method finds the correct L0 in toy models and coincides with peak sparse probing performance in LLM SAEs. We find that most commonly used SAEs have an L0 that is too low. Our work shows that practitioners must set L0 correctly to train SAEs with monosemantic features.

David Chanin, Adrià Garriga-Alonso

University College London · MATS Research

Activation Sparsity LLM Interpretability Representation Disentanglement Sparse Autoencoders Sparse Coding Hyperparameter Optimization

Ask PDF

Jul 10, 2025

A statistical physics framework for optimal learning

Learning is a complex dynamical process shaped by a range of interconnected decisions. Careful design of hyperparameter schedules for artificial neural networks or efficient allocation of cognitive resources by biological learners can dramatically affect performance. Yet, theoretical understanding of optimal learning strategies remains sparse, especially due to the intricate interplay between evolving metaparameters and nonlinear learning dynamics. The search for optimal protocols is further hindered by the high dimensionality of the learning space, often resulting in predominantly heuristic, difficult to interpret, and computationally demanding solutions. Here, we combine statistical physics with control theory in a unified theoretical framework to identify optimal learning protocols in prototypical neural network models. In the high-dimensional limit, we derive closed-form ordinary differential equations that track online stochastic gradient descent through low-dimensional order parameters. We formulate the design of learning protocols as an optimal control problem directly on the dynamics of the order parameters with the goal of minimizing the generalization error. This formulation encompasses a variety of learning scenarios, optimization constraints, and control budgets. We apply it to representative cases, including optimal curricula, adaptive dropout regularization and noise schedules in denoising autoencoders. We find nontrivial yet interpretable strategies highlighting how optimal protocols mediate learning trade-offs. Our results establish a principled foundation for understanding and designing optimal protocols and suggest a path toward a theory of meta-learning grounded in statistical physics.

Francesca Mignacco, Francesco Mori

1Joseph Henry Laboratories of Physics, Princeton University, Princeton, NJ 08544, USA · 2Graduate Center, City University of New York, New York, NY 10016, USA · Center of Mathematical Sciences and Applications, Harvard University, Cambridge, MA 02138, USA

Statistical Physics of Learning Neural Network Training Dynamics Statistical Learning Theory Optimal Control Hyperparameter Optimization

Ask PDF

Jun 4, 2025

Temporal horizons in forecasting: a performance-learnability trade-off

When training autoregressive models to forecast dynamical systems, a critical question arises: how far into the future should the model be trained to predict for optimal performance? In this work, we address this question by analyzing the relationship between the geometry of the loss landscape and the training time horizon. Using dynamical systems theory, we prove that loss minima for long horizons generalize well to short-term forecasts, whereas minima found on short horizons result in worse long-term predictions. However, we also prove that the loss landscape becomes rougher as the training horizon grows, making long-horizon training inherently challenging. We validate our theory through numerical experiments and discuss practical implications for selecting training horizons. Our results provide a principled foundation for hyperparameter optimization in autoregressive forecasting models.

Pau Vilimelis Aceituno, Jack William Miller, Noah Marti +2

Institute of Neuroinformatics, ETH Zürich and University of Zürich, Winterhurerstrasse 190, Zürich 8057, Switzerland · School of Computing, National Australian University, 108 North Rd, Acton ACT 2601, Australia · Unit of Land Change Science, Swiss Federal Research Institute for Forest, Snow and Landscape Zürcherstrasse 111, Birmensdorf 8903, Switzerland

Long-Term Time Series Forecasting Time Series Forecasting Loss Landscape Geometry Hyperparameter Optimization Autoregressive Modeling

Ask PDF

May 28, 2025

Accelerating Optimization via Differentiable Stopping Time

Optimization is an important module of modern machine learning applications. Tremendous efforts have been made to accelerate optimization algorithms. A common formulation is achieving a lower loss at a given time. This enables a differentiable framework with respect to the algorithm hyperparameters. In contrast, its dual, minimizing the time to reach a target loss, is believed to be non-differentiable, as the time is not differentiable. As a result, it usually serves as a conceptual framework or is optimized using zeroth-order methods. To address this limitation, we propose a differentiable stopping time and theoretically justify it based on differential equations. An efficient algorithm is designed to backpropagate through it. As a result, the proposed differentiable stopping time enables a new differentiable formulation for accelerating algorithms. We further discuss its applications, such as online hyperparameter tuning and learning to optimize. Our proposed methods show superior performance in comprehensive experiments across various problems, which confirms their effectiveness.

Zhonglin Xie, Yiman Fong, Haoran Yuan +1

Beijing International Center for Mathematical Research, Peking University · Department of Industrial Engineering, Tsinghua University · School of Mathematics Science, Peking University

Differentiable Optimization Hyperparameter Optimization Learning to Optimize

Ask PDF

May 28, 2025

Fully Offline Reinforcement Learning

Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.

Mattie Fellows, Clarisse Wibault, Uljad Berdica +3

Foerster Lab for AI Research (FLAIR), Department of Engineering Science, University of Oxford · Machine Learning Research Group, Department of Engineering Science, University of Oxford

Bayesian RL Offline RL Policy Learning Hyperparameter Optimization Model-Based RL

Ask PDF

Nov 25, 2024

ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networks

Hyperparameter tuning remains a significant challenge in the training of deep neural networks (DNNs), requiring manual search or time-intensive grid searches that increase resource costs and limit the accessibility of machine learning. The global initial learning rate is among the most consequential of these hyperparameters. Adaptive and scheduling-based methods manage the learning rate during training but still require manual selection of an initial global value; learning-rate-free alternatives remove this selection at the cost of performance or stability on non-convex problems. We present ExpTest, an autonomous learning-rate controller that treats the training loss curve as an online signal and performs sequential statistical tests on theoretically motivated windows to detect convergent behavior and trigger learning-rate reductions. The framework combines a covariance-based initial learning-rate estimate, curvature-motivated window sizing, and two-phase test-driven decay, drawing on the approximately exponential decay behavior predicted under linearized network dynamics. We provide a mathematical motivation for ExpTest and evaluate it on regression, classification, forecasting, and natural-language tasks across fully-connected, convolutional, transformer-based, and pretrained architectures. Across these tasks, ExpTest achieves competitive performance relative to hand-tuned SGD-based baselines and recent learning-rate-free methods, without manual initial learning-rate selection or predefined scheduling.

Zan Chaudhry, Naoko Mizuno

Laboratory of Structural Cell Biology, National Institutes of Health, Bethesda, MD, 20892 USA · Ragon Institute, Cambridge, MA, 02139

Learning Rate Scheduling Neural Network Training Dynamics Hyperparameter Optimization

Ask PDF

Jan 7, 2024

Multi-Objective Hyperparameter Search via Damped Gauss--Newton Optimization

We study hyperparameter optimization (HPO) from a numerical-optimization perspective and propose a multi-objective damped Newton--Gauss--Newton search method. Rather than perturbing each hyperparameter separately or treating model evaluations as independent trials, the method uses performance changes between successive full hyperparameter vectors to construct an iterative secant approximation of the local sensitivity matrix. Each iteration therefore requires only one new full-vector model evaluation while jointly updating all optimized hyperparameters. A Tikhonov-regularized Gauss--Newton system stabilizes the update when the number of hyperparameters exceeds the number of performance objectives. The search is initialized from readily available empirical/default settings of the underlying learner, without requiring a separate initialization search. We evaluate four-dimensional XGBoost HPO on three public classification datasets against exhaustive grid search, random search, and tree-structured Parzen estimator (TPE) optimization. On Breast Cancer Wisconsin, the proposed method matches the best validation accuracy of a 320-configuration grid search with slightly better log loss and ROC--AUC. Across three datasets and three seeds, predictive performance remains competitive with random search and TPE while using substantially fewer search iterations. A supplementary proprietary eight-dimensional threshold-optimization case study further demonstrates joint multi-parameter optimization under competing performance targets and reveals non-monotonic, oscillatory trajectories that motivate damping and best-iterate selection. Overall, the results establish iterative secant sensitivity as an evaluation-efficient local alternative to exhaustive HPO.

Qinwu Xu, Yifan Jiang

Gauss-Newton Optimization Second-Order Optimization Multi-Objective Optimization Hyperparameter Optimization

Ask PDF

Jun 26, 2023

Efficient Cross-Validation for Sparse Linear Regression

Given a high-dimensional covariate matrix and a response vector, ridge-regularized sparse linear regression selects a subset of features that explains the relationship between covariates and the response in an interpretable manner. To choose hyperparameters that control the sparsity level and amount of regularization, practitioners commonly use k-fold cross-validation. However, cross-validation substantially increases the computational cost of sparse regression as it requires solving many mixed-integer optimization problems (MIOs) for each hyperparameter combination. To address this computational burden, we derive computationally tractable relaxations of the k-fold cross-validation loss, facilitating hyperparameter selection while solving

50

80\%

fewer MIOs in practice. Our computational results demonstrate, across eleven real-world UCI datasets, that exact MIO-based cross-validation can be competitive with mature software packages such as glmnet and L0Learn.

Ryan Cory-Wright, Andrés Gómez

Department of Analytics, Marketing and Operations, Imperial Business School, London, UK · Department of Industrial and Systems Engineering, Viterbi School of Engineering, University of Southern California, CA

Model Selection Cross-Validation Mixed-Integer Linear Programming Linear Regression Hyperparameter Optimization High-Dimensional Linear Regression

Ask PDF