Hyperparameter Optimization

Also known as HPO

Hyperparameter optimization studies how to choose settings such as learning rates, regularization, architecture, batch size, and algorithm-specific controls using finite validation or compute budgets. Common approaches range from grid and random search to Bayesian optimization with Gaussian processes or tree-structured Parzen estimators, bandit and multi-fidelity methods that stop weak runs early, and evolutionary or gradient-based bilevel methods. Meta-learning, scaling laws, transferred priors, and language-model agents can reuse prior experiments or guide proposals, including for costly language-model training, reinforcement learning, and scientific applications. A central challenge is that useful settings and method rankings vary across datasets, seeds, scales, and hardware, so sample efficiency must be balanced against reliable validation, transferability, and tuning cost.

Latest papers

161 papers15 papers in the last four weeks, down 17% on the four weeks before. 0.1% of all new papers.

Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH
Jan 18, 2026Beicheng Xu, Weitong Qian, Lingching Tung +2Automated Machine Learning Hyperparameter Optimization
School of Computer Science, Peking University, Beijing, China
Survival of the fittest Cox model: Pivotal variable selection for time-to-event data
Oct 22, 2025Maxime van Cutsem, Sylvain SardyModel Selection Survival Analysis
Department of Mathematics, University of Geneva
Bayesian Optimization under Uncertainty for Training a Scale Parameter in Stochastic Models
Oct 7, 2025Akash Yadav, Ruda ZhangBayesian Optimization Hyperparameter Optimization
University of Houston
Sparse but Wrong: Incorrect L0 Leads to Incorrect Features in Sparse Autoencoders
Aug 22, 2025David Chanin, Adrià Garriga-AlonsoActivation Sparsity LLM Interpretability
University College London · MATS Research
A statistical physics framework for optimal learning
Jul 10, 2025Francesca Mignacco, Francesco MoriStatistical Physics of Learning Neural Network Training Dynamics
1Joseph Henry Laboratories of Physics, Princeton University, Princeton, NJ 08544, USA · 2Graduate Center, City University of New York, New York, NY 10016, USA · Center of Mathematical Sciences and Applications, Harvard University, Cambridge, MA 02138, USA
Temporal horizons in forecasting: a performance-learnability trade-off
Jun 4, 2025Pau Vilimelis Aceituno, Jack William Miller, Noah Marti +2Long-Term Time Series Forecasting Time Series Forecasting
Institute of Neuroinformatics, ETH Zürich and University of Zürich, Winterhurerstrasse 190, Zürich 8057, Switzerland · School of Computing, National Australian University, 108 North Rd, Acton ACT 2601, Australia · Unit of Land Change Science, Swiss Federal Research Institute for Forest, Snow and Landscape Zürcherstrasse 111, Birmensdorf 8903, Switzerland
Accelerating Optimization via Differentiable Stopping Time
May 28, 2025Zhonglin Xie, Yiman Fong, Haoran Yuan +1Differentiable Optimization Hyperparameter Optimization
Beijing International Center for Mathematical Research, Peking University · Department of Industrial Engineering, Tsinghua University · School of Mathematics Science, Peking University
Fully Offline Reinforcement Learning
May 28, 2025Mattie Fellows, Clarisse Wibault, Uljad Berdica +3Bayesian RL Offline RL
Foerster Lab for AI Research (FLAIR), Department of Engineering Science, University of Oxford · Machine Learning Research Group, Department of Engineering Science, University of Oxford
ExpTest: Loss-Curve Hypothesis Testing for Autonomous Learning-Rate Selection in Deep Neural Networks
Nov 25, 2024Zan Chaudhry, Naoko MizunoLearning Rate Scheduling Neural Network Training Dynamics
Laboratory of Structural Cell Biology, National Institutes of Health, Bethesda, MD, 20892 USA · Ragon Institute, Cambridge, MA, 02139
Multi-Objective Hyperparameter Search via Damped Gauss--Newton Optimization
Jan 7, 2024Qinwu Xu, Yifan JiangGauss-Newton Optimization Second-Order Optimization
Efficient Cross-Validation for Sparse Linear Regression
Jun 26, 2023Ryan Cory-Wright, Andrés GómezModel Selection Cross-Validation
Department of Analytics, Marketing and Operations, Imperial Business School, London, UK · Department of Industrial and Systems Engineering, Viterbi School of Engineering, University of Southern California, CA

Jan 18, 2026J/K move · Enter open · S save

Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH

Beicheng Xu, Weitong Qian, Lingching Tung, Yupeng Lu+1

School of Computer Science, Peking University, Beijing, China

Automated Machine Learning Hyperparameter Optimization Large Language Model-Guided Optimization Bayesian Optimization Monte Carlo Tree Search

Open paper AskPDF arXiv

To lower the expertise barrier in machine learning, the AutoML community has focused on the CASH problem, which jointly automates algorithm selection and hyperparameter tuning. While traditional methods like Bayesian Optimization (BO) struggle with cold-start issues, Large Language Models (LLMs) can mitigate these through semantic priors. However, existing LLM-based optimizers generalize poorly to high-dimensional, structured CASH spaces. CASH requires coordinated exploration and exploitation both across candidate algorithms and within their algorithm-specific hyperparameter spaces. Yet existing approaches often rely on flat optimization histories that mix observations across algorithms and obscure configuration-refinement trajectories. In this paper, we propose LB-MCTS, a trajectory-structured optimization framework that uses a Monte Carlo Tree Search tree as a shared state for algorithm selection, hyperparameter refinement, and BO-LLM proposer synergy. Within this shared state, BO provides algorithm-specific surrogate modeling for quantitative search, while the LLM exploits path-aware selective memory to generate semantic proposals and reflections. As the surrogate model improves, a reliability-aware proposer policy adaptively shifts from LLM-driven to BO-driven proposals within a unified search trajectory. Experiments on 104 AMLB datasets demonstrate that LB-MCTS consistently outperforms BO-based, LLM-based, and hybrid baselines. Code: https://github.com/PKU-DAIR/open-box.