Hyperparameter Optimization

Also known as HPO

Hyperparameter optimization studies how to choose settings such as learning rates, regularization, architecture, batch size, and algorithm-specific controls using finite validation or compute budgets. Common approaches range from grid and random search to Bayesian optimization with Gaussian processes or tree-structured Parzen estimators, bandit and multi-fidelity methods that stop weak runs early, and evolutionary or gradient-based bilevel methods. Meta-learning, scaling laws, transferred priors, and language-model agents can reuse prior experiments or guide proposals, including for costly language-model training, reinforcement learning, and scientific applications. A central challenge is that useful settings and method rankings vary across datasets, seeds, scales, and hardware, so sample efficiency must be balanced against reliable validation, transferability, and tuning cost.

Latest papers

161 papers16 papers in the last four weeks, down 11% on the four weeks before. 0.1% of all new papers.

REDAR: Reinforcement-Driven Adaptive Regularisation for Class Incremental Learning
Oct 22, 2026Akila Ayanthi, Darshana Priyasad, Tharindu Fernando +3Class-Incremental Learning Regularization-Based Continual Learning
SAIVT, Queensland University of Technology, Australia · CSIRO Robotics, CSIRO, Australia
Nominate-then-Adjudicate: LLM-Assisted One-Pass and Per-Instance MIP Solver Tuning
Oct 21, 2026Yifan Zhao, Qianyang Zhao, Xinyu Guan +2Large Language Model-Guided Optimization Mixed-Integer Linear Programming
Alibaba Group, Hangzhou, China
The Shape of the DPO Hyperparameter Landscape: What to Measure, How It Moves with Scale, and When Not to Transfer It
Oct 20, 2026Hannan Mahadik, David Salinas, Jacek Golebiowski +1Direct Preference Optimization Hyperparameter Optimization
ELLIS Institute Tübingen, Germany · ELLIS Institute Tübingen & Prior Labs, Germany · distil labs, Germany
IRAH: Hierarchical Agentic Tree Search for Retrieval Pipeline Optimization
Oct 16, 2026Maria Ana Cardei, Aniket Singh, Amir Moghadam +2Information Retrieval Hyperparameter Optimization
University of Virginia · University of California San Diego · Capital One
FGSS-BO: Feedback-Guided Hypothesis Management for LLM-Based Bayesian Optimization
Oct 15, 2026Hao Chen, Ziyang Ling, Ronald X. Xu +1Large Language Model-Guided Optimization Bayesian Optimization
School of Biomedical Engineering, University of Science and Technology of China, Suzhou, China · School of Engineering Science, University of Science and Technology of China, Suzhou, China
Beyond the Trial-and-Error Loop: Hybrid Projection and Automated Tuning for Distributed Training
Oct 14, 2026Anshu Raina, Peyman Razaghi, Yuankai Chen +7Distributed Training LLM Training
Advanced Micro Devices, Inc.
ATTUNE: Agentic Throughput Tuning for LLM Training
Oct 13, 2026Leo Fan, Hongyang Wang, Wenbo Chen +10Efficient Language Model Training Deep Learning Optimization
Amazon
Revisited Support Vector Machines: A Unified Joint Optimization Framework with Nyström Approximation for Scalable Adaptive Learning
Oct 11, 2026Nassou Gongon Benjamin, Noulapeu Ngaffo, Patrick Njionou SadjangNyström Method Support Vector Machines
Laboratory of Data Science and Artificial Intelligence (LSDIA), National Higher Polytechnic School of Douala (ENSPD), University of Douala, Cameroon · Department of Computer Sciences, National Higher Polytechnic School of Douala (ENSPD), University of Douala, Cameroon · Department of Basic Scientific Education, National Higher Polytechnic School of Douala (ENSPD), University of Douala, Cameroon
Towards Easier-to-Tune Training Algorithms: New Results from AlgoPerf
Oct 9, 2026Priya Kasimbeg, Ahmed Khaled, Aahlad Chandrabhatta +9Deep Learning Optimization Benchmark Design
Alice · Goodfire · University of Tübingen +1
Prior or Feedback? What an LLM Uses When Adapting Neural Operators
Oct 8, 2026Julian Chan, Javier Mora JimenezPartial Differential Equations LLM Decision-Making
Compare the Market
Pre-training of Bayesian Optimization Algorithm through Bayesian Optimization
Oct 7, 2026Satoshi Katayama, Shoyo Hunt, Shintaro Masuda +1Bayesian Optimization Black-Box Optimization
Nagoya Institute of Technology
Early Memory Selection for Balanced Adam
Oct 6, 2026Alberto Fernández-Hernández, Cristian Pérez-Corral, Jose I. Mestre +2Adaptive Gradient Methods Hyperparameter Optimization
Universitat Politècnica de València Valencia, Spain · Universitat Jaume I Castelló de la Plana, Spain
On Hyperparameter Tuning on the Test Set
Oct 5, 2026Matteo Fregonara, Tom Viering, Jan van GemertModel Selection Benchmark Validity
Delft University of Technology
Hyperparameter selection for equation learning with biologically-informed neural networks
Oct 2, 2026William Lavery, Jodie A. Cochrane, John T. Nardini +1Hyperparameter Optimization Physics-Informed ML
Department of Information Technology, Uppsala University, Sweden · Department of Mathematics and Statistics, The College of New Jersey, USA
Calibrating Prediction Timeliness Through Multi-Objective Hyperparameter Optimization for Remaining Useful Life Prediction
Oct 1, 2026Tugrul Cabir Hakyemez, Ener Uras GokhanRUL Estimation Multi-Objective Evolutionary Optimization
Department of Management Information Systems Istanbul Bilgi University Istanbul, Turkey · Beştepe Koleji Ankara, Turkey
UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?
Sep 29, 2026Puning Yang, Qizhou Wang, Junchi Yu +2LLM Unlearning Hyperparameter Optimization
Division of Computing and Mathematical Sciences, MBZUAI · RIKEN Center for Advanced Intelligence Project · TMLR Group, Department of Computer Science, Hong Kong Baptist University +1
Freeze the Decoder, Heal the Encoder: Parameter-Efficient Adaptation for SVD-Based KV-Cache Compression
Sep 25, 2026Yufeng WangKV-Cache Compression Parameter-Efficient Fine-Tuning
Anomaly-Free Self-Optimization via AUC Bounds
Sep 23, 2026Kevin Wilkinghoff, Zheng-Hua TanUnsupervised Anomaly Detection Hyperparameter Optimization
Department of Electronic Systems, Aalborg University, Denmark · Pioneer Centre for AI, Denmark
Bilevel Optimization of Topology and Hyperparameters (BOTH)
Sep 18, 2026Suryanarayanan Manoj Sanu, Miguel Anibal Bessa, Alejandro Marcos AragónHypergradient Optimization Topology Optimization
Faculty of Mechanical Engineering Delft university of technology Delft The Netherlands · School of engineering Brown University Providence United States of America
Agentic Search Spaces for Tabular Machine Learning
Sep 14, 2026Renat Sergazinov, Artem Chistyakov, Sergey Pankevich +1Large Language Model-Guided Optimization Tabular ML
† Yandex ‡ HSE University
Making Political Text Scaling Comparable: Infrastructure and Hyperparameter Sensitivity for 17 Algorithms
Sep 13, 2026Patrick ParschanComputational Social Science Hyperparameter Optimization
Department of Media and Communication LMU Munich
JumpStart Your Policy Learning with Lessons from 160,000 Training Runs
Sep 12, 2026Nabil Omi, Eric Bae, Chung Yik Edward Yeung +2Offline RL Algorithm Selection
University of Washington · Microsoft
Early-Stopping Thresholds for ES-HyperNEAT: A Data-Driven Approach from Fitness Dynamics
Sep 11, 2026Romain Claret, Arthur Gygax, Michael O'Neill +2Early Stopping Evolutionary Optimization
University of Neuchâtel, Switzerland · Information Management Institute · Computer Science Department +1
Parameter Sensitivity Analysis for Aerial LiDAR-Inertial Odometries in low-altitude flights
Sep 11, 2026Robert Milijas, Jose Ramiro Martinez-de Dios, Stjepan BogdanSimultaneous Localization and Mapping UAV Navigation
CoE MARBLE - Centre of Excellence in Maritime Robotics and Technologies for Sustainable Blue Economy, Unska 3, Zagreb, 10000,, Croatia · GRVC Robotics Laboratory, Universidad de Sevilla, Camino de los Descubrimientos, Sevilla, 41092,, Spain · University of Zagreb Faculty of Electrical Engineering and Computing, Laboratory for Robotics and Intelligent Control Systems (LARICS), Unska 3, Zagreb, 10000,, Croatia
Revisiting Multi-Object Tracking Baselines: Hyperparameter Optimization with Multi-Fidelity Greedy Coordinate Search
Sep 10, 2026Momir AdžemovićMulti-Fidelity Optimization Hyperparameter Optimization
Department of Computer Science, Faculty of Mathematics, University of Belgrade, Studentski trg 16, Belgrade, 11000, Serbia
The Art of Hierarchical Competing Patterns: Gaussian Process Optimization of Hyphenation
Sep 7, 2026Ondřej Sojka, Petr SojkaBayesian Optimization Hyperparameter Optimization
Faculty of Informatics, Masaryk University
TASTE: Throughput-Aware Batch Size Tuning for On-Device Edge Learning
Sep 7, 2026Avik Bhatnagar, Federico Nicolas Peccia, Oliver BringmannOn-Device Training Edge ML
FZI Research Center for Information Technology, University of Tübingen Germany
Constrained Bayesian Optimization for Hierarchical Federated Learning in IoT Networks for Plant Disease Classification
Sep 6, 2026Athanasios Papanikolaou, Athanasios Tziouvaras, Apostolos Xenakis +6Hierarchical Federated Learning Internet of Things
University of Zagreb, Croatia · University of Thessaly, Greece · International Hellenic University +3
Amortizing Scaling Law Construction Costs
Sep 4, 2026Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6Language Model Scaling Laws Bayesian Optimization
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +4
HyperMC: Multi-Fidelity Hyperparameter Tuning for Stochastic Gradient MCMC
Sep 2, 2026Ming Tan, Xiyun JiaoHyperparameter Optimization Markov Chain Monte Carlo
Department of Statistics and Data Science Southern University of Science and Technology Shenzhen, Guangdong, China · Department of Statistics and Data Science2026 Southern University of Science and Technology Shenzhen, Guangdong, China

Oct 22, 2026J/K move · Enter open · S save

REDAR: Reinforcement-Driven Adaptive Regularisation for Class Incremental Learning

Akila Ayanthi, Darshana Priyasad, Tharindu Fernando, Sridha Sridharan+2

SAIVT, Queensland University of Technology, Australia · CSIRO Robotics, CSIRO, Australia

Class-Incremental Learning Regularization-Based Continual Learning Hyperparameter Optimization Adaptive Knowledge Distillation

Open paper AskPDF arXiv

Regularisation-based class-incremental learning (CIL) methods control the stability-plasticity trade-off through fixed coefficients tuned offline, independently of the continual learning objective and blind to phase-specific variations in data volume, task difficulty, and class similarity. We argue this is a fundamental mismatch and propose REDAR: Reinforcement-Driven Adaptive Regularisation for Class Incremental Learning, which treats regularisation strength selection as a sequential decision problem. At each incremental phase, a lightweight reinforcement learning (RL) agent observes a compact state of current performance metrics and incoming data statistics, and outputs the distillation coefficients for that phase. To respect CIL's constraint on historical data access, the policy is trained entirely on pseudo-CIL tasks constructed from the initial phase, then deployed on the incremental sequence without further adaptation. Experiments on CIFAR-100 and Tiny-ImageNet demonstrate that REDAR reduces forgetting and improves forward transfer over fixed and heuristic baselines, and that policies trained on short pseudo-task horizons transfer effectively to longer incremental runs.