math.OCJan 7, 2024

Multi-Objective Hyperparameter Search via Damped Gauss--Newton Optimization

Authors: Qinwu Xu, Yifan Jiang

Abstract

We study hyperparameter optimization (HPO) from a numerical-optimization perspective and propose a multi-objective damped Newton--Gauss--Newton search method. Rather than perturbing each hyperparameter separately or treating model evaluations as independent trials, the method uses performance changes between successive full hyperparameter vectors to construct an iterative secant approximation of the local sensitivity matrix. Each iteration therefore requires only one new full-vector model evaluation while jointly updating all optimized hyperparameters. A Tikhonov-regularized Gauss--Newton system stabilizes the update when the number of hyperparameters exceeds the number of performance objectives. The search is initialized from readily available empirical/default settings of the underlying learner, without requiring a separate initialization search. We evaluate four-dimensional XGBoost HPO on three public classification datasets against exhaustive grid search, random search, and tree-structured Parzen estimator (TPE) optimization. On Breast Cancer Wisconsin, the proposed method matches the best validation accuracy of a 320-configuration grid search with slightly better log loss and ROC--AUC. Across three datasets and three seeds, predictive performance remains competitive with random search and TPE while using substantially fewer search iterations. A supplementary proprietary eight-dimensional threshold-optimization case study further demonstrates joint multi-parameter optimization under competing performance targets and reveals non-monotonic, oscillatory trajectories that motivate damping and best-iterate selection. Overall, the results establish iterative secant sensitivity as an evaluation-efficient local alternative to exhaustive HPO.

Explore similar work

CardsList
Jun 8, 2026cs.LG

Importance-Aware Scheduling for High-Dimensional Hyperparameter Optimization

Hyperparameter Optimization (HPO) is essential for building high-performing ML/DL models, yet conventional optimizers often struggle in high-dimensional spaces where evaluations are costly and progress is diluted across many low-impact variables. We propose Greedy Importance First (GIF), an importance-aware scheduling strategy that uses a small-sample warm start to estimate hyperparameter importance, forms importance-based groups, allocates trials proportionally, and retains a full-space fallback. We evaluate GIF under fixed evaluation budgets on five anisotropic analytic functions, Bayesmark, and NAS-Bench-301. On the higher-dimensional benchmarks, GIF reaches better incumbents with faster convergence than TPE, BOHB, Random Search, and Sequential Grouping. On Bayesmark, where the effective dimensionality is smaller, GIF remains competitive but the margins are smaller. Ablation studies show that importance estimation, proportional allocation, and the fallback step all contribute to the gains. We also verify that the HIA component recovers the intended anisotropy on the analytic benchmarks. These results suggest that GIF is a simple and plug-compatible way to improve sample efficiency in high-dimensional HPO.
Ruinan Wang, Ian Nabney, Mohammad Golbabaee
Hyperparameter OptimizationOptimal SchedulingImportance
AskPDF
Feb 5, 2026cs.LG

Selecting Hyperparameters for Tree-Boosting

Tree-boosting is a widely used machine learning technique for tabular data. However, its out-of-sample accuracy is critically dependent on multiple hyperparameters. In this article, we empirically compare several popular methods for hyperparameter optimization for tree-boosting including random grid search, the tree-structured Parzen estimator (TPE), Gaussian-process-based Bayesian optimization (GP-BO), Hyperband, the sequential model-based algorithm configuration (SMAC) method, and deterministic full grid search using 5959 regression and binary classification data sets. We find that the SMAC method clearly outperforms all the other considered methods on average, and it gives stable performance across a diverse collection of tabular data sets under a fixed tuning budget, which is relevant for users who cannot afford extensive manual trial-and-error tuning. We further observe that (i) a relatively large number of trials larger than 100100 is typically required for accurate tuning, (ii) using default values for hyperparameters or a full search over a small grid often yields very inaccurate models, (iii) all considered hyperparameters can have a material effect on the accuracy of tree-boosting, i.e., there is no small set of hyperparameters that is more important than others, and (iv) choosing the number of boosting iterations using early stopping yields more accurate results compared to including it in the search space for regression tasks.
Floris Jan Koster, Fabio Sigrist
Gradient Boosted TreeHyperparameter OptimizationDecision Trees
AskPDF
Jun 2, 2026cs.LG

How Many Trees in a Random Forest? A Revisited Approach with Plateau Search and Optuna Integration

Hyperparameter optimization (HPO) for Random Forest faces a specific difficulty in tuning the number of trees: the predictive score typically improves monotonically with ensemble size, so standard methods such as Tree-structured Parzen Estimator (TPE) and Hyperband require a predefined search range and often drive the estimate toward its right boundary. Early-stopping strategies avoid fixing such a range, but can be sensitive to score noise and prone to premature stopping. To address this, we propose an integrated triplet-based plateau-search algorithm that removes the number of trees from the direct TPE search space and still exploits information accumulated across HPO trials. The method adaptively tracks a near-minimal sufficient ensemble size by monitoring relative changes in the out-of-bag (OOB) score across a triplet of forest sizes and shifting this triplet accordingly. This yields an automated and user-interpretable procedure based on a tolerance parameter. We also provide a theoretical analysis: we relate the proposed relative OOB-score criterion to the gap between the current and limiting scores, and derive an asymptotic variance estimate for the corresponding OOB-based absolute relative difference. Experiments show that the selected number of trees can differ substantially from the common heuristic: for most classical benchmark datasets it is smaller, whereas for some high-dimensional bioinformatics datasets, such as Arcene and Dorothea, it is larger. The source code and reproducible experiments are available at https://github.com/lange-am/rf_plateau_hpo.
Vadim Porvatov, Andrey Dukhovny, Andrey Lange
Random ForestHyperparameter OptimizationTreesBoreal Forests
AskPDF