math.OCOct 30, 2026

Distributed Grid Search for Hyperparameter Tuning

Authors: Constança De Herdeiro Cachim, Andrea Martin, Nicola Bastianello

Organizations: KTH Royal Institute of Technology, Stockholm, Sweden · University College London, United Kingdom

Abstract

The performance of many distributed optimization algorithms over peer-to-peer networks strongly depends on the choice of their hyperparameters. However, existing selection rules often require global quantities, yield conservative choices, or are tied to specific algorithms and hyperparameters. We propose Distributed Grid Search (D-GS), a consensus-based search method that evaluates and selects hyperparameter configurations across the network, requiring only the ability to run the tuned algorithm and evaluate local gradients. We also propose a zoom-in variant, Distributed Zoom Grid Search (D-ZGS), which improves empirical performance at a reduced cost. We prove that both methods terminate in finite time with all agents holding the same hyperparameter configuration, and quantify the (sub)optimality of this configuration. Numerical experiments on linear and logistic regression show that D-GS closely matches a centralized reference and that tuning and deployment cost a fraction of deploying algorithms with hyperparameters tuned using theoretical results.

Figures & tables

Explore similar work

Jan 7, 2024

Multi-Objective Hyperparameter Search via Damped Gauss--Newton Optimization

We study hyperparameter optimization (HPO) from a numerical-optimization perspective and propose a multi-objective damped Newton--Gauss--Newton search method. Rather than perturbing each hyperparameter separately or treating model evaluations as independent trials, the method uses performance changes between successive full hyperparameter vectors to construct an iterative secant approximation of the local sensitivity matrix. Each iteration therefore requires only one new full-vector model evaluation while jointly updating all optimized hyperparameters. A Tikhonov-regularized Gauss--Newton system stabilizes the update when the number of hyperparameters exceeds the number of performance objectives. The search is initialized from readily available empirical/default settings of the underlying learner, without requiring a separate initialization search. We evaluate four-dimensional XGBoost HPO on three public classification datasets against exhaustive grid search, random search, and tree-structured Parzen estimator (TPE) optimization. On Breast Cancer Wisconsin, the proposed method matches the best validation accuracy of a 320-configuration grid search with slightly better log loss and ROC--AUC. Across three datasets and three seeds, predictive performance remains competitive with random search and TPE while using substantially fewer search iterations. A supplementary proprietary eight-dimensional threshold-optimization case study further demonstrates joint multi-parameter optimization under competing performance targets and reveals non-monotonic, oscillatory trajectories that motivate damping and best-iterate selection. Overall, the results establish iterative secant sensitivity as an evaluation-efficient local alternative to exhaustive HPO.
Nov 2, 2026

ExaTune: Exhaustive Hyperparameter Landscape Exploration at Scale

Hyperparameter optimization (HPO) returns a single optimal configuration, systematically discarding the explored performance landscape that encodes how results depend on configuration choices. Fitness landscape analysis comprises a robust collection of methods from evolutionary computation offering ruggedness indices, autocorrelation functions, and fitness-distance correlation that have never been applied to machine learning hyperparameter spaces at scale, given that fine-grained ground-truth enumeration data has not existed. We present ExaTune, an open-source HPC system to produce a new artifact class: the hyperparameter landscape map. ExaTune exhaustively evaluates every point on a discretized hyperparameter grid via KK-fold cross-validation, aggregates results into reproducible parquet benchmarks, and applies a suite of five formalized landscape metrics: spatial autocorrelation (Moran's II), ruggedness, local optima density, fitness-distance correlation (FDC), and variance-based parameter importance. We release a benchmark collection of 8 complete landscape maps (totalling 285,080 exhaustively evaluated configurations) spanning five model families (SVMs, random forests, XGBoost, ridge regression, and kk-nearest neighbors) across six classification and two regression benchmarks, hosted on HuggingFace. Empirical analysis demonstrates that Bayesian optimization, Optuna, Hyperband, and Random Search at standard trial budgets systematically miss instability regions, secondary optima, and parameter interaction effects that are visible in complete landscapes; information that is structurally unavailable to any optimizer by design.
Apr 18, 2026

Towards Fully Parameter-Free Stochastic Optimization: Grid Search with Self-Bounding Analysis

Parameter-free stochastic optimization aims to design algorithms that are agnostic to the underlying problem parameters while still achieving convergence rates competitive with optimally tuned methods. While some parameter-free methods do not require the specific values of the problem parameters, they still rely on prior knowledge, such as the lower or upper bounds of them. We refer to such methods as partially parameter-free''. In this work, we target achieving fully parameter-free'' methods, i.e., the algorithmic inputs do not need to satisfy any unverifiable condition related to the true problem parameters. We propose a powerful and general grid search framework, named Grasp, with a novel self-bounding analysis technique that effectively determines the search ranges of parameters, in contrast to previous work. Our method demonstrates generality in: (i) the non-convex case, where we propose a fully parameter-free method that achieves near-optimal convergence rate, up to logarithmic factors; (ii) the convex case, where our parameter-free methods are competitive with strong performance in terms of acceleration and universality. Finally, we contribute a sharper guarantee for the model ensemble, a final step of the grid search framework, under interpolated variance characterization.