math.OCOct 30, 2026

Distributed Grid Search for Hyperparameter Tuning

Authors: Constança De Herdeiro Cachim, Andrea Martin, Nicola Bastianello

Organizations: KTH Royal Institute of Technology, Stockholm, Sweden · University College London, United Kingdom

Abstract

The performance of many distributed optimization algorithms over peer-to-peer networks strongly depends on the choice of their hyperparameters. However, existing selection rules often require global quantities, yield conservative choices, or are tied to specific algorithms and hyperparameters. We propose Distributed Grid Search (D-GS), a consensus-based search method that evaluates and selects hyperparameter configurations across the network, requiring only the ability to run the tuned algorithm and evaluate local gradients. We also propose a zoom-in variant, Distributed Zoom Grid Search (D-ZGS), which improves empirical performance at a reduced cost. We prove that both methods terminate in finite time with all agents holding the same hyperparameter configuration, and quantify the (sub)optimality of this configuration. Numerical experiments on linear and logistic regression show that D-GS closely matches a centralized reference and that tuning and deployment cost a fraction of deploying algorithms with hyperparameters tuned using theoretical results.

Figures & tables

Explore similar work

CardsList
  1. ExaTune: Exhaustive Hyperparameter Landscape Exploration at Scale

    Nov 2, 2026Kevin Dick, Katie LonswayHyperparameter OptimizationFitness Landscape Analysis

  2. Towards Fully Parameter-Free Stochastic Optimization: Grid Search with Self-Bounding Analysis

    Apr 18, 2026Yuheng Zhao, Yu-Hu Yan, Amit Attia +3Stochastic OptimizationNonconvex Stochastic Optimization