Distributed Grid Search for Hyperparameter Tuning
Organizations: KTH Royal Institute of Technology, Stockholm, Sweden · University College London, United Kingdom
Abstract
The performance of many distributed optimization algorithms over peer-to-peer networks strongly depends on the choice of their hyperparameters. However, existing selection rules often require global quantities, yield conservative choices, or are tied to specific algorithms and hyperparameters. We propose Distributed Grid Search (D-GS), a consensus-based search method that evaluates and selects hyperparameter configurations across the network, requiring only the ability to run the tuned algorithm and evaluate local gradients. We also propose a zoom-in variant, Distributed Zoom Grid Search (D-ZGS), which improves empirical performance at a reduced cost. We prove that both methods terminate in finite time with all agents holding the same hyperparameter configuration, and quantify the (sub)optimality of this configuration. Numerical experiments on linear and logistic regression show that D-GS closely matches a centralized reference and that tuning and deployment cost a fraction of deploying algorithms with hyperparameters tuned using theoretical results.
Figures & tables
| Protocol | Parameters | Steps | Notes | |
|---|---|---|---|---|
| ( 4 ) | may require | |||
| FTERC | needs exact comm. | |||
| FTQC | bounds |
| Problem | Algorithm | D-GS | D-ZGS | C-GS |
|---|---|---|---|---|
| Linear | ED | 0.88 | 0.88 | 0.87 |
| ADMM | 0.76 | 0.76 | 0.75 | |
| ATG | 0.79 | 0.78 | 0.79 | |
| Logistic | ED | 0.89 | 0.89 | 0.88 |
| ADMM | 0.78 | 0.75 | 0.74 | |
| ATG | 0.82 | 0.83 | 0.81 |
| Method | Tuning | Deployment | Total | |
|---|---|---|---|---|
| Theoretical bound | — | 0 | 16,681 | 16,681 |
| D-GS | 3 | 5.4 | 16.8 | 22.3 |
| 4 | 7.2 | 5.6 | 12.9 | |
| 5 | 9.0 | 4.0 | 13.0 | |
| 10 | 18.1 | 3.6 | 21.7 | |
| 20 | 36.1 | 3.6 | 39.8 |
Hyperparameter Optimization
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.
See all 178 papersExaTune: Exhaustive Hyperparameter Landscape Exploration at Scale
Towards Fully Parameter-Free Stochastic Optimization: Grid Search with Self-Bounding Analysis
partially parameter-free''. In this work, we target achieving fully parameter-free'' methods, i.e., the algorithmic inputs do not need to satisfy any unverifiable condition related to the true problem parameters. We propose a powerful and general grid search framework, named Grasp, with a novel self-bounding analysis technique that effectively determines the search ranges of parameters, in contrast to previous work. Our method demonstrates generality in: (i) the non-convex case, where we propose a fully parameter-free method that achieves near-optimal convergence rate, up to logarithmic factors; (ii) the convex case, where our parameter-free methods are competitive with strong performance in terms of acceleration and universality. Finally, we contribute a sharper guarantee for the model ensemble, a final step of the grid search framework, under interpolated variance characterization.