The performance of many distributed optimization algorithms over peer-to-peer networks strongly depends on the choice of their hyperparameters. However, existing selection rules often require global quantities, yield conservative choices, or are tied to specific algorithms and hyperparameters. We propose Distributed Grid Search (D-GS), a consensus-based search method that evaluates and selects hyperparameter configurations across the network, requiring only the ability to run the tuned algorithm and evaluate local gradients. We also propose a zoom-in variant, Distributed Zoom Grid Search (D-ZGS), which improves empirical performance at a reduced cost. We prove that both methods terminate in finite time with all agents holding the same hyperparameter configuration, and quantify the (sub)optimality of this configuration. Numerical experiments on linear and logistic regression show that D-GS closely matches a centralized reference and that tuning and deployment cost a fraction of deploying algorithms with hyperparameters tuned using theoretical results.
Figures & tables
Protocol
Parameters
Steps
ε
Notes
( 4 )
K
K
ρKσ
may require K≫1
FTERC
Nˉ
2Nˉ
0
needs exact comm.
FTQC
Δ,K
K
O(Δ)†
Δ bounds ε
TABLE I: Comparison of (K,ε) -average consensus protocols.
Fig. 1: Depiction of D-ZGS with two rounds (in the numerical set-up of section V-B ); the shaded region is Λ(r) and the dots the points in Γ(r) .
Problem
Algorithm
D-GS
D-ZGS
C-GS
Linear
ED
0.88
0.88
0.87
ADMM
0.76
0.76
0.75
ATG
0.79
0.78
0.79
Logistic
ED
0.89
0.89
0.88
ADMM
0.78
0.75
0.74
ATG
0.82
0.83
0.81
TABLE II: Mean convergence rate under tuned hyperparameters.
Method
∣Γ∣
Tuning
Deployment
Total
Theoretical bound
—
0
16,681
16,681
D-GS
3
5.4
16.8
22.3
4
7.2
5.6
12.9
5
9.0
4.0
13.0
10
18.1
3.6
21.7
20
36.1
3.6
39.8
TABLE III: Computational cost, in units of 103 .
Fig. 2: Trajectory of the gradient norm under selected configurations.
Fig. 3: Convergence rate of the configuration selected by D-GS, as a function of the averaging error ε , with the protocols of Table I .
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.
BORN AI Foundations Lab, Better Outcomes Registry & Network (BORN) Ontario Children’s Hospital of Eastern Ontario Research Institute (CHEO-RI) Department of Systems & Computer Engineering, Carleton University Ottawa, ON
National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China · Tel Aviv University