AlignedHPO: Efficient Surrogate-Guided Loss Hyperparameter Optimization
Organizations: Sber AI Lab · HSE University · ISP RAS Research Center for Trusted Artificial Intelligence
Abstract
Modern deep learning losses involve multiple hyperparameters, such as weights, scales, and margins, typically tuned manually or through expensive black-box optimization despite their differentiability. We propose AlignedHPO, an efficient framework for tuning differentiable loss hyperparameters using gradient alignment and surrogate validation objectives. Our method employs Jacobian-vector products to reduce hypergradient computation time and memory, while differentiable metric surrogates guide optimization toward task-specific evaluation criteria. We address gradient magnitude ambiguity and stochastic instability through normalized alignment and gradient smoothing. Experiments with generalized Focal Loss and ArcFace demonstrate effective adaptation across long-tailed classification and open-set face recognition. AlignedHPO achieves competitive or improved performance compared to implicit differentiation, with nearly 12x faster hypergradient computation. These results highlight the potential of efficient surrogate-guided loss hyperparameter optimization.
Figures & tables
| Method | Application | Nonlinear param. | Tuning | Target objective | Complexity |
|---|---|---|---|---|---|
| OL-AUX [ 16 ] | AUX | – | Gradient alignment | Main loss | BP |
| Auto- [ 17 ] | MTL | – | Bilevel (FD) | Val. task loss | FP/BP |
| AMAL [ 26 ] | AUX | – | One-step bilevel | Val. task loss | BP |
| Meta-Weight-Net [ 25 ] | Design/HPO | – | One-step bilevel | Cross-Ent. | BP |
| L2T-DLF [ 27 ] | Design/HPO | Unrolled bilevel | Expected metric | HVP + mixed | |
| ML3 [ 1 ] | Design/HPO | Unrolled bilevel | Meta-task loss | BP/HVP |
| Method | CIFAR10-LT | CIFAR100-LT |
|---|---|---|
| Cross-Entropy | 71.33 | 39.92 |
| Focal ( ) | 72.39 | 39.83 |
| Implicit Diff. | 74.53 | 40.48 |
| AlignedHPO (Dot) | 74.79 | 40.65 |
| AlignedHPO (Cosine) | 74.41 | 42.00 |
| Method | LFW | AgeDB | CFP-FP |
|---|---|---|---|
| ArcFace (original) | 97.68 | 86.37 | 87.34 |
| AlignedHPO – Dot product | |||
| AlignedHPO (LFW) | 97.15 | 84.33 | 86.63 |
| AlignedHPO (AgeDB) | 96.98 | 85.87 | 79.44 |
| AlignedHPO (CFP-FP) | 96.68 | 82.45 | 89.26 |
| AlignedHPO – Cosine similarity | |||
Hyperparameter Optimization
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.
See all 181 papers