cs.LGNov 17, 2026

AlignedHPO: Efficient Surrogate-Guided Loss Hyperparameter Optimization

Authors: Ivan Karpukhin, Andrey Savchenko

Organizations: Sber AI Lab · HSE University · ISP RAS Research Center for Trusted Artificial Intelligence

Abstract

Modern deep learning losses involve multiple hyperparameters, such as weights, scales, and margins, typically tuned manually or through expensive black-box optimization despite their differentiability. We propose AlignedHPO, an efficient framework for tuning differentiable loss hyperparameters using gradient alignment and surrogate validation objectives. Our method employs Jacobian-vector products to reduce hypergradient computation time and memory, while differentiable metric surrogates guide optimization toward task-specific evaluation criteria. We address gradient magnitude ambiguity and stochastic instability through normalized alignment and gradient smoothing. Experiments with generalized Focal Loss and ArcFace demonstrate effective adaptation across long-tailed classification and open-set face recognition. AlignedHPO achieves competitive or improved performance compared to implicit differentiation, with nearly 12x faster hypergradient computation. These results highlight the potential of efficient surrogate-guided loss hyperparameter optimization.

Figures & tables

Explore similar work

CardsList
  1. Greedy Alignment Principle for Optimizer Hyperparameter Selection

    Dec 6, 2025Jaerin Lee, Kyoung Mu LeeHyperparameter OptimizationSGD with Momentum

  2. When Losses Align: Gradient-Based Composite Loss Weighting for Efficient Pretraining

    May 8, 2026Ivan Karpukhin, Andrey SavchenkoSelf-Supervised Pre-TrainingAdaptive Loss Weighting

  3. Differentiable Optimization Layers for Guaranteed Fairness in Deep Learning

    May 16, 2026David Troxell, Noah Roemer, Guido MontúfarDeep Learning OptimizationAlgorithmic Fairness