Modern deep learning losses involve multiple hyperparameters, such as weights, scales, and margins, typically tuned manually or through expensive black-box optimization despite their differentiability. We propose AlignedHPO, an efficient framework for tuning differentiable loss hyperparameters using gradient alignment and surrogate validation objectives. Our method employs Jacobian-vector products to reduce hypergradient computation time and memory, while differentiable metric surrogates guide optimization toward task-specific evaluation criteria. We address gradient magnitude ambiguity and stochastic instability through normalized alignment and gradient smoothing. Experiments with generalized Focal Loss and ArcFace demonstrate effective adaptation across long-tailed classification and open-set face recognition. AlignedHPO achieves competitive or improved performance compared to implicit differentiation, with nearly 12x faster hypergradient computation. These results highlight the potential of efficient surrogate-guided loss hyperparameter optimization.
Figures & tables
Figure 1: Computational efficiency of hypergradient estimation for an ArcFace model with a ResNet-50 backbone (batch size 32). We compare implicit differentiation using a five-step Neumann approximation, conventional reverse-over-reverse differentiation, and our forward-over-reverse implementation (AlignedHPO). Lower values are better.
Method
Application
Nonlinear param.
Tuning
Target objective
Complexity
OL-AUX [ 16 ]
AUX
–
Gradient alignment
Main loss
O(K) BP
Auto- λ [ 17 ]
MTL
–
Bilevel (FD)
Val. task loss
O(1) FP/BP
AMAL [ 26 ]
AUX
–
One-step bilevel
Val. task loss
O(1) BP
Meta-Weight-Net [ 25 ]
Design/HPO
–
One-step bilevel
Cross-Ent.
O(1) BP
L2T-DLF [ 27 ]
Design/HPO
✓
Unrolled bilevel
Expected metric
O(T) HVP + mixed
ML3 [ 1 ]
Design/HPO
✓
Unrolled bilevel
Meta-task loss
O(T) BP/HVP
Table 1: Comparison of differentiable loss optimization methods. We summarize their applications, support for nonlinear loss parameters, tuning strategies, target objectives, and computational complexity. Complexities indicate the number of differentiation operations rather than directly comparable runtimes.
Figure 2: Toy example of automatic loss hyperparameter adaptation. (a) Training loss evaluated with each method’s own class weight. (b) Validation loss evaluated with the shared weight pval=0.8 . (c) Evolution of the training class weight p . AlignedHPO automatically adapts the training objective, achieving validation performance comparable to the oracle baseline with p=0.8 . Solid lines show the mean over five independent runs; shaded regions indicate one standard deviation.
Figure 3: Evolution of the effective Focal Loss parameters α (top) and γ (bottom) for selected classes on CIFAR-10-LT. We compare AlignedHPO, its cosine variant, and implicit differentiation.
Method
CIFAR10-LT
CIFAR100-LT
Cross-Entropy
71.33
39.92
Focal ( γ=1 )
72.39
39.83
Implicit Diff.
74.53
40.48
AlignedHPO (Dot)
74.79
40.65
AlignedHPO (Cosine)
74.41
42.00
Table 2: Classification accuracy (%) on CIFAR10-LT and CIFAR100-LT. Higher is better.
Method
LFW
AgeDB
CFP-FP
ArcFace (original)
97.68
86.37
87.34
AlignedHPO – Dot product
AlignedHPO (LFW)
97.15
84.33
86.63
AlignedHPO (AgeDB)
96.98
85.87
79.44
AlignedHPO (CFP-FP)
96.68
82.45
89.26
AlignedHPO – Cosine similarity
Table 3: Face verification performance (%) on LFW, AgeDB-30, and CFP-FP. Each AlignedHPO variant is tuned using the corresponding validation dataset. The retrained variants use the parameters identified during hyperparameter optimization.
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.
Recent works have shown that gradient-update alignment is a powerful signal for modulating optimizer updates, often leading to faster training. We promote this update-wise heuristic as a mathematically grounded principle for selecting and tuning optimizer hyperparameters. By treating gradients and updates as signals and an optimizer as a causal filter that maps between them, we formulate optimizer selection as maximizing the expected drop rate in loss over a prescribed family of optimizers. We show that this objective is exactly the inner product between the optimizer filter and the gradient autocorrelation, and prove that a greedy optimum exists and has a stability bound under perturbations of the estimated gradient statistics. Specializing in momentum-based optimizers, the theory yields simple dynamic momentum selection rules for both SGD+Momentum and Adam/AdamW. Experiments across image classification, language model fine-tuning, and vision transformer fine-tuning show that the resulting dynamic momentum rules match or improve upon the best fixed hyperparameters found via manual sweeps, reducing the need for exhaustive momentum sweeps. Code is available at https://github.com/ironjr/gap
Jaerin Lee, Kyoung Mu Lee
Computer Vision Lab, ASRI, Seoul National University
Modern deep models are often pretrained on large-scale data with missing labels using composite objectives, where the relative weights of multiple loss terms act as hyperparameters. Tuning these weights with random search or Bayesian optimization is computationally expensive, as it requires many independent training runs. To address this, we propose a gradient-based bilevel method that learns pretraining loss weights online by aligning the composite pretraining gradient with a downstream objective. By exploiting the structure of the loss, the method avoids the multiple backward passes typically required by truncated backpropagation through the full model, reducing the overhead of hyperparameter tuning to approximately 30% above a single training run. We evaluate the approach on event-sequence modeling and self-supervised computer vision, where it matches or improves upon carefully tuned baselines while substantially reducing the cost of hyperparameter tuning compared to random or Bayesian search.
Ivan Karpukhin, Andrey Savchenko
Sber AI Lab · HSE University ISP RAS Research Center for Trusted Artificial Intelligence
Differentiable optimization layers are traditionally integrated in predict-then-optimize frameworks where a neural model estimates parameters that subsequently serve as fixed inputs to downstream decision-making optimization problems. In this work, we introduce the concept of a "fairness layer": a differentiable optimization layer appended to a model's output layer that guarantees a chosen notion of output parity is satisfied when integrated into a neural network. Additionally, we introduce an online primal-dual inference algorithm that provides provable aggregate fairness guarantees for streaming predictions with arbitrarily small batch sizes, where traditional per-batch constraints become overly restrictive. Numerical experiments demonstrate the effectiveness of the fairness layer and associated algorithm, and theoretical analysis characterizes the layer's differentiability and stability properties during model training and backpropagation. Our code for these experiments is publicly available on GitHub (https://github.com/dtroxell19/FairDL-ICML-2026.git) and our public Python package documentation can be found online: https://dtroxell19.github.io/fairness_training/.
David Troxell, Noah Roemer, Guido Montúfar
Department of Statistics & Data Science, University of California, Los Angeles, USA · Department of Mathematics, University of California, Los Angeles, USA · Max Planck Institute for Mathematics in the Sciences, Leipzig, Germany