Organizations: Beijing International Center for Mathematical Research, Peking University · Department of Industrial Engineering, Tsinghua University · School of Mathematics Science, Peking University
Optimization is an important module of modern machine learning applications. Tremendous efforts have been made to accelerate optimization algorithms. A common formulation is achieving a lower loss at a given time. This enables a differentiable framework with respect to the algorithm hyperparameters. In contrast, its dual, minimizing the time to reach a target loss, is believed to be non-differentiable, as the time is not differentiable. As a result, it usually serves as a conceptual framework or is optimized using zeroth-order methods. To address this limitation, we propose a differentiable stopping time and theoretically justify it based on differential equations. An efficient algorithm is designed to backpropagate through it. As a result, the proposed differentiable stopping time enables a new differentiable formulation for accelerating algorithms. We further discuss its applications, such as online hyperparameter tuning and learning to optimize. Our proposed methods show superior performance in comprehensive experiments across various problems, which confirms their effectiveness.
Figures & tables
Figure 1: Illustration of the differentiable stopping time on f(x1,x2)=0.5x12+2x22 and A(x,θ,t)=diag(1,θ)∇f(x) . Effect of θ on continuous and discrete stopping time TJ,NJ for different ε values.
Figure 2: Experimental results comparing the discrete and continuous stopping time gradients across varying problem dimensions, stopping thresholds, and time step sizes. (a) shows the relative error of the discrete gradient approximation. (b) shows the computational cost ratio.
Figure 3: Test results of different optimizers on logistic regression: Function value versus iteration.
Figure 4: Comparison of different optimizers on smooth SVM: Function value versus iteration. Here, fmin denotes the minimum function value achieved across all iterations for each optimizer.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: NFEs of different solvers.
Dataset (Experiment)
β (Learning Rate Update)
ϵ (Descent Threshold)
a1a ( exp_svm )
1×10−2
1×10−5
a2a ( exp_svm )
1×10−3
1×10−3
a3a ( exp_svm )
5×10−5
5×10−4
w3a ( exp_svm )
0.005
5×10−9
Appendix
Table 1: Hyperparameter settings for Adam-OLA on different datasets. The parameter β controls the learning rate adaptation magnitude, and ϵ specifies the sufficient decrease threshold for triggering a learning rate update.
Figure 6: Comparison of different optimizers on smooth SVM: Function value versus iteration. Here, fmin denotes the minimum function value achieved across all iterations for each optimizer.
Solving massive-scale optimization problems requires scalable first-order methods with low per-iteration cost. This tutorial highlights a shift in optimization: using differentiable programming not only to execute algorithms but to learn how to design them. Modern frameworks such as PyTorch, TensorFlow, and JAX enable this paradigm through efficient automatic differentiation. Embedding first-order methods within these systems allows end-to-end training that improves convergence and solution quality. Guided by Fenchel-Rockafellar duality, the tutorial demonstrates how duality-informed iterative schemes such as the alternating direction method of multipliers, and the primal-dual hybrid gradient can be learned and adapted through representative case studies.
Differentiable optimization layers are traditionally integrated in predict-then-optimize frameworks where a neural model estimates parameters that subsequently serve as fixed inputs to downstream decision-making optimization problems. In this work, we introduce the concept of a "fairness layer": a differentiable optimization layer appended to a model's output layer that guarantees a chosen notion of output parity is satisfied when integrated into a neural network. Additionally, we introduce an online primal-dual inference algorithm that provides provable aggregate fairness guarantees for streaming predictions with arbitrarily small batch sizes, where traditional per-batch constraints become overly restrictive. Numerical experiments demonstrate the effectiveness of the fairness layer and associated algorithm, and theoretical analysis characterizes the layer's differentiability and stability properties during model training and backpropagation. Our code for these experiments is publicly available on GitHub (https://github.com/dtroxell19/FairDL-ICML-2026.git) and our public Python package documentation can be found online: https://dtroxell19.github.io/fairness_training/.
David Troxell, Noah Roemer, Guido Montúfar
Department of Statistics & Data Science, University of California, Los Angeles, USA · Department of Mathematics, University of California, Los Angeles, USA · Max Planck Institute for Mathematics in the Sciences, Leipzig, Germany
Balancing convergence speed, generalization capability, and computational efficiency remains a core challenge in deep learning optimization. First-order gradient descent methods, epitomized by stochastic gradient descent (SGD) and Adam, serve as the cornerstone of modern training pipelines. However, large-scale model training, stringent differential privacy requirements, and distributed learning paradigms expose critical limitations in these conventional approaches regarding privacy protection and memory efficiency. To mitigate these bottlenecks, researchers explore second-order optimization techniques to surpass first-order performance ceilings, while zeroth-order methods reemerge to alleviate memory constraints inherent to large-scale training. Despite this proliferation of methodologies, the field lacks a cohesive framework that unifies underlying principles and delineates application scenarios for these disparate approaches. In this work, we retrospectively analyze the evolutionary trajectory of deep learning optimization algorithms and present a comprehensive empirical evaluation of mainstream optimizers across diverse model architectures and training scenarios. We distill key emerging trends and fundamental design trade-offs, pinpointing promising directions for future research. By synthesizing theoretical insights with extensive empirical evidence, we provide actionable guidance for designing next-generation highly efficient, robust, and trustworthy optimization methods. The code is available at https://github.com/APRIL-AIGC/Awesome-Optimizer.
Tong Zhang, Jiangning Zhang, Zhucun Xue +9
Zhejiang University, APRIL Lab · Fudan University · Shanghai Jiaotong University +1