cs.LGMay 28, 2025

Accelerating Optimization via Differentiable Stopping Time

Authors: Zhonglin Xie, Yiman Fong, Haoran Yuan, Zaiwen Wen

Organizations: Beijing International Center for Mathematical Research, Peking University · Department of Industrial Engineering, Tsinghua University · School of Mathematics Science, Peking University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Optimization is an important module of modern machine learning applications. Tremendous efforts have been made to accelerate optimization algorithms. A common formulation is achieving a lower loss at a given time. This enables a differentiable framework with respect to the algorithm hyperparameters. In contrast, its dual, minimizing the time to reach a target loss, is believed to be non-differentiable, as the time is not differentiable. As a result, it usually serves as a conceptual framework or is optimized using zeroth-order methods. To address this limitation, we propose a differentiable stopping time and theoretically justify it based on differential equations. An efficient algorithm is designed to backpropagate through it. As a result, the proposed differentiable stopping time enables a new differentiable formulation for accelerating algorithms. We further discuss its applications, such as online hyperparameter tuning and learning to optimize. Our proposed methods show superior performance in comprehensive experiments across various problems, which confirms their effectiveness.

Figures & tables

Figure 1: Illustration of the differentiable stopping time on f(x1,x2)=0.5x12+2x22f(x_{1},x_{2})=0.5x_{1}^{2}+2x_{2}^{2} and A(x,θ,t)=diag⁡(1,θ)∇f(x)\mathcal{A}(x,\theta,t)=\operatorname{diag}(1,\theta)\nabla f(x) . Effect of θ\theta on continuous and discrete stopping time TJ,NJT_{J},N_{J} for different ε\varepsilon values.
 ∂Nk∂θ=hstep∇J(xk+1)⊤(∂A(θk,xk,tk)/∂θ)J(xk+1)−J(xk).\frac{\partial N_{k}}{\partial\theta}=\frac{h_{\text{step}}\nabla J(x_{k+1})^{\top}(\partial\mathcal{A}(\theta_{k},x_{k},t_{k})/\partial\theta)}{J(x_{k+1})-J(x_{k})}. (13)
Algorithm 2 Adam-OLA
Figure 2: Experimental results comparing the discrete and continuous stopping time gradients across varying problem dimensions, stopping thresholds, and time step sizes. (a) shows the relative error of the discrete gradient approximation. (b) shows the computational cost ratio.
Figure 3: Test results of different optimizers on logistic regression: Function value versus iteration.
Figure 4: Comparison of different optimizers on smooth SVM: Function value versus iteration. Here, fmin⁡f_{\min} denotes the minimum function value achieved across all iterations for each optimizer.
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: NFEs of different solvers.
Dataset (Experiment)β\bm{\beta} (Learning Rate Update)ϵ\bm{\epsilon} (Descent Threshold)
a1a ( exp_svm )1×10−21\times 10^{-2}1×10−51\times 10^{-5}
a2a ( exp_svm )1×10−31\times 10^{-3}1×10−31\times 10^{-3}
a3a ( exp_svm )5×10−55\times 10^{-5}5×10−45\times 10^{-4}
w3a ( exp_svm )0.0055×10−95\times 10^{-9}
Appendix
Table 1: Hyperparameter settings for Adam-OLA on different datasets. The parameter β\beta controls the learning rate adaptation magnitude, and ϵ\epsilon specifies the sufficient decrease threshold for triggering a learning rate update.
Figure 6: Comparison of different optimizers on smooth SVM: Function value versus iteration. Here, fmin⁡f_{\min} denotes the minimum function value achieved across all iterations for each optimizer.
More in this topic

Differentiable Optimization

Differentiable optimization makes solvers and constrained decisions trainable with gradients for learning, control, design, and inverse problems.

See all 83 papers

Explore similar work

CardsList

  1. Learning to Optimize by Differentiable Programming

    Jan 23, 2026Liping Tao, Xindi Tong, Chee Wei TanPrimal-Dual OptimizationLearning to Optimize

    Nanyang Technological University, Singapore · Nanyang Technological University, Singapore, Singapore

  2. Differentiable Optimization Layers for Guaranteed Fairness in Deep Learning

    May 16, 2026David Troxell, Noah Roemer, Guido MontúfarDeep Learning OptimizationAlgorithmic Fairness

    Department of Statistics & Data Science, University of California, Los Angeles, USA · Department of Mathematics, University of California, Los Angeles, USA · Max Planck Institute for Mathematics in the Sciences, Leipzig, Germany

  3. Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

    Apr 14, 2026Tong Zhang, Jiangning Zhang, Zhucun Xue +9Deep Learning OptimizationAdaptive Gradient Methods

    Zhejiang University, APRIL Lab · Fudan University · Shanghai Jiaotong University +1