Understanding the Curse of Unrolling

Sheheryar Mehmood; Florian Knoll; Peter Ochs

cs.LGFeb 23, 2026

Understanding the Curse of Unrolling

Authors: Sheheryar Mehmood, Florian Knoll, Peter Ochs

Organizations: Saarland University, Saarbrücken, Germany · Friedrich-Alexander University, Erlangen, Germany

Abstract

Algorithm unrolling is ubiquitous in machine learning, particularly in hyperparameter optimization and meta-learning, where Jacobians of solution mappings are computed by differentiating through iterative algorithms. Although unrolling is known to yield asymptotically correct Jacobians under suitable conditions, recent work has shown that the derivative iterates may initially diverge from the true Jacobian, a phenomenon known as the curse of unrolling. In this work, we provide a non-asymptotic analysis that explains the origin of this behavior and identifies the algorithmic factors that govern it. We show that truncating early iterations of the derivative computation mitigates the curse while simultaneously reducing memory requirements. Finally, we demonstrate that warm-starting in bilevel optimization naturally induces an implicit form of truncation, providing a practical remedy. Our theoretical findings are supported by numerical experiments on representative examples.

Figures & tables

Figure 1: Iterate

\bm{x}^{(k)}(\bm{u})

vs derivative

D\bm{x}^{(k)}(\bm{u})

error plot for gradient descent applied to

f(\bm{x},u)\coloneqq\|A\bm{x}-\bm{b}\|^{2}/2+u\|\bm{x}\|^{2}/2

. Unlike

\bm{x}^{(k)}(\bm{u})

D\bm{x}^{(k)}(\bm{u})

initially drifts away from its limit before eventually coming back to it. We provide a non-asymptotic understanding of this transient behavior, called the curse of unrolling, and study simple ways to mitigate it.

Iterations	Iterates	Propagation
Unrolled Algorithm	$(\bm{x}^{(k)})_{k=0}^{K}$	$0\to K$
Forward Mode	$(\dot{X}^{(k)})_{k=0}^{K}$	$0\to K$
Reverse Mode	$(\bar{U}_{K}^{(k)})_{k=K}^{0}$	$K\to 0$

Table 1: Propagation of algorithm and derivative iterates.

Figure 2: Error evolution of

e^{(k)}(\bm{u})

\dot{e}^{(k)}(\bm{u})

, and

\bar{e}^{(k)}(\bm{u})

generated by gradient descent applied to

f(\bm{x},u)\coloneqq\|A\bm{x}-\bm{b}\|^{2}/2+u\|\bm{x}\|^{2}/2

. The dashed lines denote the bounds given in ( 8 ) and ( 19 ). The vertical lines denote

\dot{k}

and

\bar{k}

defined in Sections 3.1 and 3.2 respectively.

Iterations	Iterates	Propagation
Idle Algorithm	$(\bm{x}^{(k)})_{k=0}^{T^{\prime}}$	$0\to T^{\prime}$
Unrolled Algorithm	$(\bm{x}^{(k)})_{k=T^{\prime}}^{K^{\prime}}$	$T^{\prime}\to K^{\prime}$
Forward Mode	$(\dot{X}^{(k)})_{k=T^{\prime}}^{K^{\prime}}$	$T^{\prime}\to K^{\prime}$
Reverse Mode	$(\bar{U}_{K}^{(k)})_{k=K^{\prime}}^{T^{\prime}}$	$K^{\prime}\to T^{\prime}$

Table 2: Propagation of the iterates under truncation.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3: Late-start / truncation behavior for $N=2$ , $\alpha=2/(L+m)$ , and $\rho\approx 0.763526$ .

Panel of Figure 4 — Figure 4: Late-start / truncation behavior for $N=5$ , $\alpha=2/(L+m)$ , and $\rho\approx 0.921979$ .

Panel of Figure 5 — Figure 5: Late-start / truncation behavior for $N=10$ , $\alpha=2/(L+m)$ , and $\rho\approx 0.974362$ .

Panel of Figure 6 — Figure 6: Late-start / truncation behavior for $N=20$ , $\alpha=2/(L+m)$ , and $\rho\approx 0.993876$ .

Panel of Figure 7 — Figure 7: Late-start / truncation behavior for $N=30$ , $\alpha=2/(L+m)$ , and $\rho\approx 0.998509$ .

Panel of Figure 8 — Figure 8: Late-start / truncation behavior for $N=40$ , $\alpha=2/(L+m)$ , and $\rho\approx 0.999722$ .

Panel of Figure 9 — Figure 9: Late-start / truncation behavior for $N=2$ , $\alpha=1/(3L)$ , and $\rho\approx 0.763526$ .

Panel of Figure 10 — Figure 10: Late-start / truncation behavior for $N=5$ , $\alpha=1/(3L)$ , and $\rho\approx 0.921979$ .

Panel of Figure 11 — Figure 11: Late-start / truncation behavior for $N=10$ , $\alpha=1/(3L)$ , and $\rho\approx 0.974362$ .

Panel of Figure 12 — Figure 12: Late-start / truncation behavior for $N=20$ , $\alpha=1/(3L)$ , and $\rho\approx 0.993876$ .

Panel of Figure 13 — Figure 13: Late-start / truncation behavior for $N=30$ , $\alpha=1/(3L)$ , and $\rho\approx 0.998509$ .

Panel of Figure 14 — Figure 14: Late-start / truncation behavior for $N=40$ , $\alpha=1/(3L)$ , and $\rho\approx 0.999722$ .

Algorithm Unrolling

Algorithm unrolling turns iterative optimization procedures into trainable networks while retaining their explicit update structure.

See all 17 papers

Efficient Bilevel Optimization with KFAC-Based Hypergradients
Mar 31, 2026Disen Liao, Felix Dangel, Yaoliang YuKronecker-Factored Approximation Bilevel Optimization
University of Waterloo · Vector Institute
Scalable Deep Unfolding of Conic Optimizers
Jun 11, 2026Alex Oshin, Rahul Vodeb Ghosh, Evangelos A. TheodorouSemidefinite Programming Implicit Differentiation
Georgia Institute of Technology
Sharper Analysis of Single-Loop Methods for Bilevel Optimization
Jul 11, 2026Yubo Zhou, Jun Shu, Luo Luo +4Implicit Differentiation Optimization Convergence Analysis
School of Mathematics and Statistics, Xi’an Jiaotong University · School of Data Science, Fudan University · 3SGIT AI Lab, State Grid Corporation of China +1

Mar 31, 2026J/K move · Enter open · S save

Efficient Bilevel Optimization with KFAC-Based Hypergradients

Disen Liao, Felix Dangel, Yaoliang Yu

University of Waterloo · Vector Institute

Kronecker-Factored Approximation Bilevel Optimization Hypergradient Optimization Differentiable Optimization Curvature-Aware Optimization Second-Order Optimization

Open paper AskPDF arXiv

Bilevel optimization (BO) is widely applicable to many machine learning problems. Scaling BO, however, requires repeatedly computing hypergradients, which involves solving inverse Hessian-vector products (IHVPs). In practice, these operations are often approximated using crude surrogates such as one-step gradient unrolling or identity/short Neumann expansions, which discard curvature information. We build on implicit function theorem-based algorithms and propose to incorporate Kronecker-factored approximate curvature (KFAC), yielding curvature-aware hypergradients with a better performance efficiency trade-off than Conjugate Gradient (CG) or Neumann methods and consistently outperforming unrolling. We evaluate this approach across diverse tasks, including meta-learning and AI safety problems. On models up to BERT, we show that curvature information is valuable at scale, and KFAC can provide it with only modest memory and runtime overhead. Our implementation is available at https://github.com/liaodisen/NeuralBo.

Abstract

Algorithm Unrolling

Explore similar work

Efficient Bilevel Optimization with KFAC-Based Hypergradients

Scalable Deep Unfolding of Conic Optimizers

Sharper Analysis of Single-Loop Methods for Bilevel Optimization