Optimal Diagonal Preconditioning Beyond Worst-Case Conditioning: Theory and Practice of Omega Scaling

Structured preconditioners restrict optimization to a small family of positive metrics, but endpoint condition-number reachability does not measure the geometric effort required to reach a useful metric. We formulate this effort as a path-space value problem. Restricted dynamic geometric complexity is the least affine-invariant length of an admissible metric path whose endpoint reaches a Hessian-relative generalized-eigenvalue condition target. Path elimination gives an exact min-plus semigroup and Bellman principle, while fixed-horizon kinetic energy is exactly squared complexity divided by twice the horizon. The main response result is global on a Hadamard state space: geodesic convexity produces a smooth intervention-cube path branch and a uniformly coercive Jacobi form, while one Green inverse generates the value Hessian, two-sided force-to-curvature bounds, exact Möbius effects, and arbitrary prescribed finite-order responses. For the hard condition target, a bordered Jacobi--KKT theorem differentiates the moving projection endpoint and multiplier on every regular active spectral stratum; its indefinite inverse also explains why hard-target interactions need not share the unconstrained sign. The theory specializes to affine-invariant positive-definite geometry. A determinant-one two-dimensional diagonal model has an exact target interval, a closed-form forced path, and a strictly negative-definite interaction matrix. A moving diagonal Hessian gives a closed-form hard-target projection, multiplier, and pair effects of either sign, while a coordinate-sequential three-dimensional protocol yields an exact path metric strictly larger than the ambient projection distance. Thus the global Green and bordered hard-target responses are explicit laws of restricted metric-path elimination built on Bellman composition.

In this work, we study the convergence properties of the Dual Space Preconditioned Gradient Descent, encompassing optimizers such as Normalized Gradient Descent and Gradient Clipping. We consider preconditioners of the form $\nabla K$, where $K: \mathbb{R}^{d \times k} \to \mathbb{R}$ is convex and apply $\nabla K(\cdot)$ to train an over-parameterized linear model with a convex loss of the form $\ell(X W - Y)$, for weights $W \in \mathbb{R}^{d \times k}$, labels $Y \in \mathbb{R}^{n \times k}$ and data $X \in \mathbb{R}^{n \times d}$. Under the aforementioned assumptions, we prove that the iterates of the full-batch preconditioned gradient descent converge at an exponential rate to a point $W_{\infty} \in \mathbb{R}^{d \times k}$ satisfying $XW_{\infty} = Y$. We also study the implicit bias of Dual Space Preconditioned Gradient Descent. First, we demonstrate analytically and empirically that, for general $K(\cdot)$, $W_\infty$ depends on the chosen constant step size, hindering a precise characterization of the implicit bias. We also provide an approximate implicit bias property for general preconditioners, namely, $\|W_0 - W_{\infty}\|_F \le c \|W_0 - W_{\text{GD}, \infty}\|_F$ for a constant $c>0$ and $W_{\text{GD}, \infty}$ denoting the convergence point of GD initialized at $W_0$. Furthermore, for preconditioners of the form $K(G) = h(\|G\|_F)$, known as {\it isotropic preconditioners}, and for the stochastic variation of the algorithm with arbitrary batch-size, we prove linear convergence to $W_{\text{GD}, \infty}$. Finally, in the experiments, we demonstrate faster convergence on a nonlinear model obtained using the smoothed matrix elastic-net as a preconditioner.

Recently, Jiang et al. [2026] developed Leon, a practical variant of One-sided Shampoo [Xie et al., 2025a, An et al., 2025] algorithm for online convex optimization, which does not require computing a costly quadratic projection at each iteration. Unfortunately, according to the existing analysis, Leon requires tuning an additional hyperparameter in its preconditioner and cannot achieve dimension-independent convergence guarantees for convex optimization problems beyond the bounded gradients assumption. In this paper, we resolve this issue by proving certain stability properties of Leon's preconditioner. Using our improved analysis, we show that tuning the extra hyperparameter can be avoided and, more importantly, develop the first practical variant of One-sided Shampoo with Nesterov acceleration, which does not require computing projections at each iteration. As a side contribution, we obtain improved dimension-independent rates in the non-smooth non-convex setting and develop a unified analysis of the proposed algorithm, which yields accelerated projection-free adaptive SGD with (block-)diagonal preconditioners.