Multiplicative vs. Additive State Coupling in Learned Optimizer Schedules

M. Harper Langston; Pierre-David Letourneau; Dalton Jones; Roland Memisevic; Peter Scott; Mingu Lee; Richard A. Lethin; Christopher Lott

cs.LGOct 23, 2026

Multiplicative vs. Additive State Coupling in Learned Optimizer Schedules

Authors: M. Harper Langston, Pierre-David Letourneau, Dalton Jones, Roland Memisevic, Peter Scott, Mingu Lee, Richard A. Lethin, Christopher Lott,

Organizations: Qualcomm AI Research

Abstract

We learn schedules for a continuous penalty relaxation of MAX-CUT by backpropagating through an unrolled classical solver. A state-blind schedule generalizes to instances substantially larger than those used for training. A state-adaptive bilinear schedule also generalizes out of distribution, outperforming its untrained baseline across 15 seeds ( $t=4.1$ ). By contrast, an additive ablation with identical inputs and output heads is statistically indistinguishable from the bilinear model in distribution but performs worse than its own baseline out of distribution ( $t=-11.7$ ). This separation appears without regularization and persists across a $30$ round weight-decay sweep: the bilinear advantage remains significant at every tested value, whereas the additive model never recovers. The results in this setting isolate how solver state enters the schedule, rather than state adaptivity alone, as the architectural distinction associated with out-of-distribution transfer.

Figures & tables

Figure 1: State-adaptive schedule architectures. The bilinear schedule projects the context and current penalty separately, then combines them by element-wise multiplication. The additive ablation concatenates the same inputs and applies an affine map followed by

\tanh

. Both architectures use the same output heads to produce the step size

\alpha_{k}

and penalty increment

\rho_{k+1}-\rho_{k}

	$n=100$ synthetic (discrete cut)		Biq Mac ratio to optimum
Model	frozen	learned	frozen	learned
State-blind (scalar)	$1393.1\pm 6.7$	$1406.9\pm 7.4$	$0.9259\pm 0.0030$	$0.9420\pm 0.0053$
Bilinear	$1410.9\pm 6.8$	$1415.4\pm 7.3$	$0.9543\pm 0.0024$	$0.9627\pm 0.0078$
Additive (ablation)	$1411.0\pm 6.8$	$1397.9\pm 12.3$	$0.9545\pm 0.0024$	$0.9023\pm 0.0166$
Tuned fixed schedule (common reference)			$0.9624\pm 0.0024$

Table 1: Out-of-distribution performance without regularization over 15 seeds . Synthetic

n=100

results are discrete cut values (mean

\pm

standard deviation); Biq Mac results are ratios to known optima over 80-instance suite ( Rendl et al., 2010 ) . “Frozen” denotes each architecture’s untrained schedule, and “learned” denotes its trained schedule. The final row reports a common geometric schedule selected by grid search on the training distribution.

Figure 2: Change in Biq Mac ratio after training, relative to each architecture’s frozen baseline, across the weight-decay sweep. Bilinear improvement remains positive, whereas additive change remains negative, including at zero weight decay. Complete numerical results reported in Appendix C .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: PSD-by-construction of the moment matrix, shown for a general $R{=}3$ -atom marginal (this paper’s MAX-CUT relaxation is the $R{=}1$ special case). Rows: three atoms at locations $x^{r}$ with weights $w^{r}$ ; their Chebyshev basis vectors $\phi_{r}=[T_{0}(x^{r}),\dots,T_{d}(x^{r})]$ ; and the weighted rank-one outer products $w^{r}\phi_{r}\phi_{r}^{\top}$ . Bottom: their sum $M_{d}(\mu)=\sum_{r}w^{r}\phi_{r}\phi_{r}^{\top}$ (Eq. 13 ), whose eigenvalues are all nonnegative. A sum of nonnegative-weighted rank-one terms is PSD for any atoms and weights, so no semidefinite constraint is enforced during optimization.

Figure 4: The particle-parameterization solver trades a rugged original landscape (left; red circles mark a sample of its $>3^{D}$ local minima) for a smooth moment-space landscape (right) with a single global minimum reachable by gradient-based search (green trajectory), shown on an illustrative 2D objective unrelated to MAX-CUT. This paper’s MAX-CUT relaxation uses the rank-one ( $R{=}1$ ) case of the same parameterization.

Figure 5: Comparison of in-distribution and out-of-distribution behavior. Left: held-out training-distribution graphs (

n\in\{6,\ldots,15\}

). All three learned schedules perform similarly, with the bilinear and additive models falling within the observed variability. Right: out-of-distribution Biq Mac evaluation (

n=100

), shown as the change relative to each architecture’s own frozen baseline (

\Delta=R_{\mathrm{learned}}-R_{\mathrm{frozen}}

). The state-blind and bilinear schedules improve after training (

\Delta=+0.0161

and

+0.0084

, respectively), whereas the additive ablation degrades substantially (

\Delta=-0.0522

, paired

t=-11.7

). The separation between bilinear and additive schedules is therefore not visible on the training distribution and emerges only under distribution shift.

Model	$\mathrm{w_{dec}}$	$\Delta$ Biq Mac	SE	$t$	sign $(+/-)$	Hard #opt
Bilinear	$0$ (ref.)	$+0.0084$	$0.0021$	$4.1$	$13/2$	$12/15$
Bilinear	$10^{-4}$	$+0.0103$	$0.0011$	$9.8$	$14/1$	$12/15$
Bilinear	$3\times 10^{-4}$	$+0.0119$	$0.0008$	$15.3$	$15/0$	$12/15$
Bilinear	$10^{-3}$	$+0.0132$	$0.0005$	$29.3$	$15/0$	$13/15$
Bilinear	$3\times 10^{-3}$	$+0.0134$	$0.0005$	$28.3$	$15/0$	$13/15$
Additive	$0$ (ref.)	$-0.0522$	$0.0044$	$-11.7$	$0/15$	$13/15$

Appendix

Table 2: Weight-decay sweep with 15 seeds per value. The

\mathrm{w_{dec}}=0

rows repeat the unregularized results from Table 1 . Here,

\Delta=R_{\mathrm{learned}}-R_{\mathrm{frozen}}

, “sign” gives the number of seeds with positive and negative

\Delta

, and “Hard #opt” gives the number of seeds reaching the exact optimum on the fixed

n=15

instance. Statistical tests and notation are defined in Appendix C.3 .

Learning to Optimize

Learning to Optimize uses data to adapt algorithms, solver decisions, and solution maps for faster or better optimization.

See all 139 papers

Extending $μ$ P: Spectral Conditions for Feature Learning Across Optimizers
Feb 24, 2026Akshita Gupta, Marieme Ngom, Sam Foreman +1Hyperparameter Transfer Adaptive Gradient Methods
Purdue University · Argonne National Laboratory
Optimistic Dual Averaging Unifies Modern Optimizers
May 11, 2026Thomas Pethick, Wanyun Xie, Roman Machacek +1Stochastic Optimization Deep Learning Optimization
Independent Researcher · EPFL (LIONS) · University of Bern
Celo2: Towards Learned Optimization Free Lunch
Feb 22, 2026Abhinav Moudgil, Boris Knyazev, Eugene BelilovskyLearning to Optimize Deep Learning Optimization
Mila · Concordia University · Samsung AI Lab, Montreal +1

Feb 24, 2026J/K move · Enter open · S save

Extending $μ$ P: Spectral Conditions for Feature Learning Across Optimizers

Akshita Gupta, Marieme Ngom, Sam Foreman, Venkatram Vishwanath

Purdue University · Argonne National Laboratory

Hyperparameter Transfer Adaptive Gradient Methods

Open paper AskPDF arXiv

Several variations of adaptive first-order and second-order optimization methods have been proposed to accelerate and scale the training of large language models. The performance of these optimization routines is highly sensitive to the choice of hyperparameters (HPs), which are computationally expensive to tune for large-scale models. Maximal update parameterization

(μ

)

is a set of scaling rules which aims to make the optimal HPs independent of the model size, thereby allowing the HPs tuned on a smaller (computationally cheaper) model to be transferred to train a larger, target model. Despite promising results for SGD and Adam, deriving

μ

P for other optimizers is challenging because the underlying tensor programming approach is difficult to grasp. Building on recent work that introduced spectral conditions as an alternative to tensor programs, we propose a novel framework to derive

μ

P for a broader class of optimizers, including AdamW, ADOPT, LAMB, Sophia, Shampoo and Muon. We implement our

μ

P derivations on multiple benchmark models and demonstrate zero-shot learning rate transfer across increasing model width for the above optimizers. Further, we provide empirical insights into depth-scaling parameterization for these optimizers.

Abstract

Learning to Optimize

Explore similar work

Extending μμμP: Spectral Conditions for Feature Learning Across Optimizers

Optimistic Dual Averaging Unifies Modern Optimizers

Celo2: Towards Learned Optimization Free Lunch

Extending $μ$ P: Spectral Conditions for Feature Learning Across Optimizers