Recent works have shown that gradient-update alignment is a powerful signal for modulating optimizer updates, often leading to faster training. We promote this update-wise heuristic as a mathematically grounded principle for selecting and tuning optimizer hyperparameters. By treating gradients and updates as signals and an optimizer as a causal filter that maps between them, we formulate optimizer selection as maximizing the expected drop rate in loss over a prescribed family of optimizers. We show that this objective is exactly the inner product between the optimizer filter and the gradient autocorrelation, and prove that a greedy optimum exists and has a stability bound under perturbations of the estimated gradient statistics. Specializing in momentum-based optimizers, the theory yields simple dynamic momentum selection rules for both SGD+Momentum and Adam/AdamW. Experiments across image classification, language model fine-tuning, and vision transformer fine-tuning show that the resulting dynamic momentum rules match or improve upon the best fixed hyperparameters found via manual sweeps, reducing the need for exhaustive momentum sweeps. Code is available at https://github.com/ironjr/gap
Figures & tables
Figure 1: Just as optimizers train models by parameter updates u , models can also select an optimizer from a candidate set Q using the gradient statistics R .
Figure 2: Empirical instantiation of Theorem 3.1 for SGD+Momentum. The greedy optimal SGD+Momentum of Algorithm 1 is compared with fixed-hyperparameter baselines on the CIFAR-100 dataset [ 38 ] with ResNet-18 [ 30 ] . The error bars indicate the mean and standard deviation over 10 runs.
Table 3
Figure 3: Empirical instantiation of Corollary 3.3 for Adam [ 36 ] . The greedy optimal Adam of Algorithm 2 is compared with fixed-hyperparameter baselines on the CIFAR-100 dataset [ 38 ] with ResNet-18 [ 30 ] . The error bars indicate the mean and standard deviation over 10 runs.
Table 5
Table 5: AdamW on Gemma-2B with Commonsense-170K dataset. mean ± std.
Table 6: AdamW on Vision Transformer fine-tuning tasks with LoRA. mean ± std.
ResNet-18
Gemma-2B
Gemma-2B
Llama-3-8B
ViT-Base
ViT-Large
(full model)
( r=32 LoRA)
(Full FT)
( r=32 LoRA)
( r=32 LoRA)
( r=8 LoRA)
Original VRAM
0.961 GiB
16.303 GiB
16.943 GiB
19.910 GiB
1.078 GiB
2.049 GiB
K -switch VRAM
1.344 GiB
16.303 GiB
27.920 GiB
21.381 GiB
1.080 GiB
2.049 GiB
Memory overhead
+39.84%
+0.00%
+64.78%
+7.39%
+0.18%
+0.00%
# β candidates
8
6
6
6
6
6
Per-step runtime
+14.0%
−0.50%
+29.9%
+0.33%
+2.21%
+1.01%
Table 7: Hyperparameter tuning cost comparison between K -Switch and full hyperparameter sweep.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Parameter
Value(s) / Description
Dataset
MetaMathQA-395K
Training subset size
100,000
Models tested
Gemma-2B ( google/gemma-2b )
Hardware
1 × A100-80GB
Precision
Bfloat16 (BF16)
Optimizer
AdamW [ 36 , 44 ]
Appendix
Table 8: Hyperparameters and settings for math finetuning experiments on Gemma-2B [ 26 ] with LoRA [ 31 ] . Values reflect the experimental script.
Parameter
Value(s) / Description
Dataset
MetaMathQA-395K
Training subset size
100,000
Models tested
Llama-3-8B ( meta-llama/llama-3-8b )
Hardware
1 × A100-80GB
Precision
Bfloat16 (BF16)
Optimizer
AdamW [ 36 , 44 ]
Appendix
Table 9: Hyperparameters and settings for math finetuning experiments on Llama-3-8B [ 28 ] with LoRA [ 31 ] . Values reflect the experimental script.
Parameter
Value(s) / Description
Dataset
Commonsense-170K [ 32 ]
Models tested
Gemma-2B [ 26 ]
Hardware
1 × A100-80GB
Precision
Bfloat16 (BF16)
Optimizer
AdamW [ 36 , 44 ]
Optimal AdamW-type optimizer
two-option switch with β1 endpoints of 0.8 and 0.95
Appendix
Table 10: Hyperparameters and settings for the main commonsense finetuning experiments on Gemma-2B [ 26 ] with LoRA [ 31 ] .
Parameter
Value(s) / Description
Dataset
Commonsense-170K [ 32 ]
Models tested
Gemma-2B [ 26 ]
Hardware
1 × A100-80GB
Precision
Bfloat16 (BF16)
Optimizer
AdamW [ 36 , 44 ]
Optimal AdamW-type optimizer
two-option switch with β1 endpoints of 0.1 and 0.99
Appendix
Table 11: Hyperparameters and settings for the main commonsense finetuning experiments on Gemma-2B [ 26 ] with full fine-tuning.
two-option switch with β1 endpoints of 0.5 and 0.99
Appendix
Table 12: Common hyperparameters and settings for ViT-Base and ViT-Large [ 20 ] finetuning experiments across various classification datasets.
Figure 4: Demonstration of Theorem 3.1 and Corollary 3.3 . Our instantiations of optimal optimizers are compared with baselines having fixed hyperparameters on the CIFAR-100 dataset [ 38 ] with ResNet-18 [ 30 ] , following the standard settings of [ 30 ] . The line and shaded area indicate the mean and standard deviation over 10 runs. For clear visualization, each baseline plot shows only the best run. For SGD+Momentum, momentum below 0.8 showed suboptimal performance.
Table 15
Table 15: Full test results of Gemma-2B trained with MetaMathQA-395K, validated on GSM8K. mean ± std.
Table 16: Full test results of Gemma-2B trained with Commonsense-170K. mean ± std.
Figure 5: Selection ratios of the candidate hyperparameters over training for SGDM and Adam. The experiments are the same ResNet-18 / CIFAR-100 runs in Figures 2 and 3 .
Result
What it says
In plain terms
Prop. 2.1
Pt(Q)=⟨Q,Rt⟩H under A1–A3
Alignment is a filter–statistics inner product.
Thm. 2.2
maxQ∈QP is the support function of Q
A compact convex family has a greedy optimum, stable in R .
Cor. 2.3
supclco(F∪{0})P=(supFP)+
Scoring in the hyperparameterized set F is exact.
Thm. 3.1
β⋆=argmaxβ1−β2E[g⊤mβ]
Hilbert-energy-normalized SGDM selection.
Cor. 3.3
state-conditioned closed form for (β1,β2)
Hilbert-energy-normalized Adam/AdamW selection.
Thm. 3.4
∣Dη−ηA∣≤Lsη2∥u∥2/2
Alignment is the first-order loss drop; η must stay local.
Appendix
Table 17: Summary of the theorems presented in the main text. The proof of each theorem is linked next to its title with button: [ proof ]. All the proofs are located in Appendix E .
Hyperparameter optimization selects model and training settings to improve predictive performance, efficiency, or reliability under limited evaluation budgets.