ZEMA: confidence-calibrated signed descent with only one state
Organizations: Dalhousie University and Vector Institute
Abstract
Recent work characterizes the Adam optimizer as sign descent modulated by gradient uncertainty: the update is the sign of the batch gradient, scaled by how strongly the batch supports that sign. We apply this view directly and introduce ZEMA (Z-score Evidence Momentum Adaptation), which measures that support with a per-coordinate -score computed from the per-example gradients in the current batch, converts it into a sign probability, and smooths the resulting update with a single exponential moving average. Because the probability is formed before smoothing rather than after, ZEMA keeps one state per parameter instead of Adam's two, at a increase in per-step time; the per-example variance it requires can also be estimated from a small fraction of the batch. In fine-tuning and pretraining experiments, ZEMA is competitive with Adam, reduces the variability of the validation loss, and in some settings recovers the sign of the full-dataset gradient more consistently. We close with two directions this opens: extending confidence adaptation beyond the sign geometry, to methods such as Muon, and to differentially private training.
Figures & tables
| E2E NLG | SAMSum | ||||
|---|---|---|---|---|---|
| Model | Optimizer | ppl | vs. AdamW | ppl | vs. AdamW |
| Llama-3.2-1B | AdamW | 2.959 | — | 8.134 | — |
| ZEMA -CLT | 2.971 | 8.173 | |||
| ZEMA -Adam | 2.975 | 8.172 | |||
| Llama-3.2-3B | AdamW | 2.911 | — | 7.157 | — |
| ZEMA -CLT | 2.932 | 7.197 | |||
| Optimizer | states/param | learning rate | val. ppl | vs. AdamW |
|---|---|---|---|---|
| AdamW | — | |||
| ZEMA -CLT | ||||
| ZEMA -Adam | ||||
| Signum | ||||
| RMSProp | ||||
| Lion |
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Task | AdamW | ZEMA -CLT | ZEMA -Adam |
|---|---|---|---|---|
| Llama-3.2-1B | E2E | |||
| SAMSum | ||||
| Llama-3.2-3B | E2E | |||
| SAMSum | ||||
| Llama-3.1-8B | E2E | |||
| SAMSum |
| Tokens | AdamW | ZEMA -CLT | ZEMA -t2 | CI | seeds | |
|---|---|---|---|---|---|---|
| B | 0/3 | |||||
| B | 0/3 | |||||
| B | 1/3 | |||||
| B | 1/3 | |||||
| B | 3/3 | |||||
| B | 3/3 |
| Arm | validation perplexity by learning rate (seed ) |
|---|---|
| Lion, | |
| Lion, | |
| RMSProp | |
| Adagrad |
| Batch | AdamW | lr | ZEMA -CLT | lr | ||
| Tulu-3 drift, Qwen2.5-0.5B, rank-8 LoRA | ||||||
| MATH rare-token split, Llama-3.2-1B, rank-16 LoRA | ||||||
| Body QKV | Output head | |||||
|---|---|---|---|---|---|---|
| Step | AdamW | ZEMA -CLT | AdamW | ZEMA -CLT | ||
| Task | Tensor | AdamW | ZEMA -CLT | |
|---|---|---|---|---|
| E2E | L0 q_proj | |||
| E2E | L31 down_proj | |||
| SAMSum | L0 q_proj | |||
| SAMSum | L31 down_proj | |||
| E2E | pooled | — | — | |
| SAMSum | pooled | — | — |
| Optimizer | lr | body state | total state | val. ppl | vs. AdamW |
|---|---|---|---|---|---|
| AdamW | M | M | — | ||
| ZEMA -CLT | M | M | |||
| Muon | M | M | |||
| GaLore | M | M | |||
| ZEMA -CLT/GaLore | M | M | |||
| ZEMA -Adam/GaLore | M | M |
| Tokens | AdamW | ZEMA -CLT | CI | seeds | |
| , lr | |||||
| B | 0/3 | ||||
| B | 0/3 | ||||
| B | 2/3 | ||||
| B | 3/3 | ||||
| B | 3/3 | ||||
| Variant | Rule | Val. PPL | lr |
|---|---|---|---|
| AdamW | — | ||
| ZEMA -CLT | EMA of the belief (Algorithm 2.2 ) | ||
| ZEMA -CLT-z | EMA of , then link | ||
| ZEMA -pool | pooled , bias corrected by | ||
| ZEMA -logodds | accumulate log-odds, squash with | ||
| ZEMA -kalman | posterior mean, bias corrected by |
| Configuration | Train time | Tokens/s | vs. AdamW |
|---|---|---|---|
| AdamW | h | ||
| Direct implementation of Algorithm 2.2 | h | ||
| bfloat16 statistic accumulation | h | ||
| rank-local pooling (§ E.2 ) | h | ||
| overlapped reduction (§ E.4 ) | |||
| and fused per-example kernel (§ E.3 ) † | h |
| Configuration | Val. loss | Perplexity | Verdict | |
|---|---|---|---|---|
| Exact baseline | — | — | ||
| bfloat16 statistic | pass | |||
| rank-local pooling | pass | |||
| fused kernel | pass |
| Configuration | ms/step | vs. AdamW | Peak GiB |
|---|---|---|---|
| AdamW | |||
| Direct implementation of Algorithm 2.2 | |||
| AdamW | |||
| Rank-local pooling fused kernel, no overlap | |||
| All three changes |
Adaptive Gradient Methods
Adaptive Gradient Methods adjust updates using gradient history, curvature, or noise estimates to improve optimization across diverse problems.
See all 180 papers