Phase-HDC: Replacing Optimizer History with Gradient Thresholds in Discrete Phase Learning

Ahmed Nebli

cs.LGOct 7, 2026

Phase-HDC: Replacing Optimizer History with Gradient Thresholds in Discrete Phase Learning

Authors: Ahmed Nebli

Organizations: Mathalyse Research

Abstract

Training a compact model often needs far more memory than storing it, because the optimizer keeps its own records of past gradients. For a hyperdimensional classifier whose learned parameters are low-bit angles, which we call a \emph{phase memory}, these records take several times more memory than the model itself. We ask whether such a model can be trained while storing nothing but the model. The proposed method, Phase-HDC, turns each stored angle by at most one step per update, against the sign of its current gradient, and only when that gradient is large enough. We show that this simple rule is the exact solution of a first-order loss model in which every changed parameter pays a fixed cost. When everything except the update rule is held fixed, Phase-HDC matches the accuracy of Adam with 6-bit moments while storing three times less. Across eleven image, tabular, and text datasets, it stores 16--23 $\times$ less than standard float32 Adam and 4--6 $\times$ less than 8-bit Adam. The price is an average loss of about five accuracy points against float32 Adam, while Phase-HDC is more accurate than 8-bit Adam on six of the eleven datasets, including byte-level text prediction, where 8-bit Adam collapses. Instrumented training runs explain these outcomes. Once parameters must sit on a discrete grid, Adam's moments mainly decide whether a parameter moves at all, a decision that a threshold on the current gradient can make without memory, and coarse quantization of the moments breaks this decision for inputs that the data rarely contain. The storage savings are logical state rather than measured hardware memory.

Figures & tables

Figure 1: Overview of Phase-HDC, rendered from the trained digit model (seed 1). (a) An input digit selects one row of the learned position table $P$ for each pixel $p$ and one row of the shared value table $V$ for its value $x_{p}$ . Colors show stored phase indices. At each coordinate $d$ , binding adds the two angles; bundling then sums the resulting unit phasors over all 64 pixels head to tail, giving the complex coordinate $h_{d}$ (teal). (b) The bundle is compared with unit-modulus class vectors, and the magnitude of each overlap enters a softmax. The bars are the trained model’s class probabilities for this input. (c) Training moves a stored index by at most one notch per step: against the gradient sign when $|g_{j}|>\tau_{j}$ , and not at all inside the dead zone. Dials are drawn with $K=16$ so that one notch is visible; the model uses $K=64$ . (d) Bits retained per coordinate between minibatches for methods that share the same 6-bit phase parameters. Phase-HDC keeps only the parameter. Values are logical array state, excluding activations, gradients, and quantizer metadata.

Study	Data and scale	Question	Selection; seeds
Controlled comparison ( Section 5.1 )	digits; $D=64$ , $k=6$	Does removing all optimizer history cost accuracy when parameters, head, and initialization are identical?	separate validation set; 5
Screening studies ( Section 5.2 )	digits; $D=96$ , 20 epochs	Where does Phase-HDC sit among float32, 8-bit, 6-bit, and hybrid training systems? Do generated item memories and fixed-point gradients work?	tuning seed on evaluation split; 5
Six-dataset evaluation ( Section 5.3 )	six public datasets; $D\leq 384$	How large are the accuracy cost and the state saving across tasks, and how does the phase width matter?	tuning seed on evaluation split; 3
Character-level text ( Section 5.4 )	five text corpora; $D=128$ , 64-symbol context	Does the trade-off carry over to next-symbol prediction with vocabularies of up to 256 symbols?	tuning seed on test windows; 3
Instrumented replays ( Section 5.5 )	digits; controlled runs	How often and how far does each method move each coordinate, and why?	replays of the final runs; 5

Table 1: Evidence map. The five lines of study, the question each answers, and the controls under which it was run. “Selection” states which data chose the hyperparameters.

Dataset	Train	Eval.	$F$	$B$	$C$	$D$	Epochs	Batch
Letter	14,000	6,000	16	16	26	128	25	256
ISOLET	5,457	2,340	617	32	26	256	25	256
HAR	7,352	2,947	561	32	6	256	25	256
MNIST	60,000	10,000	784	16	10	384	10	128
Fashion-MNIST	60,000	10,000	784	16	10	384	10	128
CIFAR-10	50,000	10,000	1,024	64	10	384	10	64

Table 2: Settings of the six-dataset evaluation.

F

: positions;

B

: value bins;

C

: classes;

D

: phase dimension. The evaluation examples also served for hyperparameter selection.

Method	Test accuracy (%)	State (KiB)	Ratio
Phase-HDC	$\bm{92.44\pm 0.93}$	4.27	$\mathbf{1.00}\bm{\times}$
Projected Adam, 6-bit moments	$91.22\pm 1.19$	12.80	$3.00\times$
Projected Adam, 8-bit moments	$91.44\pm 0.87$	15.64	$3.67\times$
Projected Adam, float32 moments	$90.78\pm 2.05$	49.77	$11.67\times$

Table 3: Controlled digit comparison. All methods share the 6-bit phase grid, class head, initial tables, and minibatch order; only the update and its stored moments differ. Accuracy is mean

\pm

sample SD over five seeds after validation-only selection. State counts packed parameter and moment arrays. Ratio is state relative to Phase-HDC. Bold marks the best value in each column.

Figure 2: Controlled comparison on a common 6-bit phase grid. (a) Test accuracy of the five final seeds per method; diamonds and bars give mean

\pm

sample SD. (b) Persistent parameter and moment state relative to Phase-HDC. All methods share parameters, head, and initialization, and were tuned on a separate validation set. Ratios are logical storage, not measured speed or memory.

System	Accuracy (%)	Gap (pp)	State (KiB)	Ratio
Adam-fp32	$\bm{96.48\pm 0.56}$	—	113.62	1.0 $\times$
Affine Adam-8	$94.41\pm 0.65$	$-2.07$	28.41	4.0 $\times$
Affine Adam-6	$93.15\pm 2.83$	$-3.33$	21.30	5.3 $\times$
Hybrid, $k=4$	$94.89\pm 0.80$	$-1.59$	26.30	4.3 $\times$
Hybrid, $k=6$	$94.33\pm 0.67$	$-2.15$	28.20	4.0 $\times$
Phase-HDC, $k=4$	$84.81\pm 2.72$	$-11.67$	4.27	26.6 $\bm{\times}$

Table 4: Screening frontier on digits (

D=96

, five seeds). Gap is the accuracy difference to Adam-fp32 in percentage points; ratio is Adam-fp32 state divided by each system’s state. Hybrid systems train an integer encoder with Eq. 4 and a float32 class head with Adam. State follows Eq. 9 . Bold marks the best accuracy and the smallest state.

Figure 3: Screening studies on digits. (a) Accuracy against persistent training state for eight training systems (mean

\pm

SD over five seeds; logarithmic horizontal axis). The dotted line connects Phase-HDC at

k=4

, 6, and 8. (b) Share of gradient signs that agree with the exact float gradient at one frozen parameter state, as the softmax and lookup-table precisions vary. (c) Test accuracy after end-to-end training with the rounded pipeline (table bits fixed at 6); the shaded band is the float control mean

\pm

SD under the same threshold grid.

Dataset	Adam-fp32	Affine Adam-8	Affine Adam-6	Phase-HDC-6
Letter	$\bm{89.19\pm 0.18}$	$86.06\pm 0.44$	$\underline{86.45\pm 0.44}$	$79.68\pm 0.25$
ISOLET	$\bm{93.13\pm 0.31}$	$92.86\pm 0.41$	$\underline{89.05\pm 0.54}$	$77.49\pm 0.75$
HAR	$\bm{94.16\pm 0.09}$	$90.09\pm 1.73$	$42.31\pm 9.01$	$\underline{87.12\pm 4.74}$
MNIST	$\bm{93.91\pm 0.20}$	$89.72\pm 3.92$	$51.69\pm 35.47$	$\underline{91.22\pm 1.18}$
Fashion-MNIST	$83.42\pm 0.86$	$\bm{84.48\pm 0.65}$	$37.56\pm 10.25$	$\underline{80.81\pm 1.73}$
CIFAR-10	$\bm{37.04\pm 1.06}$	$23.00\pm 3.96$	$12.67\pm 0.86$	$\underline{32.54\pm 2.14}$

Table 5: Accuracy (%) on six datasets. Mean

\pm

sample SD over three seeds. Bold marks the best system per dataset; underline marks the better of the two 6-bit systems when it is not the overall best. Evaluation data were reused for tuning, and the Adam arms have free complex class vectors, so entries compare complete systems.

Dataset	Adam-fp32	Adam-8	Adam-6	Phase-HDC-6	fp32	8-bit	6-bit
	State (KiB)				Ratio to Phase-HDC-6
Letter	126.00	31.50	23.62	5.44	$23.17\times$	$5.79\times$	$4.34\times$
ISOLET	2103.00	525.75	394.31	126.56	$16.62\times$	$4.15\times$	$3.12\times$
HAR	1815.00	453.75	340.31	112.31	$16.16\times$	$4.04\times$	$3.03\times$
MNIST	3690.00	922.50	691.88	227.81	$16.20\times$	$4.05\times$	$3.04\times$
Fashion-MNIST	3690.00	922.50	691.88	227.81	$16.20\times$	$4.05\times$	$3.04\times$

Table 6: Persistent state on six datasets (KiB) and its ratio to Phase-HDC-6 for each Adam system. Ideal packed parameter and moment arrays, not measured allocations. Bold marks the smallest state per dataset.

Figure 4: Accuracy against persistent state on six datasets. Diamonds and bars show mean

\pm

sample SD; faint points are individual seeds. The horizontal axis is logical persistent state on a logarithmic scale. Hyperparameter selection reused evaluation data, and class heads differ between Phase-HDC and the Adam arms. Adam-fp32 is the most accurate system on five datasets and Affine Adam-8 on Fashion-MNIST; Affine Adam-6 shows large, dataset-dependent losses.

Corpus	$B$	$100/B$	Adam-fp32	Affine Adam-8	Phase-HDC-6
Tiny Shakespeare	65	1.5	$\bm{26.94\pm 0.48}$	$13.29\pm 0.60$	$23.82\pm 0.34$
TinyStories	70	1.4	$\bm{35.47\pm 0.16}$	$21.17\pm 0.16$	$31.61\pm 0.68$
PTB	49	2.0	$\bm{31.59\pm 0.35}$	$19.87\pm 1.15$	$28.78\pm 0.43$
text8	27	3.7	$\bm{27.70\pm 0.53}$	$26.66\pm 0.31$	$26.05\pm 0.23$
enwik8	256	0.4	$\bm{25.23\pm 0.34}$	$0.78\pm 0.72$	$20.26\pm 0.44$

Table 7: Next-symbol accuracy (%) on five text corpora. Mean

\pm

sample SD over three seeds.

B

is the vocabulary size and

100/B

the uniform-guessing level. Bold marks the best system per corpus. Selection reused the test windows.

Corpus	Adam-fp32	Adam-8	Phase-HDC-6	fp32	8-bit
	State (KiB)			Ratio to Phase-HDC-6
Tiny Shakespeare	388.5	97.1	18.2	$21.4\times$	$5.3\times$
TinyStories	411.0	102.8	19.1	$21.5\times$	$5.4\times$
PTB	316.5	79.1	15.2	$20.8\times$	$5.2\times$
text8	217.5	54.4	11.1	$19.7\times$	$4.9\times$
enwik8	1248.0	312.0	54.0	$23.1\times$	$5.8\times$

Table 8: Persistent state on five text corpora (KiB) and its ratio to Phase-HDC-6. Bold marks the smallest state per corpus.

Figure 5: Character-level text prediction on five corpora. (a) Next-symbol accuracy (mean

\pm

SD over three seeds); dotted lines mark one over the vocabulary size. (b) Accuracy against persistent training state; lines connect the three systems on the same corpus, colors follow (a), and markers identify the corpus. Affine Adam-8 collapses on byte-level enwik8.

Figure 6: Phase-width sweep of Phase-HDC on six datasets. Mean accuracy and sample SD over three seeds. Dotted lines mark

1/C

. The shaded column marks

k=8

, where the selected encoder threshold lies at the grid boundary on five of six datasets. Step size and effective gate change with

k

( Eq. 6 ), so the sweep is not an isolated capacity experiment. Exact values are in Table 14 .

Item memory	$k=2$	$k=4$	$k=6$	$k=8$
Generated, flattened 1D index	38	41	53	55
Generated, row and column frequencies	50	—	—	72 †
Dense table	96	96	95	94

Table 9: Generated against dense item memories on digits (test accuracy, %,

D=128

). A generated memory derives every symbol’s phase from one learned frequency per attribute. † At

D=256

. Bold marks the best memory per width.

Table bits	4	6	8	10	16
4	65.7	94.2	97.7	97.8	97.8
8	68.1	94.3	99.6	99.8	99.9
16	68.1	95.8	99.6	99.9	100.0

Table 10: Gradient sign agreement (%) with the exact float gradient at one frozen parameter state, by lookup-table precision (rows) and softmax precision (columns).

Softmax bits	Accuracy (%)
Float (control)	$94.96\pm 1.17$
16	$95.04\pm 0.71$
12	$95.07\pm 1.41$
10	$94.89\pm 0.25$
8	$95.00\pm 0.74$
6	$94.81\pm 0.78$

Table 11: Test accuracy (%) after end-to-end training with rounded gradient arithmetic (table bits 6, five seeds). The control computes in float with the same threshold grid.

Figure 7: Update dynamics in the controlled comparison. Instrumented replays of the 20 final runs, which reproduce the released checkpoints exactly; curves are means over five seeds, smoothed over five steps. (a) Share of all phase coordinates whose stored index changes at each step. (b) Share of coordinates displaced by more than three notches in one step; float32 moments produced no such jump. (c) For projected Adam with 6-bit moments, each point is one row of the value table, that is, one pixel value

\ell

. Rarely observed values are displaced far more often (Spearman rank correlation

-0.98

over 17 rows).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$x=(x_{1},\ldots,x_{F})$ , $y$	discretized input with values $x_{p}\in\{0,\ldots,B-1\}$ ; class label
$F$ , $B$ , $C$ , $D$	number of positions, value bins, classes; representation dimension
$p$ , $\ell$ , $c$ , $d$ , $j$	indices of position, value bin, class, dimension, and concatenated coordinate
$k$ , $K=2^{k}$ , $a=2\pi/K$	phase bit width, number of phase levels, angle of one notch
$\mathbb{Z}_{K}$	integers $\{0,\ldots,K-1\}$ with addition modulo $K$
$P$ , $V$ , $\psi$	position, value, and class phase tables

Appendix

Table 12: Symbols used in the paper. Each symbol has one meaning throughout.

Dataset	Adam-fp32	Affine Adam-8	Affine Adam-6	Phase-HDC-6
Letter	$(0.3,\ 0.1)$	$(0.3,\ 0.1)$	$(0.5,\ 0.1)$	$(10^{-4},\ 10^{-4})$
ISOLET	$(1,\ 0.03)$	$(1,\ 0.03)$	$(1,\ 0.03)$	$(3\times 10^{-5},\ 10^{-4})$
HAR	$(1,\ 0.03)$	$(0.5,\ 0.03)$	$(1,\ 0.1)$	$(3\times 10^{-5},\ 3\times 10^{-5})$
MNIST	$(0.1,\ 0.03)$	$(0.5,\ 0.1)$	$(1,\ 0.1)$	$(3\times 10^{-5},\ 3\times 10^{-4})$
Fashion-MNIST	$(0.3,\ 0.03)$	$(0.3,\ 0.03)$	$(1,\ 0.1)$	$(3\times 10^{-5},\ 3\times 10^{-4})$
CIFAR-10	$(0.5,\ 0.03)$	$(1,\ 0.1)$	$(0.3,\ 0.03)$	$(3\times 10^{-4},\ 3\times 10^{-4})$

Appendix

Table 13: Selected exploratory configurations as (encoder, class) pairs: learning rates for Adam, thresholds in phase-index units for Phase-HDC. Selection used the evaluation set on seed 0.

Dataset	$k=2$	$k=4$	$k=6$	$k=8$
Letter	$3.75\pm 0.16$	$69.63\pm 2.63$	$\bm{79.68\pm 0.25}$	$77.24\pm 0.19$
ISOLET	$4.33\pm 0.34$	$34.27\pm 15.20$	$\bm{77.49\pm 0.75}$	$57.72\pm 1.87$
HAR	$18.23\pm 0.02$	$38.36\pm 3.79$	$\bm{87.12\pm 4.74}$	$84.38\pm 0.71$
MNIST	$10.02\pm 1.23$	$16.06\pm 11.65$	$91.22\pm 1.18$	$\bm{92.05\pm 0.77}$
Fashion-MNIST	$9.95\pm 1.03$	$17.90\pm 9.61$	$80.81\pm 1.73$	$\bm{81.97\pm 0.31}$
CIFAR-10	$11.99\pm 0.72$	$14.67\pm 3.57$	$32.54\pm 2.14$	$\bm{34.02\pm 1.16}$

Appendix

Table 14: Phase-width sweep of Phase-HDC: mean

\pm

sample SD of exploratory accuracy (%) over three seeds. The six-bit column is the main exploratory record. All widths share the threshold grid and update count; the angular step differs. Bold marks the best width per dataset.

Seed	Phase-HDC	Adam, 6-bit moments	Adam, 8-bit moments	Adam, float32 moments
1	92.78	90.00	91.39	92.22
2	93.89	92.22	91.67	93.06
3	92.22	91.39	90.56	87.78
4	91.67	92.50	90.83	90.00
5	91.67	90.00	92.78	90.83
Mean	92.44	91.22	91.44	90.78

Appendix

Table 15: Test accuracy (%) of every controlled run. All seeds share the fixed data split; the projected Adam arms share the 6-bit parameters of Phase-HDC and differ only in moment precision. Bold marks the best method per seed.

Dataset	Adam-fp32	Affine Adam-8	Affine Adam-6	Phase-HDC-6
Letter	86.8–89.3	77.7–86.6	70.2–86.1	64.2–80.5
ISOLET	77.1–93.5	62.9–92.5	51.8–88.6	54.7–78.4
HAR	90.3–94.2	69.8–89.7	13.8–72.0	75.5–90.0
MNIST	85.8–93.5	48.4–93.2	9.2–72.9	45.0–91.7
Fashion-MNIST	74.3–84.0	76.4–84.6	14.2–60.7	73.1–83.8
CIFAR-10	23.5–38.7	10.1–28.1	10.0–13.9	21.7–31.7

Appendix

Table 16: Range of tuning-seed accuracy (%) across each system’s hyperparameter grid on six datasets (eight learning-rate pairs for Adam, nine threshold pairs for Phase-HDC).

Corpus	Adam-fp32	Affine Adam-8	Phase-HDC-6
Tiny Shakespeare	23.6–27.7	7.8–16.7	18.1–24.4
TinyStories	32.1–35.6	6.0–22.3	26.4–32.2
PTB	28.1–31.7	10.6–23.1	24.1–29.7
text8	25.7–27.8	24.3–27.4	23.7–26.0
enwik8	22.4–25.1	0.0–12.6	14.4–21.0

Appendix

Table 17: Range of tuning-seed accuracy (%) across each system’s four-configuration grid on five text corpora.

Explore similar work

Jun 11, 2026cs.LG

Gefen: Optimized Stochastic Optimizer

AdamW is a default optimizer for deep learning, but its moment states add two parameter-sized buffers to training memory, increasing the cost of large-scale pretraining. We propose Gefen, a memory-efficient optimizer that automatically shares second-moment estimates across parameter blocks and quantizes the first moment using a learned codebook. Gefen reduces AdamW's optimizer memory footprint by up to 8x while maintaining performance, saving 6.5 GiB per billion parameters. Prior work shares second moments across parameters grouped along the Hessian's block-diagonal structure, but relies on hand-specified architectural rules and leaves unexplained why such grouping works. We prove that large mixed Hessian entries constrain the ratio of squared gradients toward one, explaining why shared second moments are accurate when the squared gradients they pool are similar. The Hessian need not be computed: its block structure is inherited by squared gradients, allowing blocks to be found directly. Gefen therefore infers block structure from initial squared gradients, requiring no architecture-specific metadata or user-tuned hyperparameters beyond AdamW defaults. Gefen learns an exact histogram-based dynamic-programming quantization codebook and reuses the blocks for first-moment scaling. Across diverse pretraining experiments, Gefen achieves the lowest peak optimizer memory among compared methods that maintain AdamW-level performance. In single-machine and distributed training, the reduced footprint enables larger microbatches and substantially improves throughput over AdamW, making Gefen a drop-in replacement that can train larger models or use larger global batch sizes. We provide the complete Python implementation, including fused CUDA kernels at https://github.com/ndvbd/Gefen

Nadav Benedek, Tomer Koren, Ohad Fried

Reichman University · Tel Aviv University, Google Research

Deep Learning Optimization Memory-Efficient Optimization Adaptive Gradient Methods Memory-Efficient Training

Ask PDF

Aug 23, 2026cs.LG

Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers

Optimizer-state quantization is commonly designed for Adam's dense, parameter-aligned first- and second-moment arrays. This abstraction breaks for memory-efficient optimizers, whose states may be factored, confidence-modulated, or maintained in a projected space, so similar reconstruction error can produce different update error. We formulate optimizer-state quantization as a joint problem over representation, topology, and update semantics. We then introduce Adaptive Log-Space (AL) quantization for non-negative states. AL fits each block's observed nonzero logarithmic interval and reserves a separate code for exact zero, enforcing

q = 0 \Leftrightarrow x = 0

; signed momentum and state precision remain independently selectable. Controlled probes show that adaptive ranges reduce update error and temporal drift, exact-zero reservation preserves dormant states, and state topology constrains useful block granularity. End-to-end language-model training evaluates the resulting policy across dense, factored, confidence, and projected optimizer states. On TinyLlama-1.1B, AL8 with uniform 8-bit momentum reaches 72.90 perplexity versus 73.54 for bitsandbytes 8-bit AdamW, with comparable optimizer-state storage and higher throughput. CAME matches reference-level final perplexity across three seeds when its non-negative states use AL16, while a semantic grouping-and-protection policy closes most of quantized Adafactor's 100K-step late-loss gap. These results make state topology and update semantics first-class design constraints for optimizer quantization.

Yan Wang

Independent Researcher

Deep Learning Optimization Memory-Efficient Optimization Low-Bit Quantization Neural Network Quantization

Ask PDF

May 11, 2026cs.LG

PowerStep: Memory-Efficient Adaptive Optimization via $\ell_p$ -Norm Steepest Descent

Adaptive optimizers such as Adam are standard for training Transformers, but storing gradient first and second moments incurs substantial memory overhead. We introduce PowerStep, a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics. Motivated by

\ell_p

-norm steepest descent, PowerStep applies a signed-power transform directly to one momentum buffer. We establish a finite-horizon stationarity bound for exact, unregularized updates, with an

O(1/\sqrt{T})

term and a noise-dependent residual. Experiments on Transformers from 124M to 235B parameters show competitive validation quality while halving

\texttt{fp32}

optimizer-state memory relative to AdamW. Combined with uniform

\texttt{int8}

quantization, PowerStep remains numerically stable and reduces optimizer-state memory by

\sim8\times

compared to

\texttt{fp32}

AdamW. PowerStep thus provides a simple, memory-efficient alternative for large-scale training.

Yao Lu, Dengdong Fan, Shixun Zhang +1

Pengcheng Laboratory · Peking University

Gradient Descent Memory-Efficient Optimization Adaptive Gradient Methods Memory-Efficient Training

Ask PDF

Abstract

Explore similar work

Gefen: Optimized Stochastic Optimizer

Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers

PowerStep: Memory-Efficient Adaptive Optimization via ℓp\ell_pℓp​-Norm Steepest Descent

PowerStep: Memory-Efficient Adaptive Optimization via $\ell_p$ -Norm Steepest Descent