p-Spin Glass Network Efficient Single-Batch Continual Learning

Vladimer Khasia

cs.LGAug 14, 2026

p-Spin Glass Network Efficient Single-Batch Continual Learning

Authors: Vladimer Khasia

Organizations: Independent Researcher.

Abstract

Modern sequence models heavily rely on massive memory footprints and large-batch stochastic optimization, barriers that restrict sample efficiency and continual learning. We introduce the $p$ -Spin Glass Network, a novel architecture that overcomes these limitations, structurally manages optimization variance and yields four noticeable capabilities: 1. It enforces memory efficiency: native ternary quantization compresses internal parameters by $8\times$ , while exact implicit gradients strictly bound activation memory to $\mathcal{O}(B \cdot T \cdot D)$ . 2. it demonstrates sample efficiency, matching the asymptotic performance of a Transformer baseline while utilizing $8\times$ fewer training sequences. 3. Method enables single-batch stability and smooth, monotonic convergence at a stochastic micro-batch size of $1$ . 4. Finally, this stability proves modality-agnostic, maintaining robust temporal credit assignment across both discrete subword and long horizon uncompressed raw byte streams. Ultimately, this work removes large batch requirement for stable deep learning, establishing a foundation for continuous learning and edge AI.

Figures & tables

Figure 1: Architectural diagram of the $p$ -Spin Glass Network. The left column illustrates modality-agnostic input encoding. The center column details the hardware-fused continuous state-space routing and the higher-order $p$ -spin attractor search. The right column depicts the final vocabulary projection and cross-entropy evaluation.

Architecture	Modality	Batch Size	Seq. Seen	Time/Step	Final Loss ( $\mathcal{L}$ )
Transformer Base	Subword	$64$	$64,000$	$18.90$ s	$4.8025$
$p$ -Spin (Ours)	Subword	$\mathbf{1}$	$\mathbf{8,000}$	$\mathbf{0.44}$ s	$\mathbf{4.8004}$
$p$ -Spin (Ours)	Byte	$\mathbf{1}$	$\mathbf{8,000}$	$1.53$ s	$5.5839$

Table 1: Empirical evaluation of the

p

-Spin Glass models against the Transformer baseline. Equivalent Subword Loss for the byte-level model is derived via

L_{\text{sub}}\approx\text{BPB}\cdot\ln(2)\cdot 3.8

Figure 2: Validation loss as a function of total sequences processed. The $p$ -Spin model (Subword) reaches the baseline’s asymptotic performance utilizing only $12.5\%$ of the training data.

Figure 3: Raw training loss variance of $p$ -Spin Glass model. Despite utilizing a total batch size of $1$ , the $p$ -Spin Glass model exhibits significantly smooth optimization dynamics, showing the variance-damping properties of the fixed point attractor.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Parameter Count
Total Parameters	$60,569,600$
Embedding / Head (Tied)	$25,165,824$
Non-Embedding Params	$35,403,776$

Appendix

Table 2: Parameter breakdown for the standard Transformer Baseline.

Configuration	Layer Type	Params (M)	Logical Precision	Size (MB)
Subword $p$ -Spin	Embeddings (Tied)	$26.74$ M	$32$ -bit (FP32)	$102.00$ MB
	Eq. MLP (Ternary)	$21.99$ M	$1.58$ -bit (Packed 2b)	$5.24$ MB
	State-Space Attention	$8.99$ M	$16$ -bit (FP16/BF16)	$17.15$ MB
	Total / Footprint	57.72 M	Uncompressed Eq: 161.09 MB	124.40 MB
Byte $p$ -Spin	Embeddings (Tied)	$0.18$ M	$32$ -bit (FP32)	$0.69$ MB
	Eq. MLP (Ternary)	$45.09$ M	$1.58$ -bit (Packed 2b)	$10.75$ MB

Appendix

Table 3: Comprehensive Layer Capacity and Memory Footprint for the

p

-Spin Glass Sequence architectures (Subword and Byte variations). Natively quantized Ternary matrices strictly achieve an

8\times

memory reduction compared to FP16 baselines.

Memory-Efficient Training

Memory-efficient training reduces the memory needed to optimize neural networks by compressing, recomputing, or relocating training state.

See all 90 papers

Stochastic Engrams for Efficient Continual Learning
Mar 27, 2025Isabelle Aguilar, Luis Fernando Herbozo Contreras, Omid KaveheiClass-Incremental Learning Binary Neural Networks
School of Biomedical Engineering, The University of Sydney, NSW, 2006, Australia
Anytime Training with Schedule-Free Spectral Optimization
May 21, 2026Anuj Apte, Pranav Deshpande, Niraj Kumar +2Deep Learning Optimization Learning Rate Scheduling
Global Technology Applied Research, JPMorganChase New York, NY 10001, USA
From Generic to Dedicated: A Novel Optimizer for Online Continual Learning
Oct 20, 2026Yongyi Wu, Zheng Wang, Sen LinOnline Continual Learning Catastrophic Forgetting
University of Houston Houston, USA

Mar 27, 2025J/K move · Enter open · S save

Stochastic Engrams for Efficient Continual Learning

Isabelle Aguilar, Luis Fernando Herbozo Contreras, Omid Kavehei

School of Biomedical Engineering, The University of Sydney, NSW, 2006, Australia

Class-Incremental Learning Binary Neural Networks Continual Learning Catastrophic Forgetting

Open paper AskPDF arXiv

The ability to learn continuously in artificial neural networks (ANNs) is often limited by catastrophic forgetting, a phenomenon in which new knowledge becomes dominant. By taking mechanisms of memory encoding in neuroscience (i.e., engrams) as inspiration, we propose a novel approach that integrates stochastically-activated engrams as a gating mechanism for metaplastic binarized neural networks (mBNNs). This method leverages the computational efficiency of mBNNs combined with the robustness of probabilistic memory traces to mitigate forgetting and maintain the model's reliability. Previously validated metaplastic optimization techniques have been incorporated to further enhance synaptic stability. Compared to baseline binarized models and benchmark fully connected continual learning approaches, our method is the only strategy capable of achieving average accuracies over 70% in both class-incremental and domain-incremental MNIST benchmarks, matching full-precision state-of-the-art methods. Furthermore, we achieve a significant reduction in peak GPU and RAM usage, under 5% and 20%, respectively, as well as an ~8x reduction in memory footprint compared to full precision counterparts. Our findings demonstrate (A) an improved stability vs. plasticity trade-off, (B) reduced memory intensiveness, and (C) enhanced performance in binarized architectures. By uniting principles of neuroscience and efficient computing, we offer new insights into the design of scalable and robust deep learning systems.

Abstract

Memory-Efficient Training

Explore similar work

Stochastic Engrams for Efficient Continual Learning

Anytime Training with Schedule-Free Spectral Optimization

From Generic to Dedicated: A Novel Optimizer for Online Continual Learning