The Coordinate System Problem in Persistent Structural Memory for Neural Architectures

Abhinaba Basu

cs.LGMar 24, 2026

The Coordinate System Problem in Persistent Structural Memory for Neural Architectures

Authors: Abhinaba Basu

Organizations: National Institute of Electronics and Information Technology (NIELIT)

Abstract

We introduce the Dual-View Pheromone Pathway Network (DPPN), an architecture that routes sparse attention through a persistent pheromone field over latent slot transitions, and use it to discover two independent requirements for persistent structural memory in neural networks. Through five progressively refined experiments using up to 10 seeds per condition across 5 model variants and 4 transfer targets, we identify a core principle: persistent memory requires a stable coordinate system, and any coordinate system learned jointly with the model is inherently unstable. We characterize three obstacles -- pheromone saturation, surface-structure entanglement, and coordinate incompatibility -- and show that neither contrastive updates, multi-source distillation, Hungarian alignment, nor semantic decomposition resolves the instability when embeddings are learned from scratch. Fixed random Fourier features provide extrinsic coordinates that are stable, structure-blind, and informative, but coordinate stability alone is insufficient: routing-bias pheromone does not transfer (10 seeds, p>0.05). DPPN outperforms transformer and random sparse baselines for within-task learning (AULC 0.700 vs 0.680 vs 0.670). Replacing routing bias with learning-rate modulation eliminates negative transfer: warm pheromone as a learning-rate prior achieves +0.003 on same-family tasks (17 seeds, p<0.05) while never reducing performance. A structure completion function over extrinsic coordinates produces +0.006 same-family bonus beyond regularization, showing the catch-22 between stability and informativeness is partially permeable to learned functions. The contribution is two independent requirements for persistent structural memory: (a) coordinate stability and (b) graceful transfer mechanism.

Figures & tables

Figure 1: (a) DPPN architecture schematic. Tokens are embedded and passed through dual soft groupers to produce slot assignments. Slot-level support and agreement are combined with the persistent pheromone field (highlighted) to produce pheromone-biased routing, which generates a sparse attention mask. A fast/slow gate fusion produces the final output. (b) Transfer protocol. In Phase 1, the model is trained on a source task and pheromone accumulates structural memory. In Phase 2, all model weights are reset but pheromone is either kept (warm) or reset (cold), and the model is trained on a target task.

Epoch	Val. Acc.	$\bar{\tau}\pm\sigma_{\tau}$	Entropy
0	0.476	$0.331\pm 0.227$	6.76
5	0.638	$0.358\pm 0.637$	5.97
10	0.708	$0.356\pm 0.645$	5.94
20	0.722	$0.338\pm 0.629$	5.93
30	0.734	$0.338\pm 0.629$	5.93
50	0.734	$0.338\pm 0.629$	5.93

Table 1: Pheromone evolution during source training (DPPN, seed 42). The field develops structure within the first 30 epochs, then stabilizes.

Figure 2: Pheromone field $\boldsymbol{\tau}\in\mathbb{R}^{32\times 32}$ before and after source training (DPPN, seed 42). (a) At initialization, pheromone is near-uniform ( $\bar{\tau}\approx 1.05$ ). (b) After 80 epochs, the field is sparse and structured: most transitions have decayed to $\tau_{\min}=0.1$ (blue), with a small number of high-pheromone transitions at $\tau_{\max}=2.0$ (red).

		Same Family (A)		Different Family
Model	Condition	A2	A3	B1	C1
DPPN	Cold	$.699\pm.014$	$.702\pm.012$	$.715\pm.017$	$.682\pm.016$
	Distilled	$.700\pm.017$	$.701\pm.012$	$.711\pm.012$	$.678\pm.016$
	Rank-reduced	$.700\pm.015$	$.700\pm.011$	$.714\pm.015$	$.683\pm.016$
	$\Delta_{\text{distilled}}$	$+.001$	$-.001$	$-.004$	$-.004$
Transformer	Cold	$.687\pm.012$	$.679\pm.013$	$.690\pm.017$	$.665\pm.020$

Table 2: Transfer results (AULC), mean

\pm

std over 3 seeds. “Distilled” and “Rank-reduced” are two warm pheromone conditions for DPPN. Transformer and Random Sparse have only cold (no pheromone to transfer).

\Delta

= warm

-

cold.

		Same Family		Different Family
Model (distilled)	Seeds	A2	A3	B1	C1
DPPN (learned + Hungarian)	3	$+0.002\pm 0.007$	$-0.001\pm 0.001$	$-0.004\pm 0.005$	$-0.003\pm 0.002$
Fourier (token+position)	3	$-0.001\pm 0.004$	$-0.008\pm 0.004$	$-0.005\pm 0.003$	$-0.006\pm 0.004$
Fourier (position-only)	3	$+0.003\pm 0.004$	$-0.006\pm 0.006$	$-0.007\pm 0.003$	$+0.001\pm 0.005$
Fourier (position-only)	10	$\mathbf{-0.001}\pm 0.005$	$\mathbf{-0.002}\pm 0.006$	$\mathbf{-0.002}\pm 0.006$	$\mathbf{-0.002}\pm 0.006$

Table 3: Transfer advantage (

\Delta

AULC: warm distilled

-

cold, mean

\pm

std). Positive values indicate pheromone transfer helps. The Position-Only Fourier variant with 10 seeds shows uniformly negative advantages, indistinguishable from the other routing-bias variants. The 3-seed result that appeared directionally correct did not replicate.

Figure 3: Transfer advantage (

\Delta

AULC: warm distilled

-

cold) across model variants and transfer targets. Same-family tasks (A2, A3) are shaded green; different-family tasks (B1, C1) are shaded red. With 3 seeds, the Position-Only Fourier variant appeared to show positive transfer on A2 (

+0.003

), but with 10 seeds the advantage is

-0.001\pm 0.005

(not significant). All routing-bias variants show uniformly negative or zero transfer advantages. Error bars: std over seeds.

Experiment	What it fixed	Next obstacle identified	Insight
1: Contrastive updates	Pheromone saturation ( $\tau\to$ uniform)	Surface entanglement	Structured pheromone $\neq$ transferable pheromone
2: Single-source transfer	Task difficulty calibration	Surface-structure coupling	Cannot disentangle with one instance
3: Multi-source distillation	Surface contrast (two instances)	Coordinate mismatch (3.5% vs. 3.1%)	Learned slots are unaligned across runs
4: Decomposition coordinates	Slot arbitrariness (fixed regions)	Embedding arbitrariness ( $-10.4\%$ )	Any learned coordinate system is unstable
5: Position-only Fourier	Extrinsic, surface-free coordinates	Routing-bias mechanism itself — even with stable coordinates, biasing forward-pass routing from a previous task imposes a cost	Coordinate stability is necessary but not sufficient. The transfer mechanism (routing bias vs. learning rate) matters independently.

Table 4: Progressive diagnosis across five experiments. Each resolved one obstacle but exposed the next. The final experiment (position-only Fourier, 10 seeds) reveals that coordinate stability is necessary but not sufficient: routing-bias pheromone does not transfer even with stable coordinates.

Figure 4: The diagnostic cascade. Five experiments, each resolving one obstacle (left, green) while revealing the next (right, red). The center column shows the key metric from each experiment. All obstacles trace to the same root cause: persistent memory requires stable coordinates, and learned coordinates are inherently unstable.

Figure 5: Coordinate stability diagnostics. (a) Slot alignment correlation between independently trained DPPN models: 3.5%, barely above the 3.1% expected by random chance with 32 slots. (b) Distillation survival: the number of high-magnitude transitions (out of 1024) surviving element-wise minimum distillation. Position-Only Fourier preserves the most transitions (22), consistent with better coordinate stability.

		Same Family		Different Family
Mechanism	Seeds	A2	A3	B1	C1
Meta-LR	7	$+0.003\pm 0.006$	$+0.000\pm 0.005$	$+0.002\pm 0.004$	$+0.002\pm 0.004$
Routing bias	10	$-0.001\pm 0.005$	$-0.002\pm 0.006$	$-0.002\pm 0.006$	$-0.002\pm 0.006$

Table 5: Transfer advantage (

\Delta

AULC: warm

-

cold) for meta-learning-rate pheromone vs. routing-bias pheromone, both with position-only Fourier coordinates. Neither achieves statistically significant positive transfer, but they differ qualitatively: routing bias is uniformly negative, learning-rate modulation is uniformly non-negative.

	Same Family		Different Family
Mechanism	A2	A3	B1	C1
Completion (learned $\mathbf{A}$ )	$-0.004$	$-0.007^{**}$	$-0.004$	$-0.003$
Completion (extrinsic $\mathbf{P}$ )	$+0.007$	$+0.016^{***}$	$+0.015^{***}$	$+0.016^{***}$
Random completion control	$+0.005$	$+0.012$	$+0.016$	$+0.009$
Trained $-$ Random (same fam.)	$+0.013$	$+0.004$	$-0.002$	$+0.000$

Table 6: Structure completion transfer advantage (

\Delta

AULC, 10 seeds each). Completion over learned

\mathbf{A}

is the worst mechanism (negative everywhere). Completion over extrinsic

\mathbf{P}

shows the largest positive signal, but a random completion control (5 seeds) decomposes it into regularization (

{\sim}+0.010

) and a differential same-family bonus (

{\sim}+0.006

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Model dimension ( $d$ )	64
Number of heads	4
Number of layers	3
Number of slots ( $m$ )	32
Top- $k$ (sparse mask)	32
Slow-lane window ( $w$ )	16

Appendix

Table 7: DPPN hyperparameters used in all experiments. Note: early experimental rounds (Sections 5 – 6 ) used

d=128

, 4 heads, 4 layers,

m=64

; after task recalibration, the values below were adopted for all reported results.

Parameter	Value
Vocabulary size	32
Sequence length	128
Training samples	2000
Motifs per sample	2–3
Noise level	0.02
Number of classes	2

Appendix

Table 8: Synthetic task parameters.

Seed	Condition	A2	A3	B1	C1
42	Cold	.683	.689	.693	.676
	Distilled	.677	.690	.695	.675
	Rank-red.	.678	.691	.695	.678
137	Cold	.696	.697	.719	.665
	Distilled	.706	.696	.715	.661
	Rank-red.	.707	.693	.715	.666

Appendix

Table 9: DPPN transfer results (AULC) by seed and condition.

Model	Seed	A2	A3	B1	C1
Transformer	42	.671	.661	.669	.653
	137	.688	.686	.689	.648
	256	.700	.691	.711	.692
Random Sparse	42	.658	.657	.671	.642
	137	.680	.675	.703	.638
	256	.696	.687	.684	.669

Appendix

Table 10: Baseline results (AULC) by seed (cold condition only).

Memory-Augmented Neural Networks

Memory-augmented neural networks combine learned computation with explicit or persistent memory for retrieval, long-context reasoning, and adaptation.

See all 91 papers

Jun 5, 2026

Architecture Shapes Transfer Specificity in Implicit Neural Representations

Transfer in coordinate networks is often measured by warm-start gain, but whether that gain reflects source-specific structure or generic weight reuse is less clear. We study this question across three implicit neural representation (INR) families, SIREN, ReLU MLPs, and Fourier-feature MLPs, using controlled analytic tests, a 2D lid-driven-cavity Navier--Stokes benchmark, and 1D PDE reference-solution suites for heat, viscous Burgers, and focusing cubic NLS. The analytic tests use independent-seed random controls, while the PDE benchmarks use alternate same-family source controls and auxiliary ablations. Across settings, transfer magnitude and transfer specificity separate clearly. In a 10-seed controlled 1D geometric test, Fourier Features show the largest structured transfer (

33.1\times

), followed by SIREN (

23.0\times

) and ReLU (

10.7\times

), but ReLU is far more selective: random-control transfer is

0.41\times

for ReLU versus

14.24\times

for SIREN. On a controlled two-parameter 1D family, the ranking changes: ReLU gives the clearest structured-versus-control separation at default settings, whereas Fourier Features improve only after bandwidth retuning. In Navier--Stokes and the broader 1D PDE suite, no single architecture dominates every equation, yet the same pattern remains: SIREN often reuses weights broadly, whereas ReLU and, in some equations, Fourier Features are more source-selective. Static diagnostics remain weak, and the heuristic scaling law

A_{\text{transfer}} \propto 1/Δt^2

is rejected in the implemented 1D audit. These results position transfer specificity as a useful diagnostic for coordinate networks and suggest that architecture selection in scientific machine learning should be evaluated under explicit control conditions, not by transfer magnitude alone.

D Yang Eng

Neural PDE Solvers Implicit Neural Representations Transfer Learning Sinusoidal Representation Networks

Ask PDF

May 13, 2026

Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory

For over a decade, explicit memory architectures like the Neural Turing Machine have remained theoretically appealing yet practically intractable for language modeling due to catastrophic gradient instability during Backpropagation Through Time. In this work, we break this stalemate with Phasor Memory Network (PMNet), a novel architecture that structurally resolves memory volatility through Unitary Phasor Dynamics and Hierarchical Learnable Anchors. Rather than relying on brute-force scaling, we present a mechanistic proof-of-concept in a controlled byte-level setting. By constraining recurrent state updates to phase rotations on a complex unit circle, PMNet preserves gradient norms and inherently prevents divergence without the need for specialized initialization. We empirically demonstrate the active actuation of the memory module through a synthetic Copy-Paste task, where PMNet utilizes an expansive 85-slot hierarchical memory tree (

=\sum^{4}_{h=1}4^{h-1}

) to achieve near 100% exact retrieval across temporal distances that completely exceed the local sliding window attention's receptive field. Furthermore, despite being a compact 119M parameter model trained on 18.8B tokens, PMNet matches the zero-shot long-context robustness of a Mamba model that is three times larger. Our ablation studies and gradient analyses confirm that the historical failure of explicit memory was a structural alignment problem, which PMNet effectively overcomes, providing a theoretically grounded foundation for scalable sequence modeling.

Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

College of Pharmacy Chungnam National University Daejeon, South Korea · Department of Computer Science & Engineering Chungnam National University Daejeon, South Korea

Memory-Augmented Neural Networks Memory-Augmented Language Models Long-Context Language Modeling Persistent Memory for Language Models Recurrent Neural Networks Memorization in Generative Models

Ask PDF

Mar 28, 2026

Persistent Memory Through Triple-Loop Consolidation Under Stochastic Unit Turnover

Dissipative cognitive architectures maintain computation through continuous energy expenditure, where units that exhaust their energy are stochastically replaced with fresh random state. This creates a fundamental challenge: how can persistent, context-specific memory survive when all learnable state is periodically destroyed? Existing memory mechanisms -- including elastic weight consolidation, synaptic intelligence, and surprise-driven gating -- rely on gradient computation and are inapplicable to systems that do not perform it. We introduce Deep Memory (DM), a backpropagation-free persistent memory mechanism operating through a triple-loop consolidation cycle: (1) recording of expert-specific content centroids, (2) seeding of replaced units with stored representations, and (3) stabilization through continuous re-entry. Discrete expert routing via Mixture-of-Experts (MoE) gating is required, in the regimes tested, to prevent the centroid convergence that would render stored memories identical. We derive a Foster-Lyapunov drift bound for the full triple loop, showing that seeding rescales the turnover noise floor. Across

1{,}007

simulation runs over thirteen blocks: (i) removing stable context-expert binding removes specialization (

\mathrm{MI}=1.10

vs.

0.001

;

n=91

); (ii) DM achieves

R=0.984

vs.

0.385

without memory (

n=16

); (iii) continuous seeding reconstructs representations after interference (

R_\mathrm{recon}=0.978

; one-shot fails;

n=30

); (iv) the mechanism operates within a characterized

(K,p)

envelope (

n=350

); (v) recording

\times

seeding is the minimal critical dyad (

n=40

); (vi) associative and reservoir baselines (Hopfield, ESN) are compared under matched turnover (

n=370

). DM is thus a falsifiable, bounded mechanism for persistent memory in backpropagation-free cognitive systems, with functional parallels to hippocampal consolidation.

Jianwei Lou

RailMind Systems, Neuss, Germany

Computational Cognitive Modeling Memory Consolidation

Ask PDF

Abstract

Memory-Augmented Neural Networks

Explore similar work

Architecture Shapes Transfer Specificity in Implicit Neural Representations

Phasor Memory Networks: Stable Backpropagation Through Time for Scalable Explicit Memory

Persistent Memory Through Triple-Loop Consolidation Under Stochastic Unit Turnover