Adaptive Memory Crystallization for Autonomous AI Agent Learning in Dynamic Environments

Rajat Khanda; Mohammad Baqar; Sambuddha Chakrabarti; Satyasaran Changdar

cs.LGApr 2, 2026

Adaptive Memory Crystallization for Autonomous AI Agent Learning in Dynamic Environments

Authors: Rajat Khanda, Mohammad Baqar, Sambuddha Chakrabarti, Satyasaran Changdar

Organizations: Supermicro San Jose, USA · Cisco Systems San Jose, USA · Princeton University New Jersey, USA · University of Copenhagen Copenhagen, Denmark

Abstract

Autonomous AI agents operating in dynamic environments face a persistent challenge: acquiring new capabilities without erasing prior knowledge. We present Adaptive Memory Crystallization (AMC), a memory architecture for progressive experience consolidation in continual reinforcement learning. AMC is conceptually inspired by the qualitative structure of synaptic tagging and capture (STC) theory, the idea that memories transition through discrete stability phases, but makes no claim to model the underlying molecular or synaptic mechanisms. AMC models memory as a continuous crystallization process in which experiences migrate from plastic to stable states according to a multi-objective utility signal. The framework introduces a three-phase memory hierarchy (Liquid--Glass--Crystal) governed by an Itô stochastic differential equation (SDE) whose population-level behavior is captured by an explicit Fokker--Planck equation admitting a closed-form Beta stationary distribution. We provide proofs of: (i) well-posedness and global convergence of the crystallization SDE to a unique Beta stationary distribution; (ii) exponential convergence of individual crystallization states to their fixed points, with explicit rates and variance bounds; and (iii) end-to-end Q-learning error bounds and matching memory-capacity lower bounds that link SDE parameters directly to agent performance. Empirical evaluation on Meta-World MT50, Atari 20-game sequential learning, and MuJoCo continual locomotion consistently shows improvements in forward transfer (+34--43% over the strongest baseline), reductions in catastrophic forgetting (67--80%), and a 62% decrease in memory footprint.

Figures & tables

Symbol	Meaning
$\mathcal{M}=(\mathcal{S},\mathcal{A},P,R,\gamma)$	MDP tuple
$e_{i}=(s,a,r,s^{\prime})$	Experience tuple
$c_{i}(t)\in[0,1]$	Crystallization state of $e_{i}$
$U_{i}(t)\in[0,1]$	Multi-objective utility of $e_{i}$
$\delta_{i}(t)$	Temporal-difference (TD) error
$N_{i}(t)$	State-action novelty

TABLE I: Notation used throughout the paper.

Property	Liquid	Glass	Crystal
$c_{i}$ range	$[0,\tau_{L})$	$[\tau_{L},\tau_{C}]$	$(\tau_{C},1]$
Eff. learning rate	$\eta_{\mathrm{base},t}$	${\sim}0.5\,\eta_{\mathrm{base},t}$	${\sim}0.09\,\eta_{\mathrm{base},t}$
Replay frequency	High	Medium	Low
Retention policy	Utility-FIFO	Priority	Indefinite
Interference resist.	Low	Medium	High
Buffer fraction †	62.5%	31.25%	6.25%

TABLE II: Memory Phase Properties

Method	AP $\uparrow$	FT $\uparrow$	BT $\uparrow$	ME $\uparrow$	Mem.
VR	62.3 $\pm$ 5.1	+3.2 $\pm$ 1.8	$-$ 31.4 $\pm$ 4.2	0.062	1000
PER	68.7 $\pm$ 4.3	+5.7 $\pm$ 2.1	$-$ 24.1 $\pm$ 3.9	0.069	1000
HER	71.2 $\pm$ 3.8	+8.4 $\pm$ 2.4	$-$ 21.3 $\pm$ 3.5	0.071	1000
NEC	64.5 $\pm$ 5.7	+4.1 $\pm$ 2.3	$-$ 28.7 $\pm$ 4.8	0.043	1500
PackNet	73.8 $\pm$ 3.2	+9.2 $\pm$ 1.9	$-$ 8.4 $\pm$ 2.1	0.015	5000
PNN	76.4 $\pm$ 2.9	+11.3 $\pm$ 1.7	$-$ 2.1 $\pm$ 1.3	0.015	5000

TABLE III: Meta-World MT50 results. Mean

\pm

std over 50 seeds. Mem. in MB. Best result bolded . ‡ Memory-matched comparison: VR, PER, EWC run with 380 MB buffer (proportionally reduced from 1000 MB). AMC retains its advantage at equal budget. All AMC vs. best-baseline differences:

p<10^{-4}

, Holm–Bonferroni corrected.

Method	AP(%) $\uparrow$	FT $\uparrow$	BT $\uparrow$	$>$ Human
VR	124 $\pm$ 18	+2.1 $\pm$ 3.4	$-$ 47.3 $\pm$ 8.2	7/20
PER	142 $\pm$ 15	+8.3 $\pm$ 4.1	$-$ 38.6 $\pm$ 7.1	9/20
EWC	136 $\pm$ 17	+5.2 $\pm$ 3.8	$-$ 41.2 $\pm$ 7.8	8/20
PackNet	187 $\pm$ 12	+15.7 $\pm$ 3.2	$-$ 12.4 $\pm$ 4.3	14/20
AMC	201 $\pm$ 11	+22.4 $\pm$ 2.9	$-$ 9.7 $\pm$ 3.8	16/20

TABLE IV: Sequential Atari-20. AP as % of human-level performance.

Method	Avg. Return $\uparrow$	Task-1 Retention $\uparrow$
VR	3847 $\pm$ 312	25%
PER	4213 $\pm$ 278	41%
EWC	4567 $\pm$ 245	64%
AMC	5892 $\pm$ 198	86%

TABLE V: MuJoCo Continual Locomotion. Final average return

\pm

std.

Variant	AP $\uparrow$	BT $\uparrow$
AMC (full)	81.7 $\pm$ 2.4	$-$ 6.3 $\pm$ 1.8
No crystallization ( $c_{i}\equiv 0$ )	68.7 $\pm$ 4.3	$-$ 24.1 $\pm$ 3.9
No phase LR modulation	74.1 $\pm$ 3.1	$-$ 18.7 $\pm$ 3.2
No interference detection	77.3 $\pm$ 2.8	$-$ 12.4 $\pm$ 2.5
No novelty ( $w_{2}=0$ )	79.2 $\pm$ 2.6	$-$ 9.1 $\pm$ 2.2
No downstream val. ( $w_{3}=0$ )	80.1 $\pm$ 2.5	$-$ 7.8 $\pm$ 2.0

TABLE VI: Ablation on Meta-World MT50. Each row removes one AMC component.

Parameter	Values tested	AP range
$\alpha$	$\{0.005,0.01,0.05,0.1,0.5\}$	76.2–81.7
$\beta$	$\{0.0005,0.001,0.005,0.01,0.05\}$	77.4–81.7
$\sigma$	$\{0.001,0.005,0.01,0.05\}$	79.1–81.7
$\tau_{L}$	$\{0.1,0.2,0.3,0.4,0.5\}$	78.8–81.7
$\tau_{C}$	$\{0.5,0.6,0.7,0.8,0.9\}$	79.3–81.7
$w_{1}$	$\{0.3,0.4,0.5,0.6,0.7\}$	80.1–81.7

TABLE VII: Sensitivity of AP (Meta-World MT50) to AMC hyperparameters. Each row varies one parameter while holding others at default.

Fig. 1: t-SNE projection of AMC memory after 25 Meta-World tasks. Color encodes crystallization level $c_{i}$ (dark=high, light=low). Crystallized cluster centers correspond to shared motor primitives (grasping, collision avoidance) reused across tasks. Silhouette score for crystal vs. non-crystal separation: $0.61\pm 0.04$ (50 seeds; $>$ 0.5 indicates well-separated clusters).

Transition	AMC	Biological	Ratio
Liquid $\to$ Glass	$\sim$ 1k steps	1–3 h	$\approx$ 18 $\times$
Glass $\to$ Crystal	$\sim$ 10k steps	6–24 h	$\approx$ 10 $\times$
Full crystallization	$\sim$ 50k steps	Days–wks	$\approx$ 30 $\times$

TABLE VIII: AMC vs. biological consolidation timescales.

Setting	Fast-change	Moderate	Slow-change
	(Robotics)	(Games)	(Navigation)
$\alpha$	0.10	0.05	0.01
$\beta$	0.02	0.005	0.001
$\sigma$	0.01	0.005	0.002
$\tau_{L}$	0.25	0.30	0.35
$\tau_{C}$	0.65	0.70	0.80

TABLE IX: Recommended AMC settings by deployment scenario.

Operation	Complexity	$N\!=\!10^{5}$	$N\!=\!10^{6}$
FAISS index build	$O(N\log N)$	50 ms	0.5 s
kNN query (single)	$O(\log N)$	$<$ 1 ms	$<$ 1 ms
Full batch ( $N$ queries)	$O(N\log N)$	0.8 s	8.2 s

TABLE X: kNN runtime complexity. Empirical times on A100 GPU.

Benchmark	AP gain	FT gain	Forgetting red.
Meta-World MT50	$+6.9$ pp	$+34.5\%$	$67\%$ vs. best
Atari-20	$+7.5\%$	$+42.7\%$	$22\%$ vs. PackNet
MuJoCo Locomotion	$+29\%$	—	$80\%$ vs. VR

TABLE XI: Summary of AMC improvements over the strongest per-benchmark baseline.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Result	Key assumption	Implication
Thm. III.4 : Well-posedness	$U_{i},I_{i}$ bounded, measurable	SDE ( 5 ) has unique strong solution on $[0,1]$
Thm. III.8 : Fokker–Planck	Time-averaged $\bar{U},\bar{I}$	Population density $p(c,t)$ evolves as PDE ( 8 )
Thm. III.9 : Beta stationary dist.	$\sigma>0$ , $A,B>0$	$p_{\infty}=\mathrm{Beta}(A,B)$ ; mean $=\alpha\bar{U}/(\alpha\bar{U}+\beta\bar{I})$
Thm. III.12 : Exp. mean convergence	Constant $U_{i},I_{i}$	$\|\mathbb{E}[c_{i}]-c_{i}^{}\|\leq\|c_{i}(0)-c_{i}^{}\|e^{-\lambda_{i}t}$
Thm. III.13 : Forgetting bound	$c_{i}(0)>\tau_{C}$	$\mathbb{P}[\text{forget}]\leq e^{-2\lambda_{i}(c_{0}-\tau_{L})^{2}/(\sigma^{2}T)}\cdot e^{-\lambda_{i}T/2}$
Thm. V.7 : Q-learning convergence	Lipschitz $Q^{*}$ , sufficient exploration	Error $\leq O(\beta\bar{I}/(\alpha\bar{U}{+}\beta\bar{I})/\sqrt{N_{C}})$

Appendix

TABLE XII: Summary of theoretical results. All theorems are proved in full in Section III and Section V .

Continual Learning

Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.

See all 294 papers

Mar 31, 2026

Temporal Memory for Resource-Constrained Agents: Continual Learning via Stochastic Compress-Add-Smooth

An agent that operates sequentially must incorporate new experience without forgetting old experience, under a fixed memory budget. We propose a framework in which memory is not a parameter vector but a stochastic process: a Bridge Diffusion on a replay interval

[0,1]

, whose terminal marginal encodes the present and whose intermediate marginals encode the past. New experience is incorporated via a three-step Compress--Add--Smooth (CAS) recursion. We test the framework on the class of models with marginal probability densities modeled via Gaussian mixtures of fixed number of components~

K

d

dimensions; temporal complexity is controlled by a fixed number~

L

of piecewise-linear protocol segments whose nodes store Gaussian-mixture states. The entire recursion costs

O(LKd^2)

flops per day -- no backpropagation, no stored data, no neural networks -- making it viable for controller-light hardware. Forgetting in this framework arises not from parameter interference but from lossy temporal compression: the re-approximation of a finer protocol by a coarser one under a fixed segment budget. We find that the retention half-life scales linearly as

a_{1/2}\approx c\,L

with a constant

c>1

that depends on the dynamics but not on the mixture complexity~

K

, the dimension~

d

, or the geometry of the target family. The constant~

c

admits an information-theoretic interpretation analogous to the Shannon channel capacity. The stochastic process underlying the bridge provides temporally coherent

movie'' replay -- compressed narratives of the agent's history, demonstrated visually on an MNIST latent-space illustration. The framework provides a fully analytical

Ising model'' of continual learning in which the mechanism, rate, and form of forgetting can be studied with mathematical precision.

Michael Chertkov

Graduate Inter-Disciplinary Program in Applied Mathematics, & Department of Mathematics, University Arizona, Tucson, AZ 85721

Memory Compression Continual Learning Storage-Efficient Continual Learning Catastrophic Forgetting Replay-Based Continual Learning

Ask PDF

Mar 2, 2026

Position: Modular Memory is the Key to Continual Learning Agents

Foundation models have transformed machine learning through large-scale pretraining and increased test-time compute. Despite surpassing human performance in several domains, these models remain fundamentally limited in continuous operation, experience accumulation, and personalization, capabilities that are central to adaptive intelligence. While continual learning research has long targeted these goals, its historical focus on in-weight learning (IWL), i.e., updating a single model's parameters to absorb new knowledge, has rendered catastrophic forgetting a persistent challenge. Our position is that combining the strengths of In-Weight Learning (IWL) and the newly emerged capabilities of In-Context Learning (ICL) through the design of modular memory is the missing piece for continual adaptation at scale. We outline a conceptual framework for modular memory-centric architectures that leverage ICL for rapid adaptation and knowledge accumulation, and IWL for stable updates to model capabilities, charting a practical roadmap toward continually learning agents.

Vaggelis Dorovatas, Malte Schwerin, Andrew D. Bagdanov +21

Toyota Motor Europe · University of Bremen · University of Florence +17

Continual Learning for LLM Agents Memory-Augmented Agents

Ask PDF

Apr 29, 2026

When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents

Memory-augmented LLM agents offer an appealing shortcut to continual learning: rather than updating model parameters, they accumulate experience in external memory, seemingly sidestepping the stability-plasticity dilemma of parametric learning. We show that this challenge does not disappear but resurfaces at the memory level. Under a limited context window, old and new experiences compete during retrieval, relocating the continual-learning bottleneck from parameter updates to memory access. To study this phenomenon, we introduce a (k,v) framework that disentangles two fundamental design axes of external memory: how experience is represented and how it is organized for retrieval. Across sequential-task experiments in ALFWorld and BabyAI, we find that abstract procedural memories transfer more reliably than detailed trajectories, while negative transfer disproportionately harms the hard cases. Moreover, finer-grained memory organization is not universally beneficial: designs that yield strong forward transfer can simultaneously induce severe forgetting. Together, these results reveal that external memory does not resolve the continual-learning problem; it reshapes it into a problem of memory representation and retrieval design.

Qisheng Hu, Quanyu Long, Wenya Wang

Nanyang Technological University

Continual Learning for LLM Agents LLM Agent Memory Transfer Learning Agent Memory Management Catastrophic Forgetting Memory-Augmented Language Model Agents

Ask PDF

Abstract

Continual Learning

Explore similar work

Temporal Memory for Resource-Constrained Agents: Continual Learning via Stochastic Compress-Add-Smooth

Position: Modular Memory is the Key to Continual Learning Agents

When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents