Hippocampal consolidation without replay in fully fine-tuned networks
Organizations: Department of Computer Science, Hanyang University, Seoul, Republic of Korea
Abstract
Networks that keep learning overwrite what they know. Prevailing remedies replay stored data, distil from old models or freeze the backbone. The brain suggests a different division of labour: the hippocampus indexes where experience lies in cortex while the cortex keeps learning. We show that this division lets a fully trainable network keep learning without cross-task replay, a teacher or inference-time memory. After each task, a training-only hippocampal memory indexes the input geometry of every group of synapses sharing an input, residual energy included, and limits each synapse's movement per direction before and after the optimizer. This class-free geometry carries most of the protection; with conflict detection on top, the method cuts absorption of established classes by near-duplicates several-fold against every compared method. To our knowledge, this is the first demonstration of one principle keeping whole networks learning across four architecture families and models up to 8 billion parameters. Methods that freeze the backbone hold up only as far as its representation already separates the classes; here the whole network keeps learning to tell near-duplicates apart, leading the frozen-backbone baseline by 17.5 points on TACRED. Without the memory, much old knowledge stays decodable but beyond the network's reach.
Figures & tables
| Model | Method | Relies on | TACRED | FewRel | Banking77 | CLINC150 |
|---|---|---|---|---|---|---|
| Qwen3-0.6B | SEQ | none (full FT) | 37.20 ± 3.31 | 24.46 ± 0.97 | 19.90 ± 0.25 | 39.00 ± 1.22 |
| L2KD | teacher (distillation) | 10.66 ± 1.06 | 12.04 ± 0.95 | 17.52 ± 0.15 | 24.83 ± 0.45 | |
| LAMOL | generative replay + teacher | 33.42 ± 0.42 | 27.58 ± 1.44 | 29.32 ± 1.97 | 55.09 ± 0.92 | |
| PCLL | generative replay + teacher | 32.83 ± 1.25 | 24.85 ± 1.07 | 37.15 ± 2.68 | 56.01 ± 2.25 | |
| Sculpting (fixed budget) | weights’ singular subspaces; full FT | 41.33 ± 0.48 | 40.38 ± 2.23 | 37.33 ± 1.18 | 63.91 ± 3.51 | |
| CODE-CL (task-agnostic, ) | stored input subspaces (conceptors); full FT | 40.00 ± 4.39 | 34.78 ± 3.15 | 47.27 ± 3.67 | 70.59 ± 2.45 |
| Part of the hippocampus | Entry | Size | Content |
| Linear synapse group | protected subspace (orthonormal) | ||
| energy per protected direction | |||
| scalar | per-position energy outside (running mean over episodes) | ||
| scalar | noise floor of the latest episode (spectral cutoff, in energy units) | ||
| , | scalars | positions seen in all episodes / in the latest episode | |
| class-level mean input pattern per record (running sums) |
| Stage | What is computed |
|---|---|
| Forward (bf16) | gains and embedding rows: present energies on rebuild steps; LM-head input: conflicts, , , comparator positions |
| Before backward | step count; on steps 1, 2, 4, …: rebuild the gain and per-token novelty factors , |
| Backward | constrained-input weight gradients (The constrained update); joint-share target at the logits; on embedding rows |
| After backward | gains ; comparator bases , |
| Optimizer step | AdamW moments and ; ; comparator restoration; embedding-row factor and gain novelty factor ; |
| Model | Method | Relies on | TACRED | FewRel | Banking77 | CLINC150 |
|---|---|---|---|---|---|---|
| BERT | SEQ | none (full FT) | 18.04 ± 0.26 | 17.40 ± 1.39 | 20.45 ± 0.17 | 14.97 ± 0.75 |
| SEQ* | frozen backbone + replay buffer (samples) | 30.01 ± 1.23 | 48.18 ± 2.79 | 58.45 ± 2.28 | 62.50 ± 1.68 | |
| DER++ | replay buffer (samples + logits) | 42.01 ± 0.89 | 50.57 ± 1.79 | 54.29 ± 0.68 | 75.22 ± 1.29 | |
| CLS-ER | replay buffer + averaged model copies | 44.97 ± 3.76 | 52.34 ± 1.77 | 59.44 ± 0.73 | 75.25 ± 1.76 | |
| DeltaHippo | training-only geometric memory; full FT | 62.62 ± 0.63 | 66.99 ± 2.94 | 85.02 ± 0.20 | 90.42 ± 0.35 | |
| RoBERTa | SEQ | none (full FT) | 14.61 ± 0.96 | 15.45 ± 1.62 | 19.72 ± 1.12 | 14.44 ± 0.52 |
| Model | Method | Relies on | CIFAR-100 | ImageNet-R |
|---|---|---|---|---|
| ViT-B/16 | SEQ | none (full FT) | 21.25 ± 0.85 | 23.24 ± 0.60 |
| SEQ* | frozen backbone | 81.52 ± 0.34 | 64.23 ± 0.34 | |
| DER++ | replay buffer (samples + logits) | 45.83 ± 1.90 | 42.93 ± 1.17 | |
| CLS-ER | replay buffer + averaged model copies | 80.42 ± 1.03 | 51.79 ± 0.70 | |
| EASE | frozen backbone + expandable adapters | 87.02 ± 0.03 | 67.81 ± 0.17 | |
| SD-LoRA | frozen backbone + LoRA modules | 87.47 ± 0.19 | 66.57 ± 0.09 |
| Final AA | Mean acquisition, tasks 1–9 | |
|---|---|---|
| DeltaHippo (backbone learns) | 64.17 ± 0.26 | 60.0 |
| Backbone frozen after task 0, same classifier rule | 53.25 ± 0.84 | 46.2 |
| Variant | Final AA | Mean acquisition | Retention |
|---|---|---|---|
| DeltaHippo | 71.58 ± 0.49 | 77.9 | 82.1 |
| Class-free geometric core only (Box 1: protected subspace constrained exactly, residual floor outside; no class records, class-pattern constraint, conflict detection and update gate, comparator, joint-share target or erase) | 67.07 ± 2.16 | 74.4 | 80.9 |
| No update constraint (gradient constrained only) | 62.14 ± 2.16 | 87.3 | 60.3 |
| Hard constraint inside the protected subspace, residual plasticity factor kept | 71.07 ± 1.45 | 75.9 | 82.9 |
| Graded inside the protected subspace, residual subspace fully open | 71.18 ± 2.45 | 85.3 | 79.2 |
| Operator protects only the top 25% of the stored directions, residual floor kept | 68.67 ± 2.06 | 77.5 | 79.7 |
| Model, benchmark | Full method | No update constraint | No gradient constraint | Residual subspace open |
|---|---|---|---|---|
| BERT-base, TACRED | 62.69 (63.4 / 78.3) | 59.59 (85.4 / 62.0) | 52.85 (63.4 / 71.3) | 46.84 (93.1 / 50.1) |
| ResNet-50, CIFAR-100 | 68.06 (79.2 / 73.7) | 37.51 (74.5 / 44.7) | 30.70 (44.7 / 44.8) | 27.11 (93.4 / 25.1) |
| ViT-B/16, CIFAR-100 | 88.21 ± 0.25 (94.0 / 91.6) | 71.19 ± 0.36 (98.1 / 79.3) | – | – |
| Method | Relies on | Earlier classes answered correctly (%) | Earlier classes absorbed by a later near-duplicate (%) | Errors that name a near-duplicate (%) |
|---|---|---|---|---|
| SEQ | none (full FT) | 36.6 ± 8.8 | 30.8 ± 3.7 | 31.7 ± 3.2 |
| L2KD | teacher (distillation) | 0.9 ± 0.6 | 40.5 ± 0.8 | 28.6 ± 0.6 |
| LAMOL | generative replay + teacher | 63.0 ± 2.5 | 11.3 ± 2.0 | 20.7 ± 2.2 |
| PCLL | generative replay + teacher | 41.8 ± 8.2 | 29.7 ± 2.1 | 32.1 ± 1.0 |
| Sculpting Subspaces | weights’ singular subspaces; full FT | 24.8 ± 2.5 | 24.8 ± 4.8 | 22.4 ± 4.7 |
| CODE-CL | stored input subspaces (conceptors); full FT | 50.0 ± 32.4 | 21.1 ± 14.2 | 22.3 ± 4.4 |
| Model | Earlier classes answered correctly, DeltaHippo (%) | Earlier classes answered correctly, SEQ* (%) |
|---|---|---|
| Qwen3-0.6B | 74.6 ± 2.0 | 54.6 ± 1.6 |
| Llama-3.2-1B | 76.6 ± 7.1 | 56.6 ± 1.3 |
| Qwen3-4B | 74.1 ± 7.0 | 62.1 ± 1.8 |
| Llama-3.2-3B | 81.2 ± 4.4 | 51.1 ± 1.0 |
| Qwen3-8B | 73.5 ± 2.4 | 54.9 ± 2.6 |
| Llama-3.1-8B | 73.6 ± 6.9 | 64.5 ± 6.5 |
| Regime | Approach | Where stability comes from | What it gives up |
|---|---|---|---|
| Keep old data | Replay of stored samples (DER++, CLS-ER) or generated pseudo-samples (LAMOL, PCLL) | old data or a generator | the right to forget data; for generative replay, a generator to train |
| Keep old models | Distillation from a teacher (L2KD; LAMOL and PCLL in our comparison) | a frozen copy of an earlier model | a full model copy kept beside the learner |
| Keep old models | Parameter anchors (EWC, SI) and averaged model copies (CLS-ER) | a stored copy of earlier parameters | a parameter copy; free movement of important weights |
| Do not change the model | Freezing the backbone and classifying frozen features with prototypes or analytic heads (SEQ*, which on the encoders also keeps a one-sample-per-class buffer; SimpleCIL / APER 1 ; RanPAC 2 ) | the pre-trained features, untouched | any representation learning after task 0 |
| Do not change the model | Prompts (L2P 3 ; DualPrompt 4 ; CODA-Prompt 5 ; HiDe-Prompt 6 ; CPrompt 7 ), adapters (EASE 8 ; SSIAT 9 ; MOS 10 ; TUNA 11 ) or LoRA modules (InfLoRA 12 ; SD-LoRA 13 ; O-LoRA 14 ) on a frozen backbone | frozen backbone + learned prompts, adapters or low-rank modules | backbone plasticity; growth with tasks |
| Do not answer alone | Task identity or a memory at inference | task identity or an external module at test time | the model being the knowledge carrier |
| Benchmark | Task | Classes | Tasks × classes | Train / test | Notes |
|---|---|---|---|---|---|
| TACRED 48 | relation extraction | 40 | 8 × 5 | 7,391 / 1,259 | the 41 TACRED relation types except the rarest, per: country of death; no_relation is not used; 20–320 training and 5–41 test sentences per relation |
| FewRel 49 | relation extraction | 80 | 8 × 10 | 33,600 / 11,200 | 420 / 140 sentences per relation |
| Banking77 50 | intent detection | 77 | 7 × 11 | 10,003 / 3,080 | 35–187 training and 40 test utterances per intent |
| CLINC150 51 | intent detection | 150 | 15 × 10 | 15,000 / 4,500 | in-scope intents; 100 / 30 utterances per intent |
| CIFAR-100 52 | image classification | 100 | 10 × 10 | 50,000 / 10,000 | images resized from 32 to 224 pixels (bilinear) |
| ImageNet-R 53 | image classification | 200 | 10 × 20 | 80% / 20% per class | resize 256, centre crop 224 |
| Task | Relations |
|---|---|
| 0 | org: founded by; per: employee of; org: alternate names; per: city of residence; per: children |
| 1 | per: title; per: siblings; per: religion; per: age; org: website |
| 2 | per: state or provinces of residence; org: member of; org: top members/employees; per: countries of residence; org: city of headquarters |
| 3 | org: members; org: country of headquarters; per: spouse; org: state or province of headquarters; org: number of employees/members |
| 4 | org: parents; org: subsidiaries; per: origin; org: political/religious affiliation; per: other family |
| 5 | per: state or province of birth; org: dissolved; per: date of death; org: shareholders; per: alternate names |
| Earlier class (task) | Later near-duplicates |
|---|---|
| per: city of residence (0) | per: countries of residence (2), org: city of headquarters (2), per: city of death (6), per: city of birth (7) |
| per: employee of (0) | org: member of (2) |
| org: city of headquarters (2) | org: country of headquarters (3), per: city of death (6), per: city of birth (7) |
| org: country of headquarters (3) | per: country of birth (7) |
| org: state or province of headquarters (3) | per: state or province of birth (5), per: state or province of death (6) |
| per: date of death (5) | per: cause of death (6), per: city of death (6), per: date of birth (7) |
| Model | Parameters | Layers | Hidden / MLP width | LM head | Checkpoint |
|---|---|---|---|---|---|
| Qwen3-0.6B 55 | 0.6B | 28 | 1,024 / 3,072 | tied | Qwen/Qwen3-0.6B |
| Qwen3-4B | 4B | 36 | 2,560 / 9,728 | tied | Qwen/Qwen3-4B |
| Qwen3-8B | 8B | 36 | 4,096 / 12,288 | untied | Qwen/Qwen3-8B |
| Llama-3.2-1B 56 | 1B | 16 | 2,048 / 8,192 | tied | meta-llama/Llama-3.2-1B |
| Llama-3.2-3B | 3B | 28 | 3,072 / 8,192 | tied | meta-llama/Llama-3.2-3B |
| Llama-3.1-8B | 8B | 32 | 4,096 / 14,336 | untied | meta-llama/Llama-3.1-8B |
| Decoder LLMs | Text encoders | ViT-B/16 | ResNet-50 | |
|---|---|---|---|---|
| Optimizer | AdamW (0.9, 0.999), 1e-8, weight decay 5e-4 | same | same | same |
| Learning rate | 4e-5 (8B: width rule on top) | 4e-5 (heads 1e-3) | 3e-5 (heads 1e-3) | 1e-4 (heads 1e-3) |
| Batch / epochs per task | 32 / 3 | 32 / 3 | 64 / 3 | 64 / 3 |
| Seeds | 42, 11, 3 | 42, 11, 3 | 1993, 42, 3 | 1993, 42, 3 |
| Loss | answer-token cross-entropy | cross-entropy over all seen classes | same | same |
| Precision | fp32 weights and optimizer state, bf16 autocast | same | same | same |
| Item | Sculpting Subspaces | CODE-CL |
|---|---|---|
| Matrices constrained | q, k, v, o, gate, up and down projections of every block | every linear map, including the LM head (the tied input embedding matrix is constrained through it); q/k/v and gate/up share one conceptor |
| What is protected | the top singular directions of each current weight matrix | the conceptor of each map’s input moment , accumulated over tasks by the conceptor OR |
| Amount | fixed budget: the top of each matrix’s directions frozen at task ( ) | aperture , aperture-gain target 0.95, singular values of below 0.2 set to 0; 125 training sentences per conceptor |
| When | SVD of the merged weights at the start of every task after the first | conceptor after every task except the last |
| Update | the singular-vector factors of the remaining (low-singular-value) subspace are trained; their gradients are projected off the frozen singular directions; factors merged into the dense weight at the end of each task | the AdamW step (decoupled weight decay included) is right-multiplied by , so that the written change has the form the method prescribes |
| Free dimensions | none (not part of the method) | : no task-specific mixing matrices, so no task identity is needed at test time |
| Family | Models (parameters) | Units the memory indexes | How the model answers | Normalisation |
|---|---|---|---|---|
| Convolutional network | ResNet-50 (25M) | every convolution, as a linear map on its unfolded input | linear head on the global pool | BatchNorm gains (pre-trained running statistics) |
| Vision transformer | ViT-B/16 (86M) | q/k/v, attention output, MLP input and output; patch embedding | linear head on [CLS] | LayerNorm gains |
| Bidirectional text encoders | BERT, RoBERTa (110–125M) | q/k/v, attention output, intermediate, output; token, position and type embeddings | linear head on [CLS] | LayerNorm gains (post-LN) |
| Decoder language models | Qwen3, Llama-3 (0.6–8B) | q/k/v, o, gate/up, down; input embedding matrix; LM head | generation through the LM head, tied or untied | RMSNorm gains |
| Condition | What changes | Old tasks 0–2 | Task 3 | New task 4 |
|---|---|---|---|---|
| A. Normal | nothing | 83.8 ± 3.2 | 70.7 ± 3.4 | 69.3 ± 4.9 |
| B. Read off | the hippocampus is not read; plain update | 0.0 ± 0.0 | 0.0 ± 0.0 | 91.4 ± 1.4 |
| C. Scrambled content | same rank and eigenvalues, directions rotated inside each synapse group | 4.6 ± 5.5 | 0.0 ± 0.0 | 91.1 ± 0.6 |
| D. Read restored | as B, then the original read is reconnected at 25% of task 4 | 0.0 ± 0.0 | 0.0 ± 0.0 | 91.6 ± 1.6 |
| E. Last sleep blocked | task 3’s sleep is not written; the earlier memory is read | 81.3 ± 3.1 | 38.7 ± 5.8 | 79.7 ± 1.1 |
| F. Norm-matched plain | plain direction, with A’s per-synapse-group applied norm and A’s applied/skipped steps | 2.0 ± 3.4 | 0.0 ± 0.0 | 91.3 ± 1.5 |
| Guidance on | Final AA | Mean acquisition | Retention |
|---|---|---|---|
| Whole model (DeltaHippo) | 71.6 ± 0.5 | 77.9 ± 2.0 | 82.1 ± 1.7 |
| Backbone only | 53.6 ± 11.0 | 90.3 ± 1.0 | 53.4 ± 9.8 |
| Head only | 12.1 ± 0.2 | 91.3 ± 1.2 | 1.7 ± 0.7 |
| States from \ output head from | Whole model | Backbone only | Head only |
|---|---|---|---|
| Whole model | 80.4 ± 2.8 | 43.4 ± 1.8 | 80.7 ± 2.7 |
| Backbone only | 61.8 ± 20.3 | 23.8 ± 12.4 | 60.9 ± 20.1 |
| Head only | 0.6 ± 1.1 | 0.0 ± 0.0 | 0.6 ± 1.1 |
| Memory update | Final AA | Mean acquisition | Retention | Stored energy, after task 0 6 | Protected subspace (fraction of each synapse group’s dimension), after task 0 6 | Residual-to-floor ratio, after task 0 6 |
|---|---|---|---|---|---|---|
| Conflict-aware write–erase (DeltaHippo) | 71.27 | 78.8 | 80.8 | 41 54 | 0.284 0.291 | 0.35 0.49 |
| No erase (accumulate) | 71.32 | 78.4 | 81.0 | 41 96 | 0.284 0.296 | 0.35 0.49 |
| Non-selective erase, energy-matched | 70.12 | 78.3 | 80.5 | 41 54 | 0.284 0.292 | 0.35 0.49 |
| At the end of the task sequence | Trained model (all parameters, 8 tasks) | Frozen after task 0 | Pre-trained | Generated answers |
| Earlier tasks | 77.2 | 75.4 | 72.0 | 70.1 |
| Newest task | 84.0 | 83.0 | 75.5 | 79.8 |
| Layer | Parts | Where |
|---|---|---|
| Principle | a residual-aware metric of past activations constrains the gradient and the realised update | Operators |
| Core method (class-free; alone 67.1 against 71.6 over three runs) | low-rank-plus-residual activation memory; the protected subspace and a residual floor outside it (Box 1); gradient constraint and post-optimizer update constraint | What the memory stores; Operators; The constrained update |
| Class-incremental extension | class-conditioned index (class state records and input patterns, which feed the class-pattern constraint); state-only conflict detection and comparator; joint-share target (caps a new class’s share of a decision region shared with an earlier one) | Operators; Conflict detection; Learning at detected conflicts |
| Memory maintenance | sleep encoding (required, Fig. 3); the gated-delta erase policy (erase lowers the stored energy, for sequences that approach saturation) | Sleep |
| Scaling rules | relative step; width rule; protected basis orthonormalised in fp64; embedding-row and gain plasticity factors | Model training; Operators; Cost |
| # | Hard constraint | Regime it excludes | Baselines that use it |
|---|---|---|---|
| 1 | No cross-task replay of stored, generated or reconstructed samples, and no stored per-sentence activations. The only re-reading is one gradient-free encoding phase over the just-finished task, implemented with one or more forward traversals; no earlier task is revisited and none of its samples is kept afterwards | keeping old data | DER++, CLS-ER, LAMOL, PCLL; SEQ* on the encoders |
| 2 | No teacher, model copy or distillation : no previous-model snapshot or copy, even read-only, and no distillation from an earlier model | keeping old models | L2KD, LAMOL, PCLL, CLS-ER |
| 3 | No freezing of the backbone or of any module: every parameter tensor remains in the optimization; plasticity is constrained directionally and continuously rather than switched off, and selected directions or coordinates may receive an exact zero plasticity factor | not changing the model | SEQ*, EASE, SD-LoRA |
| 4 | No LoRA, prompts or adapters , and no task-specific parameters or heads beyond the standard expanding classifier | not changing the model | EASE, SD-LoRA |
| 5 | No task identity and no memory at test time : the plain model is evaluated (greedy generation for LLMs, the plain classifier for encoders and vision), with the hippocampus detached | not answering alone | none of the compared baselines |
| Synapse group | Linear maps | Input dimension (Qwen3-0.6B) |
|---|---|---|
| qkv | query, key, value | 1024 (hidden) |
| gu | MLP gate, MLP up | 1024 |
| o | attention output | 2048 (heads × head dim) |
| down | MLP down | 3072 (intermediate) |
| Model | Layers | Linear synapse groups | Linear maps | Gain synapse groups | Embedding |
|---|---|---|---|---|---|
| Qwen3-0.6B | 28 | 113 | 197 | 113 | tied |
| Llama-3.2-1B | 16 | 65 | 113 | 33 | tied |
| Llama-3.2-3B | 28 | 113 | 197 | 57 | tied |
| Qwen3-4B | 36 | 145 | 253 | 145 | tied |
| Qwen3-8B | 36 | 145 | 253 | 145 | untied |
| Llama-3.1-8B | 32 | 129 | 225 | 65 | untied |
| Response | Acts at | What it does | Evidence |
|---|---|---|---|
| Update gate | every step, through the conflicts of its answer-token positions | applies the step with probability | removed with the other class-conditioned parts: 67.1 against 71.6 over three runs (class-free core) |
| Comparator | conflicting positions whose gold does not win yet | gradient and update along the input’s mismatch with every earlier pattern | final AA 71.6 58.7 over three runs; city of death answered as city of residence 21 times instead of at most 3 |
| Joint-share target | answer-token positions after the first, for near-duplicate records seen in conflict while gold lost | drives the gold probability to instead of 1 | removed with the other class-conditioned parts: 67.1 against 71.6 over three runs (class-free core) |
| Component | LLMs | BERT / RoBERTa | ViT-B/16 | ResNet-50 |
|---|---|---|---|---|
| Backbone operator | minimum-interference | novelty factor | minimum-interference | novelty factor |
| Comparator restoration | full | minimum-interference plasticity factor | minimum-interference plasticity factor | the backbone rule’s plasticity factor |
| Comparator active at | conflicts where gold does not win | every input whose gold does not win | conflicts where gold does not win | conflicts where gold does not win |
| Joint-share target | answer-token positions | the decision feature | – | – |
| Embedding rows | (tied: with LM-head operator) | in the units of the residual stream they feed | [CLS]/position rows in residual-stream units | – |
| Weight decay | decoupled | folded into the update before the constraint | folded | folded |
| Run | Tasks | Training | Sleep | Evaluation | Wall time | Peak GPU memory (training / sleep) |
|---|---|---|---|---|---|---|
| Llama-3.2-1B, TACRED | 8 | 171 s (45.6%) | 153 s (40.8%) | 51 s (13.6%) | 0.10 h | 33.7 / 25.0 GB |
| Llama-3.2-1B, FewRel | 8 | 634 s (57.3%) | 250 s (22.5%) | 224 s (20.2%) | 0.31 h | 30.6 / 24.9 GB |
| Llama-3.1-8B, TACRED | 8 | 486 s (42.4%) | 601 s (52.4%) | 60 s (5.2%) | 0.32 h | 138.6 / 117.9 GB |
| Llama-3.1-8B, CLINC150 | 15 | 750 s (28.8%) | 1723 s (66.1%) | 134 s (5.1%) | 0.72 h | 144.7 / 137.1 GB |
| Item | Value |
|---|---|
| Wall time | 375 s = 0.10 GPU-hours: training 45%, sleep 41%, evaluation 14% |
| Sleep | 13 gradient-free passes over task data in total, 0.82 M tokens (training: 1.44 M tokens over 3 epochs) |
| Linear algebra in sleep | eigendecomposition 89 s (dominated by the 8192 × 8192 MLP down-projection moments), QR 8 s |
| Linear algebra in training | about 1.3 s over 702 steps |
| Peak GPU memory | 33.7 GB training, 25.0 GB sleep |
| Peak host memory | 7.7 GB (subspace memory 1.40 GB fp32 on the host, about 0.7 GB bf16 on the device) |
| Benchmark | Classes | Tasks | Classes per task |
|---|---|---|---|
| TACRED | 40 | 8 | 5 |
| FewRel | 80 | 8 | 10 |
| Banking77 | 77 | 7 | 11 |
| CLINC150 | 150 | 15 | 10 |
| CIFAR-100 | 100 | 10 | 10 |
| ImageNet-R | 200 | 10 | 20 |
| Function | Account | Mechanism in DeltaHippo | Evidence in this study | Where |
|---|---|---|---|---|
| Division of labour | Complementary learning systems : long-term knowledge in the cortex; a fast hippocampal trace governs how it is laid down 65–68 | The network holds all task knowledge and is trained with plain cross-entropy; the hippocampus is read at every wake step and shapes only the update; the plain model answers | Without the read, every earlier task collapses within a quarter of the next task (Fig. 3, condition B) | Model training |
| Index | Hippocampal indexing : the hippocampus stores an index to cortical patterns, not the patterns 69,70 | Per synapse group, a low-rank subspace with energies and a residual floor; per class, the mean answer state and input pattern. No sample is stored | Rotating the stored directions with the spectrum kept collapses earlier tasks (Fig. 3, condition C) | Synapse groups; What the memory stores |
| Index | Sleep-dependent consolidation : experience is reactivated offline after learning 71,72 | After each task, a gradient-free re-read of that task writes the index | Blocking one sleep makes its own episode selectively vulnerable (Fig. 3, condition E) | Sleep |
| Regulation | Memory strength and metaplasticity : consolidated memories are harder to overwrite, and a synapse’s plasticity depends on its history 73–75 | Each direction keeps the plasticity factor ; the space outside the stored span keeps ; embedding rows and gains are protected according to their earlier use | Opening the space outside the span loses retention, more so the smaller the explicitly protected span (Extended Data Table 4) | Operators |
| Conflict | Match–mismatch novelty detection 76,77 | An answer state is in conflict when it falls inside an earlier class’s record; only the state is read | Placing the comparator where conflict detection fires raises acquisition by 3.5 points over the same number of other positions (three runs) | Conflict detection |
| Conflict | Cholinergic balance of encoding and retrieval 78 | A match with a strongly recorded class lowers the update probability | Applied whole with probability ; removed with the other class-conditioned parts, final AA falls from 71.6 to 67.1 over three runs (class-free core; Extended Data Table 4) | Learning at detected conflicts |
| Model | Method | TACRED | FewRel | Banking77 | CLINC150 |
|---|---|---|---|---|---|
| Qwen3-0.6B | DeltaHippo | 70.92 / 74.43 / 72.01 | 69.39 / 71.76 / 71.88 | 83.38 / 82.60 / 82.76 | 87.91 / 83.87 / 89.62 |
| SEQ* | 52.12 / 54.09 / 53.83 | 69.17 / 69.62 / 68.85 | 72.50 / 75.49 / 75.42 | 87.96 / 87.78 / 87.62 | |
| SEQ | 40.85 / 34.41 / 36.35 | 24.22 / 25.53 / 23.62 | 19.94 / 19.64 / 20.13 | 40.40 / 38.42 / 38.18 | |
| L2KD | 11.21 / 9.44 / 11.34 | 11.23 / 13.08 / 11.79 | 17.66 / 17.53 / 17.37 | 25.09 / 24.31 / 25.09 | |
| LAMOL | 32.96 / 33.53 / 33.77 | 29.24 / 26.66 / 26.85 | 31.04 / 29.74 / 27.18 | 54.29 / 54.89 / 56.09 | |
| PCLL | 33.78 / 31.41 / 33.29 | 23.71 / 25.03 / 25.83 | 34.16 / 39.32 / 37.99 | 56.29 / 58.11 / 53.64 |
| Model | Method | TACRED | FewRel | Banking77 | CLINC150 |
|---|---|---|---|---|---|
| BERT-base | DeltaHippo | 62.76 / 63.18 / 61.93 | 69.17 / 63.64 / 68.14 | 85.07 / 85.19 / 84.81 | 90.11 / 90.36 / 90.80 |
| SEQ | 18.00 / 17.81 / 18.32 | 15.85 / 17.84 / 18.53 | 20.39 / 20.32 / 20.65 | 14.13 / 15.58 / 15.20 | |
| SEQ* | 28.98 / 31.36 / 29.69 | 47.48 / 45.81 / 51.25 | 56.62 / 61.01 / 57.73 | 63.18 / 63.73 / 60.58 | |
| DER++ | 41.44 / 43.04 / 41.55 | 50.59 / 48.78 / 52.35 | 53.70 / 55.03 / 54.12 | 75.93 / 76.00 / 73.73 | |
| CLS-ER | 45.58 / 48.39 / 40.94 | 52.71 / 53.88 / 50.41 | 58.64 / 59.61 / 60.06 | 73.22 / 76.42 / 76.11 | |
| RoBERTa-base | DeltaHippo | 55.32 / 44.52 / 59.89 | 75.26 / 67.59 / 71.79 | 83.99 / 85.68 / 83.34 | 91.49 / 92.20 / 91.98 |
| Model | Method | CIFAR-100 | ImageNet-R |
|---|---|---|---|
| ViT-B/16 | DeltaHippo | 88.49 / 88.14 / 88.00 | 63.94 / 64.12 / 64.46 |
| SEQ | 22.16 / 21.11 / 20.47 | 23.88 / 22.69 / 23.14 | |
| SEQ* | 81.81 / 81.15 / 81.59 | 64.37 / 64.47 / 63.85 | |
| DER++ | 48.01 / 45.01 / 44.48 | 41.58 / 43.46 / 43.74 | |
| CLS-ER | 79.55 / 81.56 / 80.15 | 52.58 / 51.57 / 51.23 | |
| EASE | 86.99 / 87.04 / 87.02 | 67.69 / 68.01 / 67.74 |
| Variant | Final AA | Mean acquisition | Retention |
|---|---|---|---|
| DeltaHippo | 71.27 / 72.14 / 71.31 | 78.81 / 75.50 / 79.24 | 80.80 / 84.02 / 81.46 |
| Class-free geometric core only | 69.57 / 65.82 / 65.83 | 73.96 / 70.48 / 78.64 | 82.72 / 82.16 / 77.86 |
| No update constraint (gradient constrained only) | 60.05 / 64.37 / 62.01 | 85.32 / 88.20 / 88.47 | 56.75 / 63.30 / 60.84 |
| No gradient constraint (update constraint only) | 68.23 / 42.40 / 58.62 | 87.28 / 92.15 / 92.53 | 60.05 / 50.18 / 49.11 |
| Hard constraint inside the protected subspace, residual floor kept | 70.74 / 69.82 / 72.66 | 76.40 / 73.05 / 78.34 | 82.46 / 83.18 / 82.96 |
| Graded inside, residual subspace fully open | 69.77 / 74.02 / 69.76 | 85.37 / 86.65 / 83.96 | 77.53 / 80.51 / 79.42 |
| Order | DeltaHippo | SEQ* | Margin (mean) |
|---|---|---|---|
| Protocol order | 71.27 / 72.14 / 71.31 (71.58 ± 0.49) | 55.93 / 53.33 / 55.58 (54.95 ± 1.41) | +16.6 |
| Further order 1 | 72.70 / 74.97 / 69.05 (72.24 ± 2.99) | 58.31 / 59.68 / 59.75 (59.25 ± 0.81) | +13.0 |
| Further order 2 | 71.32 / 72.42 / 74.11 (72.62 ± 1.41) | 54.41 / 54.75 / 55.14 (54.77 ± 0.36) | +17.9 |
| Condition | Generation, tasks 0–2 / 3 | Own probe, tasks 0–2 / 3 | Normal run’s probe applied, tasks 0–2 / 3 |
|---|---|---|---|
| A. Normal | 81.6 / 69.1 | 85.9 / 75.7 | 85.9 / 75.7 |
| B. Read off | 0.0 / 0.0 | 81.4 / 70.7 | 10.6 / 2.8 |
| C. Scrambled content | 1.2 / 0.0 | 87.5 / 79.6 | 27.4 / 34.3 |
| F. Norm-matched plain | 0.0 / 0.0 | 85.1 / 75.7 | 28.7 / 8.3 |
| Pre-trained model | – | 78.3 / 71.8 | – |
| Backbone frozen after task 0 | – | 83.4 / 74.6 | – |
| Condition | Own head | Bias | Bias + temperature | Rank-1 map | Rank-4 map | Rank-16 map | Re-fitted probe |
|---|---|---|---|---|---|---|---|
| Whole model | 80.6 ± 2.8 | 82.6 ± 2.0 (0.25) | 82.9 ± 1.9 (0.29) | 85.4 ± 0.7 (0.60) | 88.4 ± 1.1 (0.97) | 88.6 ± 0.1 (1.00) | 88.6 ± 0.5 |
| Backbone only | 24.7 ± 12.5 | 47.8 ± 17.8 (0.37) | 56.5 ± 17.2 (0.50) | 84.4 ± 1.7 (0.95) | 86.5 ± 1.4 (0.98) | 88.1 ± 0.2 (1.00) | 87.8 ± 0.2 |
| Head only | 0.6 ± 1.1 | 1.2 ± 1.1 (0.01) | 3.3 ± 2.8 (0.03) | 28.9 ± 6.0 (0.32) | 75.5 ± 1.6 (0.85) | 83.4 ± 0.5 (0.94) | 88.9 ± 0.2 |
| Read off | 0.0 | 0.0 (0.00) | 0.0 (0.00) | 26.8 (0.32) | 73.4 (0.87) | 76.6 (0.90) | 84.8 |
| Scrambled content | 0.9 | 3.5 (0.03) | 11.3 (0.12) | 54.6 (0.61) | 86.7 (0.97) | 86.3 (0.97) | 89.4 |
| Norm-matched plain | 0.0 | 5.4 (0.06) | 9.7 (0.11) | 56.2 (0.63) | 85.9 (0.97) | 87.6 (0.99) | 88.5 |
| Condition | Own head | Own head, restricted decoding | Re-fitted probe | Held-out rank-1 / 4 / 16 | Same-class rank-1 / 4 / 16 |
|---|---|---|---|---|---|
| Whole model | 84.9 | 92.0 | 89.4 | 33.4 / 35.6 / 34.5 | 94.2 / 95.5 / 96.3 |
| Backbone only | 29.4 | 72.6 | 89.7 | 20.5 / 17.9 / 21.5 | 94.3 / 95.3 / 95.9 |
| Head only | 2.0 | 14.9 | 90.3 | 0.3 / 0.3 / 0.0 | 45.6 / 92.3 / 95.2 |
| Read off | 0.0 | 4.3 | 86.1 | 0.0 / 0.0 / 0.0 | 44.2 / 90.7 / 92.0 |
Catastrophic Forgetting
Catastrophic forgetting is the loss of previously learned behavior when a model adapts to new tasks or changing data.
See all 374 papers