cs.LGOct 28, 2026
SaveEpisodic looking - using reinforcement learning to bias perception enhances few-shot and continual learning with protection from forgetting
Organizations: Cerenaut Melbourne Australia
Abstract
There is a fundamental contradiction between the desire for models that can generalize, and models that can continuously adapt to recent input with changing statistics. These conditions, captured in few-shot and continual learning settings, can bias or disrupt learned representations. This harms generalization to a broader context, leaving the model unable to overcome input variance. This paper proposes a solution based on the Complementary Learning System architecture (CLS), in which a Short-Term Memory (STM) delegates generalization to a slow, statistical Long-Term Memory (LTM). Delegation is achieved by working in a space of robust, generalized perceptions produced by the LTM. With this separation, the STM can focus on memorizing recent input and avoiding interference between memories. We show that on an image classification task, the resulting CLS model is more resistant to forgetting in a domain-incremental continual learning setting than a baseline ResNet LTM, and generalizes more effectively in a few-shot setting than other continual learners. In the continual setting, overall accuracy is slightly worse than existing continual learning models, but with a different mix of performance characteristics. Results analysis suggests that the model is limited by the representations and representational plasticity of the LTM. In future work, we will seek to address this by replaying STM memories to the LTM.
Figures & tables
Figure 1: Active perception within a Complementary Learning System could allow a STM to learn rapidly without needing to generalize. In this concept, generalization occurs within the LTM, which is manipulated by the STM to emit stable, familiar representations relevant to the task. Using the generalization ability within the LTM, the STM could even immediately reason about completely novel stimuli, if they are expressed in terms the STM can already understand.
Figure 2: The CIFAR-100 dataset has 20 “coarse” classes and 5 “fine” sub-classes of each coarse class, yielding a total of 100 distinct classes. The LTM is pre-trained on fine classes 1 and 2 of all coarse classes. Next, individual pairs of coarse classes are selected for continual and few-shot experiments using the unseen fine classes 3, 4 and 5.
| Pair | Coarse classes | Fine classes | |||||
|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | |||
| A | 1 | Aquatic mammals | Beaver | Dolphin | Otter | Seal | Whale |
| A | 2 | Fish | Aquarium fish | Flatfish | Rays | Shark | Trout |
| B | 3 | Flowers | Orchids | Poppies | Roses | Sunflowers | Tulips |
| B | 4 | Food containers | Bottles | Bowls | Cans | Cups | Plates |
| C | 6 | Electrical devices | Clock | Keyboard | Lamp | Telephone | TV |
Table 1: Details of the coarse classes used in reported experiments. We formed four pairs (A-D) from these 8 coarse classes.
Figure 3: Architecture of the proposed CLS model with a sparse, episodic (RL) Short-Term Memory (STM). The STM is an Advantage Actor-Critic (A2C) model. The actor produces a multiplicative bias output which gates the residual stream of the Long-Term Memory (LTM), a four-stage ResNet-18 model adapted to CIFAR images. STM input includes previous bias output, post-bias LTM encoding and LTM classification logits. These accumulate in a buffer over each episode; the initial encoding is unbiased. The STM is rewarded for producing bias which improves the accuracy of LTM classification.
Figure 4: Comparison and ablation STM variants. The RL-STM is compared to some competitive published models (FlyModel, SDMLP) and some simple alternatives such as ablation of various features (linear probe, sparse supervised STM without RL, LTM-only fine-tuning). The Nearest Class Mean (NCM) keeps a running mean of all input samples. The STM variants all consume the 512-dimensional encoding produced by global average pooling of stage 4, which is the input of the LTM’s own dense classifier. Only the RL-STM biases the LTM. Italics describe how each STM learns. The linear probe’s dense layer is initialized from the LTM’s.
| STM variant | Target labels | Label-routed updates | Differentiable task loss | Phase change input |
|---|---|---|---|---|
| RL-STM | no | no | no | no |
| Sparse supervised STM, linear probe, SDMLP | yes | no | yes | no |
| FlyModel, NCM | yes | yes | no | no |
| LTM-only fine-tuning | yes | no | yes | no |
| SDMLP+EWC or EWC (not evaluated) | yes | no | yes | yes |
Table 2: The RL-STM could potentially be applied in a wider range of tasks than the other STM variants, because it does not require target labels, a differentiable task loss (the gradient of the loss with respect to its own parameters), or information about phase changes during continual learning. Label-routed updates: the label decides which parameters change, and only the true class’s parameters do. Every model we run is phase-blind, but other continual learners such as EWC can exploit this information.
Figure 5: Continual learning results, with test set accuracy for each of the four fine-class sets plotted over the three training phases. Results are averaged over the seeds and orderings of fine classes 3, 4 and 5. Top row: Accuracy over the 4 test sets (1 & 2, 3, 4, 5). Middle row: Forgetting of any trained fine classes trained so far. Bottom row: Stability / plasticity tradeoff of each model.
Figure 6: Continual learning results, with test set accuracy for each of the four fine-class sets plotted over the three training phases. Results are averaged over the seeds and orderings of fine classes 3, 4 and 5. The different models shown here learn and forget to different degrees. LTM-only fine-tuning learns the current class more completely than the other models, but forgets a lot. The RL-STM doesn’t learn as much, but forgets a lot less. The Sparse supervised STM forgets more than the RL-STM. The NCM “forgets” the least, but also learns less than the LTM.
| Pair | Model | ACC | LA | BWT | BWT 0 | FWT | Forgetting | Budget | |
| A | RL-STM | 3 | 0.753 0.010 | 0.795 | 0.086 | 12 | |||
| Sparse supervised STM | 3 | 0.722 0.012 | 0.822 | 0.169 | 12 | ||||
| Linear probe | 5 | 0.773 0.002 | 0.813 | 0.100 | 8 | ||||
| NCM | 5 | 0.792 0.000 | 0.794 | 0.035 | 49 | ||||
| FlyModel | 5 | 0.791 0.005 | 0.771 | 0.017 | 1 | ||||
| SDMLP | 5 | 0.794 0.001 | 0.794 | 0.031 | 16 |
Table 3: Continual learning test set accuracies over 4 coarse class pairs A-D. Results are averaged over 3 class orderings per pair, and then averaged over pairs. ACC: final average accuracy over the four test sets; LA: learned accuracy on each fine class at the end of its training phase; BWT: change on fine classes 3 to 5 from the end of their phase to the end of the run; BWT 0 : change on fine classes 1 and 2 over the run; FWT: gain on a class before its own phase; Forgetting: largest drop of a class learnt in the run from its best at a phase boundary. BWT 0 and FWT are measured from the end of pre-training, so a model that starts lower scores higher on both. : seeds. ACC is mean one standard deviation over seeds; the standard deviations of the other metrics are at most 0.032 for the RL-STM and the sparse supervised STM, 0.011 for the other heads and 0.012 for LTM-only fine-tuning. Budget: training epochs per phase. Bold: best ACC, LA, BWT and Forgetting in each block.
Figure 7: Few-shot results for all STM variants and LTM-only fine-tuning. Results show mean test set accuracy on Pair A, separately trained and evaluated on fine-classes 3, 4, and 5. 10 seeds were used for and because performance is highly dependent on the images selected. 5 seeds were used for larger . a) Accuracy on the few-shot trained fine-class (mean over 3, 4 and 5). b) Retention of pre-trained fine-classes 1 and 2. c) Mean of the four fine-class sets (1 & 2, 3, 4, and 5).
| Model | Start | |||||
|---|---|---|---|---|---|---|
| Trained class | ||||||
| RL-STM | 0.471 | 0.501 0.018 | 0.658 0.041 | 0.781 0.016 | 0.804 0.010 | 0.814 0.012 |
| Sparse supervised STM | 0.640 | 0.641 0.029 | 0.669 0.030 | 0.708 0.027 | 0.717 0.036 | 0.720 0.037 |
| Linear probe | 0.506 | 0.509 0.002 | 0.684 0.021 | 0.721 0.024 | 0.764 0.006 | 0.793 0.001 |
| NCM | 0.673 | 0.673 0.000 | 0.716 0.010 | 0.758 0.011 | 0.745 0.003 | 0.767 0.000 |
| FlyModel | 0.671 | 0.672 0.005 | 0.673 0.004 | 0.677 0.006 | 0.729 0.005 | 0.778 0.005 |
Table 4: Few-shot results for all STM variants and LTM-only fine-tuning on Pair A test sets. denotes the number of training images of each fine-class 3, 4 or 5 presented per coarse class (100 images reserved for validation). All fine-classes were trained separately from checkpoints of the models pre-trained on fine classes 1 and 2. Values are test set accuracies. Top rows: final accuracy on the few-shot trained class. Middle rows: Forgetting on fine classes 1 and 2. Bottom rows: Test set averages over the four fine-classes after few-shot training. Start denotes test accuracy before few-shot training. Training epoch budgets can be found in table 6 .
Figure 8: Analysis of STM behaviour within an episode. These data were collected from processing 500 images of Pair A fine-class 3 on which the RL-STM model was already trained. Accuracy and entropy are derived from the LTM output logits.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Learning rates |
|---|---|
| RL-STM | Not tuned (0.1); learning rate 0.2 with zero momentum tested on validation set for Pair A and found to be equivalent. Few-shot training budget optimized, but not continual. |
| Sparse supervised STM | 0.1, 0.03, 0.01 |
| Linear probe | 0.0003, 0.001, 0.003 |
| NCM | none |
| FlyModel | Not tuned (0.2) |
| SDMLP | 0.015, 0.05, 0.15 |
Table 5: Learning rates explored for each model on Pair A validation split.
| Few-shot, | |||||||
|---|---|---|---|---|---|---|---|
| Model | LR | Continual | 1 | 4 | 16 | 64 | 400 |
| RL-STM | 0.1 | 12 (fixed) | 1 (0.571) | 24 (0.646) ‡ | 26 (0.721) § | 21 (0.735) § | 21 (0.731) |
| Sparse supervised STM | 0.01 | 12 (fixed) | 1 (0.714) | 8 (0.721) | 42 (0.736) | 6 (0.735) | 1 (0.740) |
| FlyModel | – | 1 (0.749) | 1 (0.745) | 1 (0.746) | 1 (0.744) | 3 (0.756) | 1 (0.770) |
| Linear probe | 0.0003 | 24 (0.770) | 1 (0.659) | 2060 (0.705) | 471 (0.721) | 126 (0.748) | 24 (0.755) |
| Linear probe | 0.001 | 8 (0.775) | 1 (0.658) | 589 (0.705) | 148 (0.722) | 36 (0.745) | 8 (0.759) |
Table 6: Training budget optimization on the Pair A validation split. For each model and learning rate, the training epoch budget is selected to optimize mean accuracy on the four fine classes. Bold: chosen. § Not plateaued at the cap (RL-STM: 48 epochs, four times its starting budget). ‡ RL-STM at : not plateaued at 24 epochs. –: not evaluated by hyperparameter search rule.
| Component | Output dimension |
|---|---|
| Input | |
| Stem: conv., 64 channels, stride 1 | |
| Stage 1: conv., 64 channels, stride 1 | |
| Stage 2: conv., 128 channels, stride 2 | |
| Multiplicative “bias” applied here. | |
| Stage 3: conv., 256 channels, stride 2 |
Table 7: Long Term Memory configuration. Note there are 2 residual blocks per stage.
| Name | Value |
|---|---|
| Pretraining mini-batch size | 128 (16 in fine-tuning) |
| Optimizer | Adam |
| Learning rate | 0.001 (varied for fine-tuning) |
Table 8: Long Term Memory hyperparameters
| Name | Value |
| Pretraining mini-batch size | 16 |
| Fine-tuning mini-batch size | 16 |
| Optimizer | SGD |
| Learning rate | 0.1 |
| Momentum | 0.5 |
| Memory capacity | 1000 |
Table 9: RL-STM hyperparameters
More in this topic
Continual Learning
Continual Learning studies how models acquire knowledge from changing data streams while retaining or selectively revising what they learned before.
See all 307 papers