Unsupervised Continual Learning with Growing Self-Organizing Maps and Synthetic Replay

Pujan Thapa; Alexander Ororbia; Travis Desell

cs.LGAug 27, 2026

Unsupervised Continual Learning with Growing Self-Organizing Maps and Synthetic Replay

Authors: Pujan Thapa, Alexander Ororbia, Travis Desell

Organizations: Department of Computer Science Rochester Institute of Technology Rochester, NY, USA

Abstract

This work presents a generative continual learning framework based on growing self-organizing maps (GSOMs) that are augmented with learned distributional statistics as well as encoder-decoder models for class-incremental learning. The proposed approach enables exemplar-free replay using distributional statistical memory, which eliminates the need to store raw data. Each GSOM unit maintains its own mean, variance, and covariance estimates, which are subsequently used to generate synthetic samples for replay; in encoder-decoder configurations, these samples are then decoded back into the input space (via ancestral sampling) for subsequent training. Our method is fully unsupervised, as it does not rely on explicit task boundaries or class labels during training. Results across multiple benchmarks show that the proposed approach achieves performance competitive even with supervised state-of-the-art memory-based methods while consistently outperforming memory-free approaches. In several settings, our framework matches or exceeds existing baselines, particularly in challenging single-class incremental scenarios. We also provide baseline results for single-class incremental TinyImageNet and MiniImageNet, offering a useful reference for future work. This work highlights the effectiveness of an unsupervised, adaptive, topology-driven neural form of statistical replay as a scalable, flexible approach to continual learning.

Figures & tables

Figure 1: CIFAR-100 samples for the first $10$ classes generated by sampling from a GSOM integrated with ResNet-18 embeddings. Each row corresponds to one class ( $0$ – $9$ ).

Method	Split-MNIST	Split-CIFAR-10	Split-CIFAR-100
iid-offline	$95.82\pm 0.33$	$80.54\pm 0.63$	$48.09\pm 0.90$
Fine-Tune	$19.68\pm 0.02$	$19.19\pm 0.06$	$8.32\pm 0.23$
EWC	$19.92\pm 0.35$	$16.18\pm 1.37$	$4.41\pm 0.37$
SI	$19.76\pm 0.01$	$17.27\pm 0.87$	$5.87\pm 0.21$
LwF	$20.54\pm 0.64$	$18.53\pm 0.12$	$6.93\pm 0.32$
GEM	$48.57\pm 5.26$	$25.54\pm 0.19$	$6.18\pm 0.20$

Table 1: Classification accuracy for the class-incremental learning experiments on Split-MNIST, Split CIFAR-10, and Split CIFAR-100 without any pre-trained models. Our results are reported as mean

\pm

standard deviation over

20

runs, whereas the compared results are over

5

runs. † denotes task-free setup and ‡ denotes unsupervised training. Methods without symbols are task-aware and supervised.

Method	Split-MNIST	Split-CIFAR10	Split-CIFAR100	Split MiniImagenet
iid-online	85.99 $\pm$ 0.03	62.23 $\pm$ 1.5	18.13 $\pm$ 0.80	17.53 $\pm$ 1.6
finetune*	19.75 $\pm$ 0.05	18.55 $\pm$ 0.34	3.53 $\pm$ 0.04	2.84 $\pm$ 0.4
GEM*	93.25 $\pm$ 0.36	24.13 $\pm$ 2.46	11.12 $\pm$ 2.48	-
iCARL*	83.95 $\pm$ 0.21	37.32 $\pm$ 2.66	10.80 $\pm$ 0.37	-
CURL* §	92.59 $\pm$ 0.66	–	–	–
CN-DPM	93.23 $\pm$ 0.09	45.21 $\pm$ 0.18	20.10 $\pm$ 0.12	27.07 $\pm$ 2.3

Table 2: Performance comparison on Split-MNIST, Split-CIFAR10, and Split-CIFAR100. All methods use a pretrained ResNet-18 encoder and are evaluated over

20

independent runs. We additionally evaluate our method using a pretrained CLIP encoder, also over

20

independent runs. Results marked with * and

\dagger

are quoted from ( Lange and Tuytelaars, 2021 ) and ( Jin et al., 2021 ) , respectively. § denotes unsupervised training; methods without § are supervised. Global denotes a frozen shared encoder, Global FT denotes a globally fine-tuned shared encoder, and BMU Specific denotes BMU-wise encoder adaptation.

Method	w/o pretraining			w/ pretraining
	MNIST	CIFAR10	CIFAR100	CIFAR10	CIFAR100
EWC	9.91	10.01	1.03	10.21	2.93
LwF	19.96	10.05	2.13	19.39	6.25
IMM	29.16	10.25	1.21	51.22	12.58
PGMA	71.36	20.08	1.86	56.22	12.37
RPSNet	40.29	16.31	1.96	55.54	4.13

Table 3: Final classification accuracy for single class incremental learning. All baseline methods in this table are task-aware and supervised; they rely on explicit class arrival signals to trigger incremental updates, as opposed to the proposed GSOM framework which is task-free and unsupervised.

Method	Split-TinyImageNet	TinyImageNet (SCIL)	MiniImageNet (SCIL)
VAE (Global FT)	6.22 $\pm$ 0.67	6.12 $\pm$ 0.58	6.28 $\pm$ 1.12
VAE (BMU Specific)	6.33 $\pm$ 0.88	6.01 $\pm$ 0.73	5.94 $\pm$ 1.36
GSOM + ResNet-18 (global)	32.12 $\pm$ 0.78	30.67 $\pm$ 1.23	24.21 $\pm$ 0.92
GSOM + ResNet-18 (global FT)	32.78 $\pm$ 1.56	31.23 $\pm$ 0.15	25.04 $\pm$ 0.71
GSOM + ResNet-18 (BMU Specific)	32.59 $\pm$ 1.89	31.61 $\pm$ 1.51	24.63 $\pm$ 0.84
GSOM + CLIP (global)	57.98 $\pm$ 1.11	40.45 $\pm$ 1.02	33.18 $\pm$ 0.58

Table 4: GSOM-based method performance on Split-TinyImageNet and single-class incremental learning (SCIL) TinyImageNet and MiniImageNet.

Figure 2: The proposed GSOM growth mechanism. When a unit (highlighted in yellow) exceeds the growth threshold, new units are added in its immediate $4$ -neighborhood, i.e., at $(x,y\pm 1)$ and $(x\pm 1,y)$ if those positions are not already occupied.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Symbol	Best value	Other values tested
Mean	$\beta_{\mu}$	$0.99$	$0.98$ , $0.95$
Variance / Cov.	$\beta_{\sigma},\beta_{\Sigma}$	$0.95$	$0.91$ , $0.99$ , $0.90$

Appendix

Table 5: Bias-correction hyperparameters with best-performing values and additional settings tested.

Component	Parameter	Value
Datasets
	MNIST	$28\times 28$ (grayscale)
	CIFAR-10 / CIFAR-100	$32\times 32\times 3$
	TinyImageNet / MiniImageNet	$64\times 64\times 3$ / $84\times 84\times 3$
Shared GSOM Parameters
	Initial Map Size	$2\times 2$ (MNIST), $5\times 5$ (CIFAR10)

Appendix

Table 6: Unified hyperparameter configuration across all experiments. Shared GSOM parameters are consistent unless otherwise specified.

SOM-only
Method	Model	Split-MNIST	Split-CIFAR-10	Split-CIFAR-100	Split-TinyImageNet
Fixed	SOM-only	92.11 $\pm$ 1.18 (1600)	–	–	–
GSOM	GSOM-only	91.22 $\pm$ 1.23 (357)	–	–	–
VAE (Global FT)
Fixed	SOM + VAE	91.23 $\pm$ 1.21 (1600)	53.01 $\pm$ 1.18 (1600)	14.63 $\pm$ 0.33 (1600)	12.74 $\pm$ 0.84 (1600)
GSOM	GSOM + VAE	92.11 $\pm$ 0.98 (376)	50.11 $\pm$ 0.89 (956)	13.81 $\pm$ 1.05 (1167)	6.22 $\pm$ 0.67 (1241)

Appendix

Table 7: Comparison of fixed SOM and GSOM (with bias) across datasets. Results are reported as accuracy (%), with the number of best-matching units (BMUs) after final training shown in parentheses. The fixed SOM uses a

40\times 40

grid (1600 units) for all datasets.

$E_{\text{grow}}$	$G_{\text{max}}$	Split-MNIST	Split-CIFAR-10	Split-CIFAR-100	Split-TinyImageNet
6	3	95.01 (432)	57.41 (1012)	35.01 (1284)	31.74 (1386)
7	3	95.26 (468)	57.72 (1089)	35.32 (1352)	32.05 (1438)
9	4	96.12 (498)	58.00 (956)	35.67 (1161)	32.12 (1312)
10	5	96.42 (548)	58.11 (1184)	35.74 (1426)	32.28 (1502)
20	14	96.45 (712)	58.18 (1621)	35.81 (1895)	32.36 (2143)

Appendix

Table 8: Growth sensitivity analysis of GSOM with global ResNet-18. Increasing growth parameters improves accuracy but increases map size. Results are reported as accuracy (%), with the number of best-matching units (BMUs) after final training shown in parentheses.

Method	MNIST	CIFAR-10	CIFAR-100
Global VAE-GSOM (Error only)	$89.42\pm 1.21$	$44.87\pm 1.03$	$10.92\pm 0.88$
Global VAE-GSOM (Surprisal only)	$91.26\pm 1.04$	$48.63\pm 0.95$	$12.97\pm 0.96$
Global VAE-GSOM (Error + Surprisal)	$\mathbf{92.11\pm 0.98}$	$\mathbf{50.11\pm 0.89}$	$\mathbf{13.81\pm 1.05}$

Appendix

Table 9: Ablation study of GSOM growth criteria across continual learning benchmarks on Split datasets.

No Bias
Setting	Model	Split-MNIST	Split-CIFAR10	Split-CIFAR100	Split-MiniImageNet
	Global	95.84 $\pm$ 0.41	57.21 $\pm$ 1.44	34.72 $\pm$ 2.03	31.58 $\pm$ 1.37
	Global FT	96.58 $\pm$ 0.52	57.34 $\pm$ 1.51	34.91 $\pm$ 1.39	32.71 $\pm$ 1.53
	BMU Specific	93.61 $\pm$ 1.28	56.32 $\pm$ 1.67	34.02 $\pm$ 1.88	31.74 $\pm$ 1.21
Bias
	Global	96.12 $\pm$ 0.30	58.00 $\pm$ 1.31	35.67 $\pm$ 1.96	32.67 $\pm$ 1.23

Appendix

Table 10: Effect of bias correction across GSOM + ResNet-18 configurations. Results are reported as accuracy (%).

Figure 3: Synthetic samples generated using Gaussian sampling from the trained SOM. Each row corresponds to one class (0–9), and each row contains 10 synthetic samples generated from BMUs labeled with that class.

Panel of Figure 4 — Figure 4: Synthetic CIFAR-10 samples generated by sampling latent vectors from a Self-Organizing Map (SOM) trained on VAE latent space. Each latent vector was sampled from the full-covariance Gaussian distribution of a Best Matching Unit (BMU) labeled with the target class. The sampled latents were then decoded into images using VAE. Each row corresponds to one class (0–9).

Panel of Figure 5 — Figure 5: Row-normalized confusion matrices for the global VAE–GSOM model across datasets. Each row shows the prediction distribution for a given true class.

t $\backslash$ j	0	1	2	3	4	5	6	7	8	9
0	1	–	–	–	–	–	–	–	–	–
1	0.823	0.781	–	–	–	–	–	–	–	–
2	0.692	0.724	0.703	–	–	–	–	–	–	–
3	0.641	0.701	0.553	0.582	–	–	–	–	–	–
4	0.613	0.693	0.427	0.521	0.503	–	–	–	–	–
5	0.592	0.674	0.391	0.402	0.481	0.434	–	–	–	–

Appendix

Table 11: Per-task class accuracy matrix

R

for CIFAR-10 across 10 sequential tasks for GSOM with a global VAE (latent dimension

32\times 2\times 2

Task	FWT	BWT	Forgetting
0	–	-0.4960	0.4960
1	-0.6810	-0.1980	0.1980
2	-0.5130	-0.3520	0.3520
3	-0.6860	-0.3290	0.3290
4	-0.4900	-0.0800	0.0800
5	-0.5540	-0.0130	0.0130

Appendix

Table 12: Per-task Forward Transfer (FWT), Backward Transfer (BWT), and Forgetting for CIFAR-10 using GSOM with a VAE trained from scratch.

t $\backslash$ j	0	1	2	3	4	5	6	7	8	9
0	0.937	–	–	–	–	–	–	–	–	–
1	0.781	0.574	–	–	–	–	–	–	–	–
2	0.384	0.356	0.342	–	–	–	–	–	–	–
3	0.342	0.351	0.329	0.318	–	–	–	–	–	–
4	0.311	0.326	0.304	0.317	0.292	–	–	–	–	–
5	0.284	0.302	0.281	0.297	0.278	0.265	–	–	–	–

Appendix

Table 13: Task accuracy matrix

R

for CIFAR-100 across 10 sequential tasks for GSOM with a global VAE (latent dimension

32\times 2\times 2

Task	FWT	BWT	Forgetting
0	–	-0.8310	0.8310
1	-0.1020	-0.4620	0.4620
2	-0.0890	-0.2410	0.2410
3	-0.0820	-0.1200	0.1200
4	-0.0790	-0.1900	0.1900
5	-0.0750	-0.1310	0.1310

Appendix

Table 14: Per-task Forward Transfer (FWT), Backward Transfer (BWT), and Forgetting for CIFAR-100 using GSOM with a global VAE.

Layer	Kernel / Type	Stride / Scale	Activation	Skip	Output Shape	Params
Encoder
Conv2d (Input)	$3\times 3$ / Conv	1	–	No	$32\times 32\times 32$	896
ResDown Block 1	$3\times 3$ / ×2	2 + 1	ELU	Yes	$64\times 16\times 16$	46.3K
ResDown Block 2	$3\times 3$ / ×2	2 + 1	ELU	Yes	$128\times 8\times 8$	184.6K
ResDown Block 3	$3\times 3$ / ×2	2 + 1	ELU	Yes	$256\times 4\times 4$	737.9K
ResDown Block 4	$3\times 3$ / ×2	2 + 1	ELU	Yes	$512\times 2\times 2$	2.36M

Appendix

Table 15: Architecture of the VAE model used for CIFAR-10/100 (input size:

3\times 32\times 32

). ResDown and ResUp denote residual blocks with downsampling and upsampling, respectively. All activations are ELU except for the output layer, which uses Tanh.

Layer	Operation / Details	Activation	Norm	Output Shape
Linear projection	FC $(512\rightarrow 512\cdot 4\cdot 4)$ , reshape	SiLU	GN	$B\times 512\times 4\times 4$
ResBlock	Conv $3{\times}3$ + Conv $3{\times}3$	SiLU	GN	$B\times 512\times 4\times 4$
Upsample Block 1	ConvTranspose2d $4{\times}4$ , stride 2	SiLU	GN	$B\times 256\times 8\times 8$
ResBlock	Conv $3{\times}3$ + Conv $3{\times}3$	SiLU	GN	$B\times 256\times 8\times 8$
Upsample Block 2	ConvTranspose2d $4{\times}4$ , stride 2	SiLU	GN	$B\times 128\times 16\times 16$
ResBlock	Conv $3{\times}3$ + Conv $3{\times}3$	SiLU	GN	$B\times 128\times 16\times 16$

Appendix

Table 16: Decoder architecture used for ResNet-18 + GSOM experiments. The encoder is an ImageNet-pretrained ResNet-18 that outputs a 512-dimensional feature vector. The decoder maps this embedding back to the dataset-native RGB image resolution (

32\times 32

for CIFAR-10/100,

64\times 64

for TinyImageNet,

84\times 84

for MiniImageNet). GN denotes GroupNorm.

Layer	Operation / Details	Blocks	Activation	GN	Output Shape
Linear (embed $\rightarrow$ 4 $\times$ 4)	FC( $D\!\rightarrow\!$ base_ch $\cdot 4\cdot 4$ )	–	–	Yes	$B\times$ 1024 $\times 4\times 4$
ResBlock	Conv $3{\times}3$ (GN+SiLU)	$1{+}1$	SiLU	Yes	$B\times$ 1024 $\times 4\times 4$
Upsample	ConvTranspose2d $4{\times}4$	$\times 2$	–	Yes	$B\times$ 1024 $\times 8\times 8$
ResBlock (ch $\downarrow$ )	Conv $3{\times}3$ (GN+SiLU)	$1{+}1$	SiLU	Yes	$B\times$ 512 $\times 8\times 8$
Upsample	ConvTranspose2d $4{\times}4$	$\times 2$	–	Yes	$B\times$ 512 $\times 16\times 16$
ResBlock (+Attn)	Conv $3{\times}3$ + SelfAttn2d	$1{+}1$	SiLU	Yes	$B\times$ 512 $\times 16\times 16$

Appendix

Table 17: Decoder Architecture used for CLIP+SOM experiments. The encoder is the CLIP ViT-B/32 visual tower (pretrained) and is frozen by default; when finetune_last is enabled, only the last transformer block of ViT-B/32 is unfrozen. The decoder upsamples a single CLIP embedding to the target RGB image in

[-1,1]

. GN denotes GroupNorm(32).

Panel of Figure 6 — Figure 6: Visualization of the GSOM unit vectors after each task(every single task) for MNIST.

Panel of Figure 7 — Figure 7: Visualization of decoded GSOM unit vectors after each task(every class) on CIFAR-10.

Panel of Figure 8 — Figure 8: Visualization of decoded GSOM unit vectors after each task(every 10 classes) on CIFAR-100.

Replay-Based Continual Learning

Replay-based continual learning preserves prior-task knowledge by revisiting stored or generated experience while adapting models to new data.

See all 132 papers

Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning
Jul 25, 2026Tao Zhang, Qixuan Fan, Yiyuan Liang +7Class-Incremental Learning Synthetic-to-Real Domain Adaptation
Huazhong University of Science and Technology, Wuhan, China · State Key Laboratory of Multispectral Information Intelligent Processing Technology, Wuhan, China · University of Science and Technology of China, Anhui, China +1
Dream2Learn: Structured Generative Dreaming for Continual Learning
Mar 2, 2026Salvatore Calcagno, Matteo Pennisi, Federica Proietto Salanitri +4Class-Incremental Learning Self-Training
PeRCeiVe Lab, University of Catania, Italy
CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning
Jun 30, 2026Julien Lefebvre, Stefan Duffner, Mathieu LefortSelf-Supervised Learning Replay-Based Continual Learning
Universite Claude Bernard Lyon 1, CNRS, INSA Lyon, LIRIS, UMR 5205, 69622 Villeurbanne, France · INSA Lyon, CNRS, Universite Claude Bernard Lyon 1, LIRIS, UMR 5205, 69621 Villeurbanne, France · Universite Claude Bernard Lyon 1 / Univ. Rennes, Inria, CNRS, LIRIS UMR 5205 / IRISA UMR 6074, France

Jul 25, 2026J/K move · Enter open · S save

Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning

Tao Zhang, Qixuan Fan, Yiyuan Liang, Yanjie Wang+6

Huazhong University of Science and Technology, Wuhan, China · State Key Laboratory of Multispectral Information Intelligent Processing Technology, Wuhan, China · University of Science and Technology of China, Anhui, China+1

Class-Incremental Learning Synthetic-to-Real Domain Adaptation Replay-Based Continual Learning Incremental Learning Shortcut Learning

Open paper AskPDF arXiv

Class-incremental learning (CIL) requires models to continuously acquire new knowledge while avoiding catastrophic forgetting. While exemplar replay is effective, it raises concerns regarding privacy and storage. Thus, generative replay has emerged as a viable alternative, synthesizing old data using frozen pretrained text-to-image (T2I) models without any extra training. However, we observe that directly mixing synthetic old-class data with real new-class data during incremental training leads to significant performance degradation. This issue stems from a "domain shortcut", where models rely on domain-discriminative features instead of semantic class cues. To address this, we propose DREAM (

\underline{\mathbf{D}}

omain-

\underline{\mathbf{R}}

egularized

\underline{\mathbf{E}}

xemplar-free

\underline{\mathbf{A}}

lignment

\underline{\mathbf{M}}

odel), which uses a training-free generator to synthesize old-class data and eliminates domain shortcut via subspace rectification and orthogonal projection, while reinforcing semantic alignment through real-anchored prototype regularization. Extensive experiments on 4 datasets demonstrate that DREAM outperforms existing exemplar-free CIL methods and achieves state-of-the-art performance. Our source code is available at https://github.com/Light-ZhangTao/DREAM.

Abstract

Replay-Based Continual Learning

Explore similar work

Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning

Dream2Learn: Structured Generative Dreaming for Continual Learning

CLIMB: Centroid-Based Hierarchical Memory for Online Continual Self-Supervised Learning