Dream2Learn: Structured Generative Dreaming for Continual Learning

Salvatore Calcagno; Matteo Pennisi; Federica Proietto Salanitri; Amelia Sorrenti; Simone Palazzo; Concetto Spampinato; Giovanni Bellitto

cs.LGMar 2, 2026

Dream2Learn: Structured Generative Dreaming for Continual Learning

Authors: Salvatore Calcagno, Matteo Pennisi, Federica Proietto Salanitri, Amelia Sorrenti, Simone Palazzo, Concetto Spampinato, Giovanni Bellitto

Organizations: PeRCeiVe Lab, University of Catania, Italy

Abstract

Continual learning requires balancing plasticity and stability while mitigating catastrophic forgetting. Inspired by human dreaming as a mechanism for internal simulation and knowledge restructuring, we introduce Dream2Learn (D2L), a framework in which a model autonomously generates structured synthetic experiences from its own internal representations and uses them for self-improvement. Rather than reconstructing past data as in generative replay, D2L enables a classifier to create novel, semantically distinct dreamed classes that are coherent with its learned knowledge yet do not correspond to previously observed data. These dreamed samples are produced by conditioning a frozen diffusion model through soft prompt optimization driven by the classifier itself. The generated data are not used to replace memory, but to expand and reorganize the representation space, effectively allowing the network to self-train on internally synthesized concepts. By integrating dreamed classes into continual training, D2L proactively structures latent features to support forward knowledge transfer and adaptation to future tasks. This prospective self-training mechanism mirrors the role of sleep in consolidating and reorganizing memory, turning internal simulations into a tool for improved generalization. Experiments on Mini-ImageNet, FG-ImageNet, and ImageNet-R demonstrate that D2L consistently outperforms strong rehearsal-based baselines and achieves positive forward transfer, confirming its ability to enhance adaptability through internally generated training signals.

Figures & tables

Panel of Figure 1 — Figure 1: Dreamed classes generated by D2L . Examples of dreamed classes synthesized from their corresponding real classes (left). These samples emerge as semantically distinct yet structurally coherent representations in the generator’s latent space, forming intermediate concepts that enhance the continual classifier’s generalization to future tasks.

Figure 2: Overview of Dream2Learn. (1) During CL training, a deep neural network (DNN) learns from real sensory images (the current task distribution plus the buffer) and dreamed samples produced by a latent diffusion model (LDM). (2) The dreaming optimization process refines the LDM prompts, with an Oracle Network providing a stopping criterion that prevents collapse. (3) Prompts generate auxiliary classes: dreamed samples are not buffered, but rather enrich the representation space with coherent latent clusters that foster knowledge reuse and adaptation.

Figure 3: Visualization of the dreaming optimization process in the latent space of a LDM. Given a real sample $\mathbf{x}$ , the optimization refines the soft prompt $\mathbf{p}_{\text{soft},c}$ to steer the diffusion model towards generating a dreamed counterpart that aligns with the target class $c$ (e.g., a green mamba in this example). The dreaming process explores latent regions where images are visually similar yet distinct from target classes, forming novel intermediate classes (violet zones).

Panel of Figure 4 — Figure 4: Examples of dreaming optimization trajectories showing collapse . From left to right, the images depict different stages of the optimization process. Each row illustrates the evolution of three example images throughout the same prompt optimization. Initially, the generated samples maintain meaningful variations. However, as optimization progresses, they become increasingly similar, reducing diversity and leading to less effective representations.

	Mini-ImageNet		FG-ImageNet		ImageNet-R
	2000	5000	2000	5000	2000	5000
ER	27.91 $\pm$ 3.49	34.21 $\pm$ 3.04	21.08 $\pm$ 2.38	22.21 $\pm$ 3.44	7.68 $\pm$ 0.97	10.69 $\pm$ 1.29
$\hookrightarrow$ D2L	31.18 $\pm$ 2.74	39.75 $\pm$ 2.61	23.53 $\pm$ 1.98	32.73 $\pm$ 3.39	8.67 $\pm$ 0.66	11.84 $\pm$ 0.95
DER++	14.74 $\pm$ 2.14	26.92 $\pm$ 4.72	14.43 $\pm$ 3.68	23.86 $\pm$ 2.54	6.08 $\pm$ 0.81	8.29 $\pm$ 1.15
$\hookrightarrow$ D2L	21.06 $\pm$ 5.45	31.91 $\pm$ 5.19	18.86 $\pm$ 3.22	25.38 $\pm$ 2.17	8.60 $\pm$ 1.00	10.89 $\pm$ 1.56
ER-ACE	33.26 $\pm$ 3.51	40.59 $\pm$ 1.20	24.79 $\pm$ 5.02	30.16 $\pm$ 4.97	7.09 $\pm$ 0.59	9.44 $\pm$ 0.70

Table 1: Class-incremental final average accuracy (FAA) of rehearsal-based methods , with and without dreaming, for buffer sizes 2000 and 5000.

	Mini-ImageNet		FG-ImageNet		ImageNet-R
	2000	5000	2000	5000	2000	5000
ER	-2.58	-1.62	-1.88	-1.52	-1.32	-0.64
$\hookrightarrow$ D2L	+0.33	+0.47	+1.79	+1.19	+0.24	+0.14
DER++	-1.55	-2.00	-1.36	-2.48	-1.03	-1.83
$\hookrightarrow$ D2L	+0.97	+0.71	-0.13	+1.86	+0.30	+0.24
ER-ACE	-1.99	-2.45	-2.00	-2.16	-2.46	-1.27

Table 2: Forward Transfer (FWT) of rehearsal-based methods , with and without dreaming, for buffer sizes 2000 and 5000.

Method	Mini-ImageNet		FG-ImageNet		ImageNet-R
Fine-tune	6.72 $\pm$ 1.20		6.98 $\pm$ 0.10		4.46 $\pm$ 0.15
	Buffer-based methods
	2000	5000	2000	5000	2000	5000
GSS	6.40 $\pm$ 0.38	5.71 $\pm$ 0.08	8.07 $\pm$ 0.26	9.23 $\pm$ 0.85	5.08 $\pm$ 0.13	4.29 $\pm$ 0.32
A-GEM	6.78 $\pm$ 1.13	7.45 $\pm$ 0.76	6.20 $\pm$ 1.11	6.11 $\pm$ 1.13	4.69 $\pm$ 0.03	6.29 $\pm$ 0.84
RPC	9.22 $\pm$ 0.30	9.02 $\pm$ 0.24	8.13 $\pm$ 0.11	7.41 $\pm$ 0.74	5.71 $\pm$ 0.03	6.32 $\pm$ 0.80

Table 3: Comparison with state-of-the-art methods, in terms of class-incremental final average accuracy (FAA) , for buffer sizes 2000 and 5000.

	Buffer size
Method	2000	5000
ER-ACE	33.26 $\pm$ 3.51	40.59 $\pm$ 1.20
$+$ Fixed optim.	38.94 $\pm$ 0.97	46.41 $\pm$ 0.55
$+$ Oracle (D2L)	40.90 $\pm$ 0.95	47.32 $\pm$ 0.89

Table 4: Ablation on the oracle. Results on Mini-ImageNet comparing our method with Fixed optimization.

	Buffer size
Dreaming	2000	5000
No dreams	33.36 $\pm$ 3.51	40.59 $\pm$ 1.20
At beginning	36.93 $\pm$ 2.09	41.59 $\pm$ 2.54
Incremental	36.85 $\pm$ 1.16	43.87 $\pm$ 2.90
D2L	40.90 $\pm$ 0.95	47.32 $\pm$ 0.89

Table 5: Impact of dream class updates. Comparison on Mini-ImageNet of different dreaming strategies.

	Buffer size
Method	2000	5000
ER-ACE	33.26 $\pm$ 3.51	40.59 $\pm$ 1.20
$+$ Mixup	36.84 $\pm$ 0.94	44.82 $\pm$ 1.27
$+$ Continual Mixup	36.05 $\pm$ 1.22	43.45 $\pm$ 0.83
$+$ Textual Mixup	31.89 $\pm$ 0.50	–
$+$ Synth Mixup	36.99 $\pm$ 0.26	–

Table 6: Comparison of alterative dream generation strategies. Evaluation on Mini-ImageNet comparing interpolation-based baselines and WSCL (auxiliary-data surrogate against our proposed D2L.

Method	Diffusion Usage	Training cost	Inference cost	Avg. Acc. ( $\uparrow$ )
ER-ACE	None	$1\times$	$1\times$	33.26 $\pm$ 3.51
+ Mixup	None	$1\times$	$1\times$	36.84 $\pm$ 0.94
+ Diffusion (Augmentation + Rehearsal)	Frozen (samples only)	$\sim 7\times$	$1\times$	36.69 $\pm$ 0.42
DDGR (ResNet)	Generator trained (Replay)	$\sim 489\times$	$1\times$	43.46 $\pm$ 0.25
D2L (Ours)	Frozen + Dream classes	$\sim 17\times$	$1\times$	40.90 $\pm$ 0.95

Table 7: Relative training cost per task normalized to ER-ACE (

1\times

) on SeqMiniImageNet. ER-ACE serves as the reference cost model. We report training cost under different diffusion usage regimes (none, augmentation/rehearsal with frozen generator, generator-trained replay, and dreaming). Inference cost remains identical across methods (

1\times

). A detailed cost decomposition is provided in Appendix Appendix B. Computational Cost Estimation .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Method	Train Cost Breakdown	TFLOPs Breakdown	Total TFLOPs	Rel. Train Cost
ER-ACE	$\mathcal{C}_{\text{CL}}$	$\mathcal{C}_{\text{CL}}{=}273.61$	$273.61$	$1\times$
$+$ Mixup	$\mathcal{C}_{\text{CL}}+\epsilon$	$\mathcal{C}_{\text{CL}}{=}273.61,\;\epsilon{\approx}0.0$	$273.61$	$1\times$
$+$ Diffusion	$\mathcal{C}_{\text{CL}}+\mathcal{C}_{\text{synth}}$	$\mathcal{C}_{\text{CL}}{=}547.21,\;\mathcal{C}_{\text{synth}}{=}1373.21$	$1920.42$	$\sim 7\times$
DDGR (ResNet)	$\mathcal{C}_{\text{CL}}+\mathcal{C}^{\mathcal{G}}_{\text{train}}+\mathcal{C}_{\text{synth}}$	$\mathcal{C}_{\text{CL}}{=}273.61,\;\mathcal{C}^{\mathcal{G}}_{\text{train}}{=}132473.48,\;\mathcal{C}_{\text{synth}}{=}1149.94$	$133897.03$	$\sim 489\times$
D2L (Ours)	$\mathcal{C}_{\text{CL}}+\mathcal{C}_{\text{CL}}^{\prime}+\mathcal{C}_{\text{opt}}+\mathcal{C}_{\text{synth}}$	$\mathcal{C}_{\text{CL}}{=}273.61,\;\mathcal{C}_{CL^{\prime}}{=}2462.45,\;\mathcal{C}_{\text{opt}}{=}551.11,\;\mathcal{C}_{\text{synth}}{=}1377.77$	$4664.93$	$\sim 17\times$

Appendix

Table 8: Computational Analysis and Paradigm Comparison. We report the Relative Training Cost normalized to the standard ER-ACE baseline (

1\times

). We also report a per-task TFLOPs Breakdown and the corresponding Total TFLOPs .

Class-Incremental Learning

Class-incremental learning studies how models can add new categories over time while retaining the ability to recognize previously learned classes.

See all 160 papers

Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories
Jun 2, 2026Ali Behrouz, Farnoosh Hashemi, Adel Javanmard +1Continual Learning for LLMs Persistent Memory for Language Models
Google
Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning
Jul 25, 2026Tao Zhang, Qixuan Fan, Yiyuan Liang +7Class-Incremental Learning Synthetic-to-Real Domain Adaptation
Huazhong University of Science and Technology, Wuhan, China · State Key Laboratory of Multispectral Information Intelligent Processing Technology, Wuhan, China · University of Science and Technology of China, Anhui, China +1
Dynamic Mixture of Latent Memories for Self-Evolving Agents
May 21, 2026Dianzhi Yu, Vireo Zhang, Hongru Wang +7Continual Learning for LLMs Mixture-of-Experts Language Models
The Chinese University of Hong Kong · Nanyang Technological University · University of Edinburgh +2

Jun 2, 2026J/K move · Enter open · S save

Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

Ali Behrouz, Farnoosh Hashemi, Adel Javanmard, Vahab Mirrokni

Google

Continual Learning for LLMs Persistent Memory for Language Models Replay-Based Continual Learning Language Model Self-Improvement Knowledge Distillation

Open paper AskPDF arXiv

The past few decades have witnessed significant advances in the design of machine learning algorithms, from early studies on task-specific shallow models to more general deep Large Language Models (LLMs). Despite showing promising results in tasks that require instant prediction or in-context learning, existing models lack the ability to continually learn and effectively transfer their temporal in-context knowledge to their long-term parameters. Inspired by human learning process, we introduce a ''Sleep'' paradigm that allows the models to continually learn, distill their short-term fragile memories into stable long-term knowledge with replay, and recursively improve themselves with ''Dreaming'' process. In more detail, sleep consists of two stages: (1) Memory Consolidation: an upward distillation process, called Knowledge Seeding, where the memories of a smaller-self are distilled into a larger network to provide more capacity while preserving the knowledge. As a proof of concept, we present a new Generalized Distillation process for {Knowledge Seeding} (i.e., the combination of on-policy distillation with Reinforcement Learning (RL)-based imitation learning); (2) Dreaming: a self-improvement phase, where the model uses RL to generate a curriculum of synthetic data to rehearse new knowledge and refine existing capabilities without human supervision. Our experiments on long-horizon, continual learning, knowledge incorporation, and few-shot generalization tasks support the importance of the sleep stage.

Abstract

Class-Incremental Learning

Explore similar work

Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories

Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning

Dynamic Mixture of Latent Memories for Self-Evolving Agents