cs.LGOct 3, 2026

Learning to Remember: Distilling Memory Retention for Compact Recurrent Neural Networks

Authors: Nilushika Udayangania, Kishor Nandakishora, Marimuthu Palaniswami

Organizations: Department of Electrical and Electronic Engineering, University of Melbourne, Parkville, 3052, VIC, Australia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Deep learning models, particularly recurrent neural networks and their variants, such as long short-term memory, have significantly advanced time series analysis. These models capture complex, sequential patterns in time series, enabling real-time assessments. However, their high computational complexity and large model sizes pose challenges for deployment in resource-constrained environments, such as wearable devices and edge computing platforms. Knowledge Distillation (KD) offers a solution by transferring knowledge from a large, complex model (teacher) to a smaller, more efficient model (student), thereby retaining high performance while reducing computational demands. Current KD methods, originally designed for computer vision tasks, neglect the unique temporal dependencies and memory retention characteristics of time series models. To bridge this gap, we propose a novel KD framework termed Memory-Discrepancy Knowledge Distillation (MemKD). MemKD leverages a specialized loss function to capture memory retention discrepancies between the teacher and student models across subsequences within time series data, ensuring that the student model effectively mimics the teacher's behaviour. This approach facilitates the development of compact, high-performing recurrent neural networks suitable for real-time, time series analysis tasks. We provide additional experiments, in-depth theoretical analysis, and insights into the proposed framework across extended time series benchmarks. Our experiments demonstrate that MemKD significantly outperforms state-of-the-art KD methods. Additionally, we demonstrate that it can match the teacher model's performance across a wide range of compression levels, achieving notable reductions in parameter count and memory usage without a significant loss in accuracy.

Figures & tables

Figure 1: An overview of the proposed knowledge distillation framework. The student network learns target task by minimizing the classification loss while mimicking the memory retention of the teacher network using intermediate memory-maps.
Figure 2: An overview illustration of a vanilla RNN. The figure depicts the basic structure and flow of information in an RNN that consists of an input layer, hidden layer and an output layer
ModelLayer1 (hidden size)Layer2 (hidden size)Layer3 (hidden size)Total param.Model size(MB)Compression ratio
SL1_44--4880.00050.0006
SL1_88--14800.00140.0018
SL1_1616--50000.00480.0062
SL2_161616-137040.01310.0169
SL1_3232--181840.01730.0224
SL2_323232-519760.04960.064
Table 1: Configurations of teacher and student networks
ScaleDatasetTrainTestClassLengthType
ShortItalyPowerDemand671029224Sensor
SyntheticControl300300660Simulated
SonyAIBORobotSurface227953265Sensor
SonyAIBORobotSurface120601270Sensor
MiddlePhalanxOutlineCorrect600291280Image
MiddlePhalanxOutlineAgeGroup400154380Image
Table 2: Summary of dataset characteristics. Datasets are categorized as short ( L<150L<150 ), medium ( 150≤L≤500150\leq L\leq 500 ), and long ( L>500L>500 ) according to their original time-series length LL . Challenging datasets with a large number of output classes are highlighted in bold.
Table 3: Performance comparison of our method with baseline methods across 56 UCR datasets.
Figure 3: AUC-PRC plot showing how the student-MemKD is performing better than baseline model
DatasetTeacherBaseBase -KDDKDAtt.RKD -DRKD -ARKD -DAFitnetDT2WOurs
Computers59.5857.166463.7863.7262.6664.1262.7658.962.1763.22
UWaveGestureLib.All91.7277.1579.0780.5879.3178.3478.9977.5181.9681.3585.88
Strawberry87.9170.6873.0671.1389.0990.0591.5392.0887.159.5490.48
BeetleFly81.6481.1786.1584.1688.0690.9893.5790.0379.7876.9991.5
wafer81.1998.6898.5598.8398.9898.798.8198.7298.9589.3599.03
CBF72.4469.175.2390.0776.689.7496.299399.1951.2993.87
Table 4: Comparison of results with state-of-the-art methods.
ModelWinAvg. rankAvg. AUC-PRC
 TeacherBaseOursTeacherBaseOursTeacherBaseOurs
SL1_4133121.862.571.5773.1961.5269.34
SL1_890191.862.821.3273.1965.574.78
SL1_16702122.751.2573.1967.6676.19
SL2_1681192.112.571.3273.1970.6276.43
SL1_32102162.072.51.4373.1970.6576.15
Table 5: Performance comparison of the teacher, student-base, and student-MemKD models across various student architecture settings on 28 datasets.
ComponenetsMetrics
MemKD- LongMemKD- ShortAvg. AUC-PRCAvg. AUC-ROCAvg. Acc.WinsTieLoseAvg. Rank
  65.578.0162.6700283.96
 x72.2483.2267.460222.43
x 74.0484.2468.670212.07
xx74.7884.8970.06150131.54
Table 6: Ablation study on MemKD loss terms
Num. of subseq.Avg. AUC-PRCAvg. AUC-ROCAvg. ACC
5073.5283.7169.37
10074.0484.2468.6
15073.8384.1868.87
20074.2184.4669.38
3007484.5969
Table 7: Sensitivity analysis of the number of chosen subsequences on MemKD-Long
Figure 4: Sensitivity analysis on β\beta for six different datasets. The AUC-PRC values are shown for different β\beta values ( α=1\alpha=1 ).
Figure 5: Feature space representations of the test set for the UWaveGestureLibraryAll dataset using t-SNE. The last five hidden states from all models were included as candidate features. SL2_16 models were used as the student models.
More in this topic

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 814 papers

Explore similar work

CardsList
Jul 7, 2026

Enhancing deep learning models for time series classification via knowledge distillation

Deep learning has achieved remarkable success in various domains including time series analysis, computer vision and natural language processing. However, high computational and memory demands of state-of-the-art architectures pose challenges for deployment in resource-limited environments. Knowledge Distillation (KD) addresses this by transferring knowledge from a large teacher model to a smaller, more efficient student model while maintaining competitive performance. In this work, we investigate the effectiveness of KD for Time Series Classification (TSC) across three architectures: the classical Fully Convolutional Network (FCN), the convolutional Inception model and the transformer-based ConvTran model. We evaluate our approach on UCR Archive, the largest benchmark repository of time series datasets, by modifying architectural components such as convolutional filters, Inception modules and attention heads across the three architectures. Our results consistently show that KD most effectively benefits student models of intermediate complexity across all three architectures, with the distilled FCN student reducing parameters by a factor of 38, the distilled Inception student achieving nearly the same performance as the teacher with 42% fewer parameters and the distilled ConvTran student with 2 attention heads showing the most significant improvement through distillation. To encourage further research and reproducibility, we provide our implementation at https://github.com/MSD-IRIMAS/KD-4-TSC.

Javidan Abdullayev, Maxime Devanne, Jonathan Weber +1

IRIMAS, University of Haute Alsace, Frères Lumière, Mulhouse 68200, France · Faculty of IT, Monash University, Wellington Road, Melbourne 3800, Australia

Time Series ClassificationKnowledge DistillationNeural Network Compression
AskPDF
Sep 5, 2026

Memory in Deep Time-Series Models

Deep learning for time series has progressed through successive architectural paradigms, from recurrent networks and transformers to structured state-space models, retrieval-augmented predictors, foundation models, and tool-using agents. These developments are typically studied in isolation, organized by architecture or modeling era. We argue that they can instead be viewed through a common question of how does a time-series model retain and access information beyond its immediate input? This question is motivated by a fundamental limitation of conventional time-series modeling: information relevant to a prediction may lie far beyond a feasible input window, while compressing history into a fixed-size state can discard information that may become useful later. We formulate this challenge as a memory problem and organize existing time-series methods along a spectrum from internal memory, encoded in parameters and fixed-size states, to external memory that is addressable, retrievable, and increasingly maintained by agents. We then develop a unified taxonomy of memory mechanisms and review three classes of external memory, including explicit modules, retrieval augmentation, and agentic stores, under a common framework for what is retained, how it is written and accessed, and how it persists. A cross-cutting analysis maps these mechanisms to time-series tasks and identifies gaps in both methods and evaluation. We conclude by outlining open problems in building memory systems that can selectively retain, retrieve, revise, and forget information as temporal environments evolve. The result is a framework for studying memory as a first-class dimension of time series modeling, independent of the underlying backbone.

Minh Hoang Nguyen, Huu Hiep Nguyen, Manh Nguyen +3

Deakin Applied AI Initiative, Deakin University, Geelong, VIC, Australia

Memory-Augmented Neural NetworksSequence ModelingRetrieval-Augmented Time Series Forecasting
AskPDF
Apr 19, 2026

SPaRSe-TIME: Saliency-Projected Low-Rank Temporal Modeling for Efficient and Interpretable Time Series Prediction

Time series forecasting is traditionally dominated by sequence-based architectures such as recurrent neural networks and attention mechanisms, which process all time steps uniformly and often incur substantial computational cost. However, real-world temporal signals typically exhibit heterogeneous structure, where informative patterns are sparsely distributed and interspersed with redundant observations. This work introduces SPaRSe-TIME, a structured and computationally efficient framework that models time series through a decomposition into three complementary components: saliency, memory, and trend. The proposed approach reformulates temporal modeling as a projection onto informative subspaces, where saliency acts as a data-dependent sparsification operator, memory captures dominant low-rank temporal patterns, and trend encodes low-frequency dynamics. These components are integrated through a lightweight, adaptive mapping that enables simplified, selective, and interpretable temporal reasoning. Extensive experiments on diverse real-world datasets demonstrate that SPaRSe-TIME achieves competitive predictive performance compared to recurrent and attention-based architectures, while significantly reducing computational complexity. The model is particularly effective in structured time series with clear temporal components and provides explicit interpretability through component-wise contributions. Furthermore, analysis reveals both the strengths and limitations of decomposition-based modeling, highlighting challenges in highly stochastic and complex multivariate settings. Overall, SPaRSe-TIME offers a principled alternative to monolithic sequence models, bridging efficiency, interpretability, and performance, and providing a scalable framework for time series learning.

K. A. Shahriar

Department of Electrical and Electronic Engineering Bangladesh University of Engineering and Technology Dhaka-1205, Bangladesh

Time Series ForecastingTime Series Decomposition
AskPDF