Continual learning (CL) enables neural networks to adapt to non-stationary data streams by mitigating catastrophic forgetting, offering a path to robust, private, and adaptable edge devices. Yet, porting CL to real-world edge devices remains challenging for two reasons: (i) online operation, where sensor data arrives one sample at a time and the device must adapt on-the-fly, and (ii) the low footprint of edge devices, which makes traditional error backpropagation training prohibitive. Inspired by the brain's wake and sleep cycles, the former for rapid adaptation with minimal resources and the latter for long-term consolidation, we introduce Hypnos, an open-source dual-phase platform for efficient CL at the edge. In the wake phase (e.g., daily operation), Hypnos relies on a low-footprint system-on-chip with a RISC-V microcontroller unit (MCU) targeting efficient inference and supporting lightweight gradient-free prototype updates. In the sleep phase (e.g., night-time charging), where wall-plug energy and larger memory budgets are available, a smart charger performs rehearsal-based gradient updates for consolidation. Deployed on a ZCU104 field-programmable gate array (FPGA) and synthesized in a 28-nm process, Hypnos is benchmarked on the CORe50 NICv2-391 protocol. It sets a new state-of-the-art accuracy of 78.6%-81.6% for edge CL systems, with an on-device latency of 7.79s-10.54s per sample depending on activation precision, and requires 3.8× fewer learning steps and 22× less learning compute than quantized latent replay at higher accuracy and comparable latency.
Figures & tables
Fig. 1: (a) Performance/availability view of continual adaptation at the edge. Top: offline adaptation, where consolidation is offloaded to the cloud and the device is unavailable (shaded) whenever new data must be learned. Middle: fully on-device learning keeps the device available with online adaptation, at the cost of increased catastrophic forgetting. Bottom: proposed hybrid scheme, lightweight on-device learning during daily use (wake) with consolidation on the smart charger (sleep), combining the availability of online methods with offline-like accuracy. (b) Proposed device–charger system realizing hybrid adaptation, combining (i) a low-footprint edge device for online adaptation with (ii) a smart charger augmented with additional compute and memory budgets for offline consolidation.
Fig. 2: Prototype-based network in the latent space. Stars represent the class prototypes μk ; the circle is the embedding zt of an incoming sample. During inference (red), each class is scored by the distance dk between zt and its prototype. On a labeled sample, the matching prototype moves as a running mean via the update of Eq. ( 1 ), represented by the blue line and the moving prototype.
Fig. 3: Latent replay. The network is split at the replay layer : the frozen extractor below the split runs once per sample and its activations are stored in the latent sample buffer. Only the layers above the split remain plastic and are trained by gradient descent, interleaving buffered activations with new ones.
Fig. 4: Product quantization. (a) Encoding: the feature vector is split channel-wise into S sub-vectors of dimension D=C/S . Each sub-vector is then mapped to the nearest of the 2k centroids of its subspace codebook (Algorithm 1 ) and only its k -bit index is stored. (b) Decoding: each stored index selects its centroid from the codebook, a table lookup which reconstructs the sub-vectors. Repeating it over all spatial positions reconstructs the full feature map.
Fig. 5: System dual-phase learning framework, consisting of two phases. (a) Wake phase: a frozen feature extractor generates features that are compressed to be used during sleep (ii), and used for online learning (iii) using the running-mean update of Eq. ( 1 ). The wake phase is handled by an MCU minimally extended with a daily sample buffer and prototypical learning. (b) Sleep phase: the online buffer is moved onto a larger buffer and the samples are decoded (iv) and fed to the same embedder and classifier, now trained using standard gradient-based methods (v). The sleep phase is handled via a smart charger, benefiting from extended memory and power budgets.
Fig. 6: Latent-layer sweep of SIESTA and LR at FP32 precision on the NICv2-391 protocol of CORe50 with a replay buffer of 7000 samples. Top: final accuracy, mean and standard deviation over five runs. Middle: estimated compute over the full protocol, split into inference FLOPs, the forward passes through the frozen layers, and learning FLOPs, the gradient-based updates of the plastic layers, which SIESTA executes during the sleep phase and LR upon every incoming batch. Bottom: size of one latent sample ( C×H×W ) stored in the replay buffer, identical for both algorithms.
Layer 15
Layer 16
Activations
w/o PQ
w/ PQ
w/o PQ
w/ PQ
FP32
83.2±1.1
80.8±1.7
82.6±1.3
80.4±0.9
INT16
83.2±0.7
81.6±2.1
82.5±1.7
81.3±0.7
INT8
79.3±1.0
78.6±1.2
79.9±1.1
77.8±1.4
TABLE I: Accuracy (%) of candidate latent splits on CORe50 NICv2-391 versus activation precision, with and without PQ-compressed replay, as mean ± standard deviation over three runs.
Fig. 7: Full system block diagram. The system consists of an edge device and a smart charger. The edge device contains a RISC-V MCU implemented through the X-HEEP platform with a dedicated matrix co-processor. The accelerator consists of four memories, a 16-lane PE array, and the post-MAC operation blocks. The scratch memory holds the input and output activations of the layer being executed, the centroid and prototype memories hold the PQ codebooks and the class prototypes, and the PQ-code memory stores the encoded samples of the online buffer. The \raisebox{-0.9pt}{1}⃝ PE array is a flexible PE-array that supports 16× INT16, 32× INT8 or 64× INT4 multiplications in parallel with 48-bit wide accumulators. The \raisebox{-0.9pt}{2}⃝ post-op blocks , Act, Argmax, and writeback, are used depending on the instruction sent to the accelerator and are responsible for applying an activation function, determining the closest prototype/centroid and whether to write back the results or not, respectively.
Fig. 8: Wake-phase clock cycles of the backbone per operation class. (a) Clock cycles breakdown on the MCU baseline (scalar core only) into point-wise (PW) convolutions, depth-wise (DW) convolutions, squeeze-and-excitation (SE) layers, the per-channel requantization of the PW expansion outputs (Requant), and the remaining operations (Other). (b) PW and DW cycles for the MCU baseline, the MCU with the Quadrilatero matrix co-processor, the accelerator executing the per-channel requantization, and the accelerator additionally executing the DW convolutions, respectively. Requantization is counted within PW. The SE layers and the remaining operations are not shown in (b): they shrink from 0.35 G cycles on the MCU baseline to 0.11 G with the co-processor and the accelerator, for a total of 2.92 G cycles per sample in the rightmost configuration. The early backbone is executed depth-first [ 52 ] in order to minimize the activation footprint and fit within the memory budget available in Hypnos.
Fig. 9: Accelerator datapath. \raisebox{-0.9pt}{1}⃝ One of the 16 PE lanes: the activation word Act in and weight word W in are read as INT16, INT8, or INT4 operands according to Op, optionally doubled by a one-bit left shift ( ≪ 1) for the distance mode, and accumulated into a 48-bit register preloaded through Bias in . \raisebox{-0.9pt}{2}⃝ The shared post-MAC unit: multiply by M , shift by n with a selectable rounding mode, add zy , optional ReLU, hard-sigmoid, or hard-swish (through the second multiplier), and saturation to the output width.
Block a
LUTs
FFs
DSPs
LUTRAM b
BRAM b
URAM b
(design relative %)
MCU baseline
22,573
16,132
5
169.0 kB
756.0 kB
3.4 MB
26.0%
29.1%
2.2%
40.2%
54.5%
100.0%
Matrix co-proc.
23,049
10,320
192
–
–
–
26.5%
18.6%
83.8%
Accelerator
16,334
8,877
32
225.0 kB
612.0 kB
–
TABLE II: Post-implementation FPGA resource utilization of the Hypnos-A8 edge device on the ZCU104, broken down per block.
Hypnos A8
Hypnos A16
QLR-CL l=19
QLR-CL l=27
Backbone
MobileNetV3
MobileNetV3
MobileNetV1
MobileNetV1
Latent split (layer)
15
15
19
27
Weight precision
INT8 a
INT8 a
INT8 b
INT8 b
Activation precision
INT8
INT16
INT8 b
INT8 b
Replay buffer precision
PQ (8-bit)
PQ (8-bit)
INT7
INT7
Learning OPs/step c
0.343 G
0.343 G
7.70 G
5 M
TABLE III: Per-sample learning cost of Hypnos and QLR-CL on NICv2-391 . Hypnos is reported in two synthesized variants: A8 , the FPGA-validated design with INT8 activations, and A16 , with a larger scratch memory that fits INT16 activations.
Several edge computing platforms, such as autonomous vehicles and smart sensing devices, need to adapt to dynamic environments in real time by learning from new data in the field. Continual learning has emerged as a promising solution for edge training, by incorporating techniques that successfully combine a highly summarized version of previously trained data (to avoid catastrophic forgetting) with recently sensed data. However, as is the case with other ML algorithms, continual learning generates significant data movement between general-purpose CPUs/GPUs and memory, impacting the suitability of continual learning for edge platforms. In-memory computing (IMC; also known as processing-using-memory) can curtail this waste and make continual learning feasible at the edge, but it faces two unique challenges: (1) IMC architectures make use of noisy computation operations that significantly harm training accuracy; and (2) IMC architectures have poor and often incomplete support for resource-efficient training. To address these challenges, we propose CLASP (the Continual Learning Acceleration System Platform), which to our knowledge is the first end-to-end system with IMC acceleration for continual learning. The hardware and software of CLASP are co-designed to support a wide range of continual learning algorithms, through software-visible assembly-level instructions that can be incorporated without constraints into ML-based algorithms. CLASP is designed around a back-end-of-line (BEOL) compatible ECRAM device that we fabricate, which can overcome the challenges of IMC-based training using other emerging memory devices. We show that CLASP with ECRAM approaches the accuracy of in-GPU training, while delivering a speedup of 67x and energy savings of 132x for learning without forgetting and experience replay using MNIST.
Continual learning (CL) on edge devices requires not only high accuracy but also training-time efficiency to support on-device adaptation under strict memory and computational constraints. While prompt-based continual learning (PCL) is parameter-efficient and achieves competitive accuracy, prior work has focused mainly on accuracy or inference-time performance, often overlooking the memory and computational costs of on-device training. In this paper, we propose CPS-Prompt, a critical patch-aware sparse prompting framework that explicitly targets training-time memory usage and computational cost by integrating critical patch sampling (CPS) for task-aware token reduction and decoupled prompt and classifier training (DPCT) to reduce backpropagation overhead. Experiments on three public benchmarks and real edge hardware show that CPS-Prompt improves peak memory, training time, and energy efficiency by about 1.6x over the balanced CODA-Prompt baseline, while maintaining accuracy within 2% of the state-of-the-art C-Prompt on average and remaining competitive with CODA-Prompt in accuracy. The code is available at https://github.com/laymond1/cps-prompt.
Wonseon Lim, Jaesung Lee, Dae-Won Kim
School of Computer Science and Engineering, Chung-Ang University · Department of Artificial Intelligence, Chung-Ang University
Edge AI must adapt to new classes and shifting data within a tight power budget, but online continual learning (learning from each sample before the next arrives) is where conventional processors are least efficient: each small update pays a full dispatch at full power. We present CLP-SNN, a spiking architecture that recasts the Continually Learning Prototypes (CLP) algorithm as event-driven neural dynamics: a self-normalizing local rule replaces CLP's global renormalization, prototype allocation becomes novelty-triggered neurogenesis, and correctness-modulated plasticity becomes reward-gated per-neuron metaplasticity. Deployed on Intel Loihi 2 as the first online continual learner on that chip, rehearsal-free CLP-SNN matches Replay and SLDA on OpenLORIS at 1-shot; at 25-shot, it stays within 0.8 percentage points of Replay and 6.2 points below covariance-based SLDA. Each prediction-and-update step on extracted features completes on-chip in 0.33 ms at 0.05 mJ, 7.3x lower latency and 341x lower energy than Replay on a Jetson Orin Nano.
Elvin Hajizada, Danielle Rager, Timothy Shea +5
Institute of Informatics, University of Munich (LMU), Munich, Germany · Intel Labs, Intel Corporation, Santa Clara, CA, USA · Southwest Research Institute, San Antonio, TX, USA +3