TrainDeeploy: Hardware-Accelerated Parameter-Efficient Fine-Tuning of Small Transformer Models at the Extreme Edge
Authors: Run Wang, Victor J. B. Jung, Philip Wiese, Francesco Conti, Alessio Burrello, Luca Benini
Organizations: Integrated Systems Laboratory (IIS), ETH Zurich, Switzerland · Department of Electrical, Electronic and Information Engineering (DEI), University of Bologna, Italy · Department of Control and Computer Engineering (DAUIN), Politecnico di Torino, Italy
On-device tuning of deep neural networks enables long-term adaptation at the edge while preserving data privacy. However, the high computational and memory demands of backpropagation pose significant challenges for ultra-low-power, memory-constrained extreme-edge devices. These challenges are further amplified for attention-based models due to their architectural complexity and computational scale. We present TrainDeeploy, a framework that unifies efficient inference and on-device training on heterogeneous ultra-low-power System-on-Chips (SoCs). TrainDeeploy provides the first complete on-device training pipeline for extreme-edge SoCs supporting both Convolutional Neural Networks (CNNs) and Transformer models, together with multiple training strategies such as selective layer-wise fine-tuning and Low-Rank Adaptation (LoRA). On a RISC-V-based heterogeneous SoC, we demonstrate the first end-to-end on-device fine-tuning of a Compact Convolutional Transformer (CCT), achieving up to 11 trained images per second. We show that LoRA reduces dynamic memory usage by 23%, decreases the number of trainable parameters and gradients by 15x, and reduces memory transfer volume by 1.6x compared to full backpropagation. TrainDeeploy achieves up to 4.6 FLOP/cycle on CCT (0.28M parameters, 71-126M FLOPs) and up to 13.4 FLOP/cycle on Deep-AE (0.27M parameters, 0.8M FLOPs), while expanding the scope of prior frameworks to support both CNN and Transformer models with parameter-efficient tuning on extreme-edge platforms.
Figures & tables
Fig. 1: Low-Rank Adaptation ( LoRA ). (a) Frozen pre-trained weight W0 with trainable low-rank matrices A and B . (b) Comparison of memory footprint over time between full-parameter fine-tuning and LoRA . The stacked areas illustrate how tensors are allocated and released during execution. By reducing gradient storage, LoRA further lowers the peak memory footprint, enabling training within tight on-chip memory budgets.
Fig. 2: Overview of the TrainDeeploy framework. (a) Models are defined in PyTorch and exported through ONNX . (b) The training engine augments the forward graph with the backward graph via autograd, producing a full training graph. (c) Deeploy extends its inference compiler to training with a frontend ( ONNX parsing), a midend (memory optimizer integrating tiling and static allocation across full forward-backward graph), and a backend (C code generation). (d) The generated code is deployed on the heterogeneous SoC leveraging hierarchical memory mapping ( L1 – L3 ) and an on-board accelerator for GEMMs .
Fig. 3: Illustration of fine-tuning strategies evaluated on the CCT-2 model for on-device training. The convolutional tokenizer ( Conv ) is frozen in all strategies, while different subsets of transformer encoder blocks ( Attn ) and the classifier head are adapted. Five representative strategies are considered: LP (linear probing), only the classifier head is trained; FT-1 , full fine-tuning of the last attention block; LoRA-1 , low-rank adaptation (LoRA, rank r=4 ) applied to the last attention block; FT-2 , full fine-tuning of the last two attention blocks; and LoRA-2 , LoRA (rank r=4 ) applied to the last two attention blocks.
Fig. 4: Hardware setup of the PULP-based SoC modeled in GVSoC. The system consists of a host and an 8-core compute cluster with shared multi-banked L1 TCDM (128 KB), a hierarchical memory with L2 SRAM (2 MB) and external L3 HyperRAM (32 MB), and the RedMulE floating-point GEMM accelerator integrated with direct low-latency access to L1.
Strategy
Training Components
LoRA
Accuracy (50-Shot Transfer)
Cost
CIFAR-10 → MNIST
CIFAR-10 → EuroSAT
FLOPs (M)
Trained Param(MB)
Full FT
Entire model
✗
92.83 ± 0.91
64.85 ± 3.11
201
1.12
LP
Classifier head
✗
88.34 ± 0.42
76.70 ± 0.55
71
0.005
FT-1
Last attention block
✗
93.54 ± 0.38
78.94 ± 0.44
96
0.38
LoRA-1
Last attention block
✓
95.38 ± 0.29
77.00 ± 0.47
86
0.026
FT-2
Last 2 attention blocks
✗
94.62 ± 0.33
81.52 ± 0.36
126
0.76
TABLE I: Comparison of fine-tuning strategies on CCT-2 and cost metrics. Accuracy is reported for 50-shot transfer (mean ± std over 30 runs). For reference, training from scratch yields 99.7% on MNIST and 94.0% on EuroSAT.
Fig. 5: End-to-end training latency across fine-tuning strategies. For each strategy, the left bar shows runtime using 8 cores without RedMulE acceleration, while the right bar shows runtime with RedMulE acceleration. In the accelerated LoRA-2 and FT-2 configurations, the measured latency corresponds to a peak throughput of up to 11 gradient updates per second under single-sample, end-to-end fine-tuning.
Fig. 6: (a) Dynamic peak L3 memory usage during forward and backward passes, including activations and gradients but excluding the fixed 1.12 MB weights and input. For each configuration, the reported peak memory corresponds to the maximum memory footprint over its training graph lifetime. (b) Aggregate off-chip data movement (L3 ↔ L2), reported as the total transferred bytes across five strategies.
System
Hardware
Model
Memory Layout 2
Params
FW+BW FLOP 1
FLOP/cyc
Training Memory 2
PULP-TrainLib [ 9 ]
PULP-SoC
Deep-AE / DS-CNN
64 KB
270K / 52.5K
0.8M / 7.6M
5.6
64 KB
POET [ 19 ]
NRF-52840
ResNet-18
256 KB + 32 GB 3
11M
4.5G
N/A
< 256 KB + N/A 4
MiniLearn [ 10 ]
NRF-52840
CNNs
256 KB + 1 MB
79K / 47K / 12K
1.9M / 2.9M / 0.11M 5
0.59 / 1.52 / 0.09
196 KB + 439 KB
TTE [ 8 ]
STM32F7
MCUNet
320 KB + 1 MB
0.48M
< 69M 6
< 0.43
173 KB
Ours
PULP-SoC
CCT
128 KB + 32 MB
0.28M
71M–126M
4.6
128 KB + 2.5 MB
Ours
PULP-SoC
Deep-AE
128 KB + 32 MB
270K
0.8M
13.4
128 KB
TABLE II: Comparison with related ultra–low-power on-device training work.
On-device model adaptation is essential to enable lifelong personalization on resource-constrained hardware, but compute, power, and memory limitations of such devices make end-to-end backpropagation impractical for modern deep neural networks. This work proposes a heterogeneous adaptation pipeline that repurposes a commercial edge AI inference accelerator, Hailo-8L, for frozen-backbone feature extraction during on-device training. The computational graph is partitioned so that the pre-trained backbone is quantized to INT8 and run on the accelerator, while only a lightweight FP32 classification head is fine-tuned on the host CPU, enabling frequent, energy-efficient in-field updates with most weights remaining fixed. Across multiple architectures and datasets, this pipeline achieves up to 15.4x faster wall-clock training time compared to a Raspberry Pi 5 CPU baseline, offers competitive throughput in favorable settings, and consistently reduces energy per sample. Post-training quantization restoration is shown to be crucial for preserving the quality of accelerator-generated features and mitigating accuracy loss in quantization-sensitive architectures. Overall, the results demonstrate a practical approach to efficient on-device adaptation using inference-oriented edge accelerators. The implementation is available at https://github.com/MatPiech/accelerator-training.
Mateusz Piechocki, Alessandro Capotondi, Marek Kraft
Poznan University of Technology, Institute of Robotics and Machine Intelligence, Poznan, Poland · University of Modena and Reggio Emilia, The High-Performance Real-Time Laboratory, Modena, Italy
On-device learning is necessary when the model encounters user-,sensor-, or environment-specific shifts after deployment. Although parameter-efficient fine-tuning (PEFT) methods, particularly Low-Rank Adaptation (LoRA) variants, enable efficient adaptation at the edge, the limiting resource for Convolutional Neural Network (CNN) adaptation is often not the number of trainable parameters but the activation state that must be retained until the backward pass. This paper introduces Memory-Floor LoRA (MemFLoRA), a low-rank CNN adapter built around a memory-first design principle rather than a direct application of transformer-oriented LoRA. Instead of merely reducing trainable weights, we define an activation-memory-floor criterion: trainable backward computations must not depend on full-width layer inputs. The resulting adapter freezes the down-projection, trains a scale-matched up-projection, and combines eval-mode backbone normalization with activation-minimal backward rules, reducing saved state to the low-rank branch. Evaluated on three Human Activity Recognition (HAR) datasets and two CNN backbones under subject, body-location, and sensor-placement shifts, MemFLoRA reduces saved-activation memory by 98.5-98.7% and peak training-state memory by 94.9-97.3% relative to full fine-tuning, while matching or exceeding CNN PEFT baselines.
Mehmet Emre Akbulut, Johannes Geier, Ulf Schlichtmann
Diffusion Transformers (DiT) have established a new state-of-the-art in high-fidelity image synthesis; however, their massive computational complexity and memory requirements hinder local deployment on resource-constrained edge devices. In this paper, we introduce EdgeDiT, a family of hardware-efficient generative transformers specifically engineered for mobile Neural Processing Units (NPUs), such as the Qualcomm Hexagon and Apple Neural Engine (ANE). By leveraging a hardware-aware optimization framework, we systematically identify and prune structural redundancies within the DiT backbone that are particularly taxing for mobile data-flows. Our approach yields a series of lightweight models that achieve a 20-30% reduction in parameters, a 36-46% decrease in FLOPs, and a 1.65-fold reduction in on-device latency without sacrificing the scaling advantages or the expressive capacity of the original transformer architecture. Extensive benchmarking demonstrates that EdgeDiT offers a superior Pareto-optimal trade-off between Frechet Inception Distance (FID) and inference latency compared to both optimized mobile U-Nets and vanilla DiT variants. By enabling responsive, private, and offline generative AI directly on-device, EdgeDiT provides a scalable blueprint for transitioning large-scale foundation models from high-end GPUs to the palm of the user.