TrainDeeploy: Hardware-Accelerated Parameter-Efficient Fine-Tuning of Small Transformer Models at the Extreme Edge

Run Wang; Victor J. B. Jung; Philip Wiese; Francesco Conti; Alessio Burrello; Luca Benini

cs.ARMar 10, 2026

TrainDeeploy: Hardware-Accelerated Parameter-Efficient Fine-Tuning of Small Transformer Models at the Extreme Edge

Authors: Run Wang, Victor J. B. Jung, Philip Wiese, Francesco Conti, Alessio Burrello, Luca Benini

Organizations: Integrated Systems Laboratory (IIS), ETH Zurich, Switzerland · Department of Electrical, Electronic and Information Engineering (DEI), University of Bologna, Italy · Department of Control and Computer Engineering (DAUIN), Politecnico di Torino, Italy

Abstract

On-device tuning of deep neural networks enables long-term adaptation at the edge while preserving data privacy. However, the high computational and memory demands of backpropagation pose significant challenges for ultra-low-power, memory-constrained extreme-edge devices. These challenges are further amplified for attention-based models due to their architectural complexity and computational scale. We present TrainDeeploy, a framework that unifies efficient inference and on-device training on heterogeneous ultra-low-power System-on-Chips (SoCs). TrainDeeploy provides the first complete on-device training pipeline for extreme-edge SoCs supporting both Convolutional Neural Networks (CNNs) and Transformer models, together with multiple training strategies such as selective layer-wise fine-tuning and Low-Rank Adaptation (LoRA). On a RISC-V-based heterogeneous SoC, we demonstrate the first end-to-end on-device fine-tuning of a Compact Convolutional Transformer (CCT), achieving up to 11 trained images per second. We show that LoRA reduces dynamic memory usage by 23%, decreases the number of trainable parameters and gradients by 15x, and reduces memory transfer volume by 1.6x compared to full backpropagation. TrainDeeploy achieves up to 4.6 FLOP/cycle on CCT (0.28M parameters, 71-126M FLOPs) and up to 13.4 FLOP/cycle on Deep-AE (0.27M parameters, 0.8M FLOPs), while expanding the scope of prior frameworks to support both CNN and Transformer models with parameter-efficient tuning on extreme-edge platforms.

Figures & tables

Fig. 1: Low-Rank Adaptation ( LoRA ). (a) Frozen pre-trained weight $W_{0}$ with trainable low-rank matrices $A$ and $B$ . (b) Comparison of memory footprint over time between full-parameter fine-tuning and LoRA . The stacked areas illustrate how tensors are allocated and released during execution. By reducing gradient storage, LoRA further lowers the peak memory footprint, enabling training within tight on-chip memory budgets.

Fig. 2: Overview of the TrainDeeploy framework. (a) Models are defined in PyTorch and exported through ONNX . (b) The training engine augments the forward graph with the backward graph via autograd, producing a full training graph. (c) Deeploy extends its inference compiler to training with a frontend ( ONNX parsing), a midend (memory optimizer integrating tiling and static allocation across full forward-backward graph), and a backend (C code generation). (d) The generated code is deployed on the heterogeneous SoC leveraging hierarchical memory mapping ( L1 – L3 ) and an on-board accelerator for GEMMs .

Fig. 3: Illustration of fine-tuning strategies evaluated on the CCT-2 model for on-device training. The convolutional tokenizer ( Conv ) is frozen in all strategies, while different subsets of transformer encoder blocks ( Attn ) and the classifier head are adapted. Five representative strategies are considered: LP (linear probing), only the classifier head is trained; FT-1 , full fine-tuning of the last attention block; LoRA-1 , low-rank adaptation (LoRA, rank

r=4

) applied to the last attention block; FT-2 , full fine-tuning of the last two attention blocks; and LoRA-2 , LoRA (rank

r=4

) applied to the last two attention blocks.

Fig. 4: Hardware setup of the PULP-based SoC modeled in GVSoC. The system consists of a host and an 8-core compute cluster with shared multi-banked L1 TCDM (128 KB), a hierarchical memory with L2 SRAM (2 MB) and external L3 HyperRAM (32 MB), and the RedMulE floating-point GEMM accelerator integrated with direct low-latency access to L1.

Strategy	Training Components	LoRA	Accuracy (50-Shot Transfer)		Cost
Strategy	Training Components	LoRA	CIFAR-10 $\rightarrow$ MNIST	CIFAR-10 $\rightarrow$ EuroSAT	FLOPs (M)	Trained Param(MB)
Full FT	Entire model	✗	92.83 $\pm$ 0.91	64.85 $\pm$ 3.11	201	1.12
LP	Classifier head	✗	88.34 $\pm$ 0.42	76.70 $\pm$ 0.55	71	0.005
FT-1	Last attention block	✗	93.54 $\pm$ 0.38	78.94 $\pm$ 0.44	96	0.38
LoRA-1	Last attention block	✓	95.38 $\pm$ 0.29	77.00 $\pm$ 0.47	86	0.026
FT-2	Last 2 attention blocks	✗	94.62 $\pm$ 0.33	81.52 $\pm$ 0.36	126	0.76

TABLE I: Comparison of fine-tuning strategies on CCT-2 and cost metrics. Accuracy is reported for 50-shot transfer (mean

\pm

std over 30 runs). For reference, training from scratch yields 99.7% on MNIST and 94.0% on EuroSAT.

Fig. 5: End-to-end training latency across fine-tuning strategies. For each strategy, the left bar shows runtime using 8 cores without RedMulE acceleration, while the right bar shows runtime with RedMulE acceleration. In the accelerated LoRA-2 and FT-2 configurations, the measured latency corresponds to a peak throughput of up to 11 gradient updates per second under single-sample, end-to-end fine-tuning.

Fig. 6: (a) Dynamic peak L3 memory usage during forward and backward passes, including activations and gradients but excluding the fixed 1.12 MB weights and input. For each configuration, the reported peak memory corresponds to the maximum memory footprint over its training graph lifetime. (b) Aggregate off-chip data movement (L3 $\leftrightarrow$ L2), reported as the total transferred bytes across five strategies.

System	Hardware	Model	Memory Layout 2	Params	FW+BW FLOP 1	FLOP/cyc	Training Memory 2
PULP-TrainLib [ 9 ]	PULP-SoC	Deep-AE / DS-CNN	64 KB	270K / 52.5K	0.8M / 7.6M	5.6	64 KB
POET [ 19 ]	NRF-52840	ResNet-18	256 KB + 32 GB 3	11M	4.5G	N/A	$<$ 256 KB + N/A 4
MiniLearn [ 10 ]	NRF-52840	CNNs	256 KB + 1 MB	79K / 47K / 12K	1.9M / 2.9M / 0.11M 5	0.59 / 1.52 / 0.09	196 KB + 439 KB
TTE [ 8 ]	STM32F7	MCUNet	320 KB + 1 MB	0.48M	$<$ 69M 6	$<$ 0.43	173 KB
Ours	PULP-SoC	CCT	128 KB + 32 MB	0.28M	71M–126M	4.6	128 KB + 2.5 MB
Ours	PULP-SoC	Deep-AE	128 KB + 32 MB	270K	0.8M	13.4	128 KB

TABLE II: Comparison with related ultra–low-power on-device training work.

On-Device Training

On-device training develops methods and systems for adapting or learning models directly on phones, sensors, and edge hardware.

See all 38 papers

Empowering On-Device Model Adaptation with an Edge AI Inference Accelerator
Jul 20, 2026Mateusz Piechocki, Alessandro Capotondi, Marek KraftOn-Device Training Edge ML
Poznan University of Technology, Institute of Robotics and Machine Intelligence, Poznan, Poland · University of Modena and Reggio Emilia, The High-Performance Real-Time Laboratory, Modena, Italy
MemFLoRA: Memory-Floor LoRA for CNN Adaptation at the Edge
Oct 6, 2026Mehmet Emre Akbulut, Johannes Geier, Ulf SchlichtmannOn-Device Training Memory-Efficient Fine-Tuning
Technical University of Munich Munich, Germany
EdgeDiT: Hardware-Aware Diffusion Transformers for Efficient On-Device Image Generation
Mar 30, 2026Sravanth Kodavanti, Manjunath Arveti, Sowmya Vajrala +2Diffusion Transformer Diffusion-Based Image Generation
Samsung Research Institute Bangalore, India

Jul 20, 2026J/K move · Enter open · S save

Empowering On-Device Model Adaptation with an Edge AI Inference Accelerator

Mateusz Piechocki, Alessandro Capotondi, Marek Kraft

Poznan University of Technology, Institute of Robotics and Machine Intelligence, Poznan, Poland · University of Modena and Reggio Emilia, The High-Performance Real-Time Laboratory, Modena, Italy

On-Device Training Edge ML

Open paper AskPDF arXiv

On-device model adaptation is essential to enable lifelong personalization on resource-constrained hardware, but compute, power, and memory limitations of such devices make end-to-end backpropagation impractical for modern deep neural networks. This work proposes a heterogeneous adaptation pipeline that repurposes a commercial edge AI inference accelerator, Hailo-8L, for frozen-backbone feature extraction during on-device training. The computational graph is partitioned so that the pre-trained backbone is quantized to INT8 and run on the accelerator, while only a lightweight FP32 classification head is fine-tuned on the host CPU, enabling frequent, energy-efficient in-field updates with most weights remaining fixed. Across multiple architectures and datasets, this pipeline achieves up to 15.4x faster wall-clock training time compared to a Raspberry Pi 5 CPU baseline, offers competitive throughput in favorable settings, and consistently reduces energy per sample. Post-training quantization restoration is shown to be crucial for preserving the quality of accelerator-generated features and mitigating accuracy loss in quantization-sensitive architectures. Overall, the results demonstrate a practical approach to efficient on-device adaptation using inference-oriented edge accelerators. The implementation is available at https://github.com/MatPiech/accelerator-training.

Abstract

On-Device Training

Explore similar work

Empowering On-Device Model Adaptation with an Edge AI Inference Accelerator

MemFLoRA: Memory-Floor LoRA for CNN Adaptation at the Edge

EdgeDiT: Hardware-Aware Diffusion Transformers for Efficient On-Device Image Generation