Efficient Few-Shot Learning for Edge AI via Knowledge Distillation on MobileViT
Authors: Shuhei Tsuyuki, Reda Bensaid, Jérémy Morlier, Mathieu Léonardon, Naoya Onizawa, Vincent Gripon, Takahiro Hanyu
Organizations: Research Institute of Electrical Communication, Tohoku University, Japan · Graduate School of Engineering, Tohoku University, Japan · IMT Atlantique, Lab-STICC, UMR CNRS 6285, F-29238 Brest, France
Efficient and adaptable deep learning models are an important area of deep learning research, driven by the need for highly efficient models on edge devices. Few-shot learning enables the use of deep learning models in low-data regimes, a capability that is highly sought after in real-world applications where collecting large annotated datasets is costly or impractical. This challenge is particularly relevant in edge scenarios, where connectivity may be limited, low-latency responses are required, or energy consumption constraints are critical. We propose and evaluate a pre-training method for the MobileViT backbone designed for edge computing. Specifically, we employ knowledge distillation, which transfers the generalization ability of a large-scale teacher model to a lightweight student model. This method achieves accuracy improvements of 14% and 6.7% for one-shot and five-shot classification, respectively, on the MiniImageNet benchmark, compared to the ResNet12 baseline, while reducing by 69% the number of parameters and by 88% the computational complexity of the model, in FLOPs. Furthermore, we deployed the proposed models on a Jetson Orin Nano platform and measured power consumption directly at the power supply, showing that the dynamic energy consumption is reduced by 37% with a latency of 2.6 ms. These results demonstrate that the proposed method is a promising and practical solution for deploying few-shot learning models on edge AI hardware.
Figures & tables
Figure 1: Few-Shot Learning uses a small support set to train and a query set to test. The number of classes is called the way, and the number of examples per class is the shots.
Figure 2: FSL consists of three steps: backbone learning for feature extraction, learning from a small amount of data using a pre-trained backbone, and evaluating the inference ability for classification tasks.
Figure 3: Overview of the proposed method: MobileViT-XXS is used as the backbone, and knowledge distillation is performed as pre-training. After this, the backbone is not updated, and training and inference ability are evaluated using a small number of samples, as in Figure 2 .
Figure 4: Performance comparison of EfficientNet (B0–B3), MobileNetV3 (S, L), and MobileViT (S, XS, XXS) backbones in the first stage of the EASY pipeline without ensemble features or augmented shots. All models were trained on 84×84 images with batch size 376 for 100 epochs using a learning rate of 0.01 with γ=0.1 . Accuracy after pre-training is reported as a proxy for backbone quality in few-shot learning.
Method
1-shot
5-shot
EASY + ResNet12
70.84 ± 0.19
86.28 ± 0.12
EASY + MobileViT-XXS (ours)
75.08 ± 0.10
88.43 ± 0.08
EASY + KD + MobileViT-XXS (ours)
85.08 ± 0.06
92.96 ± 0.05
Table 1: Few-shot classification accuracy on MiniImageNet for EASY (ResNet12), EASY + MobileViT-XXS, and KD + MobileViT-XXS.
Backbone
Param
FLOPs
ResNet12
3.109M
4.405 G
MobileViT-XXS
0.961M
0.512 G
Table 2: Comparison of the number of parameters and FLOPs of ResNet12 and MobileViT-XXS.
Dataset
EASY + MobileViT-XXS
EASY + KD + MobileViT-XXS
MiniImageNet
75.08 ± 0.10
85.08 ± 0.06
ImageNet
75.05 ± 0.13
82.61 ± 0.08
Caltech101
74.59 ± 0.10
79.03 ± 0.09
OxfordPets
71.04 ± 0.07
84.94 ± 0.14
Flowers102
68.61 ± 0.05
70.82 ± 0.14
Food101
39.18 ± 0.07
47.13 ± 0.10
Table 3: 1-shot few-shot classification accuracy on various datasets for EASY + MobileViT-XXS and KD + MobileViT-XXS.
Dataset
EASY + MobileViT-XXS
EASY + KD + MobileViT-XXS
MiniImageNet
88.43 ± 0.08
92.96 ± 0.05
ImageNet
87.70 ± 0.04
90.80 ± 0.07
Caltech101
89.09 ± 0.08
88.88 ± 0.05
OxfordPets
90.19 ± 0.10
94.07 ± 0.06
Flowers102
87.97 ± 0.06
85.14 ± 0.07
Food101
60.49 ± 0.09
62.77 ± 0.09
Table 4: 5-shot few-shot classification accuracy on various datasets for EASY + MobileViT-XXS and KD + MobileViT-XXS.
Metric
Idle
ResNet
MobileViT-XXS
Power (W)
4.0
19.9
14.0
Latency (ms)
–
3.6
2.6
Throughput (ips)
–
280
392
Energy/inf (J)
–
0.072
0.036
Table 5: Energy and performance metrics on Jetson Orin Nano at 15 W power mode (transposed).
Department of Electro-Optics, National Formosa University, Taiwan · Department of Electrical Engineering, University of Muhammadiyah Malang, Indonesia · Department of Electrical Engineering, National Taipei University, Taiwan +1
Poznan University of Technology, Institute of Robotics and Machine Intelligence, Poznan, Poland · University of Modena and Reggio Emilia, The High-Performance Real-Time Laboratory, Modena, Italy