Organizations: Department of Computer Science and Engineering, Amrita School of Computing, Amrita Vishwa Vidyapeetham, Amritapuri, Kerala, India · Amrita School of Agricultural Sciences, Amrita Vishwa Vidyapeetham, Coimbatore, 642109, Tamil Nadu, India · Intel Corporation, Santa Clara, California, USA · Department of Computer Science, Birla Institute of Technology and Science, Pilani Dubai Campus, Dubai, United Arab Emirates · Agrotechnology Division, CSIR–Institute of Himalayan Bioresource Technology, Palampur, 176061, India
Chilli (Capsicum annuum) is one of India's most economically significant crops, yet its productivity is persistently threatened by diseases that are difficult to identify without expert intervention. While Vision Transformers (ViTs) have achieved high classification accuracy, their large computational footprint makes deployment on resource constrained devices challenging. Existing compression approaches typically address pruning, quantization, and knowledge distillation in isolation, leaving the potential benefits and interactions of their combined application insufficiently explored. We propose a unified Vision Transformer compression framework that combines Hessian-Balanced Adaptive Block Pruning (H-BAC), guided by second-order sensitivity estimation, with quantization and attention-based knowledge distillation. To systematically identify the most effective configuration within each compression family, each technique is first evaluated independently through controlled ablation studies, after which the best-performing components are integrated into a sequential deployment pipeline tailored to real-world agricultural constraints. On a chilli 3-class village-split dataset with a genuine cross-village, cross-device out-of-distribution test split, the resulting compressed models match or exceed the 95.13% FP32 baseline's accuracy, alongside 74-98% model size reduction, and the fully integrated compression pipeline achieves a 54.5x size reduction (327.42 MB to 6.01 MB) at 95.13 +/- 2.32% accuracy across four tested configurations. A direct comparison further reveals that, on this dataset, a directly-trained student of the same final size, without pruning or distillation, reaches comparable accuracy of 94.87%, at the same 6.01 MB INT8 size, indicating where H-BAC and knowledge distillation are, and are not yet shown to be, worth their computational cost.
Figures & tables
Figure 1: Indigenously collected chilli leaf images captured in field conditions.
Figure 2: Per-class image counts across the four village-partitioned splits (log scale).
Metric
Value
OOD Accuracy
95.13%
Precision (weighted)
95.55%
Recall (weighted)
95.13%
F1-Score (weighted)
95.10%
Model Size
327.42 MB
Parameters
85.80M
Table 1: Baseline ViT-B/16 Performance and Deployment Characteristics
Figure 3: Workflow of a size/accuracy-budget-driven compression workflow: compress, check against target constraints, escalate if not met. Section 4 details the concrete 4-stage cascade used by the constrained search mode and its results.
Table 3: H-BAC 50% Pruning – Deployment Characteristics, uncompacted mask-based model (CPU: Apple M4 Pro via Core ML; GPU: NVIDIA RTX 4060 via PyTorch)
Stage
Accuracy (%)
Size (MB)
Active Params (M)
CPU (ms)
GPU (ms)
FP32 Baseline †
95.13
327.42
85.80
7.18
5.560
H-BAC Pruned (pre-finetune)
64.87
327.42
43.70
7.18
5.560
H-BAC Pruned (finetuned)
95.53
327.42
43.70
7.18
5.560
+ Attention-Based KD
93.68
21.15
5.52
1.02
0.660
+ PTQ-Dynamic (deployed)
91.97
6.01
5.52
1.02
–
Direct-training alternative ∗
94.87
6.01
5.52
1.02
—
Table 4: Integrated Compression Pipeline: stage-by-stage results (H-BAC 50%, OOD split; CPU: Apple M4 Pro via Core ML; GPU: NVIDIA RTX 4060 via PyTorch)
Figure 4: OOD accuracy (%) of H-BAC pruned ViT-B/16 at each global pruning ratio, before and after one epoch of recovery fine-tuning. Both curves are measured.
Pruning ratio (%)
Active params (M)
FLOPs (G)
Acc. pre-FT (%)
Acc. post-FT (%)
CPU (ms)
GPU (ms)
0 (baseline)
85.80
35.13
95.13
93.95
7.13
5.560
10
77.14
31.57
94.87
96.45
6.76
5.034
20
68.56
28.04
89.34
96.32
6.13
4.554
30
60.34
24.68
73.03
94.61
5.50
4.027
40
51.72
21.13
68.68
97.37
4.88
3.494
50
43.70
17.86
64.87
95.53
4.27
3.003
Table 5: Ablation of H-BAC pruning ratio. FLOPs and active parameters (mask-based model) decrease with pruning ratio; CPU/GPU latency measured on a structurally compacted copy (CPU: Apple M4 Pro via Core ML, 3 trials; GPU: NVIDIA RTX 4060 via PyTorch, 10 trials).
Method
Size (MB)
Acc. (%)
Δ Acc. (%)
Latency (ms)
Speedup
FP32 Baseline
327.42
95.13
—
7.18
1.00 ×
PTQ-Dynamic
84.42
94.34
− 0.79
7.21 a
1.00 ×
PTQ-Static
85.78
94.61
− 0.52
7.21 a
1.00 ×
Table 6: Ablation of quantization methods (CPU: Apple M4 Pro via Core ML, 3 trials). Both methods achieve a ∼ 74% model size reduction; the difference is in accuracy retention.
Method
Size (MB)
Compression Ratio
Acc. (%)
CPU Speedup b
Teacher (ViT-B/16)
327.42
1.00×
95.13
1.00×
TinyViT (no KD) a
21.15
15.48×
96.58±0.26
7.07 ×
Response KD
21.15
15.48×
96.27±0.65
7.07 ×
Feature KD
21.15
15.48×
96.18±0.60
7.07×
Attention KD
21.15
15.48×
96.71±1.03
7.07×
Table 7: KD Ablation Study: Model Size, Performance, and Speedup Comparison (CPU: Apple M4 Pro via Core ML, 3 trials), mean ± std over 3 runs
Params
No-KD (%)
Attn-KD (%)
Advantage (pp)
1.31 M
77.24±13.75
70.26±19.15
−6.97±32.59
1.89 M
70.57±19.35
76.80±15.86
+6.23±34.25
2.63 M
77.15±6.01
81.32±8.92
+4.17±5.41
3.16 M
90.35±0.88
82.11±8.07
−8.25±8.06
Table 8: Capacity sweep: Attention-Based KD vs. no-KD direct training (both taken through PTQ-Dynamic quantization, applied directly with no extra fine-tuning), four from-scratch (non-pretrained) student sizes below TinyViT’s 5.52 M parameters, mean ± std over 3 runs
Size budget
Stage reached
Family
Acc. (%)
Size (MB)
10–20 MB
9
1.89M-param student (FP32, no-KD)
94.08
7.27
25–30 MB
3
KD student (FP32)
93.68
21.15
Table 9: Constrained search: full 10-target grid (5 size budgets × 2 accuracy-drop budgets). All 10 combinations were satisfied.
Stage
Family
Acc. (%)
Size (MB)
Result
3
KD student (FP32)
93.68
21.15
size fails
4
KD student + INT8
91.97
6.01
drop fails
5
3.16M student (FP32)
86.32
12.11
drop fails
6
3.16M student + INT8
86.45
3.64
drop fails
7
2.63M student (FP32)
83.42
10.08
drop fails
8
2.63M student + INT8
83.29
3.08
drop fails
Table 10: Escalation trace for the tightest target in Table 9 (10 MB, 2% accuracy-drop budget), stages 3 through 9. Stages 1–2 (H-BAC pruning, with or without quantization) already fail every budget in this grid on size alone and are omitted; stages 5–8 fit the size budget but fail the accuracy budget.
Model
Mean Latency (ms)
Std. Dev. (ms)
CPU
GPU
CPU
GPU
ResNet50
2.320
1.712
0.010
0.0001
EfficientFormer
1.386
1.031
0.010
0.0002
Our Model
1.015
0.660
0.002
0.0003
Table 11: Comparison of our optimized model to existing models (CPU: Apple M4 Pro via Core ML, 3 trials; GPU: NVIDIA RTX 4060 via PyTorch, 10 trials)
Research Work
Dataset
Acc. (%)
Architecture
Current Study (Ours)
Chilli 3-Class, Village-Split Field Images (OOD split)
95.13
ViT-B/16
Deep Learning-Based Disease Detection Model in Plants
PlantVillage
99.69
GoogLeNet
Multiclass Plant Disease Detection via Dense CNNs
PlantVillage
99.25
DenseNet201
Table 12: Comparison of Existing Research on Plant Disease Detection with the Current Study
Configuration
Size (MB)
Reduction (%)
Accuracy (%)
CPU Latency (ms)
GPU Latency (ms)
ViT-B/16 (FP32)
327.42
—
95.13
7.18
5.560
H-BAC (50% pruning)
327.42
49.1 ∗
95.53
4.27 †
3.003 †
PTQ-Dynamic
84.42
74.2
94.34
7.21
—
Attention KD
21.15
93.5
96.71±1.03
1.02
0.660
Integrated Pipeline
6.01
98.2
95.13±2.32 ‡
1.02 §
—
Table 13: Summary of the best configuration from each compression family, plus the fully integrated pipeline (last row). H-BAC reduces effective computation (FLOPs) without reducing stored model size, while quantization and KD reduce both model size and inference latency. CPU: Apple M4 Pro via Core ML; GPU: NVIDIA RTX 4060 via PyTorch.
Automated leaf disease classification is critical for early disease detection in resource-constrained field environments. Vision Transformers (ViTs) provide strong representation capability by modeling long-range dependencies and inter-class relationships; however, their high computational cost makes them impractical for deployment on edge devices. As a result, existing approaches struggle to effectively transfer these rich representations to lightweight models. This paper introduces AgriKD, a cross-architecture knowledge distillation framework for efficient edge deployment, which transfers knowledge from a Vision Transformer (ViT) teacher to a compact convolutional student model. To bridge the representational gap between Transformer and CNN architectures, the proposed approach integrates multiple distillation objectives at the output, feature, and relational levels, where each objective captures a different aspect of the teacher knowledge. This enables the student model to better preserve and utilize transformer-derived global representations. Experiments on multiple leaf disease datasets show that the distilled student achieves performance comparable to the teacher while significantly improving efficiency, reducing model parameters by approximately 172 times, computational cost by 47.57 times, and inference latency by 18-22 times. Furthermore, the optimized model is deployed across multiple runtime formats, including ONNX, TFLite Float16, and TensorRT FP16, achieving consistent predictive performance with negligible accuracy degradation. Real-world deployment on NVIDIA Jetson edge devices and a mobile application demonstrates reliable real-time inference, highlighting the practicality of AgriKD for AI-powered agricultural applications in resource-constrained environments.
Minh-Dung Le, Minh-Duc Hoang, Hoang-Vu Truong +1
AIT laboratory, Faculty of Artificial Intelligence, FPT University, Da Nang, 550000, Viet Nam
Ternary Vision Transformers offer substantial model compression, however state-of-the-art methods only ternarize the encoder layers, leaving patch embeddings, LayerNorm parameters, and classifier heads in full precision. In compact models targeting resource-constrained processors, such as microcontrollers, these remaining full-precision components determine the total memory footprint, severely limiting deployment efficiency and on-device feasibility. In this work, we introduce a fully ternarized Vision Transformer in which all weight matrices and normalization parameters are ternarized (FTerViT). To this end, we introduce two novel operators : TernaryBitConv2d with per-channel scaling for patch embedding and TernaryLayerNorm. FTerViT is trained using knowledge distillation, followed by a lightweight quantization-aware recovery phase. Our ternary W2A8 DeiT-III-S at 384×384 resolution achieves 82.43% ImageNet-1K top-1 at 6.09,MB (∼15× compression, −2.42,pp vs.\ FP32), outperforming prior ternary ViTs methods up to 8 pp. Finally, we demonstrate the first implementation of ternary vision transformers on a dual cores XTensa LX7 microcontroller inside the ESP32-S3 system-on-chip. By deploying FTerViT-Small (based on DeiT-III-Small at 224×224 resolution, 5.81,MB), we achieve 79.64% ImageNet-1K top-1 accuracy.
Szymon Ruciński, Pietro Bonazzi, Engin Türetken +3
CSEM, Neuchâtel, Switzerland · ETH Zürich, Zurich, Switzerland
While prior studies have successfully compressed vision Transformers (ViTs) through various pruning techniques, most have concentrated on width pruning to achieve significant reductions in model size. Depth pruning, which removes entire layers from a ViT, is notoriously difficult for accuracy recovery despite its potential to deliver higher speedups, limiting the acceleration achieved by existing joint width-and-depth pruning methods. In this work, we reveal that the failure of existing depth pruning methods lies in their neglect of heterogeneity between different layers, and we introduce HetDPT, a heterogeneity-aware depth pruning method that avoids dimension mismatch. Comprehensive experiments on ImageNet-1K, CIFAR-100, COCO, and ADE20K validate our method: HetDPT achieves a 1.58× speedup for DeiT-B while maintaining accuracy and a 1.39× speedup for DeiT-S with nearly no accuracy degradation. Furthermore, when combined with width pruning, HetDPT+ sets a new state-of-the-art record in extreme ViT pruning, enhancing the acceleration ratio from 4.24× to 5.19× for the Isomorphic-Pruning-2.6G configuration while maintaining near-lossless accuracy; our code is available at https://github.com/Efficient-AI-for-All/HetDPT.
Zhenfeng Su, Kang Zhao, Han Bao +4
School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · Huawei Technologies Co., Ltd., Shenzhen, China · Tsinghua University, Beijing, China +1