DAC-Loop: Distillation-Aware Compression for Remote Sensing CNN

Rico Staedeli; Lorena Raichle; Konstantinos Tzevelekakis; Aron Asefaw; Damian Borth

cs.LGOct 26, 2026

DAC-Loop: Distillation-Aware Compression for Remote Sensing CNN

Authors: Rico Staedeli, Lorena Raichle, Konstantinos Tzevelekakis, Aron Asefaw, Damian Borth

Organizations: AIML Lab, School of Computer Science, University of St. Gallen

Abstract

While large-scale models achieve strong results in remote sensing scene classification, deploying them on edge devices remains difficult due to limited compute, memory, and energy budgets. Compact CNN backbones are therefore a practical choice in many real-world settings, offering efficient inference with well-optimized implementations on embedded hardware. To push this efficiency further, we investigate how far a knowledge-distilled student model can be compressed while preserving high accuracy. We propose a Distillation-Aware Compression Loop (DAC-Loop) that integrates (i) knowledge distillation applied both before and after structured channel pruning, and (ii) an iterative compression schedule that alternates structured pruning with unstructured sparsification. Across EuroSAT and NWPU-RESISC45, the DAC-Loop consistently improves the accuracy-compression trade-off relative to standard one-shot pruning and single-stage KD recovery, maintaining strong accuracy even beyond 90% parameter reduction. These results demonstrate that combining distillation and pruning in a unified iterative loop offers an effective path toward compact remote sensing models.

Figures & tables

Fig. 1: Distillation–Aware Compression Loop (DAC-Loop). A fixed, pretrained teacher network $f_{T}(\cdot;\theta_{T})$ provides supervision for iterative compression of a smaller student . Starting from a randomly initialized student $f_{S}^{\,i=0}$ , Stage 1 performs KD from $f_{T}$ to obtain a distilled student $f_{S}^{\,i+n}$ . The resulting model is then compressed in Stage 2 via structured or unstructured pruning, producing a pruned model $f_{P}^{\,i+1}$ . Since pruning typically degrades accuracy, the pruned network subsequently restores performance using the same KD procedure employed in Stage 1. The distilled-and-recovered student can either be used as the final DAC model or re-enter the loop for additional pruning cycles, enabling up to $N$ iterations. Details in Chapter III-C .

Ablation Study Results for DAC-Loop with a ResNet18
	Model	KD-1	KD-2	N	Compression	Parameters	Prun. Type	Val. Acc. (EuroSAT)	Val. Acc. (RESISC45)
KD	ResNetTiny	✓	–	1	0%	1.0 M	–	96.49 $\pm$ 0.17	82.63 $\pm$ 0.32
	ResNet10mini	✓	–	1	0%	2.0 M	–	97.11 $\pm$ 0.15	86.36 $\pm$ 0.37
	ResNet18	✓	–	1	0%	11.2 M	–	98.61 $\pm$ 0.08	91.95 $\pm$ 0.17
Unstr.	ResNet18	✓	–	1	0%	11.2 M	Unstructured	98.58 $\pm$ 0.06	92.59 $\pm$ 0.02
	ResNet18	–	✓	1	0%	11.2 M	Unstructured	98.41 $\pm$ 0.02	92.27 $\pm$ 0.20

TABLE I: KD-1 & KD-2 both represent Stage 1 Training with KD in our DAC-Loop (Figure 1 ) and indicate whether KD is applied pre (KD-1) or post (KD-2) pruning. N denotes the number of DAC-Loop iterations.

Model	Param	EuroSAT	RESISC45
Baselines - others
ResNet18 [ 12 ]	11.7 M	98.06	92.10
ResNet50 [ 11 ]	24.0 M	99.20	96.80
ResNet50 [ 5 ]	24.0 M	98.57	–
Baselines - ours
ResNetTiny	0.9 M	95.77 $\pm$ 0.06	80.15 $\pm$ 0.54

TABLE II: Baseline validation accuracy

Iteration (N)	Pruning Type	Parameter	Val. Acc.
1	Unstructured	11.2 M	92.18 $\pm$ 0.06
2	Structured	6.0 M	92.41 $\pm$ 0.11
3	Unstructured	6.0 M	92.53 $\pm$ 0.12
4	Structured	3.2 M	92.18 $\pm$ 0.09
5	Unstructured	3.2 M	92.21 $\pm$ 0.08
6	Structured	1.7 M	91.54 $\pm$ 0.14

TABLE III: Pruning schedule parameters and validation accuracy for ResNet18 on NWPU-RESISC45

Fig. 2: Effect of DAC-Loop on the accuracy–compression trade-off : comparison of validation accuracy on NWPU-RESISC45 under increasing compression for ResNet18 with (orange) and without (blue) the proposed DAC-Loop. Across all compression levels, the DAC-Loop maintains consistently higher accuracy than one-shot pruning.

Metric	Full	Compressed	Speedup
Parameters $\downarrow$	11.2 M	0.9 M	$\times 12.44$
FLOPs $\downarrow$	18.05 G	1.62 G	$\times 11.16$
MACs $\downarrow$	9.02 G	0.81 G	$\times 11.16$
Inference (Batch) $\downarrow$	288.25 ms	65.18 ms	$\times 4.42$

TABLE IV: Efficiency comparison of the full and DAC-Loop–compressed ResNet18 models on the RESISC45 validation set.

Neural Network Compression

Neural network compression reduces model size, memory, communication, or computation while preserving useful accuracy.

See all 217 papers

Improved Knowledge Distillation for Land-Use Image Classification
Jun 12, 2026Arundhuti Sur, Abhiroop Chatterjee, Susmita Ghosh +1Cross-Architecture Knowledge Distillation Knowledge Distillation
Jadavpur University, India · Rochester Institute of Technology, USA
EdgeCompress: Coupling Multidimensional Model Compression and Dynamic Inference for EdgeAI
Jul 8, 2026Hao Kong, Di Liu, Shuo Huai +5Edge Inference Convolutional Neural Networks
School of Computer Science and Engineering, Nanyang Technological University (NTU), Singapore · HP-NTU Digital Manufacturing Corporate Lab, Nanyang Technological University (NTU), Singapore · Department of Computer Science, Norwegian University of Science and Technology (NTNU), Trondheim, Norway +1
Efficient Remote Sensing Instance Segmentation with Linear-Time State Space Distilled Visual Foundation Models
Jun 24, 2026Qinzhe Yang, Keyan Chen, Jia Xu +2Remote Sensing Image Segmentation Instance Segmentation
Shen Yuan Honors College, Beihang University, Beijing 100191, China · Department of Aerospace Intelligent Science and Technology, School of Astronautics, and with the State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing 100191, China · Qian Xuesen Laboratory of Space Technology, China Academy of Space Technology, Beijing 100094, China

Jun 12, 2026J/K move · Enter open · S save

Improved Knowledge Distillation for Land-Use Image Classification

Arundhuti Sur, Abhiroop Chatterjee, Susmita Ghosh, Emmett Ientilucci

Jadavpur University, India · Rochester Institute of Technology, USA

Cross-Architecture Knowledge Distillation Knowledge Distillation Neural Network Compression Image Classification

Open paper AskPDF arXiv

In the present article, an improved Knowledge Distillation (KD) framework has been proposed for efficient compression of deep convolutional neural networks for land-use image classification task. Motivated by the need to achieve competitive classification accuracy while reducing computational complexity, a teacher-student learning paradigm is adopted in which a VGG16 network transfers knowledge to a lightweight MobileNetV2 model. The proposed framework integrates hard supervision from ground truth labels with a soft supervision strategy that combines Kullback-Leibler divergence and Cosine Similarity losses. Experiments conducted on three land-use datasets show that the proposed KD-based method yields improved performance, and achieves an accuracy of 99.04%, outperforming both baseline student training and single-loss distillation approaches, while retaining substantial model compression.

Abstract

Neural Network Compression

Explore similar work

Improved Knowledge Distillation for Land-Use Image Classification

EdgeCompress: Coupling Multidimensional Model Compression and Dynamic Inference for EdgeAI

Efficient Remote Sensing Instance Segmentation with Linear-Time State Space Distilled Visual Foundation Models