Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression

Longsheng Zhou; Yu Shen

cs.LGApr 5, 2026

Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression

Authors: Longsheng Zhou, Yu Shen

Organizations: Institute of Advanced Technology University of Science and Technology of China Hefei, China · Supercomputing Center University of Science and Technology of China Hefei, China

Abstract

Modern deployment often requires trading accuracy for efficiency under tight CPU and memory constraints, yet common compression proxies such as parameter count or FLOPs do not reliably predict wall-clock inference time. In particular, unstructured sparsity can reduce model storage while failing to accelerate (and sometimes slightly slowing down) standard CPU execution due to irregular memory access and sparse kernel overhead. Motivated by this gap between compression and acceleration, we study a practical, ordered pipeline that targets measured latency by combining three widely used techniques: unstructured pruning, INT8 quantization-aware training (QAT), and knowledge distillation (KD). Empirically, INT8 QAT provides the dominant runtime benefit, while pruning mainly acts as a capacity-reduction pre-conditioner that improves the robustness of subsequent low-precision optimization; KD, applied last, recovers accuracy within the already constrained sparse INT8 regime without changing the deployment form. We evaluate on CIFAR-10/100 using three backbones (ResNet-18, WRN-28-10, and VGG-16-BN). Across all settings, the ordered pipeline achieves a stronger accuracy-size-latency frontier than any single technique alone, reaching 0.99-1.42 ms CPU latency with competitive accuracy and compact checkpoints. Controlled ordering ablations with a fixed 20/40/40 epoch allocation further confirm that stage order is consequential, with the proposed ordering generally performing best among the tested permutations. Overall, our results provide a simple guideline for edge deployment: evaluate compression choices in the joint accuracy-size-latency space using measured runtime, rather than proxy metrics alone.

Figures & tables

Fig. 1 : Overview of the proposed hybrid compression pipeline. A dense FP32 teacher with weights $\mathbf{W}^{(0)}$ is first pruned into a sparse FP32 student ( $\mathbf{W}^{(p)}=\mathbf{M}\odot\mathbf{W}^{(0)}$ ), then optimized under INT8 quantization (PTQ/QAT) to obtain a sparse INT8 student $\mathbf{W}^{(q)}=Q_{8}(\mathbf{W}^{(p)})$ , and finally refined via knowledge distillation to produce the hybrid student $\mathbf{W}^{(h)}$ in the compressed space.

Method	INT8	Prune	Opt.	KD
Ours (P50 $\to$ INT8 $\to$ KD)	$\checkmark$	Unstr.	Pipeline	$\checkmark$
GETA	$\triangle$	Str.	Joint	$\times$
SQL	$\times$	Unstr.	Constr.	$\times$
QST	$\times$	Unstr.	Coupled	$\times$
ANNC	$\triangle$	Unstr.	Constr.	$\times$

TABLE I : Design philosophy comparison (compact).

Fig. 2 : Synergy of the ordered hybrid method. Left: limitations of single-step pruning-only, quantization-only, and KD-only strategies, each optimizing mainly one axis (accuracy or efficiency). Right: our three-stage ordering, where pruning shrinks the active set, quantization converts sparsity into real efficiency gains, and KD recovers accuracy in the compressed space, moving the model toward the Pareto front in the accuracy–efficiency plane.

(a) Main results
Method	ResNet-18 + CIFAR-10				WRN-28-10 + CIFAR-100				VGG-16-BN + CIFAR-10
Method	Acc. (%) $\uparrow$	Size (MB) $\downarrow$	Lat. (ms) $\downarrow$	Speedup $\uparrow$	Acc. (%) $\uparrow$	Size (MB) $\downarrow$	Lat. (ms) $\downarrow$	Speedup $\uparrow$	Acc. (%) $\uparrow$	Size (MB) $\downarrow$	Lat. (ms) $\downarrow$	Speedup $\uparrow$
Baseline	78.37	42.65	2.45	1.00	76.03	139.38	3.42	1.00	79.38	56.18	2.62	1.00
Prune (30%)	79.84	33.13	2.43	1.01	79.41	97.56	3.43	1.00	80.11	39.33	2.64	0.99
Prune (50%)	80.38	26.97	2.55	0.96	79.07	69.69	3.36	1.02	80.55	28.11	2.81	0.93
QAT (INT8)	77.42	10.66	0.99	2.47	71.29	35.81	1.42	2.41	77.23	14.05	1.01	2.59

TABLE II : Main results (fully trained) across backbones and datasets with controlled stage-order ablation.

Fig. 3 : Comparison of standalone baselines and the ordered hybrid pipeline on ResNet-18/CIFAR-10.

Panel of Figure 6 — Fig. 4 : Global analysis on ResNet-18/CIFAR-10: ROC/PR curves and the 3D accuracy–size–latency trade-off.

Method	Ratio	FT	Acc. (%) $\uparrow$		Efficiency
Method	Ratio	FT	Tr.	Val.	Size (MB) $\downarrow$	Compr. ( $\times$ ) $\uparrow$	Lat. (ms) $\downarrow$	Spdup ( $\times$ ) $\uparrow$
Baseline	–	20	$75.51$	$73.98$	$42.65$	$1.00$	$2.45$	$1.00$
Prune Only	30%	10	81.75	77.83	$33.13$	$1.29$	2.43	1.01
Prune Only	50%	10	81.77	78.35	$26.97$	$1.58$	$2.55$	$0.96$
QAT Only	–	8	$72.42$	$69.63$	10.66	4.00	0.99	2.48
KD Only	–	10	$70.41$	$70.10$	$42.65$	$1.00$	$2.49$	$0.99$

TABLE III : Diagnostic snapshot on ResNet-18/CIFAR-10.

Method	W-bit	Prune	Acc. (%) $\uparrow$	Rel. BOPs $\downarrow$
Baseline	32	None	91.70	100.0
SQL	MP	Unstr.	90.90	6.1
QST-P	MP	Unstr.	91.80	5.0
GETA	MP	Str.	91.42	4.5
QST-Q	MP	Unstr.	91.60	3.3
Ours (P50 $\rightarrow$ INT8QAT $\rightarrow$ KD)	8	Unstr.	91.83	3.1

TABLE IV : Literature-aligned comparison on ResNet-20/CIFAR-10.

Explore similar work

APQF: Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuning
Aug 6, 2026Sadegh Jafari, Mohiuddin Bilwal, Fan Zhou +2Model Compression Mixed-Precision Quantization
Department of Computer Science Iowa State University Ames, IA, USA · Department of Agricultural and Biosystems Engineering Iowa State University Ames, IA, USA
Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment
Aug 16, 2026Subhransu Das, Jiaming Cheng, Arnav Kumar +6Edge AI Model Compression
The Ohio State University, Department of Computer Science and Engineering, Columbus, OH, USA
LLM Compression with Jointly Optimizing Architectural and Quantization choices
Jun 2, 2026Hoang-Loc La, Truong-Thanh Le, Amir Taherkordi +1Mixed-Precision Quantization LLM Quantization
UiT The Arctic University of Norway · University of Oslo, Norway

Aug 6, 2026J/K move · Enter open · S save

APQF: Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuning

Sadegh Jafari, Mohiuddin Bilwal, Fan Zhou, Brian Gelder+1

Department of Computer Science Iowa State University Ames, IA, USA · Department of Agricultural and Biosystems Engineering Iowa State University Ames, IA, USA

Model Compression Mixed-Precision Quantization Quantization-Aware Training Neural Network Pruning

Open paper AskPDF arXiv

Modern deep neural networks achieve strong performance, but their scale makes them costly and slow, especially on resource-constrained edge devices. Pruning and quantization address this, but rely on manual, expert choices and on algorithms that are hard to apply across architectures. Uniform settings also ignore how differently individual layers respond to compression, which costs accuracy. We introduce APQF, an agentic profiling-guided framework that combines structured pruning, mixed-precision quantization-aware training, and accuracy recovery in one automated pipeline. A profiling agent measures how cost is distributed across the model and how sensitive each part is to pruning, and this evidence drives per-layer pruning ratios, per-layer bit-widths, and the recovery strategy, all proposed by LLM planners and validated before execution. To our knowledge, APQF is the first framework to combine LLM-guided, profiling-grounded decisions with a fully training-aware pruning and quantization pipeline for both CNNs and vision transformers. We evaluate APQF on ResNet, VGG7, ViT, DeiT, and Swin using ImageNet-1k and CIFAR-10. On ImageNet it cuts compute to 5.6-7.7 percent of the original bit-operations, a 13-18x reduction, while keeping accuracy close to the baseline, and under a 200K-image budget it stays roughly 17 points higher in Top-1 than existing joint pruning and quantization methods. On CIFAR-10 it compresses further than that method on four of five architectures. On VGG7 it reaches 93.15 percent using only 0.41 percent of baseline bit-operations, the only method at that compression level to improve on its full-precision baseline. Ablations show that uniform compression loses the most accuracy at matched compute, and that withholding profiling data from the planner hurts every model. Six LLM planners, including free open-weight ones, all reach 97.4-97.9 percent on Swin-Tiny.

Abstract

Explore similar work

APQF: Agentic Profiling-Guided Structured Pruning and Mixed-Precision Quantization with Adaptive Fine-Tuning

Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment

LLM Compression with Jointly Optimizing Architectural and Quantization choices