Teacher-Anchored Selection of Post-Training Quantized Models under Domain Shift

Alejandro Rodriguez Dominguez; Muhammad Shahzad; Xia Hong

cs.LGSep 25, 2026

Teacher-Anchored Selection of Post-Training Quantized Models under Domain Shift

Authors: Alejandro Rodriguez Dominguez, Muhammad Shahzad, Xia Hong

Organizations: Department of Computer Science, University of Reading, Reading, United Kingdom · Quantitative Analysis and Artificial Intelligence Department, Miralta Finance Bank S.A., Madrid, Spain · Lecturer, Albert School, Madrid, Spain

Abstract

Compressing a trained model yields a family of deployment candidates, and under domain shift the most compressed one need not be the one to deploy. We study selection over such a family, with candidates and teacher fixed and target labels absent or scarce. Two findings organize the label-free case. Minimum teacher distortion behaves almost as a constant rule, selecting the same eight-bit, per-channel, unclipped configuration in every run, which does not minimize empirical target cross-entropy. Established estimators divide sharply: in the overconfident-collapse regime of the CNN families, confidence-based estimators order the family close to backwards, and the diagnostics that identify it need the labels the setting denies, while output-distribution estimators match the teacher-relative anchor and on one architecture beat it. Distortion is nonetheless stable, so a supervised term can move selection away from it. Combining the two, we give exact quadratic identities for a canonical quadratic analogue of the family. We also show that under symmetric corruption the label-dependent part of a criterion linear in the label indicator is multiplied by one common factor whenever its coefficient sums are candidate-invariant, a class holding teacher contrasts and accuracy but not cross-entropy. These characterize the score's components without bounding selection regret. Across one hundred and thirty-four candidate families, one per independently trained convolutional or Vision Transformer teacher, anchoring reduces mean regret at the smallest label budget in every setting, an advantage that fades beyond twenty-five labels.

Figures & tables

Method	RN50	MNV2	ViT–DN	ViT–C20
Direct validation	0.182 $\pm$ 0.06	0.227 $\pm$ 0.13	0.208 $\pm$ 0.12	0.222 $\pm$ 0.10
CE-combo (ours)	0.166 $\pm$ 0.09	0.103 $\pm$ 0.07	0.139 $\pm$ 0.14	0.115 $\pm$ 0.06
Alignment (ours)	0.168 $\pm$ 0.10	0.088 $\pm$ 0.07	0.205 $\pm$ 0.33	0.150 $\pm$ 0.06
SoftmaxCorr [TMLR’24]	0.212 $\pm$ 0.18	0.061 $\pm$ 0.27	0.224 $\pm$ 0.47	0.083 $\pm$ 0.06
COT [NeurIPS’23]	0.257 $\pm$ 0.21	0.007 $\pm$ 0.01	0.278 $\pm$ 0.66	0.098 $\pm$ 0.05
Nuclear norm [ICML’23]	0.277 $\pm$ 0.20	0.013 $\pm$ 0.03	0.280 $\pm$ 0.66	0.098 $\pm$ 0.05

Table 1: Selection regret, mean

\pm

standard deviation over runs. Columns are the four architecture–dataset settings: ResNet50 (RN50) and MobileNetV2 (MNV2) on DomainNet with

30

classes, and Vision Transformers on DomainNet with

50

classes (ViT–DN) and on held-out-domain CIFAR-20 with

20

classes (ViT–C20). Every supervised selector here uses

n=10

labels , the smallest budget we evaluate, so all columns are comparable; each gets the same ten examples, including those its coefficient selection consumes. The label-free rules use none, so their values hold at any

n

. Boldface marks the lowest mean per column. Row labels give the venue and year; the corresponding references are Hendrycks and Gimpel (2017) for average confidence and entropy, Garg et al. (2022) for ATC, Deng et al. (2023) for nuclear norm, Guillory et al. (2021) for DOC, Tu et al. (2024) for SoftmaxCorr, Lu et al. (2023) for COT and Liu et al. (2023) for distortion. Runs are

45

per CNN architecture over three executions,

17

and

27

on the ViT settings; the other budgets are in Supplement Table S9 .

Panel of Figure 1 — Figure 1: Why the label-free rule needs help, and why confidence cannot supply it. (a) Two candidates on a level set of the teacher distortion, $g_{D}$ and $g_{A}$ , are indistinguishable to the label-free rule; the supervised term breaks the tie, and alignment prefers movement along $e_{y}-p_{f}$ , while an overconcentrated candidate sits near a vertex, confident and far from the teacher. (b, c) Each point is one of the $72$ candidates of a single run, positioned by its mean confidence on the unlabeled pool and its target accuracy, coloured by bit width, with the dashed line at chance. On ResNet50 confidence and accuracy are negatively related across the family ( $\rho=-0.32$ in this run); on ViT–DomainNet the relation is strongly positive ( $\rho=+0.93$ ), which is why the same estimators select well there.

Paired difference, validation CE minus distortion
Setting	Selected by	ECE	Acc.	CE
ResNet50	Distortion	0.216	0.332	2.865
	Validation CE	0.083	0.327	2.745
MobileNetV2	Distortion	0.227	0.308	3.019
	Validation CE	0.195	0.308	2.884
ResNet50		$-.122$	$-.042$	$-.208$

Table 2: Post-hoc calibration audit, computed on the first execution of each CNN setting. Upper block: medians over its

15

runs per architecture, ECE with

15

equal-width bins, accuracy top-

1

. Lower block: mean run-level differences (validation CE minus distortion) with run-bootstrap

95\%

intervals, leading zeros omitted. Test labels are used only for explanation.

Setting	$n$	Dist.	Val-CE	CE-combo	Align
ResNet50–DomainNet	10	0.2323	0.1756	0.1941	0.1870
	25	0.2323	0.0972	0.0948	0.1143
	50	0.2323	0.0571	0.0498	0.0635
	100	0.2323	0.0267	0.0271	0.0342
	150	0.2323	0.0232	0.0239	0.0270
	300	0.2323	0.0000	0.0012	0.0024

Table 3: Median target cross-entropy selection regret with clean labels, over run–labeled-subset outcomes. Dist. is label-free and constant across budgets. Boldface marks the lowest median among the supervised selectors in each row. The largest budget is the largest evaluated in each setting, not the size of its labeled pool. Runs are

45

per CNN architecture, pooling three end-to-end executions, and

17

and

27

on the two ViT settings.

		$\eta=0$			$\eta=0.2$			$\eta=0.4$
Setting	$n$	Val-CE	Val-Acc	CE-combo	Val-CE	Val-Acc	CE-combo	Val-CE	Val-Acc	CE-combo
ResNet50–DomainNet	10	0.168	0.630	0.159	0.326	1.497	0.234	0.393	2.485	0.278
	25	0.100	0.249	0.091	0.145	0.427	0.111	0.361	0.792	0.230
	50	0.051	0.262	0.053	0.118	0.269	0.083	0.317	0.262	0.216
	100	0.025	0.227	0.030	0.104	0.239	0.071	0.288	0.240	0.194
	300	0.008	0.219	0.010	0.087	0.231	0.073	0.232	0.240	0.202

Table 4: Label budget crossed with corruption rate: mean target cross-entropy regret over

15

runs per architecture and ten corruption realizations per cell. Unlike the clean-label tables this grid is computed on the first complete execution of each setting, since crossing five budgets with three corruption rates and ten realizations makes pooling all three executions prohibitively slow. Every selector receives the same

n

labels at the same rate

\eta

, and the labels its coefficient selection consumes are corrupted too. Boldface marks the lowest mean in each cell.

Setting	$\eta$	Predicted	Label comp.	Val. accuracy
ResNet50–DomainNet	10%	0.897	0.881	0.881
	20%	0.793	0.800	0.802
	30%	0.690	0.683	0.687
	40%	0.586	0.577	0.579
MobileNetV2–DomainNet	10%	0.897	0.882	0.882
	20%	0.793	0.803	0.802

Table 5: Empirical validation of the symmetric-corruption attenuation identity at the complete labeled pool, pooling all three executions,

45

runs per architecture and ten corruption realizations per rate. Entries are through-origin slopes relating the corrupted quantity to its clean counterpart. The teacher component is unchanged to numerical precision at every rate.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning	Symbol	Meaning
$f$ , $p_{f}(\cdot\mid x)$	fixed teacher and its predictive distribution	$g\in\mathcal{G}$ , $p_{g}(\cdot\mid x)$	candidate and its predictive distribution
$\mathcal{G}$ , $M$	fixed candidate family and its cardinality	$\mathcal{U}$	unlabeled target input pool
$\mathcal{S}$ , $n$	labeled target sample and its size	$\mathcal{T}$	disjoint target test set, evaluation only
$e_{y}$	one-hot vector for class $y$	$z_{f}(x)$ , $z_{g}(x)$	teacher and candidate outputs in a chosen representation
$\delta_{g}^{p}(x)$	probability change $p_{g}(x)-p_{f}(x)$	$\Phi_{\mathcal{S}}(g)$	supervised term added to the distortion
$D_{\mathcal{V}}^{d}(g,f)$	teacher distance of type $d$ on input set $\mathcal{V}$	$a_{\ell}(x,y;g,f)$	per-example alignment contribution for loss $\ell$

Appendix

Table S1: Notation used in the framework and the supplementary analyses.

Panel of Figure 2 — Figure 2: Levels of the experimental design. A base configuration is a shift–seed pair: $30$ on the two CNN settings and $44$ on the two ViT settings, $74$ in all. Each CNN configuration is executed three times with a retrained teacher on identical splits and each ViT configuration once, so the number of independently trained teachers, and hence of family-runs , is $30\times 3+44\times 1=134$ : $45$ per CNN architecture, $17$ on ViT–DomainNet and $27$ on ViT–CIFAR-20. The family-run is the statistical unit: subset outcomes are averaged within it. For paired inference the three executions of a configuration are averaged as well, leaving $15$ paired units per CNN architecture.

Setting	Cfg.	Ex.	Runs	Pool	Clean	Noise
RN50–DN	15	3	45	$150{+}150$	25 †	10
MNV2–DN	15	3	45	$150{+}150$	25 †	10
ViT–DN	17	1	17	$2526$ – $6308$	25	10
ViT–C20	27	1	27	$666{+}666$	25	10
Total	74	–	134	–	–	–

Appendix

Table S2: Accounting of the experimental design. A run contains one teacher and its fixed

72

-candidate family. Executions are independent end-to-end repetitions with retrained teachers on identical splits. Results for the two CNN settings pool all three executions, giving

45

runs per architecture; the two ViT settings have one execution each. † for

n<300

; the complete pool is evaluated once at

n=300

. On the two ViT settings every clean budget is below the pool size, so all of them use the full

25

repetitions. The counts in this table are those of the budget sweep; the budget-by-rate grid of the main text draws ten repetitions per cell at every rate, including

\eta=0

. The ViT–DomainNet selection pool varies by shift, from

2526

6308

examples; the largest supervised budget remains

n=150

Selector	Score minimized over $g\in\mathcal{G}$	Role
Distortion	$D_{\mathcal{U}}^{\mathrm{KL}}(g,f)$	label-free teacher-relative baseline
Highest precision	none (fixed configuration)	data-free baseline matching the distortion pick
Validation CE	$\widehat{\mathrm{CE}}_{\mathcal{S}}(g)$	direct supervised reference
CE-combo	$D_{\mathcal{U}}^{\mathrm{KL}}(g,f)+\alpha\widehat{\mathrm{CE}}_{\mathcal{S}}(g)$	anchored cross-entropy
Alignment	$D_{\mathcal{U}}^{\mathrm{KL}}(g,f)-\lambda A_{\mathcal{S}}^{\mathrm{B}}(g,f)$	teacher-relative directional selector
Teacher component	$D_{\mathcal{U}}^{\mathrm{KL}}(g,f)-\lambda A_{\mathcal{S}}^{\mathrm{teach}}(g,f)$	mechanism ablation, no observed class

Appendix

Table S3: Selectors and ablations. All supervised entries receive the same labeled sample

\mathcal{S}

of the stated size, including labels consumed by coefficient selection.

Data	Teacher	Cls.	Cfg.	Ex.	Runs	Pool	Max. $n$
DomainNet	ResNet50	30	15	3	45	300	300
DomainNet	MNV2	30	15	3	45	300	300
DomainNet	ViT	50	17	1	17	2.5k–6.3k	150
CIFAR-20	ViT	20	27	1	27	1332	200
Total	–	–	74	–	134	–	–

Appendix

Table S4: Experimental settings. A run contains one fixed teacher and its

72

-candidate family. The selection pool is the union of the quantization-calibration and selection splits; the largest label budget is listed separately because it is not the pool size. Exec. counts independent end-to-end repetitions of the whole pipeline with retrained teachers on identical splits.

		CE-cmb $-$ Val-CE		Align $-$ Val-CE		Align $-$ CE-cmb
Setting	$n$	Diff.	$p$	Diff.	$p$	Diff.	$p$
RN50–DN	10	-0.016	1.000	-0.014	1.000	+0.001	1.000
	25	+0.003	0.776	+0.020	0.469	+0.016	0.077
	50	+0.008	0.393	+0.026	0.112	+0.018	0.063
	100	+0.004	0.485	+0.020	0.167	+0.016	0.029
	150	-0.001	0.899	+0.012	0.324	+0.012	0.324

Appendix

Table S5: Paired run-level inference. Differences are computed after averaging within-run outcomes; negative favors the first method.

p

is the two-sided sign-flip value, obtained by complete enumeration of the sign assignments when the number of paired units is at most

22

and by a

200{,}000

-draw Monte Carlo sign-flip test otherwise, Holm-adjusted across the comparisons shown in each row, which is the family; boldface marks

p<0.05

after adjustment. Upper block: clean labels. Lower block: symmetric corruption.

Estimator	RN50–DN	MNV2–DN
Distortion	0.0032	0.0081
SoftmaxCorr	0.0092	0.0085
Nuclear norm	0.0090	0.0031
DOC	0.0033	0.0076
ATC-MC	0.3211	0.2717
ATC-NE	0.3211	0.2404

Appendix

Table S6: Accuracy regret, the quantity several of the label-free estimators were proposed to rank, over all three executions and

45

runs per architecture. Entries are the gap in target top-

1

accuracy between the best candidate and the one selected. The deployment objective in this paper is cross-entropy, so these estimators are elsewhere evaluated outside their original objective; on their own metric the confidence-based group still fails by two orders of magnitude. The anchored selectors spend labels, so they are outside this label-free comparison.

Setting	$n$	Val-CE	All $\mathcal{U}$	Same $\mathcal{S}$	Indep.	COT anchor
ResNet50–DomainNet	10	0.1801	0.1627	0.1564	0.1629	0.1888
	50	0.0608	0.0677	0.0682	0.0672	0.0757
	300	0.0129	0.0086	0.0086	0.0086	0.0094
MobileNetV2–DomainNet	10	0.2473	0.1065	0.1080	0.1101	0.0847
	50	0.0504	0.0231	0.0227	0.0219	0.0119
	300	0.0010	0.0010	0.0010	0.0010	0.0009

Appendix

Table S7: Where the anchor is evaluated, and which anchor. Mean regret over the first complete execution of each CNN setting,

15

runs per architecture, so the values are not directly comparable with the pooled means of Supplement Table S9 . Columns two to four vary only the input set on which the teacher distortion is averaged: the complete unlabeled pool, the labeled subsample itself, or an independent subsample of the same size. The last column replaces the teacher anchor with the optimal-transport estimator, rescaled so that the coefficient grid remains comparable.

	CE regret at floor			Accuracy regret
Setting	$10^{-6}$	$10^{-8}$	$10^{-12}$	Dist.	Avg. conf.
ResNet50–DomainNet	0.2158	0.2161	0.2161	0.0040	0.3212
MobileNetV2–DomainNet	0.0848	0.0848	0.0848	0.0076	0.3076

Appendix

Table S8: Two sensitivity checks. Left: distortion’s cross-entropy regret under three probability floors, showing that the magnitudes do not depend on clipping. Right: regret in accuracy , the quantity the confidence estimators were designed to rank, for the anchor and for average confidence. The confidence estimator fails on its own metric as well.

Setting	$n$	Dist.	Val-CE	CE-combo	Align
ResNet50–DomainNet	10	0.2198	0.1819	0.1664	0.1678
	25	0.2198	0.1015	0.1047	0.1211
	50	0.2198	0.0619	0.0702	0.0882
	100	0.2198	0.0334	0.0372	0.0532
	150	0.2198	0.0282	0.0276	0.0398
	300	0.2198	0.0132	0.0071	0.0121

Appendix

Table S9: Budget ablation: mean target cross-entropy regret at every clean-label budget evaluated. Distortion is label-free and constant in

n

. Boldface marks the lowest of the three supervised selectors in each row. Anchoring wins at the smallest budget in all four settings and in seven of the remaining nineteen comparisons; its advantage concentrates at ten and twenty-five labels and fades from fifty onward. Runs are

45

per CNN architecture, pooling three executions,

17

on ViT–DomainNet and

27

on ViT–CIFAR-20.

		$95$ th-percentile regret							$\Pr(R>0.1)$
Setting	Selector	10	25	50	100	150	300		10	25	50	100	150	300
RN50–DN	Val-CE	0.58	0.35	0.24	0.14	0.12	0.08	0.50	0.33	0.17	0.07	0.06	0.02
	CE-combo	0.49	0.37	0.33	0.20	0.13	0.03	0.49	0.32	0.19	0.08	0.06	0.00
	Align	0.48	0.41	0.34	0.28	0.18	0.05	0.50	0.36	0.27	0.13	0.09	0.00
MNV2–DN	Val-CE	1.18	0.61	0.13	0.06	0.04	0.00	0.30	0.15	0.06	0.02	0.01	0.00
	CE-combo	0.71	0.20	0.12	0.08	0.06	0.02	0.23	0.13	0.06	0.02	0.02	0.00

Appendix

Table S10: Upper-tail behavior across clean-label budgets, pooling all three end-to-end executions,

45

runs per architecture. Left block:

95

th percentile of target cross-entropy regret pooled over run–subset outcomes rather than over run means, since the tail is a property of the outcome distribution. Right block: the catastrophic-selection rate

\Pr(R>0.1)

. The last column is the complete labeled pool of

300

Setting	Supervision	Median $\lambda$	IQR	$\Pr(\lambda{=}0)$	$\Pr(\lambda{=}\lambda_{\max})$	SD
ResNet50–DomainNet	Clean	7.5	[5.5, 9.0]	0.111	0.156	3.12
	20% noise	9.0	[8.5, 9.5]	0.000	0.244	1.28
	40% noise	8.5	[7.5, 9.0]	0.000	0.067	1.01
MobileNetV2–DomainNet	Clean	3.5	[0.0, 5.5]	0.289	0.022	2.81
	20% noise	4.0	[0.0, 5.5]	0.289	0.000	2.91
	40% noise	10.0	[8.0, 10.0]	0.133	0.533	3.41

Appendix

Table S11: Distribution of the cross-validated alignment coefficient at the complete labeled pool.

\Pr(\lambda{=}0)

is the frequency with which the procedure declines to use the supervised direction and reverts to pure distortion;

\Pr(\lambda{=}\lambda_{\max})

the frequency of selecting the grid boundary. SD is across run medians.

Figure 3: Mean target cross-entropy regret as a function of the alignment weight

\lambda

on the two DomainNet CNN settings.

\lambda=0

recovers distortion-only selection. Mean regret decreases with

\lambda

on both settings, steeply on ResNet50 between roughly

\lambda=4

and

\lambda=8

and more gradually thereafter; the shaded bands are interquartile ranges across runs. Marked points are the medians of the cross-validated coefficient.

Setting	$\eta$	Val-CE	CE-combo	Align
RN50–DN	20%	0.064 (0.078)	0.064 (0.068)	0.065 (0.067)
	40%	0.165 (0.231)	0.127 (0.200)	0.102 (0.180)
MNV2–DN	20%	0.009 (0.181)	0.013 (0.105)	0.001 (0.015)
	40%	0.779 (0.831)	0.706 (0.770)	0.393 (0.399)

Appendix

Table S12: Symmetric label corruption at the complete labeled pool, pooling all three end-to-end executions,

45

runs per architecture: median target cross-entropy regret with mean in parentheses taken over runs after averaging corruption realizations within each run. Boldface marks the lowest median in each row.

Setting	$n$	Dist.	Align	Perm.	Teach.
ResNet50–DomainNet	10	0.2323	0.1870	0.2170	0.1445
	25	0.2323	0.1143	0.2228	0.1101
	50	0.2323	0.0635	0.2276	0.0692
	100	0.2323	0.0342	0.2304	0.0432
	150	0.2323	0.0270	0.2320	0.0360
	300	0.2323	0.0024	0.2307	0.0260

Appendix

Table S13: The permutation control and the two components of the directional term, in median target cross-entropy regret. The control receives the identical score structure, coefficient grid and cross-validation as alignment, and differs only in that the teacher-residual directions

e_{y}-p_{f}(x)

are permuted across the labeled examples before the statistic is formed. Boldface marks the lowest of the three in each row. CNN rows pool all three executions,

45

runs per architecture.

Setting	Budget	Admissible	Excess over free	Internal spread
RN50–DN	$\leq 10\%$	12	0.710	14.41
	$\leq 16\%$	36	0.103	15.02
	$\leq 22\%$	54	0.007	15.12
	none	72	0.000	15.13
MNV2–DN	$\leq 10\%$	12	0.901	10.41
	$\leq 16\%$	36	0.869	10.63

Appendix

Table S14: Selection under a deployment-cost constraint. Cost is the weight memory of a candidate relative to full precision, with layers kept in full precision counted at their true width. For each budget we report how many of the

72

candidates are admissible, its excess over the unconstrained best, and the mean spread between the worst and best admissible candidate. Means over

45

runs per architecture.

		$n{=}10$			$n{=}50$			$n{=}300$
Setting	Budget	Dist.	Val-CE	CE-combo	Dist.	Val-CE	CE-combo	Dist.	Val-CE	CE-combo
ResNet50–DomainNet	$\leq 10\%$	0.095	0.097	0.090	0.079	0.043	0.042	0.075	0.002	0.004
	$\leq 16\%$	0.305	0.168	0.160	0.304	0.040	0.056	0.304	0.014	0.016
	$\leq 22\%$	0.269	0.184	0.176	0.283	0.063	0.078	0.290	0.015	0.009
	none	0.220	0.182	0.164	0.220	0.062	0.070	0.220	0.013	0.007
MobileNetV2–DomainNet	$\leq 10\%$	0.011	0.003	0.005	0.000	0.000	0.000	0.000	0.000	0.000

Appendix

Table S15: Constrained selection. The candidate set, the coefficient search and the oracle are all restricted to

\mathcal{G}_{B}=\{g:C(g)\leq B\}

; the labeled sample is unchanged, so regret is measured against the best admissible candidate. Pooled over all three executions,

45

runs per architecture, lower of the two supervised selectors in boldface.

Setting	Runs	$K$	AvgConf	Entropy	ATC-MC	ATC-NE	$\rho$	$c_{\text{worst}}$	$c_{\text{best}}$	Distinct
ResNet50–DomainNet	45	30	15.062	15.062	15.062	15.062	$-0.38$	1.000	0.408	34%
MobileNetV2–DomainNet	45	30	10.814	10.814	4.886	4.696	$-0.09$	0.985	0.534	100%
ViT–CIFAR-20	27	20	0.124	0.167	0.100	0.099	$+0.86$	0.579	0.726	100%
ViT–DomainNet	17	50	0.062	0.339	0.020	0.020	$+0.88$	0.523	0.761	100%

Appendix

Table S16: Degradation regimes. Median target cross-entropy regret of the four confidence-based estimators, with diagnostics.

\rho

is the median Pearson correlation between test accuracy and mean confidence across the

72

candidates within a run.

c_{\text{worst}}

and

c_{\text{best}}

are the median confidences of the worst and best candidate by test cross-entropy. “Distinct” is the median fraction of distinct confidence values among collapsed candidates (accuracy

\leq 1.5\times

chance). CNN rows pool three end-to-end executions.

	RN50		MNV2		ViT–C20		ViT–DN
Bits	PT	PC	PT	PC	PT	PC	PT	PC
2	99	100	100	100	86	96	91	88
3	36	16	99	100	33	0	31	0
4	33	0	91	90	15	0	17	0
5	0	0	87	39	0	0	0	0
6	0	0	0	0	0	0	0	0

Appendix

Table S17: Collapse rate, the fraction of candidates with accuracy at most

1.5\times

chance, by bit width and scaling granularity. PT is per-tensor and PC per-channel. Percentages are over candidates and runs within each setting.

Model Selection

Model selection chooses models, features, or configurations that balance predictive fit, generalization, complexity, and deployment constraints.

See all 96 papers

Oct 13, 2026

Detecting Unreliable Classifications from Quantized Models

Post-training INT8 quantization is typically validated by checking accuracy retention, which requires ground-truth labels often unavailable in low-resource clinical settings. We show a label-free alternative: comparing a model's own output before and after quantization. We compute cosine similarity between the FP32 and INT8 models' output probability vectors for each prediction, which requires only a single forward pass. This output-agreement similarity separates lower-accuracy predictions across four evaluated architectures (ResNet-50, VGG-16-BN, Wide ResNet-50, and ViT-B/16) under fixed-percentile splits, outperforming a confidence-only baseline for all four. These results show that quantized models can be compared to their baselines to assess their drift and determine whether they are well suited for clinical deployment.

Ananya Kotla, Jorge T. Gomez

Post-Training Quantization Classification

Ask PDF

Aug 11, 2026

SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features

Quantization-Aware Training (QAT) enables the deployment of quantized models with minimal accuracy degradation. However, in practical scenarios, training labels are often unavailable due to privacy, copyright, or cost constraints. Knowledge Distillation (KD) is a common approach to address this challenge, but we observe that prior work combining QAT with KD suffers from a fundamental limitation: during distillation, the range mismatch between the teacher and the quantized student model induces an unattainable residual, resulting in an irreducible lower bound on the distillation loss. Motivated by this observation, we propose SQuaT (Student-Aware Quantized Teacher Features), a label-free QAT framework with KD that theoretically eliminates this lower bound by applying the student's quantization parameters to quantize the teacher's features during distillation. Through comprehensive experiments across diverse settings, we demonstrate that SQuaT consistently outperforms strong baselines, with particularly pronounced gains in extreme low-bit (e.g., 1- and 2-bit) settings. Furthermore, extensive evaluations across various model design choices show that our approach does not rely on specific architectural assumptions, making it broadly applicable across diverse architectures and quantization settings. The source code is available at https://github.com/lcdbsa522/SQuaT.

HyeonJun Lee, Hyeonsik Jo, Jinwoo Chung +1

Kookmin University

Quantization-Aware Training Quantization-Aware Distillation Self-Supervised Knowledge Distillation Knowledge Distillation Low-Bit Quantization

Ask PDF

Oct 2, 2026

Progressive Multi-Ancestor Bit-Depth Distillation

Model compression strategies are widely employed to reduce memory footprint and network complexity, particularly for devices with constrained computational, memory, and energy resources. Prior works that rely on simultaneous conversion from floating-point high-precision (FP32) to integer low-precision (INT4) representations and distillation into smaller models suffer from unstable training and drastic degradation of prediction performance. To address these limitations, we propose a unified framework, known as Progressive Multi-Ancestor Bit-depth Distillation (PMABD), that progressively compresses the network while transferring knowledge through a growing pool of higher-precision ancestor teachers. PMABD generates a sequence of intermediate teachers that each learn from all higher-precision ancestors and jointly supervise the final target student. This multi-ancestor, multi-stage design stabilizes ultra-low-bit quantization by lowering quantization noise profiles across training and ensuring stable quantization. Experiments on CIFAR-10/100 with ResNet-20/32/18, and Tiny-ImageNet with MobileNetV2 show that PMABD outperforms state-of-the-art compression frameworks, results in 1.06

\%

increase in performance of W2A2 (ResNet-18/CIFAR-100) student model. We show that a saturation-based stopping criterion contributes to improve the performance of our final student.

Adil Mubashir Chaudhry, Osama Ahmad, Zubair Khalid +1

LUMS School of Science and Engineering · University of Massachusetts Amherst, USA

Quantization-Aware Distillation Model Compression Neural Network Quantization Knowledge Distillation Multi-Teacher Knowledge Distillation

Ask PDF

Abstract

Model Selection

Explore similar work

Detecting Unreliable Classifications from Quantized Models

SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features

Progressive Multi-Ancestor Bit-Depth Distillation