Temperature-Adaptive Transformed Teacher Matching

Hiroaki Aizawa; Yoshikazu Hayashi

cs.LGAug 29, 2026

Temperature-Adaptive Transformed Teacher Matching

Authors: Hiroaki Aizawa, Yoshikazu Hayashi

Organizations: Graduate School of Advanced Science and Engineering Hiroshima University, Japan · Intelligent Production Technology Research & Development Center for Aerospace Gifu University, Japan

Abstract

Temperature scaling is a core component of knowledge distillation, yet its role and effect are still not fully understood. Transformed Teacher Matching (TTM) clarifies the role of temperature scaling by applying it only to the teacher distribution and interpreting the resulting objective as standard distillation with an implicit Rényi entropy regularization on the student. However, TTM still relies on a fixed temperature and does not specify how the teacher-side temperature should be adapted for individual samples. In this paper, we introduce a sample-wise inverse-temperature update for TTM by locally minimizing the Kullback-Leibler divergence between the temperature-scaled teacher distribution and the student's prediction. We derive closed-form first and second derivatives with respect to the inverse temperature, and show that they can be expressed using variance and covariance statistics of centered teacher and student logits under the transformed teacher weighting. This yields an efficient curvature-aware update that requires one softmax evaluation and a constant number of class-wise weighted sums. Experiments on standard image classification distillation benchmarks show that our temperature adaptation generally improves TTM and WTTM, while remaining competitive with or outperforming prior temperature-adaptive distillation baselines.

Figures & tables

Figure 1: Overview of the proposed Temperature-Adaptive Transformed Teacher Matching. Given an input sample, the teacher produces a predictive distribution, which is calibrated by a power transform with an adaptive inverse temperature $\gamma$ . The inverse temperature $\gamma$ is updated to obtain sample-adaptive soft targets for distillation. The student is trained using cross-entropy and a distillation loss that matches the student prediction to the adapted teacher distribution.

Feature-based Knowledge Distillation
Teacher	WRN-40-2	WRN-40-2	ResNet56	ResNet110	ResNet110	ResNet32x4	VGG13
Teacher	75.61	75.61	72.34	74.31	74.31	79.42	74.64
Student	WRN-16-2	WRN-40-1	ResNet20	ResNet20	ResNet32	ResNet8x4	VGG8
Student	73.26	71.98	69.06	69.06	71.14	72.50	70.36
FitNet	$73.58_{\pm 0.32}$	$72.24_{\pm 0.24}$	$69.21_{\pm 0.36}$	$68.99_{\pm 0.27}$	$71.06_{\pm 0.13}$	$73.50_{\pm 0.28}$	$71.02_{\pm 0.31}$
AT	$74.08_{\pm 0.25}$	$72.77_{\pm 0.10}$	$70.55_{\pm 0.27}$	$70.22_{\pm 0.16}$	$72.31_{\pm 0.08}$	$73.44_{\pm 0.19}$	$71.43_{\pm 0.09}$

Table 1: Performance comparison on CIFAR100 in the same-architecture distillation setting.

\dagger

indicates results reproduced by us using the official implementation under the protocol described in the paper [ 37 ] .

\Delta

and

\Delta^{\dagger}

denote accuracy differences from TTM/WTTM and TTM † /WTTM † , respectively; positive values are highlighted in red and negative values in blue.

Feature-based Knowledge Distillation
Teacher	VGG13	ResNet50	ResNet50	ResNet32 $\times$ 4	ResNet32 $\times$ 4	WRN-40-2
Teacher	74.64	79.34	79.34	79.42	79.42	75.61
Student	MobileNetV2	MobileNetV2	VGG8	ShuffleNetV1	ShuffleNetV2	ShuffleNetV1
Student	64.60	64.60	70.36	70.50	71.82	70.50
FitNet	$64.14_{\pm 0.50}$	$63.16_{\pm 0.47}$	$70.69_{\pm 0.22}$	$73.59_{\pm 0.15}$	$73.54_{\pm 0.22}$	$73.73_{\pm 0.32}$
AT	$59.40_{\pm 0.20}$	$58.58_{\pm 0.54}$	$71.84_{\pm 0.28}$	$71.73_{\pm 0.31}$	$72.73_{\pm 0.09}$	$73.32_{\pm 0.35}$

Table 2: Performance comparison on CIFAR100 in the cross-architecture distillation setting.

Teacher	ResNet-34 (73.31)
Student	ResNet-18 (69.76)
Method	KD	TTM	WTTM	EA-KD	\cellcolor blue!8TA-TTM	\cellcolor blue!8TA-WTTM
Top-1 Acc.	70.66	71.74	71.81	71.79	\cellcolor blue!871.92 ( $\textcolor{red!70!black}{+0.18}$ )	\cellcolor blue!872.11 ( $\textcolor{red!70!black}{+0.30}$ )

Table 3: Performance comparison on ImageNet1k. TTM and WTTM results are from our reproduced experiments. These results are obtained from a single run.

Teacher	WRN-40-2	ResNet56	VGG13
Student	WRN-16-2	ResNet20	VGG8
KD w/ CE	$75.31_{\pm 0.25}$	$71.52_{\pm 0.31}$	$73.31_{\pm 0.28}$
TTM w/o CE	$73.89_{\pm 0.19}$	$71.21_{\pm 0.38}$	$73.13_{\pm 0.30}$
WTTM w/o CE	$75.27_{\pm 0.09}$	$71.17_{\pm 0.02}$	$74.02_{\pm 0.04}$
\rowcolor blue!8 TA-TTM w/o CE	$74.25_{\pm 0.20}$	$71.29_{\pm 0.10}$	$73.48_{\pm 0.23}$
\rowcolor blue!8 TA-WTTM w/o CE	$75.59_{\pm 0.16}$	$71.54_{\pm 0.41}$	$74.14_{\pm 0.03}$

Table 4: Top-1 accuracy (%) on CIFAR100 same-architecture pairs. TTM/WTTM variants omit cross-entropy; KD with cross-entropy is included as a reference. Results are reported over three seeds.

Teacher	ViT-S (92.44)	Swin-T (89.26)	Mixer-B/16 (87.62)
Student	ResNet18 (74.01)	ResNet18 (74.01)	ResNet18 (74.01)
KD	$76.41_{\pm 0.16}$	$77.06_{\pm 0.17}$	$76.84_{\pm 0.22}$
EA-KD	$76.82_{\pm 0.03}$	$77.05_{\pm 0.18}$	$76.81_{\pm 0.17}$
TTM	$78.89_{\pm 0.19}$	$80.76_{\pm 0.15}$	$79.04_{\pm 0.20}$
WTTM	$79.16_{\pm 0.11}$	$80.37_{\pm 0.73}$	$79.03_{\pm 0.17}$
\rowcolor blue!8 TA-TTM	$79.31_{\pm 0.25}$ ( $\textcolor{red!70!black}{+0.42}$ )	$80.84_{\pm 0.07}$ ( $\textcolor{red!70!black}{+0.08}$ )	$79.29_{\pm 0.16}$ ( $\textcolor{red!70!black}{+0.25}$ )

Table 5: Performance comparison on CIFAR100 in the distillation setting from transformer-based teachers to CNN-based students.

Panel of Figure 2 — Figure 2: Temperature diagnostics for TA-TTM on the representative same-architecture pair ResNet110 $\rightarrow$ ResNet20. The panels show the evolution of the mean and standard deviation of the inverse temperature $\gamma$ , the stage-wise distribution of $\gamma$ , and the correlation between per-sample cross-entropy loss and the optimized $\gamma$ , respectively.

Panel of Figure 3 — Figure 3: Temperature diagnostics for TA-WTTM on ResNet110 $\rightarrow$ ResNet20, using the same layout as Fig. 2 .

Teacher	WRN-40-2	ResNet110	VGG13	VGG13	ResNet32 $\times$ 4	WRN-40-2
Student	WRN-16-2	ResNet32	VGG8	MobileNetV2	ShuffleNetV1	ShuffleNetV1
TA-WTTM	$76.57_{\pm 0.17}$	$74.04_{\pm 0.11}$	$74.71_{\pm 0.08}$	$68.84_{\pm 0.74}$	$74.32_{\pm 0.23}$	$75.72_{\pm 0.29}$
TA-WTTM-full	$76.54_{\pm 0.23}$	$73.64_{\pm 0.29}$	$74.56_{\pm 0.24}$	$68.74_{\pm 0.33}$	$73.83_{\pm 0.60}$	$75.53_{\pm 0.26}$
$\Delta$	$\textcolor{red!70!black}{+0.03}$	$\textcolor{red!70!black}{+0.40}$	$\textcolor{red!70!black}{+0.15}$	$\textcolor{red!70!black}{+0.10}$	$\textcolor{red!70!black}{+0.49}$	$\textcolor{red!70!black}{+0.19}$

Table 7: Ablation study of the TA-WTTM temperature update on CIFAR100. We compare TA-WTTM, which uses the decoupled update, with the full-objective update on representative same- and different-architecture pairs.

\Delta

denotes TA-WTTM minus TA-WTTM-full.

Panel of Figure 4 — Figure 4: Hyperparameter sensitivity of the proposed temperature adaptation. We evaluate the update rate $\eta$ , warm-up length, and update interval for the sample-wise inverse temperature update.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Method	ATD	TTM	WTTM	TA-TTM	TA-WTTM
Acc.	$75.83_{\pm 0.14}$	$76.23_{\pm 0.15}$	$76.37_{\pm 0.10}$	$76.42_{\pm 0.16}$	$76.57_{\pm 0.17}$

Appendix

Table 8: Comparison with ATD on CIFAR100 for WRN-40-2

\rightarrow

WRN-16-2. Results are Best Acc@1.

Panel of Figure 5 — Figure 5: Cross-architecture temperature diagnostics for WRN-40-2 $\rightarrow$ ShuffleNetV1. The first and second rows show TA-TTM and TA-WTTM, respectively; the columns show temperature dynamics, stage-wise inverse-temperature distributions, and correlations with per-sample cross-entropy loss.

Method	Epoch	Mean $h_{\mathrm{update}}$	Min. $h_{\mathrm{update}}$	$h_{\mathrm{update}}<0$	$\|h_{\mathrm{update}}\|<10^{-8}$	$\|h_{\mathrm{update}}\|<10^{-3}$
TA-TTM	100	22.46	$-35.14$	0.164%	0%	0%
TA-TTM	200	16.79	1.47	0%	0%	0%
TA-TTM	240	16.77	1.83	0%	0%	0%
TA-WTTM	100	22.02	$-32.06$	0.164%	0%	0.002%
TA-WTTM	200	17.03	$-2.88$	0.004%	0%	0%
TA-WTTM	240	17.08	1.02	0%	0%	0%

Appendix

Table 9: Curvature diagnostics for WRN-40-2

\rightarrow

WRN-16-2, evaluated over all 50,000 training samples. The last three columns report the percentage of samples satisfying each condition.

Method	Statistic	Epoch 100	Epoch 200	Epoch 240
TA-TTM	Pearson	$-0.673$	$-0.687$	$-0.713$
	Spearman	$-0.710$	$-0.722$	$-0.744$
TA-WTTM	Pearson	$-0.673$	$-0.645$	$-0.683$
	Spearman	$-0.720$	$-0.681$	$-0.709$

Appendix

Table 10: Correlation between the learned inverse temperature

\gamma

and per-sample cross-entropy loss for WRN-40-2

\rightarrow

WRN-16-2, using 1,000 samples per epoch.

Pair	Method	$\gamma_{\mathrm{init}}$	$\beta$	$\eta$
vgg13_MobileNetV2	TA-TTM	0.2	16	0.005
vgg13_MobileNetV2	TA-WTTM	0.2	3	0.004
ResNet50_MobileNetV2	TA-TTM	0.2	20	0.004
ResNet50_MobileNetV2	TA-WTTM	0.2	5	0.001
ResNet50_vgg8	TA-TTM	0.1	70	0.005
ResNet50_vgg8	TA-WTTM	0.1	2	0.005

Appendix

Table 11: Hyperparameters used for each teacher–student pair.

Knowledge Distillation

Knowledge distillation transfers useful behavior or representations from larger or specialized models into more compact or adaptable learners.

See all 783 papers

Consistently Informative Soft-Label Temperature for Knowledge Distillation
May 19, 2026Hoang-Chau Luong, Nghia Van Vo, Kaiqi Zhao +1Teacher-Student Learning Knowledge Distillation
Rochester Institute of Technology, Rochester, NY, USA · Oakland University, Rochester, MI, USA
A Unified Revisit of Temperature in Classification-Based Knowledge Distillation
Mar 2, 2026Logan Frank, Jim DavisKnowledge Distillation
Ohio State University
Rethinking the Role of Temperature in Large Language Model Distillation
May 29, 2026Hoang-Chau Luong, Lingwei ChenLanguage Model Distillation Knowledge Distillation
Golisano College of Computing and Information Sciences Rochester Institute of Technology Rochester, NY, United States

May 19, 2026J/K move · Enter open · S save

Consistently Informative Soft-Label Temperature for Knowledge Distillation

Hoang-Chau Luong, Nghia Van Vo, Kaiqi Zhao, Lingwei Chen

Rochester Institute of Technology, Rochester, NY, USA · Oakland University, Rochester, MI, USA

Teacher-Student Learning Knowledge Distillation

Open paper AskPDF arXiv

Knowledge distillation (KD) transfers knowledge from a high-capacity teacher to a compact student by matching their predictive distributions, with temperature scaling serving as a central mechanism for smoothing teacher predictions and exposing informative "dark knowledge" beyond the hard label. However, the standard fixed-temperature design is inherently sample-agnostic. Since samples differ in logit scale and learning difficulty, a single global temperature produces teacher soft labels with highly inconsistent entropy: some predictions remain overly sharp and provide limited inter-class information, whereas others become over-smoothed and lose class-discriminative information. Moreover, sharing the same temperature between teacher and student further imposes rigid logit-scale alignment despite their capacity mismatch. To address these limitations, we propose CIST (Consistently Informative Soft-label Temperature), which assigns separate sample-wise adaptive temperatures to the teacher and student. This design produces consistently informative teacher soft labels while relaxing rigid teacher--student logit-scale matching. It also reweights the distillation objective according to teacher confidence and student learning difficulty. Theoretically, we show that teacher-label entropy is largely governed by the ratio between the maximum teacher logit and the temperature, providing a principled basis for adaptive smoothing. Empirically, CIST mitigates the inconsistency induced by fixed temperature, and experiments on both vision and language distillation tasks show consistent improvements over standard KD and strong baselines with negligible computational overhead.

Abstract

Knowledge Distillation

Explore similar work

Consistently Informative Soft-Label Temperature for Knowledge Distillation

A Unified Revisit of Temperature in Classification-Based Knowledge Distillation

Rethinking the Role of Temperature in Large Language Model Distillation