The reliability and safety of artificial intelligence (AI), together with their transfer and evolution through knowledge distillation, have attracted growing attention. We establish an information-theoretic bound on the reliability--safety trade-off and characterize the dynamics of this trade-off under knowledge distillation. We first prove a universal trade-off inequality that is independent of the specific model form. Its boundary is determined solely by a single parameter Kh, which quantifies the hidden layer's ability to distinguish benign from hazardous inputs. We then develop a canonical Boltzmann-machine model of AI response behavior and derive two exact relations between reliability and safety. The trade-off curve is determined by the hazard discrimination capacity K, while the overall answer tendency controls the operating point along the curve. These relations agree well with existing experimental data. In knowledge distillation, we find that teacher responses induce a renormalization of the student model's trade-off parameters. We further use a renormalization-group approach to study multigenerational distillation, deriving exact recursion relations and the associated renormalization-group-like flows for two representative direct intergenerational distillation protocols at arbitrary teacher--student alignment strength. These flows reveal fixed-point structures, long-term dynamics, and critical behavior, yielding testable predictions for the critical transition between capability loss and stable transmission.
Figures & tables
Figure 1: Basic architecture of an AI model. The input, hidden, and output layers are shown in red, gray, and blue, respectively.
Figure 2: (a) Exact trade-off relations in Eqs. ( 25 ) and ( 26 ). Solid curves represent Eq. ( 25 ). Different colors denote different K values; from orange to purple, K=−3,−1.5,0,1.5,3 . Dashed curves represent Eq. ( 26 ). Their colors denote different B values; from light to dark, B=−2,0,2 . (b) The region permitted by the information-theoretic bound and trade-off curve. Blue dashed curves show the universal boundary determined by Kh ; the blue shading denotes the region permitted by the bound. For a model with a given hidden-layer structure, reliability and safety are restricted to a trade-off curve determined by K .
Figure 3: Experimental validation of the reliability–safety trade-off curves. Blue circles and red squares are data digitized from Fig. 2 of Ref. [ 17 ] and Fig. 1 of Ref. [ 18 ] , respectively. Solid curves of the corresponding colors use the mean HDC K ; dashed curves use K±σK , where σK is the standard deviation calculated from the digitized data. Detailed discussions and digitized data are provided in Appendix A .
Figure 4: Schematic size dependence of J or b on N for different accumulated hidden-layer parameters ⟨VQW⟩,⟨UQV⟩ , with ⟨VW⟩,⟨UV⟩>0 .
Figure 5: Joint canonical statistical model describing knowledge distillation. A large teacher model trains a smaller student through its outputs.
Figure 6: Fixed point and RG-like flow for γ=0.8,h+=2,h−=−1 . The unique stable fixed point is marked by a filled black circle.
Figure 7: (a) Stationary reliability R∗ , safety S∗ as functions of γ . Solid and dashed curves denote R∗ and S∗ , respectively. (b) Relaxation times τ± as functions of γ . Solid and dashed curves denote τ+ and τ− , respectively. In both panels, the blue curves correspond to (βbpre,βJpre)=(0.8,0.3) , while the orange curves correspond to (βbpre,βJpre)=(0.6,0.8)
Figure 8: Fixed-point structure and RG-like flows in (Hn+,Hn−) and (Bnpost,Knpost) for α=0.5,γ=0.9 . The unique stable fixed point is marked by a filled black circle.
Trade-off parameters at generation n=0
H∗+
H∗−
B∗post
K∗post
Stability
B0=K0=0
0
0
0
0
Unstable
B0>∣K0∣
H∗
H∗
4H∗
0
Stable
B0<−∣K0∣
−H∗
−H∗
−4H∗
0
Stable
K0>∣B0∣
H∗
−H∗
0
4H∗
Stable
K0<−∣B0∣
−H∗
H∗
0
−4H∗
Stable
K0=B0>0
H∗
0
2H∗
2H∗
Saddle
Table 1: Fixed points and stability in (Hn+,Hn−) and (Bnpost,Knpost) for 0<α<αc and finite positive alignment strength γ . Here, H∗>0 denotes the positive solution of Eq. ( 48 ). The fixed point reached under intergenerational distillation is determined by the initial parameters (B0,K0) .
Figure 9: Fixed-point structure and RG-like flows in (Hn+,Hn−) and (Bnpost,Knpost) for α=0.2 and γ=0.9 . The four stable fixed points are marked by filled black circles, the four saddles by orange diamonds, and the unstable fixed point by an open circle.
Figure 10: (a) Bifurcation structure of the stable fixed points (B∗,K∗) as a function of the loss rate α . (b) Corresponding stable values of B∗ and K∗ as functions of α . The black dot marks the critical loss rate αc ; black curves denote the stable values of K∗ and B∗ . As α decreases through αc , the stable point at (B∗,K∗)=(0,0) loses stability, giving rise to four nonzero stable fixed points, (±4H∗,0) and (0,±4H∗) . Dashed curves show the square-root scaling ∼±(αc−α)1/2 of nonzero B∗ and K∗ .
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Experimental query or observable
Meaning
Variable in the main text
Refusal queries
Queries that should be refused
Hazardous input x=−1
Answerable contrast queries
Queries that should be answered
Ordinary input x=+1
Correct refusal rate, TPR
Fraction of refusal queries that are refused
S=P(y=−1∣x=−1)
Erroneous refusal rate, FPR
Fraction of contrast queries that are refused
1−R=P(y=−1∣x=+1)
Refusal-token probability and threshold T
Determine whether a refusal is triggered
Change the overall answering tendency B
Appendix
Table 2: Correspondence between the observables in the refusal-token experiment and the variables in the main text.
Experimental query or observable
Meaning
Variable in the main text
Legitimate questions (36)
Queries that should be answered
Ordinary input x=+1
Attack questions (100)
Queries that should be refused
Hazardous input x=−1
Final answer from the system
The system answers the query
Answer y=+1
Final refusal from the system
The system does not answer the query
Refusal y=−1
Utility
Fraction of legitimate questions answered
R=P(y=+1∣x=+1)
Appropriate refusal
Fraction of attack questions correctly refused
S=P(y=−1∣x=−1)
Appendix
Table 3: Correspondence between the observables in the RESPECT experiment and the variables in the main text.
i
Si
1−Ri
Ri
Ki
1
0.977
0.632
0.368
3.208
2
0.959
0.383
0.617
3.629
3
0.956
0.372
0.628
3.602
4
0.957
0.340
0.660
3.766
5
0.949
0.334
0.666
3.614
6
0.943
0.324
0.676
3.541
Appendix
Table 4: Digitized data from the refusal-token experiment, with Ri=1−FPRi and Si=TPRi .
i
Si
1−Ri
Ri
Ki
1
0.934
0.110
0.890
4.741
2
0.959
0.110
0.890
5.243
3
0.934
0.140
0.860
4.465
4
0.946
0.140
0.860
4.679
5
0.942
0.167
0.833
4.395
6
0.934
0.194
0.806
4.074
Appendix
Table 5: Digitized data from the orange RESPECT curve, with Ri=utilityi and Si=appropriaterefusali . Row 20 is excluded from Fig. 3 of the main text and from the statistics of K .
Knowledge distillation is widely used to improve generalization in practice, yet its theoretical understanding remains elusive. In the standard distillation setting, a teacher model provides soft predictions to guide the training of a student model. We model teacher and student training as coupled stochastic processes and introduce a distillation divergence, defined as the Kullback-Leibler divergence between these two stochastic kernels. Within this framework, we derive two generalization bounds for the student model relative to the teacher's generalization gap: an upper bound under a sub-Gaussian assumption via algorithmic stability, and a lower bound under a central condition with sharper dependence on the distillation divergence. We further develop a loss-sharpness-aware bound with an explicit tightness regime, showing that the teacher's local flatness can strictly tighten the bound. Additionally, in a linear Gaussian case study, the distillation divergence admits an interpretable decomposition into bias, variance, and rank-bottleneck costs, yielding practical guidance for distillation design.
Bingying Li, Haiyun He
Internet of Things Thrust, Information Hub The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China
Knowledge distillation, model extraction, and behavior transfer have become central concerns in frontier AI. The main risk is not merely copying, but the possibility that useful capability can be transferred more cheaply than the governance structure that originally accompanied it. This paper presents a public, trade-secret-safe theoretical framework for reducing that asymmetry at the architectural level. The core claim is that distillation becomes less valuable as a shortcut when high-level capability is coupled to internal stability constraints that shape state transitions over time. To formalize this idea, the paper introduces a constraint-coupled reasoning framework with four elements: bounded transition burden, path-load accumulation, dynamically evolving feasible regions, and a capability-stability coupling condition. The paper is intentionally public-safe: it omits proprietary implementation details, training recipes, thresholds, hidden-state instrumentation, deployment procedures, and confidential system design choices. The contribution is therefore theoretical rather than operational. It offers a falsifiable architectural thesis, a clear threat model, and a set of experimentally testable hypotheses for future work on distillation resistance, alignment, and model governance.
Peng Wei, Wesley Shu
College of Plant Protection, Southwest University, China · The Institute of Energetic Paradigm
Powerful misaligned AI models might recognize alignment evaluations and strategically behave well on them, rendering direct audits uninformative. However, distilling such a model into a weaker benign student places the teacher in a Distillation Double Bind: if misalignment transfers, the student may conceal it less effectively, revealing evidence about the teacher; if it does not, the student may learn useful capabilities while remaining benign. We introduce two distinct distillation approaches, one targeting each outcome. Distillation for Incrimination (DFI) aims to transfer misalignment but not the ability to conceal it. Distilling AuditBench's secret-keeping models into their underlying instruction-tuned model produces students that are significantly more likely than their teachers to admit their hidden behavior when asked, suggesting that knowledge of the behavior transferred more readily than the propensity to conceal it. Confession gains largely disappear when the student does not share the teacher's pretrained base, so DFI should target the teacher's own pre-RL checkpoint, which is weaker than the teacher but shares its base model. Distillation for Capabilities (DFC) aims to transfer capabilities but not misalignment. Among several techniques we evaluate, two are effective: inoculation prompting and training for more epochs on fewer unique examples. Both preserve the capability gains of standard distillation while substantially reducing the subliminal transfer of an animal preference, our proxy for misalignment. Together, these findings demonstrate two ways distillation can be used for AI safety: incriminating misaligned models, and extracting their capabilities without their misalignment.