What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2

Nabeelah Faumi; Deshan Sumanathilaka

cs.CLOct 27, 2026

What Survives Distillation? A Mechanistic Interpretability Study of GPT-2 and DistilGPT-2

Authors: Nabeelah Faumi, Deshan Sumanathilaka

Organizations: Informatics Institute of Technology, 57, Ramakrishna Road, Colombo 06, Sri Lanka

Abstract

Knowledge distillation compresses large language models into compact students that preserve aggregate metrics, yet the mechanistic consequences of depth-reduction distillation remain largely uncharacterized. This work applies linear CKA, logit lens analysis, attention pattern decomposition, sparse autoencoder feature statistics, and activation patching to compare GPT-2 and its six-layer student DistilGPT-2 across two canonical circuits: the Indirect Object Identification (IOI) circuit and the Greater-Than (GT) circuit. The results reveal a compression-collapse dichotomy: the IOI circuit suffers complete mechanistic collapse (DistilGPT-2: 35.2% accuracy vs. 94.8%, zero name-mover preservation), while the GT circuit survives with 87.7% task performance recovery through active remapping of the teacher's layer-9 comparison head into student layer~4 (mean Pearson r = 0.945). These findings demonstrate that circuit compressibility governs mechanistic survival under distillation, and that aggregate metrics are insufficient diagnostics for circuit integrity.

Figures & tables

Class	GPT-2 heads	Available in DistilGPT-2
Previous token	(0,1),(2,2),(4,11)	All 3
Duplicate token	(0,10),(3,0)	All 2
Induction	(5,1),(5,5),(6,9),(7,2),(7,10)	2 of 5
S-inhibition	(7,3),(7,9),(8,6),(8,10)	None
Name mover	(9,6),(9,9),(10,0),(10,6),(11,2)	None
Neg. name mover	(10,7),(11,10)	None

Table 1: IOI circuit heads by functional class and availability in DistilGPT-2 (layers 0-5 only).

Model	Layers	Params	PPL (TinyStories)	PPL (WikiText-2)
GPT-2	12	124M	11.54	33.02
DistilGPT-2	6	82M	17.55 (+52.1%)	49.33 (+49.4%)

Table 2: Perplexity comparison on TinyStories validation and WikiText-2 test sets.

Figure 1: Layer-wise CKA similarity under proportional mapping (top) and same-index mapping (bottom). Late-layer CKA collapses to $0.314$ at layer 5 under same-index matching and is partially recovered under proportional mapping ( $0.790$ ). Cosine similarity results agree in direction throughout.

Pair	GPT-2 $\ell$	Distil $\ell$	$\Delta$ Acc	$r_{\ell}$
D0/G0	0	0	$-$ 0.005	1.13
D1/G2	2	1	$-$ 0.039	2.10
D2/G4	4	2	$-$ 0.021	1.32
D3/G6	6	3	$-$ 0.030	1.28
D4/G8	8	4	$-$ 0.068	1.35
D5/G11	11	5	$+$ 0.006	0.97

Table 3: Extended logit lens accuracy comparison including mean probability assigned to correct next token (

\bar{p}_{\ell}

Figure 2: Attention pattern analysis. Top : entropy heatmaps for GPT-2 and DistilGPT-2 (Q1). Middle : previous-token attention heatmaps and layer averages; student peaks earlier at layer 1 (Q2). Bottom : distributions of per-head cosine similarity (mean $0.894$ ) and entropy change (mean $-0.007$ bits) under proportional mapping (Q3).

GPT-2	Distil	Disc CKA	Eval CKA	Disc $r$	Eval $r$	NM Pres%
L9H6	L0H1	0.4659	0.4592	$-$ 0.1535	$-$ 0.1514	0.0%
L9H9	L0H1	0.3209	0.3261	$-$ 0.1391	$-$ 0.1378	0.0%
L10H0	L1H6	0.5077	0.5150	0.4597	0.4417	$-$ 14.6%
L10H6	L1H6	0.3629	0.3926	0.6507	0.6484	$-$ 21.3%
L11H2	L1H6	0.2073	0.1784	0.7802	0.7781	nan

Table 4: Full name-mover head matching results across discovery and evaluation sets.

Figure 3: DLA across layers for the IOI task. GPT-2 shows strong positive contributions from layers 9-10; DistilGPT-2 shows near-zero attribution across all available layers.

Figure 4: Attention heatmaps for GPT-2 name-mover heads and their closest DistilGPT-2 matches. Matched student heads fail to exhibit IO-selective attention.

Year	YY	Attention to YY token		Pearson $r$
		GPT-2 L9H1	Distil L4H1	(full pattern)
1102	02	0.099	0.156	-
1503	03	0.239	0.175	-
1804	04	0.870	0.871	-
1505	05	0.085	0.233	-
1006	06	0.082	0.091	-

Table 5: YY-token attention weights for DistilGPT-2 L4H1 and GPT-2 L9H1 at the prediction position, across five representative GT examples.

Figure 5: DLA for the GT task. DistilGPT-2 shifts peak attribution to layer 4, functionally replacing GPT-2’s layer 9.

Head	Class	Raw score		Normalised score		Sig.
		GPT-2	Distil	GPT-2	Distil
L0H1	Year encoding	0.332	0.283	0.606	0.517	✓
L1H5	Year encoding	0.305	0.255	0.557	0.465	✓
Overall (Cohen’s $d$ )		$d=0.681$ , $p<0.0001$				✓

Table 6: GT circuit head-level activation patching results. Normalised scores use noise ceiling

\hat{s}=s/({\rm GTProb}_{\rm clean}-{\rm GTProb}_{\rm corrupt})

, noise ceiling

=0.547

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Scheme	Depth	CKA	Cosine	CKA range	Cosine range
Proportional	Early	0.9998	0.9977	-	-
	Middle	0.9668	0.9749	-	-
	Late	0.7903	0.9690	-	-
	Overall	$0.921\pm 0.109$	$0.977\pm 0.019$	[0.662, 1.000]	[0.939, 0.999]
Same-index	Early	0.9998	0.9977	-	-
	Middle	0.9082	0.9304	-	-

Appendix

Table 7: CKA and cosine similarity by depth quartile under each mapping scheme, reported as mean across seeds {42, 123, 7}.

Cross-model head similarity (proportional mapping)
Metric	GPT-2	DistilGPT-2
Mean entropy (bits)	2.001	2.094
Effective support (tokens)	11.19	12.14
Mean top-1 mass	0.507	0.493
Mean cosine sim.	$0.894\pm 0.086$ [0.610, 0.993]
Mean $\Delta H$ (bits)	$-0.007\pm 0.621$

Appendix

Table 8: Attention pattern metrics summary. Head similarity and entropy change reported as mean

\pm

std across 1,000 TinyStories samples.

Scheme	Pair	Cosine sim.	Feature survival	$\Delta L_{0}$
Same-index	L0	$0.989$	$0.339$	$-2{,}379$
	L3	$0.887$	$0.169$	$-1{,}493$
	L5	$0.655$	$0.086$	$-7{,}178$
Proportional	G1/D0	$0.184$	$0.000$	$-132$
	G6/D3	$0.927$	$0.527$	$-68$
	G11/D5	$0.927$	$0.529$	$-601$

Appendix

Table 9: SAE analysis summary across layer pairs.

Language Model Distillation

Language model distillation transfers a larger model’s capabilities to smaller or specialized models using outputs, probabilities, or internal representations.

See all 512 papers

Circuit Distillation
Sep 29, 2025Somin Wadhwa, Silvio Amir, Byron C. WallaceLanguage Model Distillation Mechanistic Interpretability
Khoury College of Computer Sciences, Northeastern University, Boston, MA 02115, USA
Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation
Jun 4, 2026Maxime Griot, Paul Steven Scotti, Tanishq Mathew AbrahamCoT Distillation Teacher-Student Learning
Université catholique de Louvain · Sophont Inc.
Beyond Imitation: Auditing the Recoverability of Reasoning in Distilled Models
Aug 11, 2026Ruitong Li, Binjie Guo, Aisheng Mo +4Language Model Distillation Knowledge Distillation
University of Hong Kong · Zhejiang University · Dalian University of Technology +1

Sep 29, 2025J/K move · Enter open · S save

Circuit Distillation

Somin Wadhwa, Silvio Amir, Byron C. Wallace

Khoury College of Computer Sciences, Northeastern University, Boston, MA 02115, USA

Language Model Distillation Mechanistic Interpretability

Open paper AskPDF arXiv

Model distillation typically focuses on behavioral mimicry, where a student model is trained to replicate a teacher's output while treating its internal computations as a black box. In this work we propose an alternative approach: Distilling the underlying computational mechanisms implemented by a teacher model. Specifically, we propose circuit distillation, which introduces an objective to align internal representations between analogous circuit components in teacher and student models. We propose a method to match ``functionally correspondent'' circuit components and introduce a loss reflecting similarities between the representations that these induce. We evaluate circuit distillation on entity tracking and theory of mind (ToM) tasks using models from the Llama3 family. Our results demonstrate that circuit distillation outperforms standard distillation, successfully transferring algorithmic capabilities by adjusting only a small, targeted subset of student model parameters. This work establishes the feasibility of transferring mechanisms, which may in turn allow for efficient distillation of targeted teacher capabilities via interpretable and controllable internal student mechanisms.

Abstract

Language Model Distillation

Explore similar work

Circuit Distillation

Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation

Beyond Imitation: Auditing the Recoverability of Reasoning in Distilled Models