Can Cross-Layer Transcoders Replace Vision Transformer Activations? An Interpretable Perspective on Vision

Gerasimos Chatzoudis; Konstantinos D. Polyzos; Zhuowei Li; Difei Gu; Gemma E. Moran; Hao Wang; Dimitris N. Metaxas

cs.CVApr 14, 2026

Can Cross-Layer Transcoders Replace Vision Transformer Activations? An Interpretable Perspective on Vision

Authors: Gerasimos Chatzoudis, Konstantinos D. Polyzos, Zhuowei Li, Difei Gu, Gemma E. Moran, Hao Wang, Dimitris N. Metaxas

Organizations: Rutgers University · University of California San Diego

Abstract

Understanding the internal activations of Vision Transformers (ViTs) is critical for building interpretable and trustworthy models. While Sparse Autoencoders (SAEs) have been used to extract human-interpretable features, they operate on individual layers and fail to capture the cross-layer computational structure of Transformers, as well as the relative significance of each layer in forming the last-layer representation. Alternatively, we introduce the adoption of Cross-Layer Transcoders (CLTs) as reliable, sparse, and depth-aware proxy models for MLP blocks in ViTs. CLTs use an encoder-decoder scheme to reconstruct each post-MLP activation from learned sparse embeddings of preceding layers, yielding a linear decomposition that transforms the final representation of ViTs from an opaque embedding into an additive, layer-resolved construction that enables faithful attribution and process-level interpretability. We train CLTs on CLIP ViT-B/32 and ViT-B/16 across CIFAR-100, COCO, and ImageNet-100. We show that CLTs achieve high reconstruction fidelity with post-MLP activations while preserving and even improving, in some cases, CLIP zero-shot classification accuracy. In terms of interpretability, we show that the cross-layer contribution scores provide faithful attribution, revealing that the final representation is concentrated in a smaller set of dominant layer-wise terms whose removal degrades performance and whose retention largely preserves it. These results showcase the significance of adopting CLTs as an alternative interpretable proxy of ViTs in the vision domain.

Figures & tables

Figure 1: Overview of the Cross-Layer Transcoder (CLT) framework. Left : Each CLT encodes LN2 activations into sparse codes $z_{\ell}$ and reconstructs MLP outputs $y_{\ell}$ via triangular decoders. Middle : During inference, CLTs replace MLPs across layers, preserving zero-shot performance. Right : Using CLTs as an interpretable proxy for ViTs to understand the cross-layer contributions of different token types and identify the most significant layers in shaping the ViT’s final representations.

Figure 2: Reconstruction performance of Cross-Layer Transcoders (CLTs) across transformer layers on CIFAR-100 using ViT-B/32. We compare three sparsity variants, i.e., JumpReLU , ReLU-Top-

k

, and Abs-Top-

k

(with

k{=}128

) using (left) cosine similarity, (center) mean squared error (MSE per token, log scale), and (right) variance explained (

R^{2}

). Metrics are averaged over all tokens in the test set.

CIFAR-100
Dataset	Backbone	Tokens	3 $\rightarrow$ 11			7 $\rightarrow$ 11			10 $\rightarrow$ 11			All
			JR	RTK	ATK	JR	RTK	ATK	JR	RTK	ATK	JR	RTK	ATK
	ViT-B/32	CLS	61.38	61.69	61.58	61.41	61.86	61.26	61.06	61.39	61.16	61.33	61.74	61.43
		Patches	53.65	54.20	55.83	61.51	61.69	63.18	61.49	61.25	61.56	49.63	51.12	48.68
		All	54.10	54.84	55.69	61.54	62.08	62.85	61.01	61.20	61.11	49.40	51.12	48.41
	ViT-B/16	CLS	66.10	65.81	65.39	66.15	66.15	65.86	66.06	65.90	65.90	66.04	65.92	65.38

Table 3: Top-1 classification accuracy (%) across datasets, backbones, and token types. Columns correspond to layer range and sparsity combinations: JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. For reference, baseline top-1 accuracies (in %) are: CIFAR-100 (B/32: 61.65, B/16: 65.97), COCO (B/32: 43.12, B/16: 43.56), and ImageNet-100 (B/32: 80.42, B/16: 84.34).

Panel of Figure 3 — Figure 3: Cross-layer attribution scores $C^{\mathrm{proj}}_{i\rightarrow j}$ for ViT-B/16 on CIFAR-100, averaged over the validation set. Scores along each column yield a partition of attribution across depth. Patch tokens (right) exhibit strong diagonal dominance, while [CLS] (left) draws attribution broadly across source layers.

Model	Dataset	Tokens	Baseline		Full CLT		Drop Top-1		Keep Top-4
Model	Dataset	Tokens	Acc $\uparrow$	KL $\downarrow$	Acc $\uparrow$	KL $\downarrow$	Acc $\uparrow$	KL $\downarrow$	Acc $\uparrow$	KL $\downarrow$
ViT-B/32	cifar100	all (cls+patch)	61.65%	-	61.31%	0.0104	59.68%	0.0800	59.96%	0.0714
		cls	61.65%	-	61.31%	0.0104	59.91%	0.0704	60.72%	0.0582
	coco	all (cls+patch)	43.12%	-	43.26%	0.0103	42.98%	0.1619	43.02%	0.0156
		cls	43.12%	-	43.26%	0.0103	42.78%	0.1157	43.00%	0.0155
	imagenet100	all (cls+patch)	80.42%	-	80.54%	0.0073	74.94%	0.2320	80.56%	0.0109

Table 4: Faithful attribution via projection-based ablation on the final layer of ViT-B/32. Accuracy (%) and KL divergence from the unmodified logits are reported for [CLS] and all tokens ( [CLS] +patch).

Figure 4: Layerwise visual retrieval using CLT sparse codes of CLS for a test image. Each row shows the top-3 retrieved training samples across transformer layers, revealing the semantic evolution of representations.

Figure 5: Layerwise visual retrieval using CLT sparse codes of Patches for a test image. Each row shows the top-3 retrieved training samples across transformer layers, revealing the semantic evolution of representations.

	Pred. Distrib.			Top-k Agree		Embedding			Prompt Sens.
Layers	$\Delta$ Acc	Flip $\downarrow$	KL $\downarrow$	Top-1	Top-5	Cos	CKA	Spear.	$r$	KL $\downarrow$
0-11 CLS	+0.07	10.4%	.034	89.6	88.5	.984	.948	.929	.995	.036
7-11 CLS	+0.01	9.5%	.030	90.5	89.0	.985	.954	.937	.992	.031
10-11 CLS	$-$ 0.07	7.2%	.018	92.8	91.2	.991	.979	.973	.994	.019
11 CLS	$-$ 0.04	5.4%	.010	94.6	93.2	.996	.991	.991	.997	.010
0-11 Patches	$-$ 11.1	35.2%	.412	64.8	70.0	.957	.817	.759	.942	.383

Table 5: CLT surrogate faithfulness metrics.

Figure 6: Reconstruction performance of CLTs across transformer layers on CIFAR-100 using ViT-B/16. We report cosine similarity (left), MSE per token in log scale (center), and

R^{2}

(right) for JumpReLU , ReLU-Top-

k

, and Abs-Top-

k

sparsity variants (

k{=}128

), averaged across all tokens in the validation set.

Figure 7: Reconstruction performance of CLTs across transformer layers on COCO using ViT-B/32. We show cosine similarity (left), MSE per token (log scale, center), and

R^{2}

(right) for the three sparsity variants JumpReLU , ReLU-Top-

k

, and Abs-Top-

k

(

k{=}128

), averaged across all tokens in the validation set.

Figure 8: Reconstruction performance of CLTs across transformer layers on COCO using ViT-B/16. As in the main paper, we compare JumpReLU , ReLU-Top-

k

, and Abs-Top-

k

(

k{=}128

) with cosine similarity (left), MSE per token (log scale, center), and

R^{2}

(right), averaged over the validation set.

Figure 9: Reconstruction performance of CLTs across transformer layers on ImageNet-100 using ViT-B/32. We plot cosine similarity (left), MSE per token in log scale (center), and

R^{2}

(right) for the three sparsity variants JumpReLU , ReLU-Top-

k

, and Abs-Top-

k

(

k{=}128

), averaged across all tokens in the validation set.

Figure 10: Reconstruction performance of CLTs across transformer layers on ImageNet-100 using ViT-B/16. Cosine similarity (left), MSE per token (log scale, center), and

R^{2}

(right) are reported for JumpReLU , ReLU-Top-

k

, and Abs-Top-

k

sparsity (

k{=}128

), averaged across all tokens in the test set.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	61.33	61.74	61.43
1 $\rightarrow$ 11	61.43	61.76	61.56
2 $\rightarrow$ 11	61.39	61.72	61.61
3 $\rightarrow$ 11	61.38	61.69	61.58
4 $\rightarrow$ 11	61.20	61.76	61.66
5 $\rightarrow$ 11	61.21	61.59	61.48

Table 6: Top-1 classification accuracy (%) on CIFAR-100 for ViT-B/32 with CLS tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 61.65 and ViT-B/16: 65.97.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	49.63	51.12	48.68
1 $\rightarrow$ 11	49.92	51.61	52.40
2 $\rightarrow$ 11	51.95	52.86	53.46
3 $\rightarrow$ 11	53.65	54.20	55.83
4 $\rightarrow$ 11	56.59	57.02	58.86
5 $\rightarrow$ 11	58.89	59.52	61.38

Table 7: Top-1 classification accuracy (%) on CIFAR-100 for ViT-B/32 with Patches tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 61.65 and ViT-B/16: 65.97.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	49.40	51.12	48.41
1 $\rightarrow$ 11	49.90	51.91	51.97
2 $\rightarrow$ 11	51.84	53.51	53.16
3 $\rightarrow$ 11	54.10	54.84	55.69
4 $\rightarrow$ 11	56.62	57.32	58.23
5 $\rightarrow$ 11	59.21	59.96	61.19

Table 8: Top-1 classification accuracy (%) on CIFAR-100 for ViT-B/32 with All tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 61.65 and ViT-B/16: 65.97.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	66.04	65.92	65.38
1 $\rightarrow$ 11	66.05	65.82	65.62
2 $\rightarrow$ 11	66.02	65.78	65.73
3 $\rightarrow$ 11	66.10	65.81	65.39
4 $\rightarrow$ 11	66.05	65.87	65.61
5 $\rightarrow$ 11	66.15	65.80	65.65

Table 9: Top-1 classification accuracy (%) on CIFAR-100 for ViT-B/16 with CLS tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 61.65 and ViT-B/16: 65.97.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	62.40	58.57	56.59
1 $\rightarrow$ 11	62.58	59.40	60.68
2 $\rightarrow$ 11	63.15	60.07	61.97
3 $\rightarrow$ 11	63.54	60.76	63.31
4 $\rightarrow$ 11	64.29	62.32	64.85
5 $\rightarrow$ 11	65.24	63.50	66.11

Table 10: Top-1 classification accuracy (%) on CIFAR-100 for ViT-B/16 with Patches tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 61.65 and ViT-B/16: 65.97.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	62.45	58.82	56.72
1 $\rightarrow$ 11	62.70	59.23	60.30
2 $\rightarrow$ 11	62.95	59.74	61.54
3 $\rightarrow$ 11	63.71	60.75	62.87
4 $\rightarrow$ 11	64.14	62.16	64.57
5 $\rightarrow$ 11	65.20	63.60	65.86

Table 11: Top-1 classification accuracy (%) on CIFAR-100 for ViT-B/16 with All tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 61.65 and ViT-B/16: 65.97.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	43.12	43.36	43.00
1 $\rightarrow$ 11	43.14	43.32	43.08
2 $\rightarrow$ 11	43.10	43.26	42.96
3 $\rightarrow$ 11	43.08	43.22	43.12
4 $\rightarrow$ 11	43.18	43.22	43.06
5 $\rightarrow$ 11	43.22	43.30	43.06

Table 12: Top-1 classification accuracy (%) on COCO for ViT-B/32 with CLS tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 43.12 and ViT-B/16: 43.56.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	39.04	38.94	39.68
1 $\rightarrow$ 11	39.38	39.44	40.02
2 $\rightarrow$ 11	39.54	40.24	40.42
3 $\rightarrow$ 11	40.58	40.92	41.48
4 $\rightarrow$ 11	41.38	41.88	42.10
5 $\rightarrow$ 11	42.06	42.28	42.28

Table 13: Top-1 classification accuracy (%) on COCO for ViT-B/32 with Patches tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 43.12 and ViT-B/16: 43.56.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	38.60	39.12	40.06
1 $\rightarrow$ 11	39.02	39.62	40.54
2 $\rightarrow$ 11	39.98	40.10	40.44
3 $\rightarrow$ 11	41.00	41.34	41.46
4 $\rightarrow$ 11	41.62	41.68	41.92
5 $\rightarrow$ 11	41.88	42.28	42.26

Table 14: Top-1 classification accuracy (%) on COCO for ViT-B/32 with All tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 43.12 and ViT-B/16: 43.56.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	43.62	43.62	42.76
1 $\rightarrow$ 11	43.56	43.62	42.40
2 $\rightarrow$ 11	43.60	43.64	42.34
3 $\rightarrow$ 11	43.68	43.64	42.64
4 $\rightarrow$ 11	43.66	43.72	42.64
5 $\rightarrow$ 11	43.58	43.80	42.98

Table 15: Top-1 classification accuracy (%) on COCO for ViT-B/16 with CLS tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 43.12 and ViT-B/16: 43.56.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	42.72	42.00	35.46
1 $\rightarrow$ 11	42.84	41.98	37.28
2 $\rightarrow$ 11	42.68	42.20	38.36
3 $\rightarrow$ 11	43.00	42.34	39.46
4 $\rightarrow$ 11	43.00	42.96	40.86
5 $\rightarrow$ 11	42.98	43.00	42.58

Table 16: Top-1 classification accuracy (%) on COCO for ViT-B/16 with Patches tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 43.12 and ViT-B/16: 43.56.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	43.00	42.08	34.16
1 $\rightarrow$ 11	42.92	42.08	36.06
2 $\rightarrow$ 11	42.74	42.20	36.82
3 $\rightarrow$ 11	42.78	42.66	38.18
4 $\rightarrow$ 11	42.84	43.10	39.80
5 $\rightarrow$ 11	43.04	43.12	42.22

Table 17: Top-1 classification accuracy (%) on COCO for ViT-B/16 with All tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 43.12 and ViT-B/16: 43.56.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	80.92	80.86	80.26
1 $\rightarrow$ 11	80.84	80.82	80.32
2 $\rightarrow$ 11	80.82	80.86	80.24
3 $\rightarrow$ 11	80.86	80.78	80.16
4 $\rightarrow$ 11	80.80	80.86	80.24
5 $\rightarrow$ 11	80.74	80.86	80.44

Table 18: Top-1 classification accuracy (%) on ImageNet-100 for ViT-B/32 with CLS tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 80.42 and ViT-B/16: 84.34.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	71.96	68.74	60.26
1 $\rightarrow$ 11	72.34	70.10	64.80
2 $\rightarrow$ 11	73.10	72.24	67.54
3 $\rightarrow$ 11	75.58	75.26	72.46
4 $\rightarrow$ 11	77.86	77.18	75.88
5 $\rightarrow$ 11	78.92	79.10	77.54

Table 19: Top-1 classification accuracy (%) on ImageNet-100 for ViT-B/32 with Patches tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 80.42 and ViT-B/16: 84.34.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	71.60	68.90	60.26
1 $\rightarrow$ 11	71.92	70.10	63.82
2 $\rightarrow$ 11	72.88	72.32	66.72
3 $\rightarrow$ 11	75.12	75.36	71.44
4 $\rightarrow$ 11	77.56	77.10	74.68
5 $\rightarrow$ 11	78.74	78.84	77.30

Table 20: Top-1 classification accuracy (%) on ImageNet-100 for ViT-B/32 with All tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 80.42 and ViT-B/16: 84.34.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	84.54	84.02	83.28
1 $\rightarrow$ 11	84.50	84.04	83.52
2 $\rightarrow$ 11	84.44	83.98	83.50
3 $\rightarrow$ 11	84.56	84.08	83.74
4 $\rightarrow$ 11	84.44	84.04	83.82
5 $\rightarrow$ 11	84.42	84.10	84.20

Table 21: Top-1 classification accuracy (%) on ImageNet-100 for ViT-B/16 with CLS tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 80.42 and ViT-B/16: 84.34.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	83.04	81.40	73.06
1 $\rightarrow$ 11	83.00	81.44	76.74
2 $\rightarrow$ 11	83.18	81.88	77.54
3 $\rightarrow$ 11	83.16	82.76	79.36
4 $\rightarrow$ 11	83.56	83.34	80.98
5 $\rightarrow$ 11	83.78	83.88	82.50

Table 22: Top-1 classification accuracy (%) on ImageNet-100 for ViT-B/16 with Patches tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 80.42 and ViT-B/16: 84.34.

Range	JR	RTK	ATK
0 $\rightarrow$ 11	83.12	80.94	70.78
1 $\rightarrow$ 11	83.08	81.58	74.70
2 $\rightarrow$ 11	83.28	81.90	76.30
3 $\rightarrow$ 11	83.00	82.64	78.08
4 $\rightarrow$ 11	83.62	83.08	80.12
5 $\rightarrow$ 11	84.16	83.86	81.76

Table 23: Top-1 classification accuracy (%) on ImageNet-100 for ViT-B/16 with All tokens, across layer ranges

s\rightarrow 11

. JR = JumpReLU, RTK = ReLU-Top-

k

, ATK = Abs-Top-

k

. Baseline top-1 accuracies (in %) are ViT-B/32: 80.42 and ViT-B/16: 84.34.

Panel of Figure 11 — Figure 11: Cross-layer contribution scores $C_{s\rightarrow\ell}$ on CIFAR-100 with ViT-B/32. Columns vary the sparsifier (JumpReLU, ReLU-Top- $k$ , Abs-Top- $k$ ) and rows show CLS (top) and patch tokens (bottom). Each heatmap visualizes the proportional contribution of source layer $s$ to the reconstructed activation at target layer $\ell$ , averaged over the validation set.

Panel of Figure 12 — Figure 12: Cross-layer contribution scores $C_{s\rightarrow\ell}$ on CIFAR-100 with ViT-B/16. Columns vary the sparsifier (JumpReLU, ReLU-Top- $k$ , Abs-Top- $k$ ) and rows show CLS (top) and patch tokens (bottom). Each heatmap visualizes the proportional contribution of source layer $s$ to the reconstructed activation at target layer $\ell$ , averaged over the validation set.

Panel of Figure 13 — Figure 13: Cross-layer contribution scores $C_{s\rightarrow\ell}$ on COCO with ViT-B/32. Columns vary the sparsifier (JumpReLU, ReLU-Top- $k$ , Abs-Top- $k$ ) and rows show CLS (top) and patch tokens (bottom). Each heatmap visualizes the proportional contribution of source layer $s$ to the reconstructed activation at target layer $\ell$ , averaged over the validation set.

Panel of Figure 14 — Figure 14: Cross-layer contribution scores $C_{s\rightarrow\ell}$ on COCO with ViT-B/16. Columns vary the sparsifier (JumpReLU, ReLU-Top- $k$ , Abs-Top- $k$ ) and rows show CLS (top) and patch tokens (bottom). Each heatmap visualizes the proportional contribution of source layer $s$ to the reconstructed activation at target layer $\ell$ , averaged over the validation set.

Panel of Figure 15 — Figure 15: Cross-layer contribution scores $C_{s\rightarrow\ell}$ on ImageNet-100 with ViT-B/32. Columns vary the sparsifier (JumpReLU, ReLU-Top- $k$ , Abs-Top- $k$ ) and rows show CLS (top) and patch tokens (bottom). Each heatmap visualizes the proportional contribution of source layer $s$ to the reconstructed activation at target layer $\ell$ , averaged over the validation set.

Panel of Figure 16 — Figure 16: Cross-layer contribution scores $C_{s\rightarrow\ell}$ on ImageNet-100 with ViT-B/16. Columns vary the sparsifier (JumpReLU, ReLU-Top- $k$ , Abs-Top- $k$ ) and rows show CLS (top) and patch tokens (bottom). Each heatmap visualizes the proportional contribution of source layer $s$ to the reconstructed activation at target layer $\ell$ , averaged over the validation set.

Explore similar work

Mar 22, 2026

CLT-Forge: A Scalable Library for Cross-Layer Transcoders and Attribution Graphs

Mechanistic interpretability seeks to understand how Large Language Models (LLMs) represent and process information. Recent approaches based on dictionary learning and transcoders enable representing model computation in terms of sparse, interpretable features and their interactions, giving rise to feature attribution graphs. However, these graphs are often large and redundant, limiting their interpretability in practice. Cross-Layer Transcoders (CLTs) address this issue by sharing features across layers while preserving layer-specific decoding, yielding more compact representations, but remain difficult to train and analyze at scale. We introduce an open-source library for end-to-end training and interpretability of CLTs. Our framework integrates scalable distributed training with model sharding and compressed activation caching, a unified automated interpretability pipeline for feature analysis and explanation, attribution graph computation using Circuit-Tracer, and a flexible visualization interface. This provides a practical and unified solution for scaling CLT-based mechanistic interpretability. Our code is available at: https://github.com/LLM-Interp/CLT-Forge.

Florent Draye, Vedant Palit, Abir Harrasse +8

Max Planck Institute for Intelligent Systems, Tübingen, Germany · Jinesis AI Lab, University of Toronto & Vector Institute · CMU +2

LLM Interpretability Mechanistic Interpretability Feature Attribution

Ask PDF

Aug 14, 2026

SpIn-ViT: Designing a Sparsity-Induced Vision Transformer That Is Mechanistically Interpretable

Mechanistic interpretability has recently expanded to Vision Transformers (ViTs), with Sparse Autoencoders (SAEs) increasingly used as post-hoc tools to decompose internal representations into sparse and more interpretable features. However, because post-hoc SAEs are trained on frozen representations after the ViT has already been optimized, their latent features are not directly aligned with the downstream classification objective. We introduce SpIn-ViT, a framework that jointly trains a pretrained ViT and a modified SAE end-to-end, directly aligning sparse patch-level representations with image classification. SpIn-ViT learns semantically coherent neuron activations that localize meaningful image regions while maintaining competitive predictive performance. We evaluate SpIn-ViT across nine image-classification benchmarks using classification accuracy, quantitative interpretability metrics, AI-based and Human evaluations. Compared with the previous state-of-the-art post-hoc SAE method, SpIn-ViT achieves 8.84% higher average classification accuracy, an AI-based interpretability score nearly four times as high, and a human-evaluation score more than twice as high. We further extract interpretable rule-sets using the SAE neurons to create neurosymbolic models which achieve 5.97% higher average classification accuracy while requiring a 58.8% smaller rule-set than the neurosymbolic models created from the SOTA post-hoc SAE method.

Philip H. Lee, Parth Padalkar

Independent Researcher · Texas State University

Explainable Image Classification Vision Transformer Mechanistic Interpretability Sparse Autoencoders

Ask PDF

Sep 8, 2026

"World Knowledge" in the Weights: Reading Concept Circuits of Vision Transformers

Vision transformers (ViTs) have achieved remarkable generalization across visual domains, yet little is known about how they internally represent the structure of the world. To address this gap, we use Cross-Layer Transcoders (CLTs) to read concept circuits from ViTs: directed graphs whose nodes correspond to sparse, interpretable concepts and edges capture concept interactions across layers. Our method yields two complementary views of model behavior. The global concept circuit is input-invariant and can be recovered directly from learned cross-layer weights, exposing the reusable "world knowledge" encoded in the model. The instance concept circuit is input-dependent and identifies the concepts and pathways actually used for a specific prediction, enabling faithful example-level explanations. We demonstrate the utility of concept circuits in three ways: (1) Automatic spurious correlation discovery: leveraging the statistics of our global concept circuits to identify shortcut dependencies within the model. (2) Spurious correlation removal: intervening on the instance concept circuit to steer the model towards correct predictions. Empirical results show that our method outperforms existing counterparts by 11.0% on the Waterbird dataset. (3) Model comparison: contrasting the global concept circuits of different foundation models (e.g., CLIP vs. DINO) to reveal how supervision paradigms shape representational structure. Our code is available at https://github.com/deep-real/VisionCLT

Yanlin Chen, Tang Li, Xi Peng

Department of Computer & Information Sciences, University of Delaware, USA · Department of Computer Science, University of Virginia, USA

Transformer Interpretability Vision Foundation Models Vision Transformer Mechanistic Interpretability Concept-Based Explanations Spurious Correlation Robustness

Ask PDF