Transferring Visual Explainability from Self-Explaining to Prediction-Only Vision Transformers via Task Arithmetic
Organizations: STAIR Lab, Chiba Institute of Technology · ZOZO Research
Abstract
In image classification scenarios where both prediction and explanation efficiency are required, self-explaining models that perform both tasks in a single inference are effective. However, for users who already have prediction-only models, training a new self-explaining model from scratch is costly in both labeling and computation. This study proposes a method to transfer the visual explanation capability of self-explaining Vision Transformer (ViT) models learned in a source domain to prediction-only, zero-shot-capable ViT models based on vision-language models (VLMs) in a target domain via task arithmetic, without target-side explanation supervision. The proposed method endows target models with explanation capability by adding an explainability vector induced by explanation supervision in the source domain based on the task arithmetic framework. Experiments on 10 diverse target datasets show that a single explainability vector learned on ImageNet-1k augmented with patch-level explanation supervision transfers consistently, improving explanation quality while largely preserving classification accuracy. Within-model faithfulness evaluation and comparisons with control vectors (e.g., random vectors of the same norm) show that the transferred explanations are faithful to the model's own predictions and that the effect is specific to the explainability vector. Beyond the transfer itself, we further investigate whether transfer success can be anticipated a priori from model-internal features alone, and find that, for a fixed source, a small set of such features carries non-trivial signal about the transfer outcome on held-out targets. Code is available at https://github.com/yuyay/explainability_vector.
Figures & tables
| Param. | Accuracy ( ) | E-RMSE ( ) | Rank corr. ( ) | IoU@1 ( ) | IoU@5 ( ) | IoU@10 ( ) | |
|---|---|---|---|---|---|---|---|
| Zero-shot | |||||||
| Finetuned on ImageNet | |||||||
| Finetuned on ImageNet+X | |||||||
| Finetuned on target | |||||||
| Proposed | |||||||
| Finetuned on target+X |
| Added vector | Acc. | Rank corr. | IoU@1 |
|---|---|---|---|
| Proposed | |||
| No vector | |||
| Equal-norm random | |||
| Front-6 blocks | |||
| Back-6 blocks | |||
| Shuffled supervision |
| Method | E-MSE | IoU@10 | Rank corr. |
|---|---|---|---|
| Lasso | |||
| Ridge |
Appendix figures & tables39 assets
Supplementary material from the paper’s appendix.
Appendix
| Head | Accuracy ( ) | E-RMSE ( ) | Rank corr. ( ) | IoU@1 ( ) | IoU@5 ( ) | IoU@10 ( ) | |
|---|---|---|---|---|---|---|---|
| Separated | 0.0 | ||||||
| Separated | 0.1 | ||||||
| Separated | 0.2 | ||||||
| Separated | 0.3 | ||||||
| Separated | 0.4 | ||||||
| Separated | 0.5 |
| Module | Norm, blocks 1–6 | Norm, blocks 7–12 | Share in blocks 7–12 |
|---|---|---|---|
| LayerNorm before MLP | |||
| LayerNorm before attention | |||
| MLP down-projection | |||
| MLP up-projection | |||
| Attention QKV projection | |||
| Attention output projection |
| Explanation loss | Classification loss | |||||
|---|---|---|---|---|---|---|
| Target | Blocks 7–12 | Blocks 11–12 | Top-5 heads | Blocks 7–12 | Blocks 11–12 | Top-5 heads |
| Cars | ||||||
| GTSRB | ||||||
| RESISC45 | ||||||
| Mean | ||||||
| Uniform | ||||||
| # | Class | Identifier | Definition |
|---|---|---|---|
| 1 | cls-circuit | cos_tr_vs_T_ft_cls | |
| 2 | cls-circuit | jacc_tr_vs_T_ft_cls | |
| 3 | cls-circuit | cos_S_ftstar_exp_vs_T_ft_cls | |
| 4 | -shift | tau_feat_shift_l2_mean | |
| 5 | -shift | tau_feat_shift_l2_std | |
| 6 | -shift | tau_centroid_cos |
| Method | E-MSE | IoU@1 | IoU@5 | IoU@10 | Rank corr. |
|---|---|---|---|---|---|
| Lasso | |||||
| Ridge |
| Method | E-MSE | IoU@1 | IoU@5 | IoU@10 | Rank corr. |
|---|---|---|---|---|---|
| Lasso | |||||
| Ridge |
| Param. | Accuracy ( ) | E-RMSE ( ) | Rank corr. ( ) | IoU@1 ( ) | IoU@5 ( ) | IoU@10 ( ) | |
|---|---|---|---|---|---|---|---|
| Zero-shot | |||||||
| Finetuned on ImageNet | |||||||
| Finetuned on ImageNet+X | |||||||
| Finetuned on target | |||||||
| Proposed | |||||||
| Finetuned on target+X |
| no source dummies (6 features) | with source dummies (6+7) | ||||
|---|---|---|---|---|---|
| Metric | Lasso | Ridge | Lasso | Ridge | |
| E-MSE | |||||
| IoU@1 | |||||
| IoU@5 | |||||
| IoU@10 | |||||
| Rank corr. | |||||
| Lasso | Ridge | |||
|---|---|---|---|---|
| Metric | 95% CI | 95% CI | ||
| E-MSE | ||||
| IoU@1 | ||||
| IoU@5 | ||||
| IoU@10 | ||||
| Rank corr. | ||||
| Metric | Regressor | Pooled | Pair 95% CI | Cluster 95% CI | Note |
|---|---|---|---|---|---|
| E-MSE | Lasso | degenerate | |||
| Ridge | worse than constant | ||||
| IoU@1 | Lasso | degenerate | |||
| Ridge | worse than constant | ||||
| IoU@5 | Lasso | degenerate | |||
| Ridge | worse than constant |
| Stage | Item | Cost |
| Source (one-time) | Kernel SHAP supervision for the ImageNet-1k training set ( M images, ) | GPU-h |
| Training and | GPU-h | |
| Total | GPU-h | |
| Per target | Transfer by parameter addition ( 8 ) (CPU) | s |
| Per image | Proposed (one forward, prediction and explanation) | ms |
| + amortized explainer (two forwards) | ms |
| Dataset ( ) | Attribution | Fid ( ) | Fid ( ) | Ins ( ) | Del ( ) |
|---|---|---|---|---|---|
| Cars (338) | Self-explanation (ours) | ||||
| Kernel SHAP on | |||||
| GT (Kernel SHAP on ) | |||||
| Random | |||||
| CIFAR10 (492) | Self-explanation (ours) | ||||
| Kernel SHAP on |
| Added vector | Acc. ( ) | E-RMSE ( ) | Rank corr. ( ) | IoU@1 ( ) | IoU@10 ( ) |
|---|---|---|---|---|---|
| Proposed ( ) | |||||
| (e) No vector | |||||
| (a) Equal-norm random | |||||
| (b) Front-6 blocks | |||||
| (b) Back-6 blocks | |||||
| (c) Full FT, raw |
| Dataset | Vector | Accuracy ( ) | E-RMSE ( ) | Rank corr. ( ) | IoU@1 ( ) | IoU@10 ( ) |
|---|---|---|---|---|---|---|
| Cars | Proposed ( ) | |||||
| (e) No vector | ||||||
| (a) Equal-norm random | ||||||
| (b) Front-6 blocks | ||||||
| (b) Back-6 blocks | ||||||
| (c) Full FT, raw |
| Dataset | Vector | Accuracy ( ) | E-RMSE ( ) | Rank corr. ( ) | IoU@1 ( ) | IoU@10 ( ) |
|---|---|---|---|---|---|---|
| GTSRB | Proposed ( ) | |||||
| (e) No vector | ||||||
| (a) Equal-norm random | ||||||
| (b) Front-6 blocks | ||||||
| (b) Back-6 blocks | ||||||
| (c) Full FT, raw |
| Method | Explains | Acc. | Fwd. | +Params | Non-random |
|---|---|---|---|---|---|
| Proposed ( ) | finetuned target itself | ||||
| + Kernel SHAP ( ) | |||||
| + amortized ( ) | M | ||||
| TVE [ 57 ] (public checkpoint) | separate linear probe | / | M |
| E-RMSE ( ) | Rank corr. ( ) | IoU@1 ( ) | IoU@10 ( ) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Dataset | ZS | Amort. | Prop. | ZS | Amort. | Prop. | ZS | Amort. | Prop. | ZS | Amort. | Prop. |
| Cars | ||||||||||||
| CIFAR100 | ||||||||||||
| DTD | ||||||||||||
| EuroSAT | ||||||||||||
| GTSRB | ||||||||||||
| Dataset | MAE+AdamW | MAE+L-BFGS | native+AdamW | native+L-BFGS |
|---|---|---|---|---|
| Cars | ||||
| CIFAR10 | ||||
| CIFAR100 | ||||
| DTD | ||||
| EuroSAT | ||||
| GTSRB |
| Dataset | Fid ( ) | Fid ( ) | Ins ( ) | Del ( ) | Probe acc. |
|---|---|---|---|---|---|
| TVE / random | TVE / random | TVE / random | TVE / random | ||
| Cars | / | / | / | / | |
| CIFAR10 | / | / | / | / | |
| CIFAR100 | / | / | / | / | |
| DTD | / | / | / | / | |
| EuroSAT † | / | / | / | / |
| Accuracy ( ) | Rank corr. ( ) | IoU@10 ( ) | E-RMSE ( ) | |
|---|---|---|---|---|
| 0.4 | ||||
| 0.5 | ||||
| 0.6 | ||||
| 0.7 (default) | ||||
| 0.8 | ||||
| 0.9 |
| Explanation-only | Accuracy-constrained | |||
| Held-out target | Rank corr. | IoU@10 | pt | pt |
| Cars | 1.1 | 1.0 | 0.9 | 1.1 |
| CIFAR10 | 1.0 | 1.0 | 0.7 | 0.9 |
| CIFAR100 | 1.1 | 1.0 | 0.7 | 0.9 |
| DTD | 1.0 | 1.0 | 0.8 | 1.0 |
| EuroSAT | 1.1 | 1.0 | 0.7 | 0.9 |
| Digit model | Clothing model | |
|---|---|---|
| Accuracy, with / without | / | / |
| Top- patches in the digit region | ( chance) | |
| Top- patches in the clothing region | ( chance) | |
| Chance rate (digit / clothing region) | / | |
| Own-region rate without (top- ) | ||