Understanding how supervision signals shape semantic visual representations remains challenging, particularly in fine-grained, low-data settings. Standard evaluations of vision models rely on downstream accuracy, which often fails to capture structural differences in learned representations. We propose a compact explainability-driven framework for comparing encoders trained with classification-only and caption-generation supervision on CUB-200. Using a controlled two-stage protocol in which both encoders are ultimately optimized for the same classification task, we enable a direct comparison of the resulting representations. Saliency maps are generated using attention rollout and gradient-weighted rollout and evaluated with five metrics covering faithfulness (positive and negative perturbation) and localization (keypoint emphasis, background attribution, and Earth Mover's Distance). Across random seeds and explanation methods, caption supervision yields representations that are more sensitive to the removal of salient regions, better aligned with annotated object parts, and less reliant on background regions, whereas classification supervision produces representations that are slightly more robust to the removal of low-saliency regions. An ablation study on the Oxford-IIIT Pet dataset shows consistent trends across key metrics. These results demonstrate that the proposed framework makes supervision-induced representational differences observable and quantifiable beyond downstream accuracy alone. More broadly, it provides a systematic basis for assessing learned representations and comparing alternative training and transfer-learning strategies.
Figures & tables
Figure 1: Overview of the proposed two-stage training and evaluation framework. (a) Stage 1 performs task-specific base tuning via either classification or captioning objectives. (b) Stage 2 applies controlled downstream finetuning for fine-grained classification using identical optimization budgets. (c) Explainability evaluation is conducted using multi-criteria metrics including perturbation-based faithfulness and localization measures.
Figure 2: Stage 1 training progress for the classification-tuned encoder. (a) training and validation loss; (b) test accuracy each averaged over three seeds with CI.
Figure 3: Qualitative captioning results under teacher forcing. Captions are color-coded based on maximum token-level F1 score against any of the 10 ground-truth captions (green: good match, orange: partial, red: poor).
Figure 4: Second-stage training progress over iterations averaged over three seeds with CI. Top row: classification-tuned encoder. Bottom row: caption-tuned encoder. Left: loss curves. Right: test accuracy with Stage 1 baseline.
Figure 5: Positive and negative perturbation curves under progressive masking based on saliency maps from two explanation methods. Blue curves correspond to the classification-tuned encoder and red curves to the caption-tuned encoder. Solid lines indicate positive perturbation (descending saliency masking), and dashed lines indicate negative perturbation (ascending saliency masking). Curves are normalized by the unmasked prediction probability.
Figure 6: Violin plots of localization metrics across three random seeds for classification- and caption-tuned encoders. Results are shown for (a) attention rollout and (b) gradient-weighted rollout. Blue violins correspond to the classification-tuned encoder and red violins to the caption-tuned encoder. Points denote seed-wise averages and the central horizontal line indicates the median.
Saliency Method
Metric
Regimen
Mean
Std
95% CI
p-value
Cohen’s d
Grad-Weighted Attention
AUCI+
Classification
0.1762
0.1430
[0.1733, 0.1793]
5.33×10−307
0.388
Captioning
0.1328
0.1074
[0.1305, 0.1351]
AUCI−
Classification
0.7716
0.1844
[0.7675, 0.7754]
<10−308
0.405
Captioning
0.7048
0.2032
[0.7004, 0.7091]
Fbg
Classification
0.6854
0.1072
[0.6831, 0.6877]
<10−308
0.741
Captioning
0.6144
0.1115
[0.6120, 0.6168]
Table 1: Comparison between Classification- and Captioning-tuned encoder training regimens using Wilcoxon signed-rank tests ( n=8646 paired samples per metric). Effect sizes reported as paired Cohen’s d . For the five metrics, expected directions are: AUCI+ ↓; AUCI− ↑; Fbg ↓; Rkp ↑; DEMD ↓.
Figure 7: Saliency map visualizations for classification- and caption-tuned models. Each row corresponds to one test image. Column 1 shows the original image with the true label. Columns 2–5 show saliency overlays: classification-tuned and caption-tuned encoders both under attention rollout and gradient-weighted rollout. Predicted labels are shown above each heatmap (green = correct, red = incorrect). A shared colorbar indicates saliency intensity.
Classification
Captioning
Saliency Method
Metric
(Best accuracy:
(Best accuracy:
Δ
p
d
93.83%±0.0098 )
92.37%±0.0071 )
Grad-Weighted Attention
AUCI+↓
0.3747±0.2092
0.3118±0.1829
−0.0630
2.28×10−99
0.436
AUCI−↑
0.7744±0.1570
0.7571±0.1601
−0.0173
1.83×10−31
0.124
Fbg↓
0.5278±0.1600
0.4793±0.1515
−0.0485
1.16×10−93
0.559
Attention Rollout
AUCI+↓
0.3249±0.2184
0.2785±0.1836
−0.0464
6.65×10−71
0.383
Table 2: Ablation on Oxford-IIIT Pet comparing classification- and caption-tuned encoders. Each cell reports mean ± standard deviation over paired test samples. Δ denotes Captioning − Classification. Arrows indicate the preferred direction.
Classification
Captioning
Saliency Method
Metric
(Best accuracy:
(Best accuracy:
Δ
p
d
87.05% )
84.52% )
Grad-Weighted Rollout
AUCI+↓
0.1430±0.1222
0.1272±0.0944
−0.0158
2.89×10−11
0.175
AUCI−↑
0.7688±0.1919
0.7124±0.2149
−0.0563
1.64×10−144
0.313
Fbg↓
0.6528±0.0925
0.5942±0.1080
−0.0585
8.19×10−212
0.674
Rkp↑
1.2035±0.3420
1.2700±0.4101
0.0665
9.52×10−21
−0.187
Table 3: Encoder-scale ablation on CUB-200 using ViT-L/16. Each cell reports mean ± standard deviation over n=2882 paired test samples. Δ denotes Captioning − Classification. Arrows indicate the preferred direction.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Illustration of CUB-200-2011 dataset annotations, including object masks and part locations.
Figure 9: Saliency-based perturbation protocol. Positive perturbation removes the most-salient regions first, producing a rapid drop in model confidence if the explanation is faithful. Negative perturbation removes the least-salient regions first, preserving task-relevant evidence and resulting in comparatively stable confidence.
Figure 10: Illustration of the Keypoint Emphasis Ratio ( Rkp ), which quantifies the relative concentration of saliency around annotated object keypoints. Each row shows one example. Column 1 presents the original image with its ground-truth label. Columns 2–5 display saliency overlays for classification- and caption-tuned encoders under attention rollout and gradient-weighted rollout. Green circles denote annotated keypoints with a scale-adaptive radius (spatial extent 0.25 ) used to compute Rkp . The white contour defines the object boundary. Red indicates high attribution intensity and blue indicates low attribution intensity.
Figure 11: Example Gaussianized keypoint density map generated from annotated object parts. Individual keypoints are represented by Gaussian kernels and masked using the object segmentation.
Stage
Training Regimen
Iteration
Test Accuracy (mean ± 95% CI)
Stage 1
V1: Classification-base training (used for Stage 2A)
Visual representation learning develops image and video features that capture useful structure for recognition, retrieval, spatial reasoning, and action.