One of the most pressing challenges in artificial intelligence is to make models more transparent to their users. Recently, explainable artificial intelligence has come up with numerous methods to tackle this challenge. A promising avenue is to use concept-based explanations, that is, high-level concepts instead of plain feature importance scores. Among this class of methods, Concept Activation Vectors (CAVs, Kim et al., 2018) stand out as one of the main protagonists. One interesting aspect of CAVs is that their computation requires sampling random examples from the train set. Therefore, the actual vectors obtained may vary depending on the randomness of this sampling. In this paper, we propose a fine-grained theoretical analysis of CAV construction in order to quantify their variability. Our results, confirmed by experiments on several real-life datasets of four different modalities, point to an universal result: the variance of CAVs declines roughly as 1/N, where N is the number of random examples. Based on this, we give practical recommendations for a resource-efficient application of the method.
Figures & tables
Figure 1: Example visualization of the Tcav score as a function of the number of random examples N on the tabular dataset described in Section 5 , computed from the hidden_layers.0 layer for the sex=Female concept. The line shows the mean Tcav score across independently sampled random sets, and the shaded ribbon shows empirical run-to-run variability. The dashed line marks the neutral Tcav value of 0.5 , which indicates no consistent directional influence of the concept and is the value expected by chance from a random direction.
Figure 2: An overview of the first part of the Tcav operating procedure: 1) Tcav extracts the activation vectors at a specific layer ℓ . 2) The Cav vCℓ , shown by the red arrow, is learned by training a binary linear classifier to differentiate between concept examples and random examples. 3) Tcav calculates the directional derivative SC,k,ℓ(x) to measure how sensitive the model’s class prediction is to the Concept Activation Vector vCℓ .
Figure 3: Mean variability of Cav s for ResNet-50 at ‘layer2‘ as a function of the number of random examples per run ( N ), shown for “striped”-, “zigzagged”-, and “dotted” concepts. Error bars indicate ±1 standard deviation; the y -axis is log-scaled. Variance of Cav s is estimated by taking the sum of per-feature variances across ten independent runs. The dashed line shows a fit of the form y=a/N+b estimated on the average of all concepts.
Figure 4: Mean variance of concept sensitivity scores vs. number N of random samples used for the class “zebra” in the Imagenet classification setting for one input. The y -value shows the geometric mean variance of these concept sensitivity scores ±1 standard deviation, averaged over all r=10 runs for a fixed positive input of a zebra. Log scale cut at ≥3×10−9 so near‑zero bands do not dominate the picture.
Figure 5: Mean variance of Tcav scores at layer ‘layer2‘ vs. the number of random examples per concept set N for “striped”-, “zigzagged”-, and “dotted” concepts on the Imagenet dataset; error bars denote ±1 standard deviation. Y ‑axis clipped at ≥10−4 to reduce distortion from near‑zero standard deviations.
Figure 6: Variance of multi-run Tcav scores in the image setting. We use a fixed number R=2000 of random samples and divide them over a varying number of subsets s , with each subset containing N=R/s samples. To get the variance of the mean over all s Tcav scores, we repeat this r=10 times and calculate the variance for each s over the r runs. The plot shows that the variance of the mean Tcav score decreases as s increases, confirming our theoretical analysis. Finally, we repeat this experiment e=10 times and report the mean variance over those e runs ±1 geometric standard deviation.
Figure 7: Average wall-clock time per concept representation as a function of the number of random reference examples per run ( N ). We compare logistic-regression Cav s ( Logistic ), hinge-loss Cav s ( Hinge ), Difference-of-Means Cav s ( DoM ), and Concept Activation Regions ( CAR ). Shaded bands denote ±1 standard deviation across repeated runs.
Figure 8: Compute-variability trade-off for Cav estimators on tabular data. We show average compute time per Cav vs. the Cav variability across estimators. Each point corresponds to a different random-sample size N averaged over all layers.
Appendix figures & tables65 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Mean variability of Cav s for “striped”, “zigzagged,” and “dotted” as a function of random examples ( N ) for ’layer2’ and ’layer3’ of the Resnet50 model . The classifiers were trained using binary cross-entropy loss . Error bars indicate ±1 SD; the y -axis is log-scaled. We fitted a curve of the form f(N)=a/N+b to it. For layer2 the parameters were a=7.71,b=0.0999 , for layer3 they were a=5.58,b=0.0643 .
Figure 10: Variance of Tcav scores at ’layer2’ and ’layer3’ of the Resnet50 model vs. the number of examples per concept set ( N ) for the visual concepts “striped”, “zigzagged”, and “dotted”. Error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For layer2 the parameters were a=0.0396,b=0.00134 , for layer3 they were a=0.0784,b=5.83×10−4 . For layer2 the y ‑axis was clipped at ≥10−4 .
Figure 11: Mean variability of Cav s for the visual concepts “striped”, “zigzagged”, and “dotted” as a function of the number of random examples per run ( N ). Results are shown for two different layers of the Resnet50 model ( ’layer2’ on the left, ’layer3’ on the right). Error bars indicate ±1 SD; the y -axis is log-scaled. Variance is estimated by the sum of per-feature variances across runs. We fitted a curve of the form f(N)=a/N+b to it. For layer2 the parameters were a=3.01,b=0.488 , for layer3 a=2.12,b=0.538 .
Figure 12: Variance of Tcav scores at ’layer2’ and ’layer3’ of the ResNet50 model vs. the number of examples per concept set ( N ) for the visual concepts “striped”, “zigzagged”, and “dotted”. The underlying Cav s were trained using hinge loss . Error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For layer2 the parameters were a=0.0168,b=0.00307 , for layer3 they were a=0.0507,b=0.00539 . For layer3 the scale is trimmed at ≥10−4 to prevent small‑variance tails collapsing the plot. Y ‑axis clipped at ≥10−6 for ’layer2’ and ≥10−4 for layer3 to reduce distortion from near‑zero standard deviations.
Figure 13: Mean variability of Car s for the visual concepts “striped”, “zigzagged”, and “dotted” as a function of the number of random examples per run ( N ). Results are shown for two different layers of the Resnet50 model ( ’layer2’ on the left, ’layer3’ on the right). Error bars indicate ±1 SD; the y -axis is log-scaled. Variance is estimated by the sum of per-feature variances across five independent runs. We fitted a curve of the form f(N)=a/N+b to it. For layer2 the parameters were a=3.3,b=0.0225 , for layer3 they were a=1.43,b=0.0107 .
Figure 14: Variance of Tcar scores at ’layer2’ and ’layer3’ of the ResNet50 model vs. the number of examples per concept set ( N ) for the visual concepts “striped”, “zigzagged”, and “dotted”. The underlying Car s were trained using non-linear SVM. Error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For layer2 the parameters were a=0.193,b=4.55×10−11 , for layer3 they were a=0.0507,b=0.00539 .
Figure 15: Mean variability of Cav s in dependance of the number of random examples per run N for ResNet50 model . The Cav s were generated using difference-of-means . Error bars indicate ±1 SD; the y -axis is log-scaled. Variance is estimated by the sum of per-feature variances across five independent runs. We fitted a curve of the form f(N)=a/N+b to it. For layer2 the parameters were a=7.24,b=0.0106 , for layer3 they were a=4.13,b=3.18×10−8 .
Figure 16: Variance of Tcav scores at ’layer2’ and ’layer3’ of the ReseNet50 model vs. the number N of random examples used. The underlying Cav s were generated using difference-of-means . Error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For layer2 the parameters were a=0.0429,b=3.29×10−4 , for layer3 they were a=0.0548,b=1.5×10−4 .
Figure 17: Compute cost and stability of Cav generation (see legend for the selected setting, e.g. layer/method). Left: compute time per Cav vs. N . Right: compute time vs. resulting Cav variability (trace variance), illustrating the compute–stability trade-off.
Figure 18: Mean variability of Cav s as a function of the number of random examples per run ( N ) on the UCI Adult dataset . Results are shown for two different hidden layers. Error bars indicate ±1 SD; the y -axis is log-scaled. Variance is estimated by the sum of per-feature variances across ten independent runs. We fitted a curve of the form f(N)=a/N+b to it. For hidden_layers_0 they were 4.75/N+2.18×10−13 for “sex=Female” and 6.74/N+0.281 for “sex=Male”. For hidden_layers_2 they were 6.82/N+0.074 i.e. 8.46/N+0.228 .
Figure 19: Variance of Tcav scores vs. the number of examples per concept set ( N ) for the concepts “male” and “female” on the UCI Adult dataset . Error bars denote ±1 standard deviation.
Figure 20: Mean variability of CAV s as a function of the number of random examples per run ( N ) on the UCI Adult dataset . Results are shown for two different hidden layers. The linear classifiers for CAV generation were trained using hinge loss . Variance is estimated by the sum of per-feature variances across ten independent runs. We fitted a curve of the form f(N)=a/N+b to it. For fc_in the parameters were 1.41/N+0.606 for “sex=Female”, and 2.61/N+0.695 for “sex=Male”. For hidden_layers_0 they were 3.7/N+0.296 i.e. 4.63/N+0.47 .
Figure 21: Variance of TCAV scores vs. the number of examples per concept set ( N ) for the concepts “male” and “female” on the UCI Adult dataset . The underlying CAV s were trained using hinge loss . Error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For fc_in the parameters were a=0.581,b=0.0712 for“sex=Female” and a=0.693,b=0.0808 for“sex=Male”. For hidden_layers_0 they were a=1.53,b=0.0533 i.e. a=1.31,b=0.098 .
Figure 22: Mean variability of CAR s as a function of the number of random examples per run ( N ) on the UCI Adult dataset . Results are shown for two different hidden layers. The CAR s were obtained using a non-linear SVC. Variance is estimated by the sum of per-feature variances across ten independent runs. We fitted a curve of the form f(N)=a/N+b to it. For fc_in the parameters were 2.78/N+0.451 for “sex=Female” and 2.76/N+0.681 for “sex=Male”. For hidden_layers_0 they were 6.03/N+0.172 i.e. 6.37/N+0.412 .
Figure 23: Variance of TCAR scores vs. the number of examples per concept set ( N ) for the concepts “male” and “female” on the UCI Adult dataset . The TCAR scores were obtained following the framework proposed by Crabbé and van der Schaar (2022) . Error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For fc_in the parameters were a=0.121,b=2×10−8 for concept “sex=Female” and a=0.364,b=0.0025 for concept “sex=Male”. For hidden_layers_0 they were a=0.306,b=7.97×10−10 i.e. a=0.769,b=0.00526 .
Figure 24: The linear classifiers for Cav generation were trained using difference-of-means . Error bars indicate ±1 SD; the y -axis is log-scaled. Variance is estimated by the sum of per-feature variances across ten independent runs. We fitted a curve of the form f(N)=a/N+b to it. For fc_in the parameters were 6.05/N+0.0306 for “sex=Female”, 8.44/N+0.209 for “sex=Male”. For hidden_layers_0 they were 4.75/N+2.18×10−13 i.e. 9.19/N+0.105 . Moreover, for hidden_layers_0 we clipped below 3×10−3 to avoid visual distortion from extreme lows.
Figure 25: Mean geometrical variance of Tcav scores vs. the number of examples per concept set ( N ) for the concepts “male” and “female” on the UCI Adult dataset . The underlying Cav s were generated using difference-of-means . Error bars denote ±1 geometrical standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For fc_in the parameters were a=1.06,b=0.0209 , for hidden_layers_0 they were a=1.81,b=3.05×10−4 .
Figure 26: Compute cost and stability of Cav generation (see legend for the selected setting, e.g. layer/method). Left: compute time per Cav vs. N . Right: compute time vs. resulting Cav variability (trace variance), illustrating the compute-stability trade-off.
Figure 27: Mean variability of Cav s as a function of the number of random examples per run ( N ), shown for concepts at two different layers ( bert_encoder_2 on the left, bert_encoder_4 on the right). Error bars indicate ±1 SD; the y -axis is log-scaled. Variance is estimated by the sum of per-feature variances across ten runs. We fitted a curve of the form f(N)=a/N+b to it. For bert_encoder_2 the parameters were a=4.02,b=0.121 , for bert_encoder_4 they were a=4.27,b=0.0958 .
Figure 28: Variance of Tcav scores at layers bert_encoder_2 and bert_encoder_4 vs. the number of examples per concept set ( N ) for “positive”-, “negative”-, and “neutral”-adjective concepts on the IMDB dataset; error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For bert_encoder_2 the parameters were a=0.295,b=0.00654 , for bert_encoder_4 they were a=0.329,b=0.0226 . For bert_encoder_2 we cut the log‑y -axis at ≥10−4 for readability to omit tiny standard deviation values.
Figure 29: Mean variability of Cav s for SmolLM2-1.7B-Instruct as a function of the number of random examples per run ( N ) on the IMDB setting. Results are shown for decoder layers decoder_hidden_6 and decoder_hidden_12 . Error bars indicate ±1 SD; the y -axis is log-scaled.
Figure 30: Variance of Tcav scores for SmolLM2-1.7B-Instruct at decoder layers decoder_hidden_6 and decoder_hidden_12 vs. the number of examples per concept set ( N ) on the IMDB setting. Error bars denote ±1 standard deviation.
Figure 31: Mean variability of Cav s as a function of the number of random examples per run ( N ), shown for concepts at two different layers. Error bars indicate ±1 SD; the y -axis is log-scaled. Variance is estimated by the sum of per-feature variances across ten runs. We fitted a curve of the form f(N)=a/N+b to it. For bert_encoder_2 the parameters were a=2.69,b=0.215 , for bert_encoder_4 they were a=3.02,b=0.298 .
Figure 32: Variance of Tcav scores at layers bert_encoder_2 and bert_encoder_4 vs. the number of examples per concept set ( N ) for “positive”-, “negative”-, and “neutral”-adjective concepts on the IMDB dataset; error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For bert_encoder_2 the parameters were a=0.372,b=0.00391 and cut the log scale at ≥10−4 so near‑zero bands do not dominate. For bert_encoder_4 the fitted parameters were a=0.235 and b=0.0368 .
Figure 33: Mean variability of Cav s for SmolLM2-1.7B-Instruct as a function of the number of random examples per run ( N ) on the IMDB setting. Results are shown for decoder layers decoder_hidden_6 and decoder_hidden_12 . Error bars indicate ±1 SD; the y -axis is log-scaled.
Figure 34: Variance of Tcav scores for SmolLM2-1.7B-Instruct at decoder layers decoder_hidden_6 and decoder_hidden_12 vs. the number of examples per concept set ( N ) on the IMDB setting. Error bars denote ±1 standard deviation.
Figure 35: Mean variability of Car s as a function of the number of random examples per run ( N ), shown for concepts at two different layers. Error bars indicate ±1 SD; the y -axis is log-scaled. Variance is estimated by the sum of per-feature variances across ten runs.
Figure 36: Variance of Tcar scores at layers bert_encoder_2 and bert_encoder_4 vs. the number of examples per concept set ( N ) for “positive”-, “negative”-, and “neutral”-adjective concepts on the IMDB dataset; error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For bert_encoder_2 the parameters were a=0.372,b=0.00391 and cut the log scale at ≥10−4 so near‑zero bands do not dominate. For bert_encoder_4 the fitted parameters were a=0.235 and b=0.0368 .
Figure 37: Mean variability of Car s for SmolLM2-1.7B-Instruct as a function of the number of random examples per run ( N ) on the IMDB setting. Results are shown for decoder layers decoder_hidden_6 and decoder_hidden_12 . Error bars indicate ±1 SD; the y -axis is log-scaled.
Figure 38: Variance of Tcar scores for SmolLM2-1.7B-Instruct at decoder layers decoder_hidden_6 and decoder_hidden_12 vs. the number of examples per concept set ( N ) on the IMDB setting. Error bars denote ±1 standard deviation.
Figure 39: Mean variability of Cav s for the concepts “positive”, “negative” and “neutral” adjectives as a function of the number of random examples per run ( N ). Results are shown for two different hidden layers. The Cav are generated using difference-of-means . Error bars indicate ±1 SD; the y -axis is log-scaled. Variance is estimated by the sum of per-feature variances across ten independent runs. We fitted a curve of the form f(N)=a/N+b to it. For bert_encoder_2 the parameters were a=2.42,b=1.19×10−7 , for bert_encoder_4 they were a=3.27,b=3.24×10−10 .
Figure 40: Variance of Tcav scores vs. the number of examples per concept set ( N ) for the concepts “positive”, “negative” and “neutral” adjectives. The underlying Cav s were generated using difference-of-means . Error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For bert_encoder_2 the parameters were a=0.16,b=2.7×10−4 , for bert_encoder_4 they were a=0.449,b=1.59×10−8 . We clipped below 10−6 respectively 10−4 to avoid visual distortion from extreme lows.
Figure 41: Mean variability of Cav s for SmolLM2-1.7B-Instruct as a function of the number of random examples per run ( N ) on the IMDB setting. Results are shown for decoder layers decoder_hidden_6 and decoder_hidden_12 . Error bars indicate ±1 SD; the y -axis is log-scaled.
Figure 42: Variance of Tcav scores for SmolLM2-1.7B-Instruct at decoder layers decoder_hidden_6 and decoder_hidden_12 vs. the number of examples per concept set ( N ) on the IMDB setting. Error bars denote ±1 standard deviation.
Figure 43: Mean variability of Cav s for the visual concepts “striped”, “zigzagged”, and “dotted” as a function of random examples ( N ) for CLIP hidden layers 4 and 8 . Error bars indicate ±1 SD; the y -axis is log-scaled. We fitted a curve of the form f(N)=a/N+b to it. For the layer vision_hidden_4 we found 6.72/N+0.0788 , for vision_hidden_8 4.81/N+0.0731 .
Figure 44: Variance of Tcav scores at CLIP hidden layers 4 and 8 vs. the number of examples per concept set ( N ) for the visual concepts “striped”, “zigzagged”, and “dotted”. Error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For the layer vision_hidden_4 we found 0.103/N+7.27×10−4 , for vision_hidden_8 we found 0.163/N+0.00133 .
Figure 45: Mean variability of Cav s for LLaVA-1.5 as a function of the number of random examples per run ( N ) for Broden-style visual concepts. Results are shown for vision layers vision_hidden_7 and vision_hidden_15 . Error bars indicate ±1 SD; the y -axis is log-scaled.
Figure 46: Variance of Tcav scores for LLaVA-1.5 at vision layers vision_hidden_7 and vision_hidden_15 vs. the number of examples per concept set ( N ) for Broden-style visual concepts. Error bars denote ±1 standard deviation.
Figure 47: Mean variability of Cav s for the visual concepts “striped”, “zigzagged”, and “dotted” as a function of the number of random examples per run ( N ). Results are shown for CLIP hidden layers 4 and 8 . Error bars indicate ±1 SD; the y -axis is log-scaled. Variance is estimated by the sum of per-feature variances across five independent runs. We fitted a curve of the form f(N)=a/N+b to it.
Figure 48: Variance of Tcav scores at CLIP hidden layers 4 and 8 vs. the number of examples per concept set ( N ) for the visual concepts “striped”, “zigzagged”, and “dotted”. The underlying Cav s were trained using hinge loss . Error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it. For vision_hidden_4 0.122/N+0.00496 , for vision_hidden_8 0.223/N+0.0183 .
Figure 49: Mean variability of Cav s for LLaVA-1.5 as a function of the number of random examples per run ( N ) for Broden-style visual concepts. Results are shown for vision layers vision_hidden_7 and vision_hidden_15 . Error bars indicate ±1 SD; the y -axis is log-scaled.
Figure 50: Variance of Tcav scores for LLaVA-1.5 at vision layers vision_hidden_7 and vision_hidden_15 vs. the number of examples per concept set ( N ) for Broden-style visual concepts. Error bars denote ±1 standard deviation.
Figure 51: Mean variability of Car s for the visual concepts “striped”, “zigzagged”, and “dotted” as a function of the number of random examples per run ( N ). Results are shown for CLIP hidden layers 4 and 8 . Error bars indicate ±1 SD; the y -axis is log-scaled. Variance is estimated by the sum of per-feature variances across five independent runs. We fitted a curve of the form f(N)=a/N+b to it. For vision_hidden_4 2.91/N+0.0168 , for vision_hidden_8 1.09/N+0.0135 .
Figure 52: Variance of Tcar scores at CLIP hidden layers 4 and 8 vs. the number of examples per concept set ( N ) for the visual concepts “striped”, “zigzagged”, and “dotted”. The underlying Car s were trained using a non-linear SVM. Error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it.
Figure 53: Mean variability of Car s for LLaVA-1.5 as a function of the number of random examples per run ( N ) for Broden-style visual concepts. Results are shown for vision layers vision_hidden_7 and vision_hidden_15 . Error bars indicate ±1 SD; the y -axis is log-scaled.
Figure 54: Variance of Tcar scores for LLaVA-1.5 at vision layers vision_hidden_7 and vision_hidden_15 vs. the number of examples per concept set ( N ) for Broden-style visual concepts. Error bars denote ±1 standard deviation.
Figure 55: Mean variability of Cav s in dependence of the number of random examples per run N for the CLIP model at hidden layers 4 and 8 . The Cav s were generated using difference-of-means . Error bars indicate ±1 SD; the y -axis is log-scaled. Variance is estimated by the sum of per-feature variances across five independent runs. We fitted a curve of the form f(N)=a/N+b to it.
Figure 56: Variance of Tcav scores at CLIP hidden layers 4 and 8 vs. the number N of random examples used for the visual concepts “striped”, “zigzagged”, and “dotted”. The underlying Cav s were generated using difference-of-means . Error bars denote ±1 standard deviation. We fitted a curve of the form f(N)=a/N+b to it.
Figure 57: Mean variability of Cav s for LLaVA-1.5 as a function of the number of random examples per run ( N ) for Broden-style visual concepts. Results are shown for vision layers vision_hidden_7 and vision_hidden_15 . Error bars indicate ±1 SD; the y -axis is log-scaled.
Figure 58: Variance of Tcav scores for LLaVA-1.5 at vision layers vision_hidden_7 and vision_hidden_15 vs. the number of examples per concept set ( N ) for Broden-style visual concepts. Error bars denote ±1 standard deviation.
Figure 59: Compute cost and stability of Cav generation (see legend for the selected setting, e.g. layer/method). Left: compute time per Cav vs. N . Right: compute time vs. resulting Cav variability (trace variance), illustrating the compute-stability trade-off.
Figure 60: Concept-set-size ablation for image data at ResNet layer2 . We vary the number M of concept examples while keeping the reference protocol fixed. Cav variability and concept-sensitivity variance decrease as more concept examples are used. Tcav -score variance is less smooth because it depends on the sign of the sensitivity scores.
Figure 61: Concept-set-size ablation for image data at ResNet layer3 . At layer3 , increasing the number of concept examples again stabilizes the Cav direction.
Figure 62: Concept-set-size ablation for tabular data at fc_in . For the input projection layer, increasing M reduces the variability of the learned concept direction. Since the tabular concepts are defined directly from an interpretable attribute, this setting isolates the effect of concept-set size particularly clearly.
Figure 63: Concept-set-size ablation for tabular data at hidden_layers.0 . At the first hidden layer, the same qualitative trend appears. Larger concept sets stabilize the Cav and the sensitivity values, whereas the Tcav score can remain noisy because it aggregates signs rather than magnitudes.
Figure 64: Concept-set-size ablation for text data at bert_encoder_2 . At bert_encoder_2 , increasing the number of adjective concept examples stabilizes the Cav direction. This supports the interpretation that concept-set sampling contributes to Cav variability in a way analogous to reference-set sampling.
Figure 65: Concept-set-size ablation for text data at bert_encoder_4 . At bert_encoder_4 , the same qualitative pattern is visible. The layer-dependent constants differ, but increasing M reduces the variability of the continuous quantities.
Figure 66: Concept-set-size ablation for multimodal data at vision_hidden_4 . At CLIP vision_hidden_4 , larger concept sets reduce Cav variability and sensitivity variance. This shows that the concept-set-size effect also appears in the vision–language setting.
Figure 67: Concept-set-size ablation for multimodal data at vision_hidden_8 . At CLIP vision_hidden_8 , the qualitative stabilization trend remains visible. As in the other modalities, TCAV-score variance is less directly tied to Cav variance because of the hard threshold in the Tcav definition.
Figure 68: Layer ablation. We vary the layer used to compute CAVs in the tabular model. Cav variability decreases with N across all layers, while the constants differ. Tcav score variance is more sensitive to the chosen layer because it depends on the signs of the concept sensitivities.
Figure 69: Normalization ablation. We compare raw activations, max-absolute normalization, and z-score normalization at hidden_layers.0 . The qualitative decrease of Cav variability with N remains visible, but normalization changes the constants and can strongly affect sensitivity variance.
Figure 70: Regularization ablation. We vary the inverse regularization parameter C of the logistic-regression Cav estimator. Smaller C corresponds to stronger regularization. Stronger regularization lowers Cav variability and improves finite-sample stability, while the qualitative dependence on N remains unchanged.
Figure 71: CAV-initialization ablation. We vary the random seed used for Cav initialization. The resulting curves nearly overlap, indicating that initialization has little effect compared to the variability induced by sampling random reference examples.
Figure 72: Optimizer ablation. We compare Adam, AdamW, and SGD for fitting the Cav estimator. All optimizers show the same qualitative stabilization trend with increasing N . The optimizer mainly changes constants and finite-sample behaviour rather than the overall dependence on the number of random reference examples.
Figure 73: Geometric visualization of the surround assumption (Assumption 1 ) in the latent space. The embeddings of the “Male” and “Female” concepts are projected onto their first two principal components. The points are distributed on all sides of the sample mean (red ’X’ ), illustrating that they are not confined to a single half-space. The empirical check is provided in the code.
Concept Activation Vectors (CAV) and the Testing with CAVs (TCAV) method are fundamental tools for concept-based explainability in deep learning, yet their practical utility is limited by the variability across different samples of concept and non-concept examples. We analyze this stochastic nature of CAVs, and identify a fundamental flaw in the standard TCAV score, its instability by using a discontinuous indicator function. To address this, we introduce α-TCAV, a parameterized unified framework that replaces the discontinuous indicator with a smooth function, which recovers TCAV as a limiting case. Under realistic assumptions, we characterize the score distributions and derive principled parameter choices: one matches the mean of its averaged counterpart, Multi-TCAV, with no larger variance using a single CAV; the other provides a calibrated probability that the concept positively influences the model output. Our analysis and experiments provide practical guidance and challenge the established practice of splitting the sampling budget across multiple CAVs.
Ekkehard Schnoor, Jawher Said, Malik Tiomoko +2
Department of Computer Science, Aalto University, Espoo, Finland · Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute, Berlin, Germany · Huawei Noah’s Ark Lab, Huawei Technologies, Paris, France +1
Image similarity underlies many computer vision applications, yet it is often unclear why two images receive a high or low similarity score. Existing explainability methods often rely on gradient-based attribution maps to provide local justifications for similarity. These approaches struggle to provide global insights into what specifically drives similarity in regions of an embedding space, such as texture, shape, or color. We introduce a model- and metric-agnostic framework that explains image similarity using Concept Activation Vectors (CAVs) extracted automatically via Sparse Autoencoders (SAEs). Given a pair of images, we perturb their embeddings along discovered concept directions and measure the resulting change in a chosen similarity function, yielding concept importances. For image pairs, we provide localization with concept attribution maps. We extend this procedure to group-level settings, explaining what drives similarity across a cluster of images rather than a single pair, and further, we introduce Exemplar Retrieval, aiming to recover samples with similar reasons contributing to similarity. Our experiments show that our latent perturbations are more faithful to the underlying data distribution than pixel-space baselines, and that concept importances linearly recover the true similarity score. Qualitative results further confirm the usefulness of our methods in understanding a model's individual and group similarity judgments.
Isaac Roberts, Petra Bevandic, Alexander Schulz +1
Bielefeld University - Faculty of Technology · University of Zagreb - Faculty of Electrical Engineering and Computing
Perturbation-based explainability methods face criticism due to their reliance on out-of-distribution mutants. This raises doubts about the quality of the explanations. In this paper, we introduce a novel forward pass paradigm, Activation-Deactivation (AD), which obviates the need for perturbation of the input. AD replaces perturbation of input features with switching off parts of the model corresponding to to the intended perturbations. We implement ConvAD, an AD approximation algorithm for CNNs. ConvAD is a drop-in mechanism that can be easily added to any trained CNN and, without any additional training, generates more robust and more transferable explanations. We provide evaluation results across multiple architectures, datasets, methods and perturbation strategies, demonstrating the superior quality of ConvAD compared to the SOTA.
Akchunya Chanchal, David A. Kelly, Hana Chockler
Department of Informatics King’s College London Strand, WC2R 2LS, London, UK