Hide and Seek: HIDE Concepts to SEEK explanations for black box models

Bhanu Tokas; Rahul Nair; Rockwell Jackson; Hannah Kerner

cs.CVOct 9, 2026

Hide and Seek: HIDE Concepts to SEEK explanations for black box models

Authors: Bhanu Tokas, Rahul Nair, Rockwell Jackson, Hannah Kerner

Organizations: Arizona State University

Abstract

Interpreting machine learning model predictions is a central challenge, particularly for black-box vision models. Concept-based attribution methods interpret models using human-understandable concepts. Existing concept-based methods can only interpret classification and regression models. They require access to model weights or large concept-annotated datasets, and most methods cannot provide image-level attribution. We propose Hide and Seek (HnS), a new method for concept-level attribution that does not require access to model weights or labeled concept datasets. HnS can interpret a wide range of vision models. Given a concept and image set, HnS retrieves concept-aligned images using a vision-language model, localizes the concept within each image, masks the localized region, and measures attribution from the change in the black-box model's output between the original and masked images. This enables faithful attribution by directly measuring the model's behavior and supporting local attributions. We validated HnS on CelebA and COCO-Stuff. Compared to existing methods, HnS showed better agreement with ground-truth attributions. We demonstrated the applicability of HnS on diverse tasks, including image captioning and satellite image segmentation, where no existing concept attribution method can be used. We provided our anonymous code in the supplementary material.

Figures & tables

Figure 1: Overview of Hide and Seek ( HnS ). For a concept $c$ (smiling), we encode its text prompt (“photo of a person smiling”) into a text vector $v_{c}$ using SigLIP (a vision-language model). We pre-process $v_{c}$ (using modality correction and orthogonalization) to obtain $v_{c}^{*}$ . From an image database, we select a set of images $R_{c}$ that show the highest cosine similarity with $v_{c}^{*}$ . For each image $x_{i}$ in $R_{c}$ , HnS identifies the region $c$ in $x_{i}$ and hides this region to create a masked image $\tilde{x}_{i}$ . We pass $x_{i}$ and $\tilde{x}_{i}$ to our model $M$ to obtain predicted probabilities for a class $y$ (male): $P(x_{i})$ and $P(\tilde{x}_{i})$ , respectively. We measure attribution for the concept $c$ (on class $y$ ) as $P(x_{i})$ - $P(\tilde{x}_{i})$ .

Method	Black Box	Concept Label-Free	Surrogate Free	Local Attribution	Wide Task Support
TCAV ( Kim et al., 2018 )	✗	✗	✓	✓	✗
ConceptSHAP ( Yeh et al., 2020 )	✗	✗	✓	✓	✗
PACE ( Wang et al., 2024 )	✗	✗	✓	✓	✗
PCBM ( Yuksekgonul et al., 2023 )	✓	✓	✗	✗	✗
Hide and Seek (ours)	✓	✓	✓	✓	✓

Table 1: Comparison of Concept Attribution Techniques

	CelebA			COCO-Stuff
Method	ResNet18	ViT	ConvNeXt	ResNet18	ViT	ConvNeXt
TCAV	0.319	0.213	0.192	0.030	0.085	0.117
PCBM (ResNet-18)	0.180	0.164	0.258	0.058	0.018	0.059
PCBM (SigLIP)	0.208	0.197	0.291	0.163	0.211	0.214
PCBM (CLIP-RN50)	0.149	0.142	0.201	0.168	0.199	0.214
HnS (ours)	0.615	0.507	0.421	0.315	0.342	0.257

Table 2: For three models, we reported the correlation (

\rho

) between ground-truth global attribution scores and the global attribution scores from HnS , TCAV, and PCBM. A higher

\rho

value for a method indicates that it shows better agreement with ground-truth attributions.

Panel of Figure 2 — Figure 2: For each class-concept pair, we ranked images by magnitude of local attribution scores of Hide and Seek ( HnS ) and TCAV. For each method, we visualized the top $3$ and bottom $3$ images. (a)-(b) are for CelebA, (c)-(d) are for COCO-Stuff. We show more examples in Appendix I .

Male-centric concepts	Number of models	Female-centric concepts	Number of models
tie	9/10	cake	7/10
toaster	8/10	hair dryer	6/10
backpack	7/10	teddy bear	6/10
sandwich	7/10	carrot	6/10
bottle	6/10	umbrella	6/10

Table 3: Across all models, we identified concepts that frequently occurred in the list of top

20

concepts with the highest positive attribution (male-centric concepts), and in the list of top

20

concepts with the highest negative attribution (female-centric concepts). For instance, tie was a male-centric concept in

9

out of

10

models.

Figure 3: HnS attributions for PRUE, a model used to segment crop fields. For each concept $c$ (e.g., inland waters), we reported three attribution scores: $(1)$ mask window A, with a blue bar ( $c$ is only hidden in window A), $(2)$ mask window B, with a orange bar ( $c$ is only hidden in window B), and $(3)$ mask both, with an green bar ( $c$ is hidden in both windows).

Appendix figures & tables39 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Reusable (encoder-side)	Black-Box-Dependent	Total
Hide and Seek (ours)	232.0s	6.0s	238.0s
TCAV	-	349.5s	349.5s
PCBM (SigLIP)	844.7s	391.7s	1236.4s
PCBM (CLIP)	516.6s	348.1s	864.7s
PCBM (ResNet-18)	615.2s	347.3s	962.5s

Appendix

Table 4: Wall-clock cost with an Nvidia A-100 for different techniques, split into encoder-side cost that is reusable across black-box models and cost that depends on the specific black-box model being explained.

Method	Reusable (encoder-side)	Black-Box-Dependent	Total
Hide and Seek (ours)	338.1s	10.6s	348.7s
TCAV	-	405.8s	405.8s
PCBM (SigLIP)	1187.0s	461.5s	1648.5s
PCBM (CLIP)	678.5s	429.6s	1108.1s
PCBM (ResNet-18)	664.1s	481.6s	1145.7s

Appendix

Table 5: Wall-clock cost with an Nvidia RTX-4090 for different techniques, split into encoder-side cost that is reusable across black-box models and cost that depends on the specific black-box model being explained.

Table 6: Joint ablation of encoder, orthogonalization, and Modality Shift for CelebA dataset with ResNet18 black box classifier.

Table 7: Joint ablation of encoder, orthogonalization, and Modality Shift for COCO-Stuff dataset with ResNet18 black box classifier.

Dataset	SigLIP	CLIP	OpenCLIP-H	Perception
CelebA	0.633	0.409	0.354	0.573
COCO-Stuff	0.318	0.331	0.197	0.245

Appendix

Table 8: Each encoder’s own best setting (over the orthogonalize x modality-correction grid)

Dataset	K=15	K=30	K=50	K=75	K=100
CelebA	0.603	0.607	0.615	0.604	0.594
COCO-Stuff	0.269	0.297	0.315	0.311	0.305

Appendix

Table 9: K (retrieval pool size) ablation.

Figure 4: Spearman $\rho$ reported on CelebA and COCO-stuff dataset with ResNet-18 black box model and SigLIP encoder. We observe that for both datasets, the curve remains relatively flat until $\alpha=1.75$ , after which it declines sharply. As such, we suggest keeping the default $\alpha=1.5$ for real-world usage.

Prompt	CelebA	COCO-Stuff
a photo of concept	0.621	0.308
a photo showing concept	0.634	0.311
an image of concept	0.605	0.306
Ensemble	0.615	0.315

Appendix

Table 10: Prompt ablation: 3 phrasing templates and ensemble, for HnS on CelebA classifier and COCO-Stuff dataset. Class-pooled Spearman

\rho

is reported against ground truth annotations.

Figure 5: Modality Shift: image embeddings (blue) and text embeddings (green) occupy two separate regions, each offset from the origin by its own center, $\mu_{img}$ and $\mu_{text}$ respectively, rather than being centered at zero. For a given concept $C$ , the full image embedding $V_{C,img}$ and text embedding $V_{C,text}$ (solid arrows from the origin) each decompose into their ellipsoid’s center plus a concept-specific offset $V_{C}$ (dotted) that is shared between the two modalities. Because $\mu_{img}\neq\mu_{text}$ , directly comparing $V_{C,img}$ and $V_{C,text}$ conflates this modality offset with the concept-specific signal we actually want, motivating the modality shift correction.

Panel of Figure 6 — Figure 6: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 7 — Figure 7: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 8 — Figure 8: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 9 — Figure 9: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 10 — Figure 10: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 11 — Figure 11: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 12 — Figure 12: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 13 — Figure 13: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 14 — Figure 14: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 15 — Figure 15: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 16 — Figure 16: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 17 — Figure 17: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 18 — Figure 18: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 19 — Figure 19: A random sample of images labeled as each concept, shown to illustrate that the labels are unreliable.

Panel of Figure 20 — Figure 20: The accuracy gap $\Delta(s)$ is shown in the black dotted line (left axis, real scale). Average absolute concept attributions for each method are shown on the right-hand axis, min-max-normalized for visual comparability.

Metric	COCO-Stuff	CelebA
TCAV	-0.42	0.27
PCBM (Resnet-18)	-0.28	-0.84
PCBM (CLIP-RN50)	-0.62	0.71
PCBM (SigLIP)	-0.63	0.66
Hide and Seek (ours)	0.82	0.85

Appendix

Table 11: Pearson correlation between the accuracy gap

\Delta(s)

and mean absolute concept attributions across the 11 shortcut rates (

s\in\{0\%,10\%,\ldots,100\%\}

Method	Attractive	Male	Average
TCAV	0.478	0.160	0.319
PCBM (ResNet-18)	0.131	0.228	0.180
PCBM (CLIP-RN50)	0.267	0.032	0.149
PCBM (SigLIP)	0.320	0.096	0.208
Hide and Seek (UTKFace)	0.501	0.426	0.463
Hide and Seek (LFW)	0.641	0.502	0.571

Appendix

Table 12: Concept attribution agreement with the official-train classifier’s masking ground truth, averaged across the Attractive and Male tasks (previously_used ground truth).

Panel of Figure 21 — Figure 21: Example images from UTKFace (left) and CelebA (right). UTKFace’s crops are tight to the face, leaving little to no visible neck or hairline, so concepts like Bald , Necklace , and Receding Hairline are frequently absent from the frame entirely. CelebA’s native crops are looser, consistently including hair, neck, and shoulders, so these same concepts remain visible.

Panel of Figure 22 — Figure 22: Top-3 (highest $|\mathrm{score}|$ ) and bottom-3 (lowest $|\mathrm{score}|$ ) Male-task images for HnS and TCAV, for each concept shown.

Panel of Figure 23 — Figure 23: Top-3 (highest $|\mathrm{score}|$ ) and bottom-3 (lowest $|\mathrm{score}|$ ) Male-task images for HnS and TCAV, for each concept shown.

Panel of Figure 24 — Figure 24: Top-3 (highest $|\mathrm{score}|$ ) and bottom-3 (lowest $|\mathrm{score}|$ ) Male-task images for HnS and TCAV, for each concept shown.

Panel of Figure 25 — Figure 25: Top-3 (highest $|\mathrm{score}|$ ) and bottom-3 (lowest $|\mathrm{score}|$ ) Male-task images for HnS and TCAV, for each concept shown.

Panel of Figure 26 — Figure 26: Top-3 (highest $|\mathrm{score}|$ ) and bottom-3 (lowest $|\mathrm{score}|$ ) Male-task images for HnS and TCAV, for each concept shown.

Panel of Figure 27 — Figure 27: Top-3 (highest $|\mathrm{score}|$ ) and bottom-3 (lowest $|\mathrm{score}|$ ) images for HnS and TCAV, for each (task class, concept) pair shown.

Panel of Figure 28 — Figure 28: Top-3 (highest $|\mathrm{score}|$ ) and bottom-3 (lowest $|\mathrm{score}|$ ) images for HnS and TCAV, for each (task class, concept) pair shown.

Panel of Figure 29 — Figure 29: Top-3 (highest $|\mathrm{score}|$ ) and bottom-3 (lowest $|\mathrm{score}|$ ) images for HnS and TCAV, for each (task class, concept) pair shown.

Panel of Figure 30 — Figure 30: Top-3 (highest $|\mathrm{score}|$ ) and bottom-3 (lowest $|\mathrm{score}|$ ) images for HnS and TCAV, for each (task class, concept) pair shown.

Panel of Figure 31 — Figure 31: Top-3 (highest $|\mathrm{score}|$ ) and bottom-3 (lowest $|\mathrm{score}|$ ) images for HnS and TCAV, for each (task class, concept) pair shown.

Panel of Figure 32 — Figure 32: Per-model attribution for the top consensus masculine-leaning concept ( tie , left) and top consensus feminine-leaning concept ( cake , right). Bar color shows sign (blue = masculine-leaning, pink = feminine-leaning). 9/10 models agree on tie ’s masculine direction (BakLLaVA is the lone, near-zero exception); 7/10 models agree on cake ’s feminine direction, with the 3 disagreeing models (LLaVA, BakLLaVA, GPT-4o) all showing a positive value instead.

Figure 33: Hide and Seek’s land-cover concept attributions for the FTW field-segmentation model, across the 14 countries with sufficient retrieved locations. Each cell is the mean drop in the model’s predicted field score when that concept’s most-associated region is masked (normalized by 255), colored on a diverging scale: red indicates the concept supports a “field” prediction (e.g., Arable land), blue indicates it suppresses one (e.g., Urban fabric). Concepts are ordered left to right by their mean effect across countries; countries are ordered top to bottom by mean $|\text{effect}|$ . The agriculture-supports/urban-suppresses gradient holds consistently across most countries.

Explore similar work

VisionLogic: Discovering and Grounding Decision-Relevant Visual Concepts
Mar 13, 2025Chuqin Geng, Yuhe Jiang, Li Zhang +5Explainable Artificial Intelligence Concept-Based Explanations
School of Computer Science, McGill University, Montreal, Canada · Department of Computer Science, University of Toronto, Toronto, Canada
Concept-Based Abductive and Contrastive Explanations for Behaviors of Vision Models
May 7, 2026Ronaldo Canizales, Divya Gopinath, Corina Păsăreanu +1Abductive Reasoning Concept-Based Explanations
Colorado State University, Fort Collins CO, USA · KBR Inc., NASA Ames, Moffett Field CA, USA · Carnegie Mellon University, Pittsburgh PA, USA
ConEx: Human-Interpretable Saliency Maps via Concept-Aware Attribution
Oct 3, 2026Yehonatan Elisha, Oren Barkan, Ziv Weiss Haddad +1Concept-Based Explanations Explainable Image Classification
Tel Aviv University · The Open University

Mar 13, 2025, cs.CVJ/K move · Enter open · S save

VisionLogic: Discovering and Grounding Decision-Relevant Visual Concepts

Chuqin Geng, Yuhe Jiang, Li Zhang, Zhaoyue Wang+4

School of Computer Science, McGill University, Montreal, Canada · Department of Computer Science, University of Toronto, Toronto, Canada

Explainable Artificial Intelligence Concept-Based Explanations Explainable Image Classification Visual Grounding Explainability Evaluation

Open paper AskPDF arXiv

Concept-based explanations help users understand vision models through recognizable visual patterns. However, existing methods often rely on correlational signals without directly validating which image cues support prediction-relevant internal features. To this end, we introduce VisionLogic, a post-hoc framework that grounds these features in visual concepts through intervention-based validation. VisionLogic first identifies compact sets of features whose contributions reproduce the model's original prediction. It then represents their activation states as predicates using class-specific thresholds. An iterative refinement procedure grounds these predicates in visual regions through ablation tests. A region is accepted when its removal deactivates the corresponding predicate, linking the feature's numerical role to visual evidence in the input. The same predicates allow us to examine how features are activated, selected, and reused across images and classes. Across CNNs and vision transformers on ImageNet-1k, we find that only a few features are selected to explain each prediction, and frequently active features are not always selected. In a large-scale human evaluation with 465 participants, VisionLogic significantly improves participants' understanding of model behavior over established methods ACE and CRAFT. Code is available at https://github.com/allengeng123/VisionLogic.

Abstract

Explore similar work

VisionLogic: Discovering and Grounding Decision-Relevant Visual Concepts

Concept-Based Abductive and Contrastive Explanations for Behaviors of Vision Models

ConEx: Human-Interpretable Saliency Maps via Concept-Aware Attribution