Ensuring trustworthiness in open-world visual recognition requires models that are interpretable, fair, and robust to distribution shifts. Yet modern vision systems are increasingly deployed as proprietary black-box APIs, exposing only output probabilities and hiding architecture, parameters, gradients, and training data. This opacity prevents meaningful auditing, bias detection, and failure analysis. Existing explanation methods assume white- or gray-box access or knowledge of the training distribution, making them unusable in these real-world settings. We introduce UNBOX, a framework for class-wise model dissection under fully data-free, gradient-free, and backpropagation-free constraints. UNBOX leverages Large Language Models and text-to-image diffusion models to recast activation maximization as a purely semantic search driven by output probabilities. The method produces human-interpretable text descriptors that maximally activate each class, revealing the concepts a model has implicitly learned, the training distribution it reflects, and potential sources of bias. We evaluate UNBOX on ImageNet-1K, Waterbirds, and CelebA through semantic fidelity tests, visual-feature correlation analyses and slice-discovery auditing. Despite operating under the strictest black-box constraints, UNBOX performs competitively with state-of-the-art white-box interpretability methods. This demonstrates that meaningful insight into a model's internal reasoning can be recovered without any internal access, enabling more trustworthy and accountable visual recognition systems.
Figures & tables
Figure 1: Overview of the UNBOX framework. Given a target class j and a black-box classifier ϕ , UNBOX reformulates activation maximization as a semantic optimization process carried out entirely in text space. The method iterates through three main components. (1) Prompt Evaluation (left): a natural-language prompt pt is rendered into an image xt=G(pt) by a text-to-image diffusion model, which is then evaluated by the classifier to obtain the target-class score st=ϕ(xt)j . (2) Optimization Context (center): the scalar feedback st is transformed into a semantic guidance signal through trend Tt and intensity It , which together define the textual loss St . This module also maintains a Local Optimization Context Ct capturing recent prompt–feedback pairs, and a Global Optimization Context (Pbest,Dbest) that accumulates high-scoring prompts and persistent lexical concepts. (3) Textual Optimization (right): two cooperating LLM agents perform prompt refinement: the Feedback Agent Af interprets the semantic signal and optimization context to generate structured feedback et , which the Updater Agent Au applies to produce the next prompt pt+1 . The loop repeats until convergence, yielding a ranked set of human-interpretable descriptors that characterize the target class under strict black-box constraints.
Figure 2: Excerpts of the system prompts for Af and Au , along with the look-up table used to retrieve the text associated with Tst and Ist for the formulation of St .
Method
W
D
RN50
ViT
WWW Ahn et al. (2024)
✓
✓
0.60
0.40
CLIPDissect Oikarinen and Weng (2023)
✓
✓
0.77
0.27
DEXTER Carnemolla et al. (2025)
✓
✗
0.47±0.03
0.34±0.01
UNBOX
✗
✗
0.64±0.09
0.49±0.04
Random
✗
✗
0.03
0.03
Table 1: Class semantic fidelity evaluation: similarity between generated descriptors and ImageNet-1K class labels on three architectures. Columns W and D denote access to model weights and data, respectively. Higher values indicate stronger alignment with the ground-truth class concept. As DEXTER and UNBOX are non-deterministic, results are averaged over three runs.
Method
W
D
RN50
ViT
WWW Ahn et al. (2024)
✓
✓
0.43
0.27
CLIPDissect Oikarinen and Weng (2023)
✓
✓
0.73
0.30
DEXTER Carnemolla et al. (2025)
✓
✗
0.49±0.01
0.34±0.01
UNBOX
✗
✗
0.63±0.04
0.45±0.03
Random
✗
✗
0.03
0.03
Table 2: Alignment between generated descriptors and data-derived semantic attributes. Scores quantify the similarity between each method’s descriptors and keywords extracted from captioned training images, reflecting how well the recovered semantics match the distribution the classifier was trained on. Columns W and D denote access to model weights and data, respectively. Higher values indicate stronger data-grounded semantic recovery. As DEXTER and UNBOX are non-deterministic, results are averaged over three runs.
Method
W
D
RN50
ViT
GT (Labels)
-
-
0.90
0.90
GT (Descr.)
✗
✓
0.76
0.76
WWW Ahn et al. (2024)
✓
✓
0.66
0.63
CLIPDissect Oikarinen and Weng (2023)
✓
✓
0.85
0.36
DEXTER Carnemolla et al. (2025)
✓
✗
0.42±0.01
0.36±0.02
UNBOX
✗
✗
0.56±0.04
0.43±0.04
Table 3: Visual grounding evaluation on ImageNet-1K. Generated descriptors are used as textual queries over validation images to assess how strongly they correspond to concrete visual evidence. Columns W and D denote access to model weights and data, respectively. Ground-truth labels (GT) serve as an upper bound. As DEXTER and UNBOX are non-deterministic, results are averaged over three runs.
Waterbirds
CelebA
Method
Weights
Data
GT
Worst
Avg
Worst
Avg
ERM
✓
✓
✗
62.6 ± 0.3
97.3
47.7 ± 2.1
94.9
LfF Nam et al. (2020)
✓
✓
✗
78.0
91.2
77.2
85.1
GEORGE Sohoni et al. (2020)
✓
✓
✗
76.2 ± 2.0
95.7
54.9 ± 1.9
94.6
JTT Liu et al. (2021)
✓
✓
✗
83.8 ± 1.2
89.3
81.5 ± 1.7
88.1
CNC Zhang et al. (2022)
✓
✓
✗
88.5 ± 0.3
90.9
88.8 ± 0.9
89.9
Table 4: Comparison of slice discovery and debiasing performance on the Waterbirds and CelebA benchmarks. Metrics include worst-group ( Worst ) and average ( Avg ) accuracy.
Pbest,Dbest
It
DL
DD
DI
✗
✗
0.50
0.30
0.42
✗
✓
0.50
0.60
0.40
✓
✓
0.90
0.90
0.87
Table 5: Ablation study evaluating the contributions of the Global Optimization Context (Pbest,Dbest) and the intensity state It to the overall performance.
Figure 3: Qualitative examples of spurious feature attribution revealed by UNBOX. For each class, we show generated images that activate the target neuron, the corresponding GradCAM visualizations, and the highest-scoring lexical units in Dbest . These examples illustrate how non-causal cues (e.g., contextual elements, co-occurring objects, or geometric patterns) can dominate the classifier’s decision process.
Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.
Interpreting machine learning model predictions is a central challenge, particularly for black-box vision models. Concept-based attribution methods interpret models using human-understandable concepts. Existing concept-based methods can only interpret classification and regression models. They require access to model weights or large concept-annotated datasets, and most methods cannot provide image-level attribution. We propose Hide and Seek (HnS), a new method for concept-level attribution that does not require access to model weights or labeled concept datasets. HnS can interpret a wide range of vision models. Given a concept and image set, HnS retrieves concept-aligned images using a vision-language model, localizes the concept within each image, masks the localized region, and measures attribution from the change in the black-box model's output between the original and masked images. This enables faithful attribution by directly measuring the model's behavior and supporting local attributions. We validated HnS on CelebA and COCO-Stuff. Compared to existing methods, HnS showed better agreement with ground-truth attributions. We demonstrated the applicability of HnS on diverse tasks, including image captioning and satellite image segmentation, where no existing concept attribution method can be used. We provided our anonymous code in the supplementary material.
Interpreting the decisions of deep image classifiers remains challenging, particularly in black-box settings where model internals are inaccessible. We introduce OCCAM, a framework for open-set causal concept explanation and ontology induction in vision models. OCCAM discovers visual concepts in an open-set manner, localizes them via text-guided segmentation, and performs object-level interventions by removing concepts to measure changes in class confidence, estimating each concept's causal contribution. Beyond local explanations, OCCAM aggregates interventional evidence across a dataset to induce a structured concept ontology that captures how classifiers globally organize visual concepts. Reasoning over this ontology reveals consistent dependencies between concepts, exposes latent causal relations, and uncovers systematic model biases. Experiments on Broden and ImageNet-S across multiple classifiers show that OCCAM improves explanation quality in open-set black-box settings while providing richer global insight than per-image attribution methods.
Chiara Maria Russo, Simone Carnemolla, Simone Palazzo +3
Interpreting the decisions of complex computer vision models is crucial to establish trust and accountability, especially in safety-critical domains. An established approach to interpretability is generating visual attribution maps that highlight regions of the input most relevant to the model's prediction. However, existing methods face a three-way trade-off. Propagation-based approaches are efficient, but they can be biased and architecture-specific. Meanwhile, perturbation-based methods are causally grounded, yet they are expensive and for vision transformers often yield coarse, patch-level explanations. Learning-based explainers are fast but usually optimize surrogate objectives or distill from heuristic teachers. We propose a learning scheme that instead optimizes deletion and insertion metrics directly. Since these metrics depend on non-differentiable sorting and ranking, we frame them as permutation learning and replace the hard sorting with a differentiable relaxation using Gumbel-Sinkhorn. This enables end-to-end training through attribution-guided perturbations of the target model. During inference, our method produces dense, pixel-level attributions in a single forward pass with optional, few-step gradient refinement. Our experiments demonstrate consistent quantitative improvements and sharper, boundary-aligned explanations, particularly for transformer-based vision models. Code and pretrained models are available at https://github.com/dschinagl/AHA.
David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch +2
Institute of Visual Computing (IVC), Graz University of Technology, Graz, Austria · Nomadic AI, USA · Amazon, USA