UNBOX: Unveiling Black-box visual models with Natural-language

Simone Carnemolla; Chiara Russo; Simone Palazzo; Quentin Bouniot; Daniela Giordano; Zeynep Akata; Matteo Pennisi; Concetto Spampinato

cs.CVMar 9, 2026

UNBOX: Unveiling Black-box visual models with Natural-language

Authors: Simone Carnemolla, Chiara Russo, Simone Palazzo, Quentin Bouniot, Daniela Giordano, Zeynep Akata, Matteo Pennisi, Concetto Spampinato

Organizations: University of Catania. · Technical University of Munich. · Helmholtz Munich.

Abstract

Ensuring trustworthiness in open-world visual recognition requires models that are interpretable, fair, and robust to distribution shifts. Yet modern vision systems are increasingly deployed as proprietary black-box APIs, exposing only output probabilities and hiding architecture, parameters, gradients, and training data. This opacity prevents meaningful auditing, bias detection, and failure analysis. Existing explanation methods assume white- or gray-box access or knowledge of the training distribution, making them unusable in these real-world settings. We introduce UNBOX, a framework for class-wise model dissection under fully data-free, gradient-free, and backpropagation-free constraints. UNBOX leverages Large Language Models and text-to-image diffusion models to recast activation maximization as a purely semantic search driven by output probabilities. The method produces human-interpretable text descriptors that maximally activate each class, revealing the concepts a model has implicitly learned, the training distribution it reflects, and potential sources of bias. We evaluate UNBOX on ImageNet-1K, Waterbirds, and CelebA through semantic fidelity tests, visual-feature correlation analyses and slice-discovery auditing. Despite operating under the strictest black-box constraints, UNBOX performs competitively with state-of-the-art white-box interpretability methods. This demonstrates that meaningful insight into a model's internal reasoning can be recovered without any internal access, enabling more trustworthy and accountable visual recognition systems.

Figures & tables

Figure 1: Overview of the UNBOX framework. Given a target class j and a black-box classifier $\phi$ , UNBOX reformulates activation maximization as a semantic optimization process carried out entirely in text space. The method iterates through three main components. (1) Prompt Evaluation (left): a natural-language prompt $p_{t}$ is rendered into an image $x_{t}=G(p_{t})$ by a text-to-image diffusion model, which is then evaluated by the classifier to obtain the target-class score $s_{t}=\phi(x_{t})_{j}$ . (2) Optimization Context (center): the scalar feedback $s_{t}$ is transformed into a semantic guidance signal through trend $T_{t}$ and intensity $I_{t}$ , which together define the textual loss $\mathcal{S}_{t}$ . This module also maintains a Local Optimization Context $C_{t}$ capturing recent prompt–feedback pairs, and a Global Optimization Context $(P_{\text{best}},D_{\text{best}})$ that accumulates high-scoring prompts and persistent lexical concepts. (3) Textual Optimization (right): two cooperating LLM agents perform prompt refinement: the Feedback Agent $A_{f}$ interprets the semantic signal and optimization context to generate structured feedback $e_{t}$ , which the Updater Agent $A_{u}$ applies to produce the next prompt $p_{t+1}$ . The loop repeats until convergence, yielding a ranked set of human-interpretable descriptors that characterize the target class under strict black-box constraints.

Figure 2: Excerpts of the system prompts for $A_{f}$ and $A_{u}$ , along with the look-up table used to retrieve the text associated with $T_{s_{t}}$ and $I_{s_{t}}$ for the formulation of $\mathcal{S}_{t}$ .

Method	W	D	RN50	ViT
WWW Ahn et al. (2024)	✓	✓	0.60	0.40
CLIPDissect Oikarinen and Weng (2023)	✓	✓	0.77	0.27
DEXTER Carnemolla et al. (2025)	✓	✗	$0.47\pm 0.03$	$0.34\pm 0.01$
UNBOX	✗	✗	$0.64\pm 0.09$	$0.49\pm 0.04$
Random	✗	✗	0.03	0.03

Table 1: Class semantic fidelity evaluation: similarity between generated descriptors and ImageNet-1K class labels on three architectures. Columns W and D denote access to model weights and data, respectively. Higher values indicate stronger alignment with the ground-truth class concept. As DEXTER and UNBOX are non-deterministic, results are averaged over three runs.

Method	W	D	RN50	ViT
WWW Ahn et al. (2024)	✓	✓	0.43	0.27
CLIPDissect Oikarinen and Weng (2023)	✓	✓	0.73	0.30
DEXTER Carnemolla et al. (2025)	✓	✗	$0.49\pm 0.01$	$0.34\pm 0.01$
UNBOX	✗	✗	$0.63\pm 0.04$	$0.45\pm 0.03$
Random	✗	✗	0.03	0.03

Table 2: Alignment between generated descriptors and data-derived semantic attributes. Scores quantify the similarity between each method’s descriptors and keywords extracted from captioned training images, reflecting how well the recovered semantics match the distribution the classifier was trained on. Columns W and D denote access to model weights and data, respectively. Higher values indicate stronger data-grounded semantic recovery. As DEXTER and UNBOX are non-deterministic, results are averaged over three runs.

Method	W	D	RN50	ViT
GT (Labels)	-	-	0.90	0.90
GT (Descr.)	✗	✓	0.76	0.76
WWW Ahn et al. (2024)	✓	✓	0.66	0.63
CLIPDissect Oikarinen and Weng (2023)	✓	✓	0.85	0.36
DEXTER Carnemolla et al. (2025)	✓	✗	$0.42\pm 0.01$	$0.36\pm 0.02$
UNBOX	✗	✗	$0.56\pm 0.04$	$0.43\pm 0.04$

Table 3: Visual grounding evaluation on ImageNet-1K. Generated descriptors are used as textual queries over validation images to assess how strongly they correspond to concrete visual evidence. Columns W and D denote access to model weights and data, respectively. Ground-truth labels (GT) serve as an upper bound. As DEXTER and UNBOX are non-deterministic, results are averaged over three runs.

				Waterbirds		CelebA
Method	Weights	Data	GT	Worst	Avg	Worst	Avg
ERM	✓	✓	✗	62.6 $\pm$ 0.3	97.3	47.7 $\pm$ 2.1	94.9
LfF Nam et al. (2020)	✓	✓	✗	78.0	91.2	77.2	85.1
GEORGE Sohoni et al. (2020)	✓	✓	✗	76.2 $\pm$ 2.0	95.7	54.9 $\pm$ 1.9	94.6
JTT Liu et al. (2021)	✓	✓	✗	83.8 $\pm$ 1.2	89.3	81.5 $\pm$ 1.7	88.1
CNC Zhang et al. (2022)	✓	✓	✗	88.5 $\pm$ 0.3	90.9	88.8 $\pm$ 0.9	89.9

Table 4: Comparison of slice discovery and debiasing performance on the Waterbirds and CelebA benchmarks. Metrics include worst-group ( Worst ) and average ( Avg ) accuracy.

$P_{best},D_{best}$	$I_{t}$	DL	DD	DI
✗	✗	0.50	0.30	0.42
✗	✓	0.50	0.60	0.40
✓	✓	0.90	0.90	0.87

Table 5: Ablation study evaluating the contributions of the Global Optimization Context

(P_{best},D_{best})

and the intensity state

I_{t}

to the overall performance.

Figure 3: Qualitative examples of spurious feature attribution revealed by UNBOX. For each class, we show generated images that activate the target neuron, the corresponding GradCAM visualizations, and the highest-scoring lexical units in $D_{\text{best}}$ . These examples illustrate how non-causal cues (e.g., contextual elements, co-occurring objects, or geometric patterns) can dominate the classifier’s decision process.

Neural Network Interpretability

Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.

See all 326 papers

Hide and Seek: HIDE Concepts to SEEK explanations for black box models
Oct 9, 2026Bhanu Tokas, Rahul Nair, Rockwell Jackson +1Interpretable ML Concept-Based Explanations
Arizona State University
OCCAM: Open-set Causal Concept explAnation and Ontology induction for black-box vision Models
May 18, 2026Chiara Maria Russo, Simone Carnemolla, Simone Palazzo +3Explainable Artificial Intelligence Concept-Based Explanations
University of Catania Catania, Italy
Learn to Rank: Visual Attribution by Learning Importance Ranking
Apr 7, 2026David Schinagl, Christian Fruhwirth-Reisinger, Alexander Prutsch +2Feature Attribution Explainable Image Classification
Institute of Visual Computing (IVC), Graz University of Technology, Graz, Austria · Nomadic AI, USA · Amazon, USA

Oct 9, 2026J/K move · Enter open · S save

Hide and Seek: HIDE Concepts to SEEK explanations for black box models

Bhanu Tokas, Rahul Nair, Rockwell Jackson, Hannah Kerner

Arizona State University

Interpretable ML Concept-Based Explanations

Open paper AskPDF arXiv

Interpreting machine learning model predictions is a central challenge, particularly for black-box vision models. Concept-based attribution methods interpret models using human-understandable concepts. Existing concept-based methods can only interpret classification and regression models. They require access to model weights or large concept-annotated datasets, and most methods cannot provide image-level attribution. We propose Hide and Seek (HnS), a new method for concept-level attribution that does not require access to model weights or labeled concept datasets. HnS can interpret a wide range of vision models. Given a concept and image set, HnS retrieves concept-aligned images using a vision-language model, localizes the concept within each image, masks the localized region, and measures attribution from the change in the black-box model's output between the original and masked images. This enables faithful attribution by directly measuring the model's behavior and supporting local attributions. We validated HnS on CelebA and COCO-Stuff. Compared to existing methods, HnS showed better agreement with ground-truth attributions. We demonstrated the applicability of HnS on diverse tasks, including image captioning and satellite image segmentation, where no existing concept attribution method can be used. We provided our anonymous code in the supplementary material.

Abstract

Neural Network Interpretability

Explore similar work

Hide and Seek: HIDE Concepts to SEEK explanations for black box models

OCCAM: Open-set Causal Concept explAnation and Ontology induction for black-box vision Models

Learn to Rank: Visual Attribution by Learning Importance Ranking