LINE: LLM-based Iterative Neuron Explanations for Vision Models

Vladimir Zaigrajew; Michał Piechota; Gaspar Sekula; Paweł Gelar; Przemysław Biecek

cs.CVApr 9, 2026

LINE: LLM-based Iterative Neuron Explanations for Vision Models

Authors: Vladimir Zaigrajew, Michał Piechota, Gaspar Sekula, Paweł Gelar, Przemysław Biecek

Organizations: Centre for Credible AI Warsaw University of Technology · Warsaw University of Technology Warsaw, Poland · Centre for Credible AI, Warsaw University of Technology University of Warsaw, Poland

Abstract

Interpreting individual neurons in deep neural networks is a crucial step towards understanding their complex decision-making processes and ensuring AI safety. Despite recent progress in neuron labeling, existing methods often limit the search space to predefined concept vocabularies or produce overly specific descriptions that fail to capture higher-order, global concepts. We introduce LINE, a novel, training-free iterative approach tailored for open-vocabulary concept labeling in vision models. Operating in a strictly black-box setting, LINE leverages a large language model and a text-to-image generator to iteratively propose and refine concepts in a closed loop, guided by activation history. LINE achieves state-of-the-art performance across multiple model architectures, yielding AUC improvements of up to 0.11 on ImageNet and 0.05 on Places365, while discovering, on average, 27% of new concepts missed by predefined vocabularies. Beyond identifying the top concept, LINE provides a complete generation history, enabling polysemanticity evaluation and producing visual explanations that rival gradient-dependent activation maximization methods. The source code will be made available soon.

Figures & tables

Figure 1: Qualitative comparison of neuron descriptions. We show the top four activating images from the ImageNet for three selected neurons from the DINOv3 norm layer ( left ) and the ResNet50 avgpool layer ( right ). The descriptions are provided from our method ( LINE ), CLIP-Dissect ( Oikarinen and Weng, 2023 ) , and INVERT ( Bykov et al., 2024 ) alongside their CoSy benchmark ( Kopf et al., 2024 ) AUC scores. The best method according to the benchmark AUC score is bold for each neuron. Extended results for different layers and models are reported in Appendix E .

Figure 2: Overview of the LINE iterative framework. Step 1: An LLM proposes a new concept $t$ based on the descriptions from the scoreboard $\mathcal{H}$ (e.g., proposing strawberry and pomegranate into red fruit ). Step 2: A Text-to-Image (T2I) model generates a batch of diverse synthetic images illustrating the concept $t$ . Step 3: These images are processed by the target vision model, extracting concept activations $A_{t}$ . Step 4: A scoring function ( $\psi_{avg}(A)$ ) converts $A_{t}$ into score $s_{t}$ , and the the $\mathcal{H}$ is updated with the result. Upon reaching the final iteration $N$ , an additional final summary iteration ( $N+1$ ) evaluates the higher-order global concept reasoned from top-scoreboard descriptions; here, Step 1 is replaced by a Summary Concept step, while the remaining steps are unchanged. The top-scoring concept from $\mathcal{H}$ is then returned. The LINE algorithm is summarized in Appendix C.1 .

Model	Dataset	Metric	LINE	CLIP-Dissect	INVERT	DnD
ResNet50	ImageNet	AUC	$\mathbf{0.96}_{\pm 0.08}$	$0.87_{\pm 0.18}$	$\underline{0.88}_{\pm 0.15}$	$0.85_{\pm 0.18}$
ResNet50	ImageNet	MAD	$\mathbf{4.99}_{\pm 2.08}$	$\underline{3.54}_{\pm 2.58}$	$2.80_{\pm 1.93}$	$3.39_{\pm 2.97}$
ResNet50	Places365	AUC	$\mathbf{0.94}_{\pm 0.08}$	$\underline{0.89}_{\pm 0.14}$	$0.54_{\pm 0.23}$	$0.84_{\pm 0.18}$
ResNet50	Places365	MAD	$\mathbf{4.27}_{\pm 2.77}$	$\underline{3.69}_{\pm 2.87}$	$0.13_{\pm 0.76}$	$2.77_{\pm 2.59}$
ResNet18	ImageNet	AUC	$\mathbf{0.97}_{\pm 0.05}$	$\underline{0.90}_{\pm 0.14}$	$0.85_{\pm 0.15}$	$0.78_{\pm 0.21}$
ResNet18	ImageNet	MAD	$\mathbf{4.72}_{\pm 1.94}$	$\underline{3.40}_{\pm 2.20}$	$2.12_{\pm 1.59}$	$1.97_{\pm 1.97}$

Table 1: Evaluation on the CoSy benchmark. We compare LINE against CLIP-Dissect, INVERT, and DnD. Evaluations are performed on the following layers: avgpool for ResNet50 and ResNet18 (trained on ImageNet-1k and Places365), encoder for ViT-B/16 (ImageNet-1k), and norm for DINOv3. We report CoSy metrics MAD and AUC; for both metrics, higher is better (

\uparrow

). Scores are averaged over 100 neurons. The best and second-best results in each row are bold and underlined , respectively. The CoSy benchmark and its evaluation metrics are detailed in Appendix B.2 .

Model	Dataset	Layer	# Neurons	Predefined	Generated
ResNet50	ImageNet	avgpool	100	67	33
ResNet50	Places365	avgpool	100	70	30
ResNet18	ImageNet	avgpool	100	84	16
ResNet18	Places365	avgpool	100	64	36
ViT-B/16	ImageNet	encoder	100	75	25
DINOv3	LVD-1689M	norm	100	78	22

Table 2: Proportion of predefined vs. generated labels in LINE . For the models, layers and neurons evaluated in Table 1 , we report, a breakdown of the final top-scoring concepts, selected from the initial vocabulary (Predefined) or newly proposed during the iterative loop (Generated). Even with a strong initialization vocabulary, LINE ’s iterative refinement discovers novel, higher-scoring concepts, highlighting the limitation of the methods relying solely on predefined vocabularies.

Figure 3: Causal impact of concept ablation on neuron activation. Using image-to-image generative models, we remove visual concepts associated with the neuron description from ResNet50 avgpool layer identified by LINE . Original ( left) and ablated ( right ) images are shown side-by-side, with normalized neuron activations displayed below each pair. Original images are outlined in blue , ablated versions in orange , and the resulting relative drop in activation is marked in green . Extended results featuring failure cases are provided in the Appendix D.2 .

Figure 4: Visual explanation comparison on Salient ImageNet. We present visual explanations generated by LINE , DiffExplainer, and DEXTER for the top-5 bias features of the “Bathing Cup” class in RobustResNet50. Neuron activation values are displayed above each visual explanation, with the corresponding LINE neuron description provided at the bottom. Compared to AM methods, LINE produces explanations without visual artifacts common in DiffExplainer while achieving higher activations than DEXTER. Appendix E provides extended results for core and bias features on additional classes from Salient ImageNet, alongside Grad-CAM-like maps ( Selvaraju et al., 2017 ) .

Panel of Figure 5 — Figure 5: Performance across iteration steps. We extend the maximum number of iterations from 10 to 20 for 100 random neurons from the ResNet50 and ResNet18 (Places365) avgpool layers. Over successive iterations, we report the relative average best activation score ( left panel ) and the discovery rate of the optimal description ( right panel ) at each step. The “S” label denotes the final summary iteration. On average, the iteration loop consistently refines concepts, yielding rapid improvements up to steps 5-6, steady gains through step 14, and plateauing afterward.

Panel of Figure 6 — Figure 6: Impact of text-to-image (T2I) models on LINE performance. We evaluate 100 random neurons from the avgpool layer of ResNet50 trained on Places365. The box plots ( left ) comparing the highest concept activation scores indicate that the FLUX model produces slightly higher activations. Corresponding descriptions and visual explanations for neuron 153 ( right ) illustrate the distinct generative priors of each T2I model: SD1.5 tends toward photorealistic outputs, whereas SDXL and FLUX lean into stylized aesthetics, with FLUX producing the most pronounced cinematic results.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Models & representations
Notation	Meaning
$f$	Target vision model
$\mathcal{X}$	Input image domain
$\mathcal{A}$	Activation space of a specific model layer
$p$	Spatial global pooling function
$D$	Dimensionality of the pooled activation vector

Appendix

Table 3: Main notations used in this work.

Method	Explanation Format	Neuron Type	Target	Method Dependency	Training Free	Model Agnostic
Network Dissection [ Bau et al., 2017 ]	fixed-label	conv.	IoU	seg. masks	✓	✓
MILAN [ Hernandez et al., 2022 ]	free-form	conv.	WPMI	labeled corpus*, trained LSTM	✗	✓
CLIP-Dissect [ Oikarinen and Weng, 2023 ]	open-vocabulary	scalar	SoftWPMI	CLIP	✓	✓
FALCON [ Kalibhat et al., 2023 ]	open-vocabulary	pre-det.	avg. CLIP score	CLIP	✓	✗
INVERT [ Bykov et al., 2024 ]	compositional	scalar	AUC	multi-labeled data	✓	✓
Describe-and-Dissect (DnD) [ Bai et al., 2025 ]	free-form	scalar	avg. rank	I2T, LLM, T2I	✓	✓

Appendix

Table 4: Comparison of neuron labeling methods. We contrast prior approaches with LINE across several key characteristics: explanation output format, targeted neuron representation, optimization metric, and external method dependencies. We also denote whether each method is training-free and architecture-agnostic. Notably, LINE leverages LLMs and T2I models to generate rich, free-form explanations without imposing architectural constraints or requiring additional training pipelines. ∗ indicates the dependency is only required during specific phases of the method.

Figure 7: CoSy evaluation framework. A schematic illustration of the CoSy framework for Neuron 80 in ResNet18’s avgpool layer. The figure is sourced from the original paper [ Kopf et al., 2024 ] .

Concept	Score
pool table	0.96
barbell	0.67
exercise mat	0.59
dumbbell	0.46
parallel bars	0.35
leonberg	0.31

Appendix

Table 5: Initial Scoreboard for neuron 1255 from ResNet50 avgpool . We provide the starting scoreboard

\mathcal{H}

used at the beginning of the LINE refinement process. The scoreboard contains concepts closely related to the final concept ( strength training ) like barbell , alongside lower-scoring, diverse concepts that facilitate broader exploration. We bold the best score and underline the second best.

Label	pool table	barbell	gym	billiards	strength training	weightlifting	physical exercise
Novel	✗	✗	✓	✓	✓	✓	✓
Step	0	0	4	5	6	7	S
Score	0.96	0.67	1.91	0.71	2.08	1.47	1.33

Appendix

Table 6: Concept scoreboard for the strength training neuron. The table presents the final scoreboard for neuron 1255 from ResNet50 avgpool , tracking the evolution from the initial vocabulary to the final summary. The Novel row indicates whether the proposed concept was newly generated by the LLM during the iterative process ( ✓ ) or selected from the initialization vocabulary ( ✗ ). Step “S” denotes the final summary step. We bold the best score and underline the second best.

Step	Concept	Score	Reasoning
1	exercise mat	0.59	Top concepts all feature a flat rectangular surface — pool table and barbell share a common shape.
2	flat surface	0.11	Top concepts relate to sports and fitness equipment, often with a flat rectangular surface.
3	weight plate	0.08	High scores for pool table, barbell, dumbbell suggest strength training equipment with flat surfaces.
4	gym	1.91	Top concepts are objects found in a specific environment — pool table and barbell are often in a gym; pier is a common exercise location.
5	billiards	0.71	Top concepts relate to fitness, but pool table being second-highest suggests a non-traditional gym activity.
6	strength training	2.08	High scores for gym, pool table, barbell suggest strength training; “gym” is now forbidden.

Appendix

Table 7: Iteration process of LINE . We provide the outputs from the LLM, the T2I, and the scoring function for each iterative step, building from them the Table 5 . The best and second-best scores are highlighted in bold and underlined , respectively.

Panel of Table 8 — Table 8: Iteration process of LINE (synthesized images). Expanding on Table 7 , we present the full set of images generated by the T2I model for each step.

Model	Dataset	Method of reference	AUC	MAD
ResNet50	ImageNet	CLIP-Dissect	$<10^{-4}$	$<10^{-4}$
		INVERT	$<10^{-4}$	$<10^{-4}$
		DnD	$<10^{-4}$	$<10^{-4}$
ResNet50	Places365	CLIP-Dissect	$0.0009$	$0.0101$
		INVERT	$<10^{-4}$	$<10^{-4}$
		DnD	$<10^{-4}$	$<10^{-4}$

Appendix

Table 9: Statistical significance test results comparing LINE and prior methods. Expanding upon the CoSy benchmark results from Table 1 , we calculate the statistical significance of the differences between LINE and the baselines (CLIP-Dissect, INVERT, and DnD). The

p

-values confirm that LINE achieves statistically significant improvements in almost all configurations, with the sole exception being the comparison against CLIP-Dissect on the ResNet18 Places365 dataset. Underlined values indicate a lack of statistical significance (

p<0.05

Method of reference	p -value
FLUX	0.7965
SD1.5	0.0375

Appendix

Table 10: Statistical significance tests across T2I models. Building on Figure 6 , we use a Wilcoxon signed-rank test to calculate the statistical significance of activation differences for 100 random ResNet50 avgpool neurons when replacing LINE ’s default SDXL model with FLUX.1[dev] or SD1.5. The

p

-values confirm no significant difference between SDXL and FLUX (

p>0.05

), while SDXL significantly outperforms SD1.5 (

p<0.05

Figure 8: Ablation of visual concepts derived from LINE . We show side-by-side comparisons demonstrating the effect of removing the LINE -defined concept $t$ from highly activating images for the ResNet50 avgpool layer using an image-editing generative model. Concept removal via generative models is not always perfect and still requires manual inspection; for instance, removal failures can be observed in the 4th and 6th row for neuron 168 and the 5st row for neuron 1099, whereas neuron 489 demonstrates largely successful ablations, but causing an increase in activation for two of five samples. Original images are outlined in blue , ablated versions in orange , and the resulting relative change in activation is marked green if there has been a decrease, or red if there has been an increase.

Class (idx)	Non-Bias	# Feature	DEXTER	LINE
Jeep (609)	✓	1067	tractor	agricultural vehicle
		1100	truck	european fire salamander
		1208	tractor	agricultural machinery
		1515	truck	ambulance
		691	ford	limousine
Daisy (985)	✓	1105	flower	sulphur-crested cockatoo

Appendix

Table 11: Detailed feature description and label comparison on Salient ImageNet. Expanding upon the visualizations presented in Figure 4 , we summarize the class and top feature descriptions while indicating whether the evaluated classes exhibit bias ( ✗ ) or not ( ✓ ). Alongside each evaluated feature, we report the labels produced by DEXTER (using the prompt "a picture of a {label}" ) and LINE . The generated labels confirm the visual explanations provided in Figure 4 , and notably, descriptions from both methods exhibit strong alignment for the non-biased classes.

Panel of Figure 9 — Figure 9: Extended visual explanations on Salient ImageNet. Extending the Figure 4 , we present results for 3 core classes ( left ) and 3 biased classes ( right ). Each class subfigure displays the top-5 features identified in the dataset, along with their attribution maps and visual from DiffExplainer, DEXTER, and LINE . Activation magnitudes appear above each image; LINE feature descriptions are provided below.

Figure 10: Qualitative comparison of neuron descriptions in ImageNet ResNet18. We extend the qualitative analysis from Figure 1 on the ImageNet ResNet18 layer4 and avgpool layers. We show the top four activating images from ImageNet-1K alongside descriptions from LINE , CLIP-Dissect, and INVERT, and their CoSy AUC scores. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .

Figure 11: Qualitative comparison of neuron descriptions in ImageNet ResNet50. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .

Figure 12: Qualitative comparison of neuron descriptions in Places365 ResNet18. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .

Figure 13: Qualitative comparison of neuron descriptions in Places365 ResNet50. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .

Figure 14: Qualitative comparison of neuron descriptions in ViT-B/16. The ln corresponds to the encoder layer reported in the Table 1 . The best-performing method for each neuron, based on the AUC score, is highlighted in bold .

Figure 15: Comparison of explanation methods on lower layers. We evaluate CoSy AUC scores for LINE , CLIP-Dissect, INVERT, and DnD across five models and their respective layers from shallow Layer 1 to deep Avgpool/Encoder , averaged over up to 100 neurons per layer. Despite employing image cropping to better capture low-level features, DnD consistently underperforms. Notably, LINE is the only method that maintains an AUC at or above 0.8 across all evaluated layers, including the earliest ones.

Figure 16: Qualitative comparison of neuron descriptions in ImageNet ResNet18. We extend the qualitative analysis from Figure 1 and Appendix E on the lower layers of ImageNet ResNet18 layer1 and layer3 . We show the top four activating images from ImageNet-1K alongside descriptions from LINE , CLIP-Dissect, and INVERT, and their CoSy AUC scores. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .

Figure 17: Qualitative comparison of neuron descriptions in ImageNet ResNet50. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .

Figure 18: Qualitative comparison of neuron descriptions in Places365 ResNet18. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .

Figure 19: Qualitative comparison of neuron descriptions in Places365 ResNet50. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .

Figure 20: Qualitative comparison of neuron descriptions in ViT-B/16. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .

Model	Dataset	# Neurons	Time/Neuron (s)	Total Time (s)
DINOv3	LVD-1689M	$100$	$534.09$	$53408.89$
ResNet18	Places365	$564$	$500.60$	$282338.24$
ResNet18	ImageNet	$564$	$521.39$	$294065.45$
ResNet50	Places365	$600$	$410.64$	$246384.12$
ViT-B/16	ImageNet	$600$	$380.91$	$228548.40$

Appendix

Table 12: Timing statistics for LINE across models and datasets. Runtimes are normalized per neuron due to batched processing. DINOv3 was run exclusively on A100 GPUs; all other models had some layers processed on H100 GPUs, which accounts for their comparatively lower average runtimes. ResNet50 ImageNet results are omitted due to an iteration count inconsistency during re-runs on failed experiments.

Explore similar work

Naming the Concepts Classifiers Rely On: Language-Anchored Decomposition for Faithful Explanation
Jul 8, 2026Ahsan Habib Akash, Dipkamal Bhusal, Stacey Jones +3Concept-Based Explanations Neural Network Interpretability
West Virginia University, USA · Rochester Institute of Technology, USA · O Analytics +3
NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning
Sep 29, 2026Ruiyu Yan, Bowen Chen, Shaowen Wan +1VLM Interpretability VLM Reasoning
New York University · New Jersey Institute of Technology
Concept-Based Abductive and Contrastive Explanations for Behaviors of Vision Models
May 7, 2026Ronaldo Canizales, Divya Gopinath, Corina Păsăreanu +1Abductive Reasoning Concept-Based Explanations
Colorado State University, Fort Collins CO, USA · KBR Inc., NASA Ames, Moffett Field CA, USA · Carnegie Mellon University, Pittsburgh PA, USA

Jul 8, 2026J/K move · Enter open · S save

Naming the Concepts Classifiers Rely On: Language-Anchored Decomposition for Faithful Explanation

Ahsan Habib Akash, Dipkamal Bhusal, Stacey Jones, Donald A. Adjeroh+2

West Virginia University, USA · Rochester Institute of Technology, USA · O Analytics+3

Concept-Based Explanations Neural Network Interpretability Explainable Image Classification Classification

Open paper AskPDF arXiv

Deep neural networks are widely deployed in high-stakes visual applications where interpretability is critical, yet existing explanations face a trade-off: post-hoc concept methods recover factors that are faithful to a model's behavior but unnamed, while naming and by-design methods attach human-readable concepts only by retraining or altering the classifier. We propose Language-Anchored Decomposition (LAD), a post-hoc framework that delivers concepts which are simultaneously named, faithful, and obtained without modifying the model. For each class, a large language model proposes a concept vocabulary that CLIP-based similarity maps localize across image regions. Inverting standard non-negative matrix factorization, LAD fixes these language-grounded maps as the coefficient matrix and learns only a concept basis that reconstructs the frozen encoder's activations, so naming becomes a structural constraint and the model's own feature geometry determines which concepts are retained. Removing this anchor preserves accuracy but collapses attribution faithfulness. Across natural-image, scene, and medical-imaging benchmarks, LAD produces spatially precise explanations that are decision-relevant under both concept insertion and deletion, while uniquely providing stable, human-interpretable concept names.

Abstract

Explore similar work

Naming the Concepts Classifiers Rely On: Language-Anchored Decomposition for Faithful Explanation

NeuronEye: Query-Guided Visual Concept Activation for Vision-Language Reasoning

Concept-Based Abductive and Contrastive Explanations for Behaviors of Vision Models