LINE: LLM-based Iterative Neuron Explanations for Vision Models
Authors: Vladimir Zaigrajew, Michał Piechota, Gaspar Sekula, Paweł Gelar, Przemysław Biecek
Organizations: Centre for Credible AI Warsaw University of Technology · Warsaw University of Technology Warsaw, Poland · Centre for Credible AI, Warsaw University of Technology University of Warsaw, Poland
Interpreting individual neurons in deep neural networks is a crucial step towards understanding their complex decision-making processes and ensuring AI safety. Despite recent progress in neuron labeling, existing methods often limit the search space to predefined concept vocabularies or produce overly specific descriptions that fail to capture higher-order, global concepts. We introduce LINE, a novel, training-free iterative approach tailored for open-vocabulary concept labeling in vision models. Operating in a strictly black-box setting, LINE leverages a large language model and a text-to-image generator to iteratively propose and refine concepts in a closed loop, guided by activation history. LINE achieves state-of-the-art performance across multiple model architectures, yielding AUC improvements of up to 0.11 on ImageNet and 0.05 on Places365, while discovering, on average, 27% of new concepts missed by predefined vocabularies. Beyond identifying the top concept, LINE provides a complete generation history, enabling polysemanticity evaluation and producing visual explanations that rival gradient-dependent activation maximization methods. The source code will be made available soon.
Figures & tables
Figure 1: Qualitative comparison of neuron descriptions. We show the top four activating images from the ImageNet for three selected neurons from the DINOv3 norm layer ( left ) and the ResNet50 avgpool layer ( right ). The descriptions are provided from our method ( LINE ), CLIP-Dissect ( Oikarinen and Weng, 2023 ) , and INVERT ( Bykov et al., 2024 ) alongside their CoSy benchmark ( Kopf et al., 2024 ) AUC scores. The best method according to the benchmark AUC score is bold for each neuron. Extended results for different layers and models are reported in Appendix E .
Figure 2: Overview of the LINE iterative framework. Step 1: An LLM proposes a new concept t based on the descriptions from the scoreboard H (e.g., proposing strawberry and pomegranate into red fruit ). Step 2: A Text-to-Image (T2I) model generates a batch of diverse synthetic images illustrating the concept t . Step 3: These images are processed by the target vision model, extracting concept activations At . Step 4: A scoring function ( ψavg(A) ) converts At into score st , and the the H is updated with the result. Upon reaching the final iteration N , an additional final summary iteration ( N+1 ) evaluates the higher-order global concept reasoned from top-scoreboard descriptions; here, Step 1 is replaced by a Summary Concept step, while the remaining steps are unchanged. The top-scoring concept from H is then returned. The LINE algorithm is summarized in Appendix C.1 .
Model
Dataset
Metric
LINE
CLIP-Dissect
INVERT
DnD
ResNet50
ImageNet
AUC
0.96±0.08
0.87±0.18
0.88±0.15
0.85±0.18
MAD
4.99±2.08
3.54±2.58
2.80±1.93
3.39±2.97
ResNet50
Places365
AUC
0.94±0.08
0.89±0.14
0.54±0.23
0.84±0.18
MAD
4.27±2.77
3.69±2.87
0.13±0.76
2.77±2.59
ResNet18
ImageNet
AUC
0.97±0.05
0.90±0.14
0.85±0.15
0.78±0.21
MAD
4.72±1.94
3.40±2.20
2.12±1.59
1.97±1.97
Table 1: Evaluation on the CoSy benchmark. We compare LINE against CLIP-Dissect, INVERT, and DnD. Evaluations are performed on the following layers: avgpool for ResNet50 and ResNet18 (trained on ImageNet-1k and Places365), encoder for ViT-B/16 (ImageNet-1k), and norm for DINOv3. We report CoSy metrics MAD and AUC; for both metrics, higher is better ( ↑ ). Scores are averaged over 100 neurons. The best and second-best results in each row are bold and underlined , respectively. The CoSy benchmark and its evaluation metrics are detailed in Appendix B.2 .
Model
Dataset
Layer
# Neurons
Predefined
Generated
ResNet50
ImageNet
avgpool
100
67
33
ResNet50
Places365
avgpool
100
70
30
ResNet18
ImageNet
avgpool
100
84
16
ResNet18
Places365
avgpool
100
64
36
ViT-B/16
ImageNet
encoder
100
75
25
DINOv3
LVD-1689M
norm
100
78
22
Table 2: Proportion of predefined vs. generated labels in LINE . For the models, layers and neurons evaluated in Table 1 , we report, a breakdown of the final top-scoring concepts, selected from the initial vocabulary (Predefined) or newly proposed during the iterative loop (Generated). Even with a strong initialization vocabulary, LINE ’s iterative refinement discovers novel, higher-scoring concepts, highlighting the limitation of the methods relying solely on predefined vocabularies.
Figure 3: Causal impact of concept ablation on neuron activation. Using image-to-image generative models, we remove visual concepts associated with the neuron description from ResNet50 avgpool layer identified by LINE . Original ( left) and ablated ( right ) images are shown side-by-side, with normalized neuron activations displayed below each pair. Original images are outlined in blue , ablated versions in orange , and the resulting relative drop in activation is marked in green . Extended results featuring failure cases are provided in the Appendix D.2 .
Figure 4: Visual explanation comparison on Salient ImageNet. We present visual explanations generated by LINE , DiffExplainer, and DEXTER for the top-5 bias features of the “Bathing Cup” class in RobustResNet50. Neuron activation values are displayed above each visual explanation, with the corresponding LINE neuron description provided at the bottom. Compared to AM methods, LINE produces explanations without visual artifacts common in DiffExplainer while achieving higher activations than DEXTER. Appendix E provides extended results for core and bias features on additional classes from Salient ImageNet, alongside Grad-CAM-like maps ( Selvaraju et al., 2017 ) .
Figure 5: Performance across iteration steps. We extend the maximum number of iterations from 10 to 20 for 100 random neurons from the ResNet50 and ResNet18 (Places365) avgpool layers. Over successive iterations, we report the relative average best activation score ( left panel ) and the discovery rate of the optimal description ( right panel ) at each step. The “S” label denotes the final summary iteration. On average, the iteration loop consistently refines concepts, yielding rapid improvements up to steps 5-6, steady gains through step 14, and plateauing afterward.
Figure 6: Impact of text-to-image (T2I) models on LINE performance. We evaluate 100 random neurons from the avgpool layer of ResNet50 trained on Places365. The box plots ( left ) comparing the highest concept activation scores indicate that the FLUX model produces slightly higher activations. Corresponding descriptions and visual explanations for neuron 153 ( right ) illustrate the distinct generative priors of each T2I model: SD1.5 tends toward photorealistic outputs, whereas SDXL and FLUX lean into stylized aesthetics, with FLUX producing the most pronounced cinematic results.
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Notation
Meaning
Models & representations
f
Target vision model
X
Input image domain
A
Activation space of a specific model layer
p
Spatial global pooling function
D
Dimensionality of the pooled activation vector
Appendix
Table 3: Main notations used in this work.
Method
Explanation Format
Neuron Type
Target
Method Dependency
Training Free
Model Agnostic
Network Dissection [ Bau et al., 2017 ]
fixed-label
conv.
IoU
seg. masks
✓
✓
MILAN [ Hernandez et al., 2022 ]
free-form
conv.
WPMI
labeled corpus*, trained LSTM
✗
✓
CLIP-Dissect [ Oikarinen and Weng, 2023 ]
open-vocabulary
scalar
SoftWPMI
CLIP
✓
✓
FALCON [ Kalibhat et al., 2023 ]
open-vocabulary
pre-det.
avg. CLIP score
CLIP
✓
✗
INVERT [ Bykov et al., 2024 ]
compositional
scalar
AUC
multi-labeled data
✓
✓
Describe-and-Dissect (DnD) [ Bai et al., 2025 ]
free-form
scalar
avg. rank
I2T, LLM, T2I
✓
✓
Appendix
Table 4: Comparison of neuron labeling methods. We contrast prior approaches with LINE across several key characteristics: explanation output format, targeted neuron representation, optimization metric, and external method dependencies. We also denote whether each method is training-free and architecture-agnostic. Notably, LINE leverages LLMs and T2I models to generate rich, free-form explanations without imposing architectural constraints or requiring additional training pipelines. ∗ indicates the dependency is only required during specific phases of the method.
Figure 7: CoSy evaluation framework. A schematic illustration of the CoSy framework for Neuron 80 in ResNet18’s avgpool layer. The figure is sourced from the original paper [ Kopf et al., 2024 ] .
Concept
Score
pool table
0.96
barbell
0.67
exercise mat
0.59
dumbbell
0.46
parallel bars
0.35
leonberg
0.31
Appendix
Table 5: Initial Scoreboard for neuron 1255 from ResNet50 avgpool . We provide the starting scoreboard H used at the beginning of the LINE refinement process. The scoreboard contains concepts closely related to the final concept ( strength training ) like barbell , alongside lower-scoring, diverse concepts that facilitate broader exploration. We bold the best score and underline the second best.
Label
pool table
barbell
gym
billiards
strength training
weightlifting
physical exercise
Novel
✗
✗
✓
✓
✓
✓
✓
Step
0
0
4
5
6
7
S
Score
0.96
0.67
1.91
0.71
2.08
1.47
1.33
Appendix
Table 6: Concept scoreboard for the strength training neuron. The table presents the final scoreboard for neuron 1255 from ResNet50 avgpool , tracking the evolution from the initial vocabulary to the final summary. The Novel row indicates whether the proposed concept was newly generated by the LLM during the iterative process ( ✓ ) or selected from the initialization vocabulary ( ✗ ). Step “S” denotes the final summary step. We bold the best score and underline the second best.
Step
Concept
Score
Reasoning
T2I Image
1
exercise mat
0.59
Top concepts all feature a flat rectangular surface — pool table and barbell share a common shape.
2
flat surface
0.11
Top concepts relate to sports and fitness equipment, often with a flat rectangular surface.
3
weight plate
0.08
High scores for pool table, barbell, dumbbell suggest strength training equipment with flat surfaces.
4
gym
1.91
Top concepts are objects found in a specific environment — pool table and barbell are often in a gym; pier is a common exercise location.
5
billiards
0.71
Top concepts relate to fitness, but pool table being second-highest suggests a non-traditional gym activity.
6
strength training
2.08
High scores for gym, pool table, barbell suggest strength training; “gym” is now forbidden.
Appendix
Table 7: Iteration process of LINE . We provide the outputs from the LLM, the T2I, and the scoring function for each iterative step, building from them the Table 5 . The best and second-best scores are highlighted in bold and underlined , respectively.
Table 8: Iteration process of LINE (synthesized images). Expanding on Table 7 , we present the full set of images generated by the T2I model for each step.
Model
Dataset
Method of reference
AUC
MAD
ResNet50
ImageNet
CLIP-Dissect
<10−4
<10−4
INVERT
<10−4
<10−4
DnD
<10−4
<10−4
ResNet50
Places365
CLIP-Dissect
0.0009
0.0101
INVERT
<10−4
<10−4
DnD
<10−4
<10−4
Appendix
Table 9: Statistical significance test results comparing LINE and prior methods. Expanding upon the CoSy benchmark results from Table 1 , we calculate the statistical significance of the differences between LINE and the baselines (CLIP-Dissect, INVERT, and DnD). The p -values confirm that LINE achieves statistically significant improvements in almost all configurations, with the sole exception being the comparison against CLIP-Dissect on the ResNet18 Places365 dataset. Underlined values indicate a lack of statistical significance ( p<0.05 ).
Method of reference
p -value
FLUX
0.7965
SD1.5
0.0375
Appendix
Table 10: Statistical significance tests across T2I models. Building on Figure 6 , we use a Wilcoxon signed-rank test to calculate the statistical significance of activation differences for 100 random ResNet50 avgpool neurons when replacing LINE ’s default SDXL model with FLUX.1[dev] or SD1.5. The p -values confirm no significant difference between SDXL and FLUX ( p>0.05 ), while SDXL significantly outperforms SD1.5 ( p<0.05 ).
Figure 8: Ablation of visual concepts derived from LINE . We show side-by-side comparisons demonstrating the effect of removing the LINE -defined concept t from highly activating images for the ResNet50 avgpool layer using an image-editing generative model. Concept removal via generative models is not always perfect and still requires manual inspection; for instance, removal failures can be observed in the 4th and 6th row for neuron 168 and the 5st row for neuron 1099, whereas neuron 489 demonstrates largely successful ablations, but causing an increase in activation for two of five samples. Original images are outlined in blue , ablated versions in orange , and the resulting relative change in activation is marked green if there has been a decrease, or red if there has been an increase.
Class (idx)
Non-Bias
# Feature
DEXTER
LINE
Jeep (609)
✓
1067
tractor
agricultural vehicle
1100
truck
european fire salamander
1208
tractor
agricultural machinery
1515
truck
ambulance
691
ford
limousine
Daisy (985)
✓
1105
flower
sulphur-crested cockatoo
Appendix
Table 11: Detailed feature description and label comparison on Salient ImageNet. Expanding upon the visualizations presented in Figure 4 , we summarize the class and top feature descriptions while indicating whether the evaluated classes exhibit bias ( ✗ ) or not ( ✓ ). Alongside each evaluated feature, we report the labels produced by DEXTER (using the prompt "a picture of a {label}" ) and LINE . The generated labels confirm the visual explanations provided in Figure 4 , and notably, descriptions from both methods exhibit strong alignment for the non-biased classes.
Figure 9: Extended visual explanations on Salient ImageNet. Extending the Figure 4 , we present results for 3 core classes ( left ) and 3 biased classes ( right ). Each class subfigure displays the top-5 features identified in the dataset, along with their attribution maps and visual from DiffExplainer, DEXTER, and LINE . Activation magnitudes appear above each image; LINE feature descriptions are provided below.
Figure 10: Qualitative comparison of neuron descriptions in ImageNet ResNet18. We extend the qualitative analysis from Figure 1 on the ImageNet ResNet18 layer4 and avgpool layers. We show the top four activating images from ImageNet-1K alongside descriptions from LINE , CLIP-Dissect, and INVERT, and their CoSy AUC scores. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .
Figure 11: Qualitative comparison of neuron descriptions in ImageNet ResNet50. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .
Figure 12: Qualitative comparison of neuron descriptions in Places365 ResNet18. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .
Figure 13: Qualitative comparison of neuron descriptions in Places365 ResNet50. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .
Figure 14: Qualitative comparison of neuron descriptions in ViT-B/16. The ln corresponds to the encoder layer reported in the Table 1 . The best-performing method for each neuron, based on the AUC score, is highlighted in bold .
Figure 15: Comparison of explanation methods on lower layers. We evaluate CoSy AUC scores for LINE , CLIP-Dissect, INVERT, and DnD across five models and their respective layers from shallow Layer 1 to deep Avgpool/Encoder , averaged over up to 100 neurons per layer. Despite employing image cropping to better capture low-level features, DnD consistently underperforms. Notably, LINE is the only method that maintains an AUC at or above 0.8 across all evaluated layers, including the earliest ones.
Figure 16: Qualitative comparison of neuron descriptions in ImageNet ResNet18. We extend the qualitative analysis from Figure 1 and Appendix E on the lower layers of ImageNet ResNet18 layer1 and layer3 . We show the top four activating images from ImageNet-1K alongside descriptions from LINE , CLIP-Dissect, and INVERT, and their CoSy AUC scores. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .
Figure 17: Qualitative comparison of neuron descriptions in ImageNet ResNet50. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .
Figure 18: Qualitative comparison of neuron descriptions in Places365 ResNet18. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .
Figure 19: Qualitative comparison of neuron descriptions in Places365 ResNet50. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .
Figure 20: Qualitative comparison of neuron descriptions in ViT-B/16. The best-performing method for each neuron, based on the AUC score, is highlighted in bold .
Model
Dataset
# Neurons
Time/Neuron (s)
Total Time (s)
DINOv3
LVD-1689M
100
534.09
53408.89
ResNet18
Places365
564
500.60
282338.24
ImageNet
564
521.39
294065.45
ResNet50
Places365
600
410.64
246384.12
ViT-B/16
ImageNet
600
380.91
228548.40
Appendix
Table 12: Timing statistics for LINE across models and datasets. Runtimes are normalized per neuron due to batched processing. DINOv3 was run exclusively on A100 GPUs; all other models had some layers processed on H100 GPUs, which accounts for their comparatively lower average runtimes. ResNet50 ImageNet results are omitted due to an iteration count inconsistency during re-runs on failed experiments.
Deep neural networks are widely deployed in high-stakes visual applications where interpretability is critical, yet existing explanations face a trade-off: post-hoc concept methods recover factors that are faithful to a model's behavior but unnamed, while naming and by-design methods attach human-readable concepts only by retraining or altering the classifier. We propose Language-Anchored Decomposition (LAD), a post-hoc framework that delivers concepts which are simultaneously named, faithful, and obtained without modifying the model. For each class, a large language model proposes a concept vocabulary that CLIP-based similarity maps localize across image regions. Inverting standard non-negative matrix factorization, LAD fixes these language-grounded maps as the coefficient matrix and learns only a concept basis that reconstructs the frozen encoder's activations, so naming becomes a structural constraint and the model's own feature geometry determines which concepts are retained. Removing this anchor preserves accuracy but collapses attribution faithfulness. Across natural-image, scene, and medical-imaging benchmarks, LAD produces spatially precise explanations that are decision-relevant under both concept insertion and deletion, while uniquely providing stable, human-interpretable concept names.
Ahsan Habib Akash, Dipkamal Bhusal, Stacey Jones +3
West Virginia University, USA · Rochester Institute of Technology, USA · O Analytics +3
Current vision-language models (VLMs) encode visual information in dense hidden states where object identity, spatial layout, and local attributes are implicitly entangled rather than explicitly disentangled, limiting their ability to isolate and modulate the specific visual evidence required by a given language query. Inspired by sparse population coding and top-down modulation in biological vision, we introduce NeuronEye, a plug-in framework that constructs a sparse, concept-level neuron vocabulary from intermediate VLM representations and selectively activates query-relevant visual concepts during inference. NeuronEye decomposes vision-token states into an overcomplete sparse basis organized by concept-level clusters, uses the language query to activate relevant clusters and localize the patches where selected concepts are expressed, and injects the focused evidence back into vision tokens. A complementary suppression mechanism attenuates dominant perceptual directions to preserve weaker but relevant cues. All operations run in a single forward pass over a frozen VLM backbone. On Qwen2.5-VL-7B, NeuronEye raises CV-Bench overall accuracy by +3.1 with gains of +9.5 on Distance, and improves BLINK Multi-view by +8.3, with similar trends on LLaVA-1.6-7B. These results suggest that sparse neuron vocabularies can serve not only as post-hoc interpretability tools but also as active interfaces for concept-level visual reasoning.
Ruiyu Yan, Bowen Chen, Shaowen Wan +1
New York University · New Jersey Institute of Technology
Concept-based explanations offer a promising approach for explaining the predictions of deep neural networks in terms of high-level, human-understandable concepts. However, existing methods either do not establish a causal connection between the concepts and model predictions or are limited in expressivity and only able to infer causal explanations involving single concepts. At the same time, the parallel line of work on formal abductive and contrastive explanations computes the minimal set of input features causally relevant for model outcomes but only considers low-level features such as pixels. Merging these two threads, in this work, we propose the notion of concept-based abductive and contrastive explanations that capture the minimal sets of high-level concepts causally relevant for model outcomes. We then present a family of algorithms that enumerate all minimal explanations while using concept erasure procedures to establish causal relationships. By appropriately aggregating such explanations, we are not only able to understand model predictions on individual images but also on collections of images where the model exhibits a user-specified, common behavior. We evaluate our approach on multiple models, datasets, and behaviors, and demonstrate its effectiveness in computing helpful, user-friendly explanations.
Ronaldo Canizales, Divya Gopinath, Corina Păsăreanu +1
Colorado State University, Fort Collins CO, USA · KBR Inc., NASA Ames, Moffett Field CA, USA · Carnegie Mellon University, Pittsburgh PA, USA