Organizations: Computer Science and Engineering, Wright State University, Dayton, OH, USA · Computer Science, Kansas State University, Manhattan, KS, USA
Convolutional Neural Networks (CNNs) can classify images using background-dominated patches that contain little or none of the target object, showing that they exploit contextual information. Previous studies detected this behavior through classification accuracy but say little about what CNN hidden layers represent. We investigate whether Concept Induction can reveal interpretable visual properties associated with hidden-neuron activations on contextual image patches. Using five agricultural datasets with high, moderate, and low contextual bias, we analyze four CNN architectures at three network depths and five patch locations. Quantitative image descriptors of each patch are encoded in a knowledge graph and used as background knowledge for Concept Induction, which derives logical expressions characterizing neuron activation. These expressions are tested on held-out images using the Mann-Whitney U test and Cliff's delta. Of 48,013 tested neuron-feature pairs, 13,164 (27.4%) were validated. Validation was most frequent at intermediate depths (mean 40.2% at mid layers versus 12.0% at deep layers), and no patch location dominated across datasets. Among the tested descriptors, edge, Gabor, and Tamura texture features had the highest validation rates. Stronger contextual bias did not correspond to higher neuron-feature validation rates: moderately biased Sugarcane had the highest rate (40.5%) and highly biased Cassava the lowest (19.2%). These results suggest that behavioral context dependence and the semantic interpretability of hidden neurons capture different aspects of CNN behavior, and that Concept Induction can complement patch-based bias detection.
Figures & tables
Feature family
#Values
Visual property quantified
Intensity statistics
8
Global distribution of pixel intensities (e.g., mean, median, standard deviation, entropy)
Color histogram
19
Distribution of colors in the image
Combined first four moments
48
Mean, standard deviation, skewness and kurtosis of intensity over image regions
Multiscale histograms
24
Intensity histograms computed at several bin resolutions
Edge statistics
28
Magnitude and direction of intensity gradients (means, medians, variances, histograms, homogeneity)
Object statistics
34
Properties of segmented objects: count, area, distances, Euler number, centroid
Table 1: UDAT feature families considered in this study, the number of numerical descriptors associated with each family, and the visual properties they quantify.
Dataset
Tasks
# Images
Acquisition Setting
# Classes
Random Chance
Patch Acc. Range
Margin
Context-Bias Category
PlantVillage
Disease classification
∼ 54,000
Controlled
15
6.7%
9.1–19.1%
6.2 pp
Moderate
PlantDoc
Disease classification
2,569
Field
27
3.7%
2.0–7.1%
0.03 pp
Low
DeepWeeds
Species classification
17,509
Field
9
11.1%
51.3–55.0%
40.6 pp
High
Cassava
Disease classification
21,397
Field
5
20%
16.0–55.8%
13.6 pp
High
WeedCrop
Crop/weed classification
2,822
Field/UAV
2
50%
85.6–92.4%
41.1 pp
High
OpenSprayer
Weed classification
∼ 10,000
UAV / aerial
2
50%
58.4–80.6%
20.2 pp
High
Table 2: Summary of dataset characteristics and patch-based context-bias results reported in the previous study. “Chance Acc.” denotes random-chance accuracy, and “Patch Acc.” denotes the reported patch accuracy range.
Figure 1: Overview of the neuro-symbolic analysis pipeline for interpreting CNN hidden-neuron activations from contextual image patches.
Architecture
Mid
Mid-deep
Deep
ResNet50V2
conv3_block4_out
conv4_block6_out
conv5_block3_out
DenseNet121
conv3_block12_concat
conv4_block24_concat
relu
InceptionV3
mixed5
mixed7
mixed10
VGG16
block3_conv3
block4_conv3
block5_conv3
Table 3: Hidden layers selected to represent mid, mid-deep, and deep stages of each CNN architecture
Architecture
Depth
Layer
Initial neurons/ channels
After variance filter
After correlation ranking
Final selected
ResNet50V2
Mid
conv3_block4_out
512
384
100
50
ResNet50V2
Mid-deep
conv4_block6_out
1,024
768
100
50
ResNet50V2
Deep
conv5_block3_out
2,048
1,536
100
50
DenseNet121
Mid
conv3_block12_concat
512
384
100
50
DenseNet121
Mid-deep
conv4_block24_concat
1,024
768
100
50
DenseNet121
Deep
relu
1,024
768
100
50
Table 4: Reduction in candidate neurons across the three-stage neuron-selection procedure for the analyzed CNN layers.
Figure 2: Ontology pattern for one patch image and one UDAT sub-feature (Gabor filters, sub-feature Bin2). The patch individual is linked by hasGaborFilters_Bin2 to a value individual (value 0.1225), which is typed with the interval class for [0.105,0.150] ; the five equal-width interval classes are subclasses of the sub-feature class GaborFilters_Bin2. The same pattern is repeated for each of the seven Gabor sub-features.
Figure 3: Progressive reduction of neuron-feature evaluation cases. Percentages indicate the proportion retained relative to the initial 105,000 evaluation cases.
Dataset
Patch
All Cases
Concept Found
Constraints Parsed
MW Test Completed
Significant
Validated
Cassava
BL
4200
3219
2356
2260
1200
362
BR
4200
3184
2318
2223
1213
387
C
4200
3155
2241
2170
1131
433
TL
4200
3227
2351
2296
1236
487
TR
4200
3242
2350
2302
1266
495
OpenSprayer
BL
4200
3005
2367
2003
791
468
Table 5: Dataset-patch-wise progression of neuron-feature evaluation cases. Patch locations are denoted as BL (Bottom-Left), BR (Bottom-Right), C (Center), TL (Top-Left), and TR (Top-Right).
Layer depth
Configurations
Mean validation rate (%)
SD
Min (%)
Max (%)
Mid
25
40.21
10.38
24.49
61.83
Mid-deep
25
26.91
12.89
14.43
54.02
Deep
25
11.99
5.65
4.41
29.07
Table 6: Summary of neuron-feature validation rates across the three analyzed CNN depth levels.
Figure 4: Validation rates of context-sensitive neuron-feature pairs across different layers for each dataset-patch configuration.
Figure 5: Dataset-wise validation rates of context-sensitive neuron-feature representations across the five predefined spatial patch locations.
Patch position
Observations
Mean validation rate (%)
SD
Min (%)
Max (%)
Top-left
15
27.39
16.25
8.87
57.32
Top-right
15
29.53
16.08
6.77
56.20
Center
15
25.61
14.85
6.96
52.43
Bottom-left
15
25.25
15.97
4.41
61.83
Bottom-right
15
24.06
14.77
6.23
50.37
Table 7: Patch-wise summary of neuron-feature validation rates across datasets and analyzed layer depths.
Figure 6: Overall validation outcomes for the analyzed UDAT feature families.
Figure 7: Feature-family validation rates grouped by dataset across the analyzed patch locations, architectures, and layer depths.
Figure 8: Feature-family validation rates grouped by spatial patch position across the analyzed datasets, architectures, and layer depths.
Figure 9: Validation rates of neuron-feature pairs across feature families and CNN layer depths.
Dataset
Bias category
Mean patch accuracy
Random chance
Above-chance patch accuracy
Tested
Validated
Validation rate
Cassava
High
33.62%
20.00%
13.62%
11,251
2,164
19.23%
OpenSprayer
High
70.18%
50.00%
20.18%
9,989
2,195
21.97%
Oxford
Low
1.70%
0.98%
0.72%
11,115
3,107
27.95%
Sugarcane
Moderate
16.92%
9.1%
7.83%
10,006
4,057
40.55%
WeedCrop
High
91.06%
50.00%
41.06%
5,652
1,641
29.03%
Table 8: Dataset-level bias and neuron-feature validation statistics.
Convolutional neural networks (CNNs) learn spatial and temporal patterns using shared filters, with applications from image recognition to scientific prediction.
Convolutional neural networks (CNNs) are typically evaluated using held-out classification accuracy, an approach that presupposes predictions are based primarily on the intended object of interest rather than incidental surrounding context. We test this assumption in CNN-based agricultural image classification by comparing model performance on original images with performance on background-dominated patches extracted from the same images across eight publicly available agricultural benchmark datasets and four widely used CNN architectures. Background-dominated patches were classified above dataset-specific random chance for six of the eight datasets, and substantially above chance for four of them, indicating that contextual information contributes to model predictions for the majority of datasets evaluated. For these four datasets, we further evaluated whether this behavior reflected genuine class-discriminative information or was primarily attributable to class imbalance using macro-averaged precision, recall, and F1 together with class-balanced test subsets. The results show that contextual reliance does not admit a single explanation: class imbalance accounts for a substantial portion of the observed signal for some datasets and architectures, whereas above-chance contextual classification persists after balancing for others. Together with previous evidence from curated object recognition and cancer pathology imaging, these findings support the growing view that contextual bias is a recurring characteristic of CNN-based image classification rather than a phenomenon confined to a single application domain. More broadly, this work provides a systematic framework for quantifying contextual bias across heterogeneous image datasets by combining dataset-specific random-chance baselines, contextual bias categorization, macro-averaged evaluation, and class-balanced robustness analysis.
Abhilekha Dalal, Michael Okonoda, Eder Martinez +1
Deep Neural Networks (DNNs) have advanced applications in domains such as healthcare, autonomous systems, and scene understanding, yet the internal semantics of their hidden neurons remain poorly understood. Prior work introduced a Concept Induction-based framework for hidden neuron analysis and demonstrated its effectiveness on the ADE20K dataset. In this case study, we investigate whether the approach generalizes by applying it to the SUN2012 dataset, a large-scale scene recognition benchmark. Using the same workflow, we assign interpretable semantic labels to neurons and validate them through web-sourced images and statistical testing. Our findings confirm that the method transfers to SUN2012, showing its broader applicability.
Moumita Sen Sarma, Samatha Ereshi Akkamahadevi, Pascal Hitzler
Department of Computer Science, Kansas State University, Manhattan, Kansas, USA
We present Feature-Align CNN (FA-CNN), a prototype CNN architecture with intrinsic class attribution through end-to-end feature alignment. Our intuition is that the use of unordered operations such as Linear and Conv2D layers cause unnecessary shuffling and mixing of semantic concepts, thereby making raw feature maps difficult to understand. We introduce two new order preserving layers, the dampened skip connection, and the global average pooling classifier head. These layers force the model to maintain an end-to-end feature alignment from the raw input pixels all the way to final class logits. This end-to-end alignment enhances the interpretability of the model by allowing the raw feature maps to intrinsically exhibit class attribution. We prove theoretically that FA-CNN penultimate feature maps are identical to Grad-CAM saliency maps. Moreover, we prove that these feature maps slowly morph layer-by-layer over network depth, showing the evolution of features through network depth toward penultimate class activations. FA-CNN performs well on benchmark image classification datasets. Moreover, we compare the averaged FA-CNN raw feature maps against Grad-CAM and permutation methods in a percent pixels removed interpretability task. We conclude this work with a discussion and future, including limitations and extensions toward hybrid models.