High-capacity neural network models have achieved state-of-the-art performance across diverse classification tasks, yet they frequently operate as black-box models, lacking the transparency necessary for critical decision-making. Such opacity creates a persistent trade-off between performance and explainability. This paper proposes a solution to address this gap: the NeuroRule knowledge distillation framework that results in explainable rule-sets from neural network models. NeuroRule adapts the EVOTER rule-set evolution infrastructure to treat neural networks as targets for the evolution process, distilling their performance into concise sets of propositional logic expressions. There are three primary contributions: (1) an evolutionary method for distilling black-box neural network models into explicit rule-set models; (2) a method for making rule sets more explainable by including a conciseness objective to evolution; and (3) a demonstration that the distillation is viable even without access to the original neural network training data. The paper thus establishes that black-box neural network models can be made explainable and therefore useful in real-world applications where trustworthiness is paramount.
Figures & tables
Figure 1. The grammar for the rules. Rules are written in propositional logic, and include feature comparison, feature exponentiation, and time lags. For simplicity, this paper does not use time lags; they are shown for completeness. This representation allows for easy understanding. Figure from ( shahrzad2024evoterevolutiontransparentexplainable ) .
Figure 2. An example rule with elements color-coded from the grammar. Rules can contain multiple conditions, and the rule-set includes a default rule that is used if none of the non-default rules are applied. This diagram demonstrates the standard structure followed by all rules in the rule-sets. Figure from ( shahrzad2024evoterevolutiontransparentexplainable ) .
Figure 3. The NeuroRule method with an Accuracy objective. A target neural network (NN) is first trained to optimize accuracy in predicting the labels in the original dataset. Rule-sets are then evolved through a genetic algorithm to match the behavior of the NN. Whereas the NN is a black box, the rule-sets are explainable. Knowledge in the network is thus distilled into an explainable model.
Figure 4. The NeuroRule method with Accuracy and Conciseness objectives. Conciseness of the solutions is measured through Condition Counts. Optimizing both objectives together results in solutions that are less complex and therefore more explainable than when optimizing for accuracy alone.
Layer
Type
Dimensions
Activation
Input
Fully Connected
d×32
ReLU
Hidden 1
Fully Connected
32×16
ReLU
Output
Fully Connected
16×2
Softmax
Table 1. Architectural specifications of the black-box NN. This network is a multi-layer perceptron (MLP) where d represents the input dimensionality of the specific dataset. The design utilizes decreasing hidden layer sizes ( 32 to 16 ) and a final Softmax output that is discretized into binary ( 0 or 1 ) labels. The labels in the original dataset are replaced by the labels generated by the NN to create a fitness function dataset for distillation.
Parameter
Setting
Optimizer
Adam ( β1=0.9,β2=0.999 )
Learning Rate
1×10−3
Loss Function
Cross-Entropy Loss
Batch Size
32
Training Epochs
100
Weight Initialization
He Normal
Table 2. Configuration used to train the black-box NN. To reflect standard machine learning practices, the network is trained using empirical defaults established in modern deep learning literature.
Hyperparameter
Setting
Population Size
100
Generations
200
Mutation Probability
0.7
Selection Method
Tournament
Novelty Selection Multiplier
1.5
Table 3. Parameters for rule-set evolution. The same set of parameters was used for the EVOTER baseline and NeuroRule. They are designed to favor aggressive exploration, which is likely to result in diverse, creative solutions.
In-Distribution (ID)
Out-of-Distribution (OOD)
ID/OOD split
NN
EVOTER
NeuroRule
NN
EVOTER
NeuroRule
80%
0.749
0.727
0.720
0.692
0.679
0.700
90%
0.726
0.708
0.712
0.697
0.688
0.706
95%
0.726
0.716
0.721
0.717
0.705
0.708
Table 4. Experiment 1: Accuracy of NN, EVOTER, and NeuroRule models on ID and OOD data at different ID/OOD splits on the three clinical benchmarks. The numbers are averages of 10 runs; the bolded values identify the best performance in each category. In ID, the NN is the most accurate, suggesting that NeuroRule models indeed evolve to approximate the NN. However, NeuroRule is usually more accurate than EVOTER, suggesting that the NN provides more regularized targets for rule evolution than the original data. In OOD, rule-set networks in general and NeuroRule in particular are more accurate than the NN, suggesting that they have learned the underlying principles of the data, allowing them to extrapolate better. The learning curves with standard deviations are shown in Figures 5 , 9 , and 10 .
Figure 5. Learning curves for EVOTER and NeuroRule compared to the NN accuracy at different ID/OOD splits on the Diabetes dataset. The curves are averages over 10 runs, and the bands indicate standard deviation. The ID graphs (left) demonstrate that NeuroRule rule-sets (green) generally achieved higher accuracy than the EVOTER baseline (blue). This result suggests that the NN helps in avoiding overfitting to noisy samples. Both rule-set approaches initially exhibit a period of rapid improvement and then plateau at a level slightly below the NN (red), suggesting that the rule-sets have a slight performance cost. With OOD (right), NeuroRule (and EVOTER in other datasets) actually performs better than the NN. This result suggests that the rule-sets discover underlying principles of data rather than overfit to noise. Results for the Breast Cancer and Heart Disease datasets are available in Figures 9 and 10 .
Figure 6. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Diabetes dataset under multi-objective optimization of accuracy and conciseness. The curves are averages over 10 runs, and the bands indicate standard deviation. Compared to single-objective optimization, accuracy has improved significantly, suggesting that conciseness encourages rule-set evolution to find representations that are more principled and generalize better. Also, the standard deviation is smaller, presumably because search focuses on a more limited set of solutions. The results for Breast Cancer and Heart Disease datasets are in Figures 11 and 12 .
Figure 7. Accuracy vs. conciseness in single and multi-objective NeuroRule evolutionary runs at different ID/OOD splits on the Diabetes dataset. The model accuracy is on the x -axis and conciseness on the y -axis; the scatter plot shows all individuals in the final population of all 10 single-objective (blue) and multi-objective (green) runs. The multi-objective solutions are concentrated at the bottom, indicating that they have much lower complexity than single-objective solutions. Notably, the most accurate solutions (furthest to the right) are also often among the most concise (low). Results for the Breast Cancer and Heart Disease datasets are in Figures 13 and 14 .
Figure 8. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Diabetes dataset when NeuroRule was evolved with synthetic data only. The curves are averages over 10 runs and the bands indicate standard deviation. The accuracy of NeuroRule is lower than when evolved with samples from the original dataset, but still viable, demonstrating that the network represents a portable source of knowledge about the domain even when the original dataset is not available. The results for Breast Cancer and Heart Disease datasets are in Figures 15 and 16 .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Breast Cancer Dataset. This dataset is smaller than Diabetes, but the results are similar. However, likely due to the small amount of data, the EVOTER accuracy has more variance.
Figure 10. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Heart Disease Dataset. This dataset is also smaller, but it turned out to be a particularly good fit for the rule-set approaches: They are more accurate in five of the six cases, including the 80% and 90% ID cases.
Figure 11. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Breast Cancer dataset under multi-objective optimization of accuracy and conciseness. The ID results initial form vertical lines due to limited data but gradually become more scattered like the OOD results as more data is introduced in the training interval. The results are similar to those with the Diabetes dataset, demonstrating significant improvement over single-objective evolution, and less variance.
Figure 12. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Heart Disease dataset under multi-objective optimization of accuracy and conciseness. The results are similar to those with the Diabetes dataset, demonstrating significant improvement over single-objective evolution, and less variance.
Figure 13. Accuracy vs. conciseness in single and multi-objective NeuroRule evolutionary runs at different ID/OOD splits on the Breast Cancer dataset. The ID results initial form vertical lines due to limited data but gradually become more scattered like the OOD results as more data is introduced in the training interval. Otherwise, the results are similar to those with the Diabetes dataset
Figure 14. Accuracy vs. conciseness in single and multi-objective NeuroRule evolutionary runs at different ID/OOD splits on the Heart Disease dataset. As in Figure 13 , the ID results initial form vertical lines due to limited data but gradually become more scattered like the OOD results as more data is introduced in the training interval. The results are otherwise similar to the Diabetes dataset.
Figure 15. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Breast Cancer dataset when NeuroRule was evolved with synthetic data only. The results are similar to those with the Diabetes dataset, demonstrating lower but still viable accuracy even when the original data is not available.
Figure 16. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Heart Disease dataset when NeuroRule was evolved with synthetic data only. The results are similar to those with the Diabetes dataset, demonstrating lower but still viable accuracy even when the original data is not available.
Logic-based methods for explaining neural network decisions offer formal guarantees of correctness and non-redundancy, but they often suffer from high computational costs, especially for large networks. In this work, we improve the efficiency of such methods by combining bound propagation with constraint simplification. These simplifications, derived from the propagation, tighten neuron bounds and eliminate unnecessary binary variables, making the explanation process more efficient. Our experiments suggest that combining these techniques reduces explanation time by up to 89.26%, particularly for larger neural networks.
Ronaldo Gomes, Jairo Ribeiro, Luiz Queiroz +1
Department of Computer Science, Instituto Federal do Ceará (IFCE), Maracanaú, Brazil
Formal explainability of classifying neural networks (NNs) is an active area of research, providing explanations with provable guarantees of the classification within continuous regions of the input feature space. However, the existing techniques are either limited to individual input features without guarantees on their relations or the provided solutions fail to scale to deep architectures. This paper addresses these issues by introducing a flexible symbolic framework for an efficient, guided computation of explanations of the NN behavior, parametrized by the activations of internal neurons, and using logical engines such as SMT solvers. Unlike prior methods that rely on specialized NN verifiers, our method yields explanations that are not restricted in shape. Our algorithm is implementable on top of a general-purpose logical solver, isolating the NN-specific encoding from the algorithmic framework. We experimented with a wide range of benchmarks from the domains of image recognition and medicine, illustrating the advantages of the new method, particularly in computational efficiency. Notably, our approach enables logical explanation of deep networks not amenable to prior logic-based methods.
Tomáš Kolárik, Faezeh Labbaf, Fabrizio Leopardi +3
University of Lugano (USI), Lugano, Switzerland · Florida State University, United States of America · SUPSI, IDSIA, Lugano, Switzerland
Despite significant progress in post-hoc explanation methods for neural networks, many remain heuristic and lack provable guarantees. A key approach for obtaining explanations with provable guarantees is by identifying a cardinally-minimal subset of input features which by itself is provably sufficient to determine the prediction. However, for standard neural networks, this task is often computationally infeasible, as it demands a worst-case exponential number of verification queries in the number of input features, each of which is NP-hard. In this work, we show that for Neural Additive Models (NAMs), a recent and more interpretable neural network family, we can efficiently generate explanations with such guarantees. We present a new model-specific algorithm for NAMs that generates provably cardinally-minimal explanations using only a logarithmic number of verification queries in the number of input features, after a parallelized preprocessing step with logarithmic runtime in the required precision is applied to each small univariate NAM component. Our algorithm not only makes the task of obtaining cardinally-minimal explanations feasible, but even outperforms existing algorithms designed to find the relaxed variant of subset-minimal explanations - which may be larger and less informative but easier to compute - despite our algorithm solving a much more difficult task. Our experiments demonstrate that, compared to previous algorithms, our approach provides provably smaller explanations than existing works and substantially reduces the computation time. Moreover, we show that our generated provable explanations offer benefits that are unattainable by standard sampling-based techniques typically used to interpret NAMs.