cs.NESep 22, 2026
SaveNeuroRule: Making Black-Box Neural Networks Explainable through Rule-set Evolution
Organizations: The University of Texas at Austin, Austin, Texas, USA · Cognizant AI Lab, San Francisco, California, USA
Abstract
High-capacity neural network models have achieved state-of-the-art performance across diverse classification tasks, yet they frequently operate as black-box models, lacking the transparency necessary for critical decision-making. Such opacity creates a persistent trade-off between performance and explainability. This paper proposes a solution to address this gap: the NeuroRule knowledge distillation framework that results in explainable rule-sets from neural network models. NeuroRule adapts the EVOTER rule-set evolution infrastructure to treat neural networks as targets for the evolution process, distilling their performance into concise sets of propositional logic expressions. There are three primary contributions: (1) an evolutionary method for distilling black-box neural network models into explicit rule-set models; (2) a method for making rule sets more explainable by including a conciseness objective to evolution; and (3) a demonstration that the distillation is viable even without access to the original neural network training data. The paper thus establishes that black-box neural network models can be made explainable and therefore useful in real-world applications where trustworthiness is paramount.
Figures & tables
Figure 1. The grammar for the rules. Rules are written in propositional logic, and include feature comparison, feature exponentiation, and time lags. For simplicity, this paper does not use time lags; they are shown for completeness. This representation allows for easy understanding. Figure from ( shahrzad2024evoterevolutiontransparentexplainable ) .
Figure 2. An example rule with elements color-coded from the grammar. Rules can contain multiple conditions, and the rule-set includes a default rule that is used if none of the non-default rules are applied. This diagram demonstrates the standard structure followed by all rules in the rule-sets. Figure from ( shahrzad2024evoterevolutiontransparentexplainable ) .
Figure 3. The NeuroRule method with an Accuracy objective. A target neural network (NN) is first trained to optimize accuracy in predicting the labels in the original dataset. Rule-sets are then evolved through a genetic algorithm to match the behavior of the NN. Whereas the NN is a black box, the rule-sets are explainable. Knowledge in the network is thus distilled into an explainable model.
Figure 4. The NeuroRule method with Accuracy and Conciseness objectives. Conciseness of the solutions is measured through Condition Counts. Optimizing both objectives together results in solutions that are less complex and therefore more explainable than when optimizing for accuracy alone.
| Layer | Type | Dimensions | Activation |
|---|---|---|---|
| Input | Fully Connected | ReLU | |
| Hidden 1 | Fully Connected | ReLU | |
| Output | Fully Connected | Softmax |
Table 1. Architectural specifications of the black-box NN. This network is a multi-layer perceptron (MLP) where represents the input dimensionality of the specific dataset. The design utilizes decreasing hidden layer sizes ( to ) and a final Softmax output that is discretized into binary ( or ) labels. The labels in the original dataset are replaced by the labels generated by the NN to create a fitness function dataset for distillation.
| Parameter | Setting |
|---|---|
| Optimizer | Adam ( ) |
| Learning Rate | |
| Loss Function | Cross-Entropy Loss |
| Batch Size | 32 |
| Training Epochs | 100 |
| Weight Initialization | He Normal |
Table 2. Configuration used to train the black-box NN. To reflect standard machine learning practices, the network is trained using empirical defaults established in modern deep learning literature.
| Hyperparameter | Setting |
|---|---|
| Population Size | 100 |
| Generations | 200 |
| Mutation Probability | 0.7 |
| Selection Method | Tournament |
| Novelty Selection Multiplier | 1.5 |
Table 3. Parameters for rule-set evolution. The same set of parameters was used for the EVOTER baseline and NeuroRule. They are designed to favor aggressive exploration, which is likely to result in diverse, creative solutions.
| In-Distribution (ID) | Out-of-Distribution (OOD) | |||||
| ID/OOD split | NN | EVOTER | NeuroRule | NN | EVOTER | NeuroRule |
| 80% | 0.749 | 0.727 | 0.720 | 0.692 | 0.679 | 0.700 |
| 90% | 0.726 | 0.708 | 0.712 | 0.697 | 0.688 | 0.706 |
| 95% | 0.726 | 0.716 | 0.721 | 0.717 | 0.705 | 0.708 |
Table 4. Experiment 1: Accuracy of NN, EVOTER, and NeuroRule models on ID and OOD data at different ID/OOD splits on the three clinical benchmarks. The numbers are averages of 10 runs; the bolded values identify the best performance in each category. In ID, the NN is the most accurate, suggesting that NeuroRule models indeed evolve to approximate the NN. However, NeuroRule is usually more accurate than EVOTER, suggesting that the NN provides more regularized targets for rule evolution than the original data. In OOD, rule-set networks in general and NeuroRule in particular are more accurate than the NN, suggesting that they have learned the underlying principles of the data, allowing them to extrapolate better. The learning curves with standard deviations are shown in Figures 5 , 9 , and 10 .
Figure 5. Learning curves for EVOTER and NeuroRule compared to the NN accuracy at different ID/OOD splits on the Diabetes dataset. The curves are averages over 10 runs, and the bands indicate standard deviation. The ID graphs (left) demonstrate that NeuroRule rule-sets (green) generally achieved higher accuracy than the EVOTER baseline (blue). This result suggests that the NN helps in avoiding overfitting to noisy samples. Both rule-set approaches initially exhibit a period of rapid improvement and then plateau at a level slightly below the NN (red), suggesting that the rule-sets have a slight performance cost. With OOD (right), NeuroRule (and EVOTER in other datasets) actually performs better than the NN. This result suggests that the rule-sets discover underlying principles of data rather than overfit to noise. Results for the Breast Cancer and Heart Disease datasets are available in Figures 9 and 10 .
Figure 6. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Diabetes dataset under multi-objective optimization of accuracy and conciseness. The curves are averages over 10 runs, and the bands indicate standard deviation. Compared to single-objective optimization, accuracy has improved significantly, suggesting that conciseness encourages rule-set evolution to find representations that are more principled and generalize better. Also, the standard deviation is smaller, presumably because search focuses on a more limited set of solutions. The results for Breast Cancer and Heart Disease datasets are in Figures 11 and 12 .
Figure 7. Accuracy vs. conciseness in single and multi-objective NeuroRule evolutionary runs at different ID/OOD splits on the Diabetes dataset. The model accuracy is on the -axis and conciseness on the -axis; the scatter plot shows all individuals in the final population of all 10 single-objective (blue) and multi-objective (green) runs. The multi-objective solutions are concentrated at the bottom, indicating that they have much lower complexity than single-objective solutions. Notably, the most accurate solutions (furthest to the right) are also often among the most concise (low). Results for the Breast Cancer and Heart Disease datasets are in Figures 13 and 14 .
Figure 8. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Diabetes dataset when NeuroRule was evolved with synthetic data only. The curves are averages over 10 runs and the bands indicate standard deviation. The accuracy of NeuroRule is lower than when evolved with samples from the original dataset, but still viable, demonstrating that the network represents a portable source of knowledge about the domain even when the original dataset is not available. The results for Breast Cancer and Heart Disease datasets are in Figures 15 and 16 .
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Breast Cancer Dataset. This dataset is smaller than Diabetes, but the results are similar. However, likely due to the small amount of data, the EVOTER accuracy has more variance.
Figure 10. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Heart Disease Dataset. This dataset is also smaller, but it turned out to be a particularly good fit for the rule-set approaches: They are more accurate in five of the six cases, including the 80% and 90% ID cases.
Figure 11. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Breast Cancer dataset under multi-objective optimization of accuracy and conciseness. The ID results initial form vertical lines due to limited data but gradually become more scattered like the OOD results as more data is introduced in the training interval. The results are similar to those with the Diabetes dataset, demonstrating significant improvement over single-objective evolution, and less variance.
Figure 12. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Heart Disease dataset under multi-objective optimization of accuracy and conciseness. The results are similar to those with the Diabetes dataset, demonstrating significant improvement over single-objective evolution, and less variance.
Figure 13. Accuracy vs. conciseness in single and multi-objective NeuroRule evolutionary runs at different ID/OOD splits on the Breast Cancer dataset. The ID results initial form vertical lines due to limited data but gradually become more scattered like the OOD results as more data is introduced in the training interval. Otherwise, the results are similar to those with the Diabetes dataset
Figure 14. Accuracy vs. conciseness in single and multi-objective NeuroRule evolutionary runs at different ID/OOD splits on the Heart Disease dataset. As in Figure 13 , the ID results initial form vertical lines due to limited data but gradually become more scattered like the OOD results as more data is introduced in the training interval. The results are otherwise similar to the Diabetes dataset.
Figure 15. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Breast Cancer dataset when NeuroRule was evolved with synthetic data only. The results are similar to those with the Diabetes dataset, demonstrating lower but still viable accuracy even when the original data is not available.
Figure 16. Learning curves for EVOTER and NeuroRule compared to NN accuracy at different ID/OOD splits on the Heart Disease dataset when NeuroRule was evolved with synthetic data only. The results are similar to those with the Diabetes dataset, demonstrating lower but still viable accuracy even when the original data is not available.
More in this topic
Explainable Artificial Intelligence
Explainable artificial intelligence develops ways to make AI decisions understandable, testable, and useful to people.
See all 327 papers