Understanding the features captured by the hidden layers of neural networks is a fundamental challenge in machine learning, despite their widespread success across various classification problems. In this work, we propose an algebraic framework for examining neural networks that model classification problems. Certain results, such as the correspondence between the neural network and neural ideals, algorithms for computing the neural ideals, and a stabilization theorem that enables approximation of the neural ideals, are first established. As an application to the framework, we present algorithms to identify and interpret the features captured by each hidden-layer neuron. Along with these theoretical developments, the practical performance has been demonstrated on the MNIST digit dataset, and the results highlight the pivotal role of neural ideals as a mathematical and computational tool for analyzing the features captured by neural networks. Further, we develop an interactive software that builds on the presented framework to visualize the features captured by each neuron. This tool is available at https://github.com/yvs1967/neural-network-representation-explorer
Figures & tables
Figure 1. The 5-neuron feedforward network layout optimized for XOR classification, including corresponding weights ( wij ) and activation thresholds ( θi ).
Input
Class
Neural code
(0,0)
0
(0,0,0,0)
(1,0)
1
(1,0,1,0)
(0,1)
1
(0,1,1,0)
(1,1)
0
(1,1,1,1)
Table 1. Neural codes of XOR
Figure 2. Visualization of learned features and output-layer weights.
Figure 3. Graphical interface of the Neural Network Representation Explorer implementing the proposed framework. The interface visualizes network connectivity, activation states, threshold-map information, allowed code spaces, and class-specific feature representations.
Figure 4. Feature balance maps generated using the proposed framework for different digit classes. Red regions indicate activation-supporting structures, whereas blue regions correspond to suppressive regions. The visualizations provide an interpretable description of the representations learned by the network.
Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.
Laboratory of Parallel and Distributed Systems Institute for Computer Science and Control (SZTAKI) · John von Neumann Faculty of Informatics Obuda University Budapest, Hungary