Modern machine learning models, such as artificial neural networks, achieve high predictive performance through increasingly complex black-box architectures, limiting their use in settings where transparency, interpretability, and fairness are required. Here we present the R package ONAM, which provides a novel and practical implementation of functional decomposition, a technique to facilitate the interpretability of black-box models through structured effect representations. ONAM implements an algorithmic framework for the post-hoc decomposition of predictions into components associated with different input features. The package is designed for usability and efficiency, including functionalities for visualization as well as quantitative measures to assess interpretability. As a model-agnostic method, it can be applied to a wide range of prediction models. We describe the mathematical foundations, algorithmic design, and software architecture of ONAM. A central result of the proposed methodology is that the resulting decomposition satisfies the condition of hierarchical orthogonality, thereby guaranteeing that all components are identifiable and uniquely defined.
Figures & tables
Figure 1: Prediction of median housing prices in the Boston area [ 10 ] . Panel A shows a scatterplot of the true median housing prices (outcome variable) vs. the in-sample predictions of an XGBoost model with socioeconomic features (see Section 3 for a detailed specification of the model). Panels B and C show the main effects of the features “pupil/teacher ratio” and “distance to nearest employment center”, as obtained from decomposing the XGBoost predictions with the ONAM package. Analogously, Panel D shows the interaction effect of the features “average number of rooms per dwelling” and “percentage of houses built prior to 1940”.
Figure 2: A: Illustration of the neural additive model in ( 3 ). In the example considered here, there are two features X1 and X2 . Accordingly, there are two main effects f10 , f20 and the two-way interaction f120 . Each effect is represented by a fully connected artificial neural network (ANN). The units in the penultimate layers of the ANNs are denoted by U1∈Rn×b1 , U2∈Rn×b2 and U12∈Rn×b12 , where b1 , b2 and b12 are the widths of the layers. The outputs of the ANNs are given by the dot products U1w10 , U2w20 and U12w120 , where w10 , w20 and w120 are vectors of weights. These effects can be interpreted using line plots and heatmaps. The values of the prediction function F are given by the sum of the three dot products (hence the term neural additive model). The parameters of the ANNs are estimated jointly using backpropagation. B: Visualization of Step 3 of the algorithm described in Section 2.2 . Here, the orthogonalization of effect f12 onto the effects f1,f2 is depicted. The matrices U1 and U2 are concatenated to Uo . The projection matrix P is computed from Uo . The output of higher-order term f120 is projected onto Uo by left multiplication with P . To obtain the orthogonalized effects, the projection is subtracted from f120 and added to f10 and f20 , corresponding to left multiplication with (I−P) in the former, and weight updates in the latter case (see Equation ( 5 )). Note that the predicted values F remain unchanged, as information is only shifted between subfunctions f10,f20 and f120 .
Figure 3
Figure 3: Example figures for the visualization of an onam model. A: Main effect of the average number of rooms per dwelling on the prediction of the median value of owner-occupied homes in $$$ 1000’s in the Boston area. B: Main effect of the proportion of owner-occupied units built prior to 1940. C: Plot of the interaction between the average room number and the proportion of owner-occupied units built prior to 1940.
Figure 4: Main effect of the categorical feature “District adjacent to Charles River” on the prediction of the median value of owner-occupied homes in $$$ 1000’s in the Boston area.
Figure 5: Example figures for the decomposition of an XGBoost model for the diagnosis of the binary outcome prevalent heart disease (defined by the presence or absence of > 50% coronary artery narrowing) in the Cleveland Heart Disease dataset [ 8 ] . A: Main effect of serum cholesterol on the logit scale. B: Main effect of resting electrocardiographic results on the logit scale. C: Plot of the interaction effect between age and maximum heart rate during exercise test on logit scale.
Neural network interpretability studies how models represent information and make predictions, and how to explain, test, or control their internal computations.
The functional ANOVA, or Hoeffding decomposition, provides a principled framework for interpretability by decomposing a model prediction into main effects and higher-order interactions. For independent inputs, this classical decomposition is explicit. It is closely connected to SHAP values, generalized additive models, and orthogonal polynomial expansions, and therefore constitutes a fundamental tool for additive explainability. In the more general and realistic dependent setting, however, obtaining a tractable representation and estimating the decomposition from data remain challenging. In this work, we address this problem for continuous inputs. By combining Hilbert space methods with the generalized functional ANOVA, we build an explicit decomposition Riesz Basis allowing to easily compute the decomposition. Our formulation recovers the classical independent case and its associated orthogonal decomposition. Building on this representation, we propose a simple but mighty algorithm to estimate the decomposition from a data sample in a model-agnostic setting and we compare it empirically with several state-of-the-art explanation methods, demonstrating the power of the approach.
Baptiste Ferrere, Nicolas Bousquet, Fabrice Gamboa +1
Post-hoc explanation methods for neural networks are often heuristic and lack provable guarantees. A common alternative to such methods that was studied in recent years is to compute a cardinally-minimal subset of input features that is provably sufficient to determine the model's prediction. For general neural networks, however, finding such explanations is computationally intractable. In contrast, recent work showed that this problem becomes more tractable for a particular family of neural networks, namely neural additive models (NAMs), though standard NAMs are highly restrictive because they exclude feature interactions altogether. In this work, we study the computation of provably minimal and sufficient explanations for neural networks with restricted feature interactions, bridging the gap between fully general neural networks and purely additive models. We first prove that even with constant-order interactions, the problem is Σ2P-hard, implying that worst-case exponential complexity is unavoidable. We then identify two provably tractable settings: one based on sparsity in the interaction structure and another based on a stronger interaction-aware notion of sufficiency. Under either of these settings, we develop algorithms that compute provable explanations for neural networks with any-order interactions while retaining much of the computational efficiency of additive models. Empirically, we show that these explanations are significantly smaller and faster to obtain than those produced by standard algorithms, while being derived from models that achieve much higher accuracy than standard NAMs. Overall, our results help expand the class of neural networks that admit efficient provable explanations, while clarifying the fundamental role of feature interactions in shaping their complexity.
Idan Refaeli, Yizhak Yisrael Elboher, Guy Katz +1
The Hebrew University of Jerusalem · Masaryk University
Despite significant progress in post-hoc explanation methods for neural networks, many remain heuristic and lack provable guarantees. A key approach for obtaining explanations with provable guarantees is by identifying a cardinally-minimal subset of input features which by itself is provably sufficient to determine the prediction. However, for standard neural networks, this task is often computationally infeasible, as it demands a worst-case exponential number of verification queries in the number of input features, each of which is NP-hard. In this work, we show that for Neural Additive Models (NAMs), a recent and more interpretable neural network family, we can efficiently generate explanations with such guarantees. We present a new model-specific algorithm for NAMs that generates provably cardinally-minimal explanations using only a logarithmic number of verification queries in the number of input features, after a parallelized preprocessing step with logarithmic runtime in the required precision is applied to each small univariate NAM component. Our algorithm not only makes the task of obtaining cardinally-minimal explanations feasible, but even outperforms existing algorithms designed to find the relaxed variant of subset-minimal explanations - which may be larger and less informative but easier to compute - despite our algorithm solving a much more difficult task. Our experiments demonstrate that, compared to previous algorithms, our approach provides provably smaller explanations than existing works and substantially reduces the computation time. Moreover, we show that our generated provable explanations offer benefits that are unattainable by standard sampling-based techniques typically used to interpret NAMs.