Projected Neural Additive Models as Universal Approximators
Organizations: Terminal Effects Division, DEVCOM Army Research Laboratory, Aberdeen Proving Ground, MD 21005, USA · Department of Civil Engineering & Engineering Mechanics, and Data Science Institute, Columbia University, New York, NY 10027, USA
Abstract
Neural additive models (NAMs) offer interpretability but lack expressivity due to their rigid additive structure, whereas multi-layer perceptrons (MLPs) achieve universal approximation at the cost of transparency. We establish the first theoretical foundation for projected neural additive models (PNAMs), which augment NAMs with a learnable linear transformation to capture cross-variable interactions. While standard NAMs cannot approximate even the simplest interaction term because their cross-derivatives are identically zero, we prove that PNAMs are universal approximators and characterize the minimum number of projections required for exact polynomial representation. Furthermore, we derive an approximation rate of for functions (i.e., functions with continuous derivatives), where denotes the number of projections and the input dimension; this rate is known to be optimal for ridge function approximation. To recover interpretability from the projected inputs, we introduce structured sparsity regularizers and post-hoc symbolic regression techniques that enable global input ranking, parameter pruning, and the conversion of learned bases into compact analytical expressions. Experiments on knot theory invariants, MNIST, and phase field fracture mechanics demonstrate that PNAMs match or surpass similar architectures in accuracy while discovering salient features and yielding symbolic models with orders of magnitude fewer parameters.
Figures & tables
| Method | Architecture | Parameter count | Test acc. |
| DM’s MLP | 4 layers: [17, 300, 300, 300, 14] | 78.0% | |
| KAN | 2 layers: [17, 1, 14] ( , ) | 81.6% | |
| Our MLP | 3 layers: [17, 64, 32, 14] | % | |
| NAM | 3 layers: | % | |
| PNAM | 3 layers: | % | |
| PNAM | 3 layers: | % |
| Rank | Input feature | Symbol | Frequency | Test acc. |
| 1 | Re(meridional translation) | 10/10 | % | |
| 2 | Longitudinal translation | 10/10 | % | |
| 3 | Im(meridional translation) | 7/10 | % | |
| Cusp volume | 1/10 | 80.5% | ||
| Im(short geodesic) | 1/10 | 73.5% | ||
| Volume | 1/10 | 70.5% |
Appendix figures & tables25 assets
Supplementary material from the paper’s appendix.
Appendix
| Remove | Test acc. | |
|---|---|---|
| Top | Bottom | |
| 0 (baseline) | % | % |
| 1 | % | % |
| 2 | % | % |
| 3 | % | % |
| Test acc. | |
|---|---|
| Random, fixed at initialization | % |
| Identity (= NAM, Table 1 ) | % |
| Learned, regularized (Table 1 ) | % |
| Learned, unregularized ( ) | % |
| Method | Architecture | Parameter count | Test acc. |
|---|---|---|---|
| MLP | 3 layers: [17, 64, 32, 1] | % | |
| NAM | 3 layers: | % | |
| PNAM ( ) | 3 layers: | % | |
| PNAM ( ) | 3 layers: | % | |
| PNAM ( ) | 3 layers: | % |
| Rank | Input feature | Symbol | Frequency | Test acc. |
| 1 | Re(meridional translation) | 10/10 | % | |
| 2 | Longitudinal translation | 6/10 | % | |
| Torsion degree | 4/10 | % | ||
| 3 | Torsion degree | 3/10 | % | |
| Re(short geodesic) | 3/10 | % | ||
| Longitudinal translation | 2/10 | % |
| ID | Expression | PC | Discovered by | Eval. of test acc. | Total acc. | |
| Reported | Our | |||||
| A | 3 | Human (DM) | 83.1% | 74.5% | 73.8% | |
| B | 12 | [3, 1] KAN | 62.6% | 27.0% | 26.8% | |
| C | 17 | [3, 1, 1] KAN | 71.9% | 41.7% | 41.5% | |
| Method | Training time (s/epoch) | Inference time (ns/sample) | Test acc. |
|---|---|---|---|
| MLP | 2.1 | 290 | % |
| NAM | 14 | 4300 | % |
| PNAM | 8.9 | 2100 | % |
| Expression G | – | 78 | 75.9% |
| Expression H | – | 130 | 81.2% |
| Expression I | – | 120 | 81.4% |
| Method | Architecture | Parameter count | Training acc. | Test acc. |
|---|---|---|---|---|
| MLP | 3 layers: [784, 64, 32, 10] | 99.9% | 96.1% | |
| MLP | 3 layers: [784, 128, 64, 10] | 99.9% | 96.3% | |
| MLP | 4 layers: [784, 128, 64, 32, 10] | 99.9% | 96.7% | |
| MLP | 4 layers: [784, 256, 128, 64, 10] | 99.9% | 97.1% | |
| NAM | 3 layers: | 95.0% | 93.0% | |
| PNAM | 3 layers: | 96.0% | 94.7% |
| Training acc. | Test acc. | Percent of zero | |
|---|---|---|---|
| 0.001 | 95.6% | 94.3% | 14.1% |
| 0.01 | 96.0% | 94.7% | 32.8% |
| 0.1 | 95.7% | 94.7% | 68.9% |
| 1 | 90.9% | 91.3% | 76.5% |
| Method | Architecture | Parameter count | Test MSE |
|---|---|---|---|
| MLP | 4 layers: [3, 256, 256, 256, 2] | ||
| NAM | 4 layers: | ||
| PNAM | 4 layers: | ||
| PNAM | 4 layers: | ||
| PNAM | 4 layers: | ||
| PNAM | 4 layers: |
| Bases | Projected variables | ||
|---|---|---|---|
| Basis | Expression | Variable | Expression |
| Method | Training time (s/epoch) | Inference time ( s/sample) | Test MSE |
|---|---|---|---|
| MLP | 1.1 | 27 | |
| NAM | 3.0 | 61 | |
| PNAM ( ) | 3.2 | 58 | |
| PNAM ( ) | 6.8 | 150 | |
| PNAM ( ) | 13 | 320 | |
| PNAM ( ) | 26 | 610 |