TreePIN: Tree-Prior Initialized Neural Networks
Organizations: Department of Computer Science, National University of Singapore, Singapore, Singapore
Abstract
Fitted decision trees provide predictive knowledge for neural network initialization. We present Tree-Prior Initialized Neural Networks (TreePIN), a method that translates split rules, decision paths and leaf predictions into initial network weights and biases for classification and regression. Sigmoid units encode split responses, hidden units combine them along decision paths, and leaf predictions initialize the output layer. We calibrate split softness using local training margins, sample support and impurity reduction, relating the initial responses to the data at each node. All weights and biases remain trainable, allowing gradient-based optimization to refine the transferred decisions and their combinations. Across six classification and six regression tasks with twenty paired seeds, TreePIN improves mean initial and early validation performance over initialization baselines on most tasks. After fifty epochs, it achieves the best or tied-best mean validation performance among the six compared methods on ten tasks. Paired ablations show how split calibration and leaf-informed output initialization contribute to the starting predictor and its subsequent learning.
Figures & tables
| Initialization | Bank | Banknote | Breast Cancer | Digits | Spambase | Wine |
|---|---|---|---|---|---|---|
| Random | ||||||
| He | ||||||
| Xavier | ||||||
| STBI | ||||||
| DJINN | ||||||
| TreePIN 2 |
| Initialization | Digits F1 (%) | Spambase F1 (%) | Concrete RMSE (MPa) | Protein RMSE (RMSD) | Gas CO RMSE (mg/m 3 ) |
|---|---|---|---|---|---|
| Without adaptive scales and leaf readout | |||||
| Without adaptive scales | |||||
| Without leaf readout | |||||
| TreePIN 2 |
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Task | Source | Split | Target / units | ||
| Classification | |||||
| Bank | UCI 222 (full dataset) | 45,211 | 16 | G | subscription, 2 classes |
| Banknote | UCI 267 | 1,372 | 4 | C | authenticity, 2 classes |
| Breast Cancer | Wisconsin Diagnostic Breast Cancer (scikit-learn) | 569 | 30 | C | diagnosis, 2 classes |
| Digits | scikit-learn Digits | 1,797 | 64 | C | digit, 10 classes |
| Spambase | UCI 94 | 4,601 | 57 | C | spam, 2 classes |
| Initial | Early (1–10) | Epoch 50 | ||||
|---|---|---|---|---|---|---|
| Task | TreePIN 2 | He | TreePIN 2 | He | TreePIN 2 | He |
| Bank | ||||||
| Banknote | ||||||
| Breast Cancer | ||||||
| Digits | ||||||
| Spambase | ||||||
| Initial | Early (1–10) | Epoch 50 | ||||
|---|---|---|---|---|---|---|
| Task | TreePIN 2 | He | TreePIN 2 | He | TreePIN 2 | He |
| California ( USD) | ||||||
| Diabetes (score) | ||||||
| Airfoil (dB) | ||||||
| Concrete (MPa) | ||||||
| Protein (RMSD) | ||||||
| Task | Initial difference | Early difference | Epoch-50 difference | Wins / ties |
| Classification (pp) | ||||
| Bank | 17/20, 0 | |||
| Banknote | 0/20, 20 | |||
| Breast Cancer | 14/20, 3 | |||
| Digits | 18/20, 0 | |||
| Spambase | 16/20, 0 | |||
| Initialization | Digits F1 (%) | Spambase F1 (%) | Concrete RMSE (MPa) | Protein RMSE (RMSD) | Gas CO RMSE (mg/m 3 ) |
|---|---|---|---|---|---|
| Before optimization | |||||
| Without both | |||||
| Without adaptive scales | |||||
| Without leaf readout | |||||
| TreePIN 2 | |||||
| Mean over epochs 1–10 | |||||
| (a) Twenty-seed factorial: epoch 50 | |||||
|---|---|---|---|---|---|
| Task | Scale/leaf | Scale/random | Leaf/adaptive | Leaf/fixed | Interaction |
| Digits | |||||
| Spambase | |||||
| Concrete (MPa) | |||||
| Protein (RMSD) | |||||
| Gas CO (mg/m 3 ) | |||||
| Task | Metric | Scale effect leaf readout | Scale effect random readout | Readout effect adaptive scales | Readout effect fixed scales |
|---|---|---|---|---|---|
| Digits | F1 pp | ||||
| Spambase | F1 pp | ||||
| Concrete | RMSE (MPa) | ||||
| Protein | RMSE (RMSD) | ||||
| Gas CO | RMSE (mg/m 3 ) |
| Task | Fixed-scale CE | TreePIN 2 CE | CE difference | Fixed-scale F1 | TreePIN 2 F1 | F1 difference |
|---|---|---|---|---|---|---|
| Bank | ||||||
| Banknote | ||||||
| Breast Cancer | ||||||
| Digits | ||||||
| Spambase | ||||||
| Wine |
| Task | Control CE | TreePIN 2 CE | CE difference | Control F1 | TreePIN 2 F1 | F1 difference |
|---|---|---|---|---|---|---|
| Digits | ||||||
| Spambase |
| Configuration | Superconductor (K) | Diamonds (log target) | Airbnb (USD) |
|---|---|---|---|
| Glorot/Tanh | |||
| Mixed ReLU | SF | ||
| Same-graph He | |||
| STBI | |||
| DJINN | SF | ||
| TreePIN 2 |
| Task | effect | Early effect | effect | 95% CI | W/T/L | |
| Classification (Macro-F1 pp) | ||||||
| Bank | 14/0/6 | |||||
| Banknote | 1/19/0 | |||||
| Breast Cancer | 10/8/2 | |||||
| Digits | 10/0/10 | |||||
| Spambase | 13/0/7 | |||||
| Task | Metric | Source tree | Network | Network | Agreement (%) |
|---|---|---|---|---|---|
| Bank | Macro-F1 | ||||
| Accuracy | – | ||||
| Banknote | Macro-F1 | ||||
| Accuracy | – | ||||
| Breast Cancer | Macro-F1 | ||||
| Accuracy | – |
| Task | Metric | Source tree | Network | Network | Normalized error |
|---|---|---|---|---|---|
| California | RMSE | ||||
| – | |||||
| Diabetes | RMSE | ||||
| – | |||||
| Airfoil | RMSE | ||||
| – |
| Task | Stage | Random | He | Xavier | STBI | DJINN |
|---|---|---|---|---|---|---|
| Bank | 1 | 1 | 1 | 1 | 1 | |
| Bank | 1 | 1 | 1 | 1 | 1 | |
| Bank | 1 | 1 | 1 | 1 | 1 | |
| Banknote | 1 | 1 | 1 | 1 | 1 | |
| Banknote | 1 | 1 | 1 | 0.0021 | 0.0003 | |
| Banknote | 1 | 1 | 1 | 1 | 1 |
| Method | Bank | Banknote | Breast Cancer | Digits | Spambase | Wine | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| /20 | KM | /20 | KM | /20 | KM | /20 | KM | /20 | KM | /20 | KM | |
| Random | 19 | 1060 | 19 | 91 | 17 | 45 | 15 | 144 | 18 | 154 | 10 | 49 |
| He | 20 | 1696 | 20 | 133 | 20 | 69 | 20 | 252 | 20 | 154 | 20 | 48 |
| Xavier | 19 | 1272 | 18 | 98 | 18 | 42 | 16 | 126 | 19 | 132 | 12 | 44 |
| STBI | 18 | 1484 | 8 | NR | 8 | NR | 0 | NR | 19 | 44 | 11 | 44 |
| DJINN | 15 | 2120 | 16 | 182 | 18 | 84 | 0 | NR | 18 | 176 | 3 | NR |
| Task | Random | He | Xavier | STBI | DJINN |
|---|---|---|---|---|---|
| Bank | 1 | 1 | 1 | 1 | 1 |
| Banknote | 1 | 0.1810 | 1 | 0.0765 | 0.1810 |
| Breast Cancer | 1 | 0.4230 | 1 | 0.8556 | 0.1229 |
| Digits | 1 | 0.8556 | 1 | 0.0002 | 0.0002 |
| Spambase | 1 | 1 | 1 | 1 | 1 |
| Wine | 0.0177 | 0.2808 | 0.0620 |
Neural Network Initialization
Neural network initialization studies how starting weights shape signal propagation, optimization, generalization, and reproducibility.
See all 56 papers