Quantifying structural stability of transparent models for general insurance pricing
Organizations: École d’actuariat, Université Laval, Canada
Abstract
The short duration of general insurance contracts leads to rapid distributional shifts in the data, warranting regular model updates. This process may yield a post-update model that differs considerably from the pre-update model in both predictions and structure. Structural stability is upstream of predictive stability, yet existing structural stability metrics are mostly restricted to simple decision trees. Thus, we first introduce the notion of functional transparency, characterising models with a prediction function that can be expressed as a sum of univariate functions for main effects and bivariate functions for pairwise interactions. Then, we propose the Mean Squared Shape function distance, a metric to quantify the structural stability of functionally transparent models, which are particularly relevant in actuarial science. Using three public auto insurance datasets, we present a benchmark study to demonstrate the usefulness of our metric in identifying a stable and accurate transparent model. Generalised linear models with centred numerical covariates are stable, and explainable boosting machines (EBMs) offer a compromise between stability and performance. We also find that stabilisation strategies, such as sequential modelling and fixed hyperparameters, can substantially improve stability for both EBM and XGBoost models, with negligible impact on performance.
Figures & tables
| Dataset | |||
| norauto | 183 999 | 4 | 4 |
| beMTPL97 | 163 212 | 11 | 5 |
| pg15training | 100 021 | 13 | 7 |
| Model | RMSS | MRD | Dev |
| GLM | 1.0 | 1.0 | 3.3 |
| GAM | 3.0 | 2.7 | 2.7 |
| EBM | 2.0 | 3.0 | 2.0 |
| XGBoost | 3.3 | 2.0 |
| norauto | beMTPL97 | pg15training | ||||||||
| Model | Metric | Frequency | Severity | Loss cost | Frequency | Severity | Loss cost | Frequency | Severity | Loss cost |
| EBM | RMSS | 68.8 | 3.5 | 54.4 | 52.2 | 70.6 | 59.3 | 56.2 | 61.4 | 49.5 |
| MRD | 72.4 | 66.8 | 48.3 | 60.6 | 64.6 | 63.2 | 81.2 | 70.2 | 88.4 | |
| Dev | 0.054 | 0.255 | 0.060 | 0.099 | 0.791 | 2.145 | 0.681 | 0.071 | 0.765 | |
| XGBoost | MRD | 31.5 | 59.3 | 31.1 | 72.5 | 50.3 | 60.2 | 75.9 | 36.3 | 73.6 |
| Dev | 0.007 | 0.406 | 0.050 | 0.030 | 0.620 | 0.259 | 0.308 | 0.128 | 0.209 | |
| norauto | beMTPL97 | pg15training | ||||||||
| Model | Metric | Frequency | Severity | Loss cost | Frequency | Severity | Loss cost | Frequency | Severity | Loss cost |
| EBM | RMSS | 9.1 | 2.6 | 0.4 | 33.5 | 10.6 | 41.4 | 8.6 | 28.0 | 3.9 |
| MRD | 4.6 | 2.9 | 0.9 | 14.7 | 6.5 | 37.6 | 15.7 | 20.5 | 3.4 | |
| Dev | 0.006 | 0.023 | 0.014 | 0.097 | 0.053 | 1.136 | 0.246 | 0.012 | 0.009 | |
| XGBoost | MRD | 1.7 | 0.2 | 4.9 | 46.4 | 0.4 | 9.6 | 2.6 | 1.2 | 0.3 |
| Dev | 0.001 | 0.021 | 0.003 | 0.021 | 0.091 | 0.025 | 0.143 | 0.057 | 0.284 | |
| norauto | beMTPL97 | pg15training | |||||||
| Frequency | Severity | Loss cost | Frequency | Severity | Loss cost | Frequency | Severity | Loss cost | |
| RMSS | 0.8 | 0.2 | 0.7 | 16.5 | 0.1 | 7.3 | 5.5 | 4.1 | 3.9 |
| MRD | 0.1 | 2.5 | 0.9 | 17.1 | 0.1 | 8.5 | 7.4 | 2.7 | 5.1 |
| Dev | 0.001 | 0.027 | 0.012 | 0.008 | 0.002 | 0.167 | 0.003 | 0.051 | 0.008 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Grid |
| smoothing_rounds | |
| interactions | |
| max_bins | |
| max_leaves |
| Hyperparameter | Grid |
| max_depth | |
| iteration_range |
| norauto | beMTPL97 | pg15training | |||||
| Model | Hyperparameter | Pre | Post | Pre | Post | Pre | Post |
| Frequency | smoothing_rounds | 100 | 1000 | 500 | 500 | 100 | 100 |
| interactions | 0.9 | 0.9 | 0.75 | 0.75 | 0.9 | 0.5 | |
| max_bins | 32 | 32 | 32 | 1024 | 4096 | 1024 | |
| max_leaves | 3 | 4 | 3 | 4 | 3 | 4 | |
| Severity | smoothing_rounds | 500 | 500 | 500 | 100 | 1000 | 100 |
| norauto | beMTPL97 | pg15training | |||||
| Model | Hyperparameter | Pre | Post | Pre | Post | Pre | Post |
| Frequency | max_depth | 3 | 3 | 3 | 5 | 5 | 5 |
| iteration_range | 1700 | 1700 | 1900 | 1900 | 1700 | 1900 | |
| Severity | max_depth | 3 | 3 | 3 | 3 | 3 | 3 |
| iteration_range | 1500 | 1500 | 1100 | 1100 | 1100 | 1100 | |
| Loss cost | max_depth | 3 | 3 | 3 | 3 | 3 | 3 |
| Dataset | Model | RMSS | MRD | Dev |
| norauto | GLM | 2.0 | 1.3 | 1.3 |
| EBM | 1.0 | 2.0 | 2.3 | |
| XGBoost | 2.7 | 2.3 | ||
| beMTPL97 | GLM | 1.0 | 1.0 | 3.3 |
| GAM | 3.0 | 2.7 | 2.7 | |
| EBM | 2.0 | 3.0 | 2.0 |
| Frequency | Severity | Loss cost | ||||||||
| Dataset | Model | RMSS | MRD | Dev | RMSS | MRD | Dev | RMSS | MRD | Dev |
| norauto | GLM | 0.0424 | 0.0229 | 0.2696 | 0.0391 | 0.0241 | 1.0620 | 0.0830 | 0.0415 | 200.67 |
| EBM | 0.0323 | 0.0241 | 0.2775 | 0.0230 | 0.0147 | 1.0596 | 0.0626 | 0.0523 | 206.78 | |
| XGBoost | — | 0.0273 | 0.2698 | — | 0.0318 | 1.0623 | — | 0.0501 | 200.77 | |
| beMTPL97 | GLM | 0.0208 | 0.0167 | 0.5371 | 0.0650 | 0.0440 | 2.0530 | 0.0623 | 0.0442 | 75.29 |
| GAM | 0.1592 | 0.0322 | 0.5368 | 0.0917 | 0.0640 | 2.0454 | 0.2131 | 0.0787 | 103.58 | |
| Dataset | Model | RMSS | MRD | Dev |
| norauto | GLM | 2.0 | 3.0 | 1.0 |
| EBM | 1.0 | 1.0 | 2.7 | |
| XGBoost | 2.0 | 2.3 | ||
| beMTPL97 | GLM | 1.3 | 2.0 | 2.0 |
| EBM | 1.7 | 2.3 | 2.3 | |
| XGBoost | 1.7 | 1.7 |
| Frequency | Severity | Loss cost | ||||||||
| Dataset | Model | RMSS | MRD | Dev | RMSS | MRD | Dev | RMSS | MRD | Dev |
| norauto | GLM | 0.0424 | 0.0229 | 0.2696 | 0.0391 | 0.0241 | 1.0620 | 0.0830 | 0.0415 | 200.67 |
| EBM | 0.0101 | 0.0066 | 0.2776 | 0.0238 | 0.0049 | 1.0623 | 0.0285 | 0.0270 | 206.91 | |
| XGBoost | — | 0.0187 | 0.2699 | — | 0.0130 | 1.0667 | — | 0.0345 | 200.87 | |
| beMTPL97 | GLM | 0.0208 | 0.0167 | 0.5371 | 0.0650 | 0.0440 | 2.0530 | 0.0623 | 0.0442 | 75.29 |
| EBM | 0.0518 | 0.0241 | 0.5352 | 0.0211 | 0.0192 | 2.0631 | 0.0693 | 0.0468 | 76.88 | |
| Frequency | Severity | Loss cost | ||||||||
| Dataset | Model | RMSS | MRD | Dev | RMSS | MRD | Dev | RMSS | MRD | Dev |
| norauto | EBM | 0.0294 | 0.0230 | 0.2775 | 0.0236 | 0.0142 | 1.0598 | 0.0623 | 0.0519 | 206.81 |
| XGBoost | — | 0.0268 | 0.2698 | — | 0.0318 | 1.0626 | — | 0.0476 | 200.76 | |
| beMTPL97 | EBM | 0.0720 | 0.0521 | 0.5352 | 0.0642 | 0.0508 | 2.0480 | 0.0996 | 0.0793 | 76.12 |
| XGBoost | — | 0.0400 | 0.5326 | — | 0.0569 | 2.0587 | — | 0.0837 | 74.90 | |
| pg15training | EBM | 0.1311 | 0.0893 | 0.4876 | 0.0448 | 0.0357 | 1.0421 | 0.1732 | 0.1384 | 46.00 |
| Frequency | Severity | Loss cost | ||||||||
| Dataset | Fixed hyperparameters | RMSS | MRD | Dev | RMSS | MRD | Dev | RMSS | MRD | Dev |
| norauto | No | 0.0329 | 0.0241 | 0.2775 | 0.0230 | 0.0145 | 1.0601 | 0.0619 | 0.0514 | 206.79 |
| Yes | 0.0291 | 0.0230 | 0.2775 | 0.0236 | 0.0146 | 1.0601 | 0.0619 | 0.0514 | 206.79 | |
| beMTPL97 | No | 0.1230 | 0.0569 | 0.5348 | 0.0711 | 0.0530 | 2.0473 | 0.1710 | 0.1411 | 75.16 |
| Yes | 0.0601 | 0.0432 | 0.5351 | 0.0642 | 0.0509 | 2.0480 | 0.0924 | 0.0725 | 75.99 | |
| pg15training | No | 0.1404 | 0.0875 | 0.4870 | 0.0415 | 0.0345 | 1.0418 | 0.1664 | 0.1313 | 46.01 |
| norauto | beMTPL97 | pg15training | |||||||
| Frequency | Severity | Loss cost | Frequency | Severity | Loss cost | Frequency | Severity | Loss cost | |
| RMSS | 1.8 | 0.1 | 1.1 | 13.7 | 0.9 | 0.6 | 2.0 | 33.5 | 7.7 |
| MRD | 0.1 | 1.0 | 1.8 | 6.9 | 2.4 | 11.0 | 17.4 | 23.1 | 8.4 |
| Dev | 0.001 | 0.049 | 0.002 | 0.025 | 0.017 | 0.132 | 0.130 | 0.015 | 0.018 |
Interpretable ML
Interpretable machine learning develops models and explanations that make predictions, evidence, and decision processes understandable to people.
See all 329 papers