The short duration of general insurance contracts leads to rapid distributional shifts in the data, warranting regular model updates. This process may yield a post-update model that differs considerably from the pre-update model in both predictions and structure. Structural stability is upstream of predictive stability, yet existing structural stability metrics are mostly restricted to simple decision trees. Thus, we first introduce the notion of functional transparency, characterising models with a prediction function that can be expressed as a sum of univariate functions for main effects and bivariate functions for pairwise interactions. Then, we propose the Mean Squared Shape function distance, a metric to quantify the structural stability of functionally transparent models, which are particularly relevant in actuarial science. Using three public auto insurance datasets, we present a benchmark study to demonstrate the usefulness of our metric in identifying a stable and accurate transparent model. Generalised linear models with centred numerical covariates are stable, and explainable boosting machines (EBMs) offer a compromise between stability and performance. We also find that stabilisation strategies, such as sequential modelling and fixed hyperparameters, can substantially improve stability for both EBM and XGBoost models, with negligible impact on performance.
Figures & tables
Dpre={(yi,xi,,ti):ti≤T,i=1,…,npost}
Algorithm 1 Construction of the pre- and post-update training and test datasets with time stamps.
Figure 1: Construction of the pre-update and post-update training and test datasets using a time stamp.
Figure 2: Simplified comparison of standard retraining and sequential modelling.
Dataset
n
p
pcat
norauto
183 999
4
4
beMTPL97
163 212
11
5
pg15training
100 021
13
7
Table 1: Overview of the datasets.
Model
RMSS
MRD
Dev
GLM
1.0
1.0
3.3
GAM
3.0
2.7
2.7
EBM
2.0
3.0
2.0
XGBoost
−
3.3
2.0
Table 2: Average rank of each metric over the three responses on beMTPL97 .
Figure 3: Relation between the MRD and the RMSS coloured by Poisson deviance for each dataset.
Figure 4: Relative dislocation in terms of the logarithm of individual MSS for the GLM (cross), GAM (triangle), and EBM (circle) frequency models on beMTPL97 (left). Corresponding boxplot of relative dislocation by model class (right). Both panels are based on the same subset of observations from the post-update test dataset.
Figure 5: Main effect MSS for frequency GLM and GAM on beMTPL97 .
Figure 6: Main effect and interaction MSS for frequency EBM on beMTPL97 .
Figure 7: Feature density (top), pre- and post-update shape functions from the three transparent models (middle), and pre- and post-update partial dependence plots from XGBoost (bottom) for agec with the frequency in beMTPL97 .
Figure 8: Feature density (top), pre- and post-update shape functions from the three transparent models (middle), and pre- and post-update partial dependence plots from XGBoost (bottom) for lat with the frequency in beMTPL97 .
norauto
beMTPL97
pg15training
Model
Metric
Frequency
Severity
Loss cost
Frequency
Severity
Loss cost
Frequency
Severity
Loss cost
EBM
RMSS
− 68.8
+ 3.5
− 54.4
− 52.2
− 70.6
− 59.3
− 56.2
− 61.4
− 49.5
MRD
− 72.4
− 66.8
− 48.3
− 60.6
− 64.6
− 63.2
− 81.2
− 70.2
− 88.4
Dev
+ 0.054
+ 0.255
+ 0.060
+ 0.099
+ 0.791
+ 2.145
+ 0.681
+ 0.071
+ 0.765
XGBoost
MRD
− 31.5
− 59.3
− 31.1
− 72.5
− 50.3
− 60.2
− 75.9
− 36.3
− 73.6
Dev
+ 0.007
+ 0.406
+ 0.050
− 0.030
+ 0.620
+ 0.259
+ 0.308
− 0.128
+ 0.209
Table 3: Variations (in %) of each metric when using sequential modelling.
norauto
beMTPL97
pg15training
Model
Metric
Frequency
Severity
Loss cost
Frequency
Severity
Loss cost
Frequency
Severity
Loss cost
EBM
RMSS
− 9.1
+ 2.6
− 0.4
− 33.5
− 10.6
− 41.4
− 8.6
− 28.0
− 3.9
MRD
− 4.6
− 2.9
− 0.9
− 14.7
− 6.5
− 37.6
− 15.7
− 20.5
− 3.4
Dev
+ 0.006
+ 0.023
+ 0.014
+ 0.097
+ 0.053
+ 1.136
+ 0.246
+ 0.012
+ 0.009
XGBoost
MRD
− 1.7
+ 0.2
− 4.9
− 46.4
− 0.4
− 9.6
− 2.6
− 1.2
+ 0.3
Dev
+ 0.001
+ 0.021
− 0.003
− 0.021
+ 0.091
− 0.025
+ 0.143
− 0.057
+ 0.284
Table 4: Variations (in %) of each metric when fixing hyperparameters.
norauto
beMTPL97
pg15training
Frequency
Severity
Loss cost
Frequency
Severity
Loss cost
Frequency
Severity
Loss cost
RMSS
− 0.8
− 0.2
− 0.7
− 16.5
+ 0.1
− 7.3
− 5.5
− 4.1
− 3.9
MRD
+ 0.1
+ 2.5
− 0.9
− 17.1
+ 0.1
− 8.5
− 7.4
− 2.7
− 5.1
Dev
+ 0.001
+ 0.027
− 0.012
− 0.008
+ 0.002
− 0.167
− 0.003
− 0.051
+ 0.008
Table 5: Variations (in %) of each metric for EBMs when fixing hyperparameters and interaction terms.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Grid
smoothing_rounds
{100,500,1000}
interactions
{0.5,0.75,0.9}
max_bins
{32,1024,4096}
max_leaves
{2,3,4}
Appendix
Table 6: Hyperparameter grid used to tune EBM models.
Hyperparameter
Grid
max_depth
{3,5,7}
iteration_range
{100,300,…,1900}
Appendix
Table 7: Hyperparameter grid used to tune XGBoost models.
norauto
beMTPL97
pg15training
Model
Hyperparameter
Pre
Post
Pre
Post
Pre
Post
Frequency
smoothing_rounds
100
1000
500
500
100
100
interactions
0.9
0.9
0.75
0.75
0.9
0.5
max_bins
32
32
32
1024
4096
1024
max_leaves
3
4
3
4
3
4
Severity
smoothing_rounds
500
500
500
100
1000
100
Appendix
Table 8: Optimal hyperparameters for EBM models.
norauto
beMTPL97
pg15training
Model
Hyperparameter
Pre
Post
Pre
Post
Pre
Post
Frequency
max_depth
3
3
3
5
5
5
iteration_range
1700
1700
1900
1900
1700
1900
Severity
max_depth
3
3
3
3
3
3
iteration_range
1500
1500
1100
1100
1100
1100
Loss cost
max_depth
3
3
3
3
3
3
Appendix
Table 9: Optimal hyperparameters for XGBoost models.
Dataset
Model
RMSS
MRD
Dev
norauto
GLM
2.0
1.3
1.3
EBM
1.0
2.0
2.3
XGBoost
−
2.7
2.3
beMTPL97
GLM
1.0
1.0
3.3
GAM
3.0
2.7
2.7
EBM
2.0
3.0
2.0
Appendix
Table 10: Average rank of each metric over the three responses.
Frequency
Severity
Loss cost
Dataset
Model
RMSS
MRD
Dev
RMSS
MRD
Dev
RMSS
MRD
Dev
norauto
GLM
0.0424
0.0229
0.2696
0.0391
0.0241
1.0620
0.0830
0.0415
200.67
EBM
0.0323
0.0241
0.2775
0.0230
0.0147
1.0596
0.0626
0.0523
206.78
XGBoost
—
0.0273
0.2698
—
0.0318
1.0623
—
0.0501
200.77
beMTPL97
GLM
0.0208
0.0167
0.5371
0.0650
0.0440
2.0530
0.0623
0.0442
75.29
GAM
0.1592
0.0322
0.5368
0.0917
0.0640
2.0454
0.2131
0.0787
103.58
Appendix
Table 11: Model stability and performance across datasets for all three responses.
Figure 9: Feature density (top), pre- and post-update shape functions from the three transparent models (middle), and pre- and post-update partial dependence plots from XGBoost (bottom) for ageph with the frequency in beMTPL97 .
Dataset
Model
RMSS
MRD
Dev
norauto
GLM
2.0
3.0
1.0
EBM
1.0
1.0
2.7
XGBoost
−
2.0
2.3
beMTPL97
GLM
1.3
2.0
2.0
EBM
1.7
2.3
2.3
XGBoost
−
1.7
1.7
Appendix
Table 12: Average rank of each metric over the three responses under sequential modelling.
Frequency
Severity
Loss cost
Dataset
Model
RMSS
MRD
Dev
RMSS
MRD
Dev
RMSS
MRD
Dev
norauto
GLM
0.0424
0.0229
0.2696
0.0391
0.0241
1.0620
0.0830
0.0415
200.67
EBM
0.0101
0.0066
0.2776
0.0238
0.0049
1.0623
0.0285
0.0270
206.91
XGBoost
—
0.0187
0.2699
—
0.0130
1.0667
—
0.0345
200.87
beMTPL97
GLM
0.0208
0.0167
0.5371
0.0650
0.0440
2.0530
0.0623
0.0442
75.29
EBM
0.0518
0.0241
0.5352
0.0211
0.0192
2.0631
0.0693
0.0468
76.88
Appendix
Table 13: Model stability and performance across datasets for all three responses under sequential modelling.
Frequency
Severity
Loss cost
Dataset
Model
RMSS
MRD
Dev
RMSS
MRD
Dev
RMSS
MRD
Dev
norauto
EBM
0.0294
0.0230
0.2775
0.0236
0.0142
1.0598
0.0623
0.0519
206.81
XGBoost
—
0.0268
0.2698
—
0.0318
1.0626
—
0.0476
200.76
beMTPL97
EBM
0.0720
0.0521
0.5352
0.0642
0.0508
2.0480
0.0996
0.0793
76.12
XGBoost
—
0.0400
0.5326
—
0.0569
2.0587
—
0.0837
74.90
pg15training
EBM
0.1311
0.0893
0.4876
0.0448
0.0357
1.0421
0.1732
0.1384
46.00
Appendix
Table 14: Model stability and performance across datasets for all three responses using fixed hyperparameters.
Frequency
Severity
Loss cost
Dataset
Fixed hyperparameters
RMSS
MRD
Dev
RMSS
MRD
Dev
RMSS
MRD
Dev
norauto
No
0.0329
0.0241
0.2775
0.0230
0.0145
1.0601
0.0619
0.0514
206.79
Yes
0.0291
0.0230
0.2775
0.0236
0.0146
1.0601
0.0619
0.0514
206.79
beMTPL97
No
0.1230
0.0569
0.5348
0.0711
0.0530
2.0473
0.1710
0.1411
75.16
Yes
0.0601
0.0432
0.5351
0.0642
0.0509
2.0480
0.0924
0.0725
75.99
pg15training
No
0.1404
0.0875
0.4870
0.0415
0.0345
1.0418
0.1664
0.1313
46.01
Appendix
Table 15: Model stability and performance across datasets for all three responses using EBMs with fixed features.
norauto
beMTPL97
pg15training
Frequency
Severity
Loss cost
Frequency
Severity
Loss cost
Frequency
Severity
Loss cost
RMSS
+ 1.8
− 0.1
− 1.1
+ 13.7
− 0.9
+ 0.6
− 2.0
− 33.5
− 7.7
MRD
+ 0.1
− 1.0
− 1.8
− 6.9
− 2.4
+ 11.0
− 17.4
− 23.1
− 8.4
Dev
+ 0.001
+ 0.049
+ 0.002
+ 0.025
+ 0.017
− 0.132
+ 0.130
− 0.015
+ 0.018
Appendix
Table 16: Variations (in %) of each metric for EBMs when fixing interaction terms.
With the rapid development of machine learning and deep learning techniques, actuaries and the broader insurance industry face a persistent trade-off between predictive accuracy and interpretability. This paper provides a comprehensive applied assessment of Explainable Boosting Machines (EBM) in a car insurance framework, focusing on claim frequency and severity modeling. EBM combines the additive structure of generalized additive models (GAM) with a cyclic gradient boosting algorithm, resulting in a glass-box model whose predictions are interpretable by design. Using real-world data, we empirically illustrate its practical relevance and compare EBM with modern benchmark models used in non-life insurance pricing. The evaluation considers (i) out-of-sample predictive accuracy, including Murphy diagrams and Bregman dominance tests, and (ii) calibration assessment using T-reliability diagrams and Murphy's score decomposition. Finally, we highlight the link between EBM predictions and Shapley values, showing how predictions can be transparently decomposed into exact main and pairwise interaction effects, providing actionable insights beyond predictive performance.
Markéta Krúpová, Nabil Rachdi, Quentin Guibert
CEREMADE · 1ADDACTIS France, 46 bis Chemin du Vieux Moulin, 69160 Tassin-la-Demi-Lune, France · 2CEREMADE, Universit´e Paris-Dauphine, PSL University, CNRS, 75016 Paris, France
This position paper argues that claims about explanation stability are scientifically invalid without cross method validation. Just as statistical significance requires the test statistic to be specified, stability should either be evaluated across multiple attribution paradigms or explicitly scoped to the computational objective of a single method. In controlled chest X ray experiments, DenseNet201, ResNet50V2, and InceptionV3 achieved AUC values above 99%, yet their stability rankings reversed across attribution methods. LayerCAM ranked InceptionV3 as the most stable model, with an IoU of 0.777, whereas GradCAM++ favored DenseNet201 and reduced InceptionV3 stability score by 17.3%. These findings demonstrate that explanation stability is an emergent property of the model method pair rather than an intrinsic characteristic of the model alone. We therefore argue that explanation based claims should be validated across multiple attribution methods and that regulatory submissions should explicitly specify the attribution operators used to avoid creating illusory safety assurances.
Kabilan Elangovan, Daniel Ting
Singapore Health Services and Singapore Eye Research Institute, Singapore.
Attribution methods (AMs) assign an importance score to each feature and are widely adopted to explain black-box models. However, most methods can produce variable attribution scores due to stochastic components in their definition. In this paper, we propose a distribution-based framework to capture the stability of attribution scores. In particular, our approach allows to understand the degree of separability in the ranked attribution vector and obtain the largest index for which a feature ranking remains reliable. We further extend this framework to compare AMs based on the robustness of their rankings across a dataset. Through experiments, we demonstrate how to apply our method to evaluate explainer stability. Overall, our approach provides a complementary criterion for evaluating the stability of AMs.