stat.MEOct 21, 2026
SaveMastering Model Fit: Applications of the InterModel Vigorish for Interpretable Machine Learning
Organizations: Leverhulme Centre for Demographic Science, University of Oxford · Nuffield College, University of Oxford · Data Science Institute, University of Chicago · NYU Steinhardt, New York University ·
Abstract
Accuracy alone gives little information about which features or model components improve probabilistic fit, which outcomes benefit, or how fit changes during training. We introduce the InterModel Vigorish (IMV) into the Machine Intelligence space in order to express differences in fit as an expected betting advantage over a reference model. SHAP-IMV attributes gains to features through Shapley-weighted subset comparisons, Multi-IMV evaluates one-vs-rest and pairwise class discrimination, and Ablate-IMV assesses architectural components and feature groups by ablation. We evaluate these applications over ten random seeds using tabular data, images, sensor sequences, and graphs. Feature rankings vary across estimators. High one-vs-rest accuracy can accompany negligible improvement over a null, while ablation can preserve or improve fit under a fixed training budget. On six graph-classification benchmarks, we time-profile IMV over 200 training epochs, reporting seed means and ranges. The open-source imvpy library and accompanying notebooks provide code to replicate our work and to use the IMV more widely.
Figures & tables
Figure 1: IMV calculation and applications. Steps 1–3 start with the probabilities that reference model and candidate model assign to the same observed outcomes. We aggregate these as geometric mean likelihoods. When both are at least , Equation ( 2 ) gives equivalent coin weights and . The directional score measures the relative change in coin weight; positive values indicate higher mean log-likelihood for . ( A ) SHAP-IMV attributes gains in fit to features through Shapley-weighted marginal contributions from refitted subset models, using a constant-only reference. ( B ) Multi-IMV reuses multiclass predictions for one-vs-all and pairwise comparisons, renormalizing probabilities within each class pair. ( C ) Ablate-IMV compares full and refitted ablated models on the same held-out observations to assess architectural components or feature groups. Scores depend on the reference and evaluation data. All data, probabilities, scores, and networks shown are schematic.
Figure 2: SHAP-IMV feature contributions for Titanic ( a. – c. ), Wisconsin Diagnostic Breast Cancer ( d. – f. ), and Adult Income ( g. – i. ). Columns show logistic regression, XGBoost, and LightGBM; each panel ranks eight predictors by mean signed SHAP-IMV. All 256 subsets are evaluated by five-fold stratified cross-validation against the same estimator fitted to a constant input. Shapley contributions use the mean IMV across folds. Bars and labels give means across ten seeds (42–51), with whiskers spanning one sample standard deviation rather than confidence intervals. Standardization uses the full dataset before cross-validation. Y-scales are shared within rows, and the contributions describe predictive fit rather than causal effects.
| Methods | Titanic | Breast Cancer | Adult Income | Average | |||||
|---|---|---|---|---|---|---|---|---|---|
| Accuracy | Precision | Accuracy | Precision | Accuracy | Precision | Accuracy | Precision | ||
| LR | SHAP-IMV | 79.50 | 74.36 | 93.92 | 94.27 | 83.78 | 70.94 | 85.73 | 79.85 |
| SHAP | 79.74 | 73.95 | 94.10 | 94.46 | 83.78 | 70.94 | 85.87 | 79.78 | |
| LIME | 79.74 | 73.95 | 94.10 | 94.46 | 84.16 | 71.35 | 86.00 | 79.92 | |
| Anchors | 79.50 | 74.36 | 93.92 | 94.27 | 83.78 | 70.94 | 85.73 | 79.85 | |
| XGB | SHAP-IMV | 81.16 | 79.33 | 93.76 | 93.11 | 85.97 | 79.42 | 86.96 | 83.95 |
Table 1: Predictive performance after top-5 feature selection by SHAP-IMV, SHAP, LIME, and Anchors. LR, XGB, and LGBM denote logistic regression, XGBoost, and LightGBM. Each dataset uses a fixed set of 8 variables drawn from the source data; Breast Cancer refers to the Wisconsin Diagnostic dataset. Rankings are averaged over seeds to select a fixed subset for refitting the same estimator. Accuracy and positive-class precision are percentages averaged over 10 stratified folds for each of 10 seeds; Average gives equal weight to the 3 datasets. Preprocessing and feature selection precede refit cross-validation, so the results describe these fixed subsets rather than a nested evaluation of the selection procedures. Within each estimator and column, bold and underlining mark the best and second-best distinct values at the displayed precision; ties share formatting, and columns in which all methods agree are unmarked.
| Dataset | Model | Rank by mean SHAP-IMV | |||||||
|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | ||
| Titanic | LR | Sex (0.147) | Title (0.065) | Class (0.041) | Age Class (0.035) | Fare (0.018) | Embarked (0.013) | Alone (0.011) | Age (0.004) |
| XGB | Title (0.113) | Sex (0.105) | Fare (0.060) | Class (0.045) | Age Class (0.022) | Embarked (0.013) | Alone (0.009) | Age (-0.000) | |
| LGBM | Title (0.113) | Sex (0.104) | Fare (0.053) | Class (0.046) | Age Class (0.025) | Embarked (0.012) | Alone (0.009) | Age (0.001) | |
| Breast Cancer | LR | Concave points (0.129) | Radius (0.124) | Concavity (0.096) | Compactness (0.072) | Texture (0.050) | Smoothness (0.031) | Symmetry (0.021) | Fractal dim. (0.016) |
| XGB | Concave points (0.129) | Radius (0.128) | Concavity (0.115) | Compactness (0.071) | Texture (0.051) | Smoothness (0.027) | Symmetry (0.013) | Fractal dim. (0.003) | |
Table 2: Complete SHAP-IMV feature rankings for the three binary classification examples. LR, XGB, and LGBM denote logistic regression, XGBoost, and LightGBM. Each dataset uses a fixed set of 8 variables drawn from the source data; Breast Cancer refers to the Wisconsin Diagnostic dataset. Predictors are ordered by decreasing mean exact Shapley contribution to model IMV; parentheses give means to three decimals. Attribution uses 5-fold stratified cross-validation for each of 10 seeds. The 10-fold refit evaluation in Table 1 is separate and evaluates the fixed subset in ranks 1–5 for its SHAP-IMV rows. Figure 2 plots the same means with one sample standard deviation across seeds. Rankings describe predictive contributions within each specification, not causal effects.
Figure 3: Multi-IMV on Dry Bean ( a. – f. ; seven classes, 16 features), Car Evaluation ( g. – l. ; four classes, six features), and Nursery ( m. – r. ; eight features, retaining not_recom, priority, and spec_prior). Columns show logistic regression, XGBoost, and LightGBM, each compared with its constant-input counterpart. Heatmaps restrict held-out observations to class pairs and renormalize both models’ probabilities without binary refitting. Matrices are symmetric with conventional zero diagonals and one colour scale per dataset. Bars show one-vs-rest IMV from unrenormalized class probabilities on all held-out observations; the two views have different populations and reference coin weights. Five-fold stratified scores are averaged within each of ten seeds (42–51). Cells, bars, and labels show seed means; whiskers show one sample standard deviation, not confidence intervals. Dashed lines show the ceiling of the One-vs-rest IMV for the targeted category.
| Methods | Unacc | Acc | Good | Very good |
|---|---|---|---|---|
| Multi-IMV (LR) | 0.15 0.00 | 0.05 0.00 | 0.00 0.00 | 0.02 0.00 |
| Brier Loss (LR) | 0.17 0.00 | 0.16 0.00 | 0.04 0.00 | 0.03 0.00 |
| Accuracy (LR) | 72.16 0.31 | 74.30 0.34 | 96.01 0.00 | 95.48 0.08 |
| Precision (LR) | 74.51 0.19 | 34.99 1.34 | 0.00 0.00 | 33.91 1.74 |
| Recall (LR) | 91.58 0.33 | 18.23 0.78 | 0.00 0.00 | 21.23 1.21 |
| Multi-IMV (XGB) | 0.40 0.00 | 0.25 0.00 | 0.03 0.00 | 0.04 0.00 |
Table 3: One-vs-rest Multi-IMV and classification performance on Car Evaluation. LR, XGB, and LGBM denote logistic regression, XGBoost, and LightGBM. Values are means one sample standard deviation across 10 seeds, each using 5-fold stratified cross-validation; these are not confidence intervals. IMV compares the model using the predictors with the same estimator fitted to a constant input only. We average IMV over folds within each seed and calculate the other metrics from pooled out-of-fold predictions. Accuracy, precision, and recall are percentages for each class versus the rest, based on the multiclass argmax prediction; undefined precision or recall is set to zero. Brier Loss is the Brier score loss on class probabilities. Rounding to two decimals can produce a displayed zero standard deviation even when runs differ. Unacc and Acc denote Unacceptable and Acceptable. Global metrics (LR): F1 Score: 63.84 0.32; AUC-ROC: 77.29 0.12; Log Loss: 0.69 0.00; Hamming Loss: 0.31 0.00; Cohen’s Kappa: 16.31 0.78; Jaccard: 52.42 0.31. Global metrics (XGB): F1 Score: 96.69 0.29; AUC-ROC: 99.71 0.05; Log Loss: 0.14 0.00; Hamming Loss: 0.03 0.00; Cohen’s Kappa: 92.79 0.63; Jaccard: 93.76 0.51. Global metrics (LGBM): F1 Score: 97.89 0.18; AUC-ROC: 99.90 0.04; Log Loss: 0.08 0.00; Hamming Loss: 0.02 0.00; Cohen’s Kappa: 95.38 0.40; Jaccard: 95.94 0.34. F1, one-vs-rest AUC-ROC, and Jaccard use support-weighted class averages. These three metrics and Cohen’s kappa are multiplied by 100; log loss and Hamming loss are unscaled.
| Methods | Not recommend | Priority acceptance | Special acceptance |
|---|---|---|---|
| Multi-IMV (LR) | 0.52 0.00 | 0.31 0.00 | 0.28 0.00 |
| Brier Loss (LR) | 0.00 0.00 | 0.13 0.00 | 0.13 0.00 |
| Accuracy (LR) | 100.00 0.00 | 78.75 0.05 | 78.75 0.05 |
| Precision (LR) | 100.00 0.00 | 68.20 0.07 | 67.16 0.10 |
| Recall (LR) | 100.00 0.00 | 69.48 0.12 | 65.82 0.10 |
| Multi-IMV (XGB) | 0.52 0.00 | 0.49 0.00 | 0.45 0.00 |
Table 4: Multi-IMV results for Nursery using logistic regression, XGBoost, and LightGBM. Means and sample standard deviations are calculated across ten seeds, each evaluated with five stratified folds. Metric definitions and units follow Table 3 . Global metrics (LR): F1 Score: 78.74 0.05; AUC-ROC: 91.77 0.01; Log Loss: 0.39 0.00; Hamming Loss: 0.21 0.00; Cohen’s Kappa: 68.11 0.08; Jaccard: 67.87 0.06. Global metrics (XGB): F1 Score: 99.40 0.05; AUC-ROC: 99.99 0.00; Log Loss: 0.08 0.00; Hamming Loss: 0.01 0.00; Cohen’s Kappa: 99.10 0.08; Jaccard: 98.82 0.10. Global metrics (LGBM): F1 Score: 99.76 0.06; AUC-ROC: 100.00 0.00; Log Loss: 0.06 0.00; Hamming Loss: 0.00 0.00; Cohen’s Kappa: 99.65 0.09; Jaccard: 99.53 0.12.
| Methods | Seker | Barbunya | Bombay | Cali | Dermason | Horoz | Sira |
|---|---|---|---|---|---|---|---|
| Multi-IMV (LR) | 0.17 0.00 | 0.10 0.00 | 0.04 0.00 | 0.13 0.00 | 0.32 0.00 | 0.16 0.00 | 0.21 0.00 |
| Brier Loss (LR) | 0.01 0.00 | 0.01 0.00 | 0.00 0.00 | 0.01 0.00 | 0.03 0.00 | 0.01 0.00 | 0.04 0.00 |
| Accuracy (LR) | 98.45 0.02 | 98.55 0.03 | 100.00 0.00 | 98.53 0.01 | 95.83 0.03 | 98.71 0.01 | 94.76 0.04 |
| Precision (LR) | 94.93 0.10 | 94.22 0.13 | 100.00 0.00 | 93.41 0.10 | 92.28 0.06 | 95.47 0.10 | 85.61 0.12 |
| Recall (LR) | 94.68 0.08 | 90.63 0.22 | 99.96 0.08 | 94.36 0.06 | 91.66 0.06 | 95.41 0.06 | 87.69 0.12 |
| Multi-IMV (XGB) | 0.17 0.00 | 0.10 0.00 | 0.04 0.00 | 0.13 0.00 | 0.33 0.00 | 0.16 0.00 | 0.21 0.00 |
Table 5: Multi-IMV results for Dry Bean using logistic regression, XGBoost, and LightGBM. Means and sample standard deviations are calculated across ten seeds, each evaluated with five stratified folds. Metric definitions and units follow Table 3 . Global metrics (LR): F1 Score: 92.43 0.04; AUC-ROC: 99.34 0.00; Log Loss: 0.21 0.00; Hamming Loss: 0.08 0.00; Cohen’s Kappa: 90.83 0.05; Jaccard: 86.10 0.06. Global metrics (XGB): F1 Score: 92.75 0.11; AUC-ROC: 99.42 0.01; Log Loss: 0.19 0.00; Hamming Loss: 0.07 0.00; Cohen’s Kappa: 91.23 0.14; Jaccard: 86.65 0.19. Global metrics (LGBM): F1 Score: 92.75 0.11; AUC-ROC: 99.41 0.01; Log Loss: 0.20 0.00; Hamming Loss: 0.07 0.00; Cohen’s Kappa: 91.23 0.13; Jaccard: 86.66 0.18.
Figure 4: Ablate-IMV for MNIST odd/even digits (CNN; a. , d. ), UCI HAR upstairs/downstairs activity (bidirectional GRU; b. , e. ), and MAGIC Gamma gamma/hadron events (MLP; c. , f. ). Heatmaps compare enhanced rows with basic columns; positive values favor rows, and reverse comparisons need not be negatives. Diagonals are zero; heatmaps share a zero-centred colour scale. Bars compare models with constant 0.5 predictions, not the full model shown first. Cells, bars, and labels show ten-seed means (42–51); whiskers show one sample standard deviation, not confidence intervals. MNIST/HAR ablate architecture; MAGIC removes feature groups with hidden architecture fixed. Training lasts one, 15, and 12 epochs, respectively. Official MNIST/HAR test sets and stratified 75/25 MAGIC splits align observations across variants within seeds.
Figure 5: Test performance during training of a three-layer GCN on a. PROTEINS (1,113 graphs), b. NCI1 (4,110), c. NCI109 (4,127), d. Mutagenicity (4,337), e. AIDS (2,000), and f. DD (1,178). Counts refer to full datasets. Lines show means across ten seeds (42–51): solid for IMV, dashed for accuracy, dash-dotted for precision, and dotted for recall. Shading spans the minimum and maximum across seeds at each epoch, not confidence intervals. Each seed defines a stratified 70/15/15 training/validation/test split; the same test graphs are evaluated from untrained epoch 0 through epoch 200. IMV compares probabilities with the training-prevalence null; classification uses a 0.5 threshold with class 1 positive. Curves are not smoothed, and negative IMV values are retained.
| Dataset | Graphs | Mean nodes | Mean edges |
|---|---|---|---|
| PROTEINS | 1,113 | 39.06 | 72.82 |
| NCI1 | 4,110 | 29.87 | 32.30 |
| NCI109 | 4,127 | 29.68 | 32.13 |
| Mutagenicity | 4,337 | 30.32 | 30.77 |
| AIDS | 2,000 | 15.69 | 16.20 |
| DD | 1,178 | 284.32 | 715.66 |
Table 6: Datasets used in the GNN experiments. Mean edge counts refer to undirected edges before GCN self-loops are added.
| IMV | Median (ms) | IQR (ms) | Time/obs. ( s) | |
|---|---|---|---|---|
| 300 | 0.282 | 0.078 | 0.002 | 0.260 |
| 3,000 | 0.279 | 0.124 | 0.000 | 0.041 |
| 30,000 | 0.303 | 0.719 | 0.010 | 0.024 |
| 300,000 | 0.303 | 6.987 | 0.319 | 0.023 |
Table 7: Runtime of vanilla_imv for the simulation in Box 3.3 . The median and interquartile range (IQR) summarize 9 batches of repeated calls; is the number of test observations. Timing excludes fitting and prediction.
More in this topic
Interpretable ML
Interpretable machine learning develops models and explanations that make predictions, evidence, and decision processes understandable to people.
See all 310 papers