Beyond Interaction Scores: Selecting Feature Interactions via Independent Replication
Organizations: HKUST(GZ)
Abstract
Learning feature interactions is essential for accurate prediction and interpretable modeling, yet a high interaction score may reflect fitting noise rather than a reproducible pattern. Ranking methods score feature pairs but leave the cutoff to the user; testing methods set the cutoff but rely on repeated model refits or a model of the joint feature distribution. We present LENS, a model-agnostic framework for selecting feature interactions through independent replication. LENS trains two models with the same learning algorithm on disjoint data and accesses them only through predictions. For each feature pair, it removes main effects from the joint-effect surface to isolate an interaction pattern, ranks pairs by its strength, and tests whether the second model reproduces it at a meaningful strength. A single multiple-testing correction determines which pairs to retain, using only two model fits regardless of the number of features. We prove that a replication ratio alone can retain arbitrarily weak patterns shared by both fits, whereas a strength floor restores selection consistency, and we characterize when normalization in Friedman's reverses interaction rankings. On 16 synthetic designs, LENS matches leading model-specific ranking methods (e.g., TreeSHAP) and recovers 91% of material interactions, against 45% for the knockoff-based DIAMOND, without selecting false pairs; with 1,225 candidate pairs and no interactions, it makes substantially fewer false discoveries. On large-scale French motor insurance data, it identifies interactions documented in the actuarial literature and recovers 90% of the deviance gain obtained using all pairs. On two credit datasets, LENS selects few pairs while maintaining or slightly improving predictive performance over the additive baseline.
Figures & tables
| Additive | Interaction | |||||
|---|---|---|---|---|---|---|
| 100 | 120 | 20 | 100 | 120 | 20 | |
| 90 | 110 | 20 | 90 | 110 | 20 | |
| 70 | 90 | 20 | 70 | 130 | 60 | |
| Symbol | Meaning |
|---|---|
| , | a pair of features; number of pairs, |
| , , | discovery, confirmation, and evaluation parts |
| , | context surface of row ; its average |
| projection that removes both main effects | |
| residual table of the discovery model | |
| residual table of the confirmation model |
| Term | A pair must reproduce | This rules out |
|---|---|---|
| a quarter of its own strength | a strong pair that barely replicates | |
| a quarter of the typical strength | a weak pair that replicates a negligible pattern | |
| 2% of the spread of the predictions | a model that learned no interaction, where every table shrinks |
| Method | Model | Recall@15 | AP |
|---|---|---|---|
| LENS | XGBoost | 0.88 | 0.90 |
| TreeSHAP [ 8 ] | XGBoost | 0.87 | 0.88 |
| [ 11 ] | XGBoost | 0.48 | 0.38 |
| LENS | MLP | 0.95 | 0.96 |
| Integrated Hessians [ 10 ] | MLP | 0.94 | 0.95 |
| NID [ 9 ] | MLP | 0.67 | 0.62 |
| Additive | Additive-corr | Sparse | Pair-corr | F1–F10 | |||||
|---|---|---|---|---|---|---|---|---|---|
| Method | Model | FDP | FDP | FDP | power | FDP | power | FDP | power |
| LENS | XGBoost | 0.10 | 0.05 | 0.01 | 1.00 | 0.67 | 1.00 | 0.00 | 0.62 |
| LENS | MLP | 0.00 | 0.15 | 0.00 | 1.00 | 0.03 | 1.00 | 0.00 | 0.66 |
| DIAMOND | XGBoost | 0.90 | 0.90 | 0.18 | 0.93 | 0.35 | 0.55 | 0.07 | 0.31 |
| DIAMOND | MLP | 0.65 | 0.90 | 0.14 | 0.93 | 0.12 | 0.20 | 0.10 | 0.29 |
| , normal refit null | XGBoost | 1.00 | 1.00 | 0.96 | 1.00 | 0.99 | 1.00 | 0.72 | 0.96 |
| F1–F10 FDP | F1–F10 power | Additive FDP | Sparse FDP | Pair-corr FDP | List size | ||||||||
| Setting | XGB | MLP | XGB | MLP | XGB | MLP | XGB | MLP | XGB | MLP | freMTPL2 | HELOC | UCI |
| 0.24 | 0.02 | 1.00 | 1.00 | 0.93 | 1.00 | 0.54 | 0.20 | 0.78 | 0.90 | 13.0 | 23.2 | 35.2 | |
| 0.06 | 0.00 | 1.00 | 1.00 | 0.49 | 0.97 | 0.28 | 0.20 | 0.71 | 0.90 | 13.0 | 19.0 | 33.0 | |
| 0.00 | 0.00 | 1.00 | 1.00 | 0.33 | 0.45 | 0.09 | 0.00 | 0.70 | 0.36 | 13.0 | 10.4 | 17.0 | |
| (LENS) | 0.00 | 0.00 | 0.91 | 1.00 | 0.06 | 0.07 | 0.01 | 0.00 | 0.67 | 0.03 | 12.0 | 2.0 | 5.2 |
| 0.00 | 0.00 | 0.82 | 0.88 | 0.00 | 0.02 | 0.00 | 0.00 | 0.67 | 0.00 | 9.0 | 0.2 | 3.6 | |
| freMTPL2 | HELOC | UCI credit | ||||
| List | pairs | loss | pairs | loss | pairs | loss |
| None (additive) | 0 | 0.3128 | 0 | 0.5508 | 0 | 0.4312 |
| LENS | 12.0 | 0.3095 | 2.0 | 0.5508 | 5.2 | 0.4303 |
| No floor | 13.0 | 0.3095 | 23.2 | 0.5541 | 35.2 | 0.4318 |
| Top by energy | 7 | 0.3108 | 10 | 0.5518 | 10 | 0.4303 |
| All pairs | 36 | 0.3091 | 253 | 0.5915 | 253 | 0.4421 |
| Pair | kept | rank | range | S&W | H&H | ||
|---|---|---|---|---|---|---|---|
| VehAge VehBrand | 5 | 1.0 | 0.247 | 0.90 | 0.78 to 1.06 | ✓ | |
| DrivAge BonusMalus | 5 | 2.2 | 0.141 | 0.62 | 0.43 to 0.89 | ✓ | ✓ |
| DrivAge Region | 5 | 4.4 | 0.108 | 0.50 | 0.31 to 0.67 | ✓ | |
| VehAge VehGas | 5 | 11.4 | 0.062 | 0.96 | 0.47 to 1.45 | ✓ | ✓ |
| VehPower VehAge | 5 | 13.0 | 0.049 | 0.98 | 0.55 to 1.46 | ✓ | |
| VehPower DrivAge | 5 | 19.6 | 0.033 | 0.80 | 0.54 to 1.35 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Model | size | FDP | power | size | FDP | power | size | FDP | power |
|---|---|---|---|---|---|---|---|---|---|---|
| LENS (default, ) | XGBoost | 3.0 | 0.01 | 1.00 | 3.0 | 0.01 | 1.00 | 3.0 | 0.01 | 1.00 |
| LENS, | XGBoost | 3.0 | 0.01 | 1.00 | 3.2 | 0.06 | 1.00 | 3.4 | 0.09 | 1.00 |
| LENS, | XGBoost | 3.0 | 0.01 | 1.00 | 3.5 | 0.12 | 1.00 | 7.2 | 0.54 | 1.00 |
| LENS, | XGBoost | 8.3 | 0.61 | 1.00 | 79.0 | 0.96 | 1.00 | 233.8 | 0.99 | 1.00 |
| LENS (default, ) | MLP | 3.0 | 0.00 | 1.00 | 3.0 | 0.00 | 1.00 | 3.0 | 0.00 | 1.00 |
| Method | Model | size | FDP | power | size | FDP | power | size | FDP | power |
|---|---|---|---|---|---|---|---|---|---|---|
| LENS (default, ) | XGBoost | 3.0 | 0.67 | 1.00 | 3.0 | 0.67 | 1.00 | 3.0 | 0.67 | 1.00 |
| LENS, | XGBoost | 3.0 | 0.67 | 1.00 | 3.2 | 0.69 | 1.00 | 3.4 | 0.70 | 1.00 |
| LENS, | XGBoost | 3.0 | 0.67 | 1.00 | 3.5 | 0.71 | 1.00 | 4.9 | 0.78 | 1.00 |
| LENS, | XGBoost | 8.6 | 0.88 | 1.00 | 76.8 | 0.99 | 1.00 | 240.8 | 1.00 | 1.00 |
| LENS (default, ) | MLP | 1.1 | 0.03 | 1.00 | 1.1 | 0.05 | 1.00 | 1.1 | 0.03 | 1.00 |
| Method | Model | size | any | size | any | size | any |
|---|---|---|---|---|---|---|---|
| LENS (default, ) | XGBoost | 0.1 | 0.05 | 0.1 | 0.05 | 0.1 | 0.10 |
| LENS, | XGBoost | 0.1 | 0.15 | 0.2 | 0.20 | 0.2 | 0.25 |
| LENS, | XGBoost | 0.9 | 0.65 | 7.5 | 1.00 | 17.2 | 1.00 |
| LENS, | XGBoost | 8.3 | 1.00 | 76.5 | 1.00 | 219.6 | 1.00 |
| LENS (default, ) | MLP | 0.0 | 0.00 | 0.0 | 0.00 | 0.0 | 0.00 |
| Method | Model | size | any | size | any | size | any |
|---|---|---|---|---|---|---|---|
| LENS (default, ) | XGBoost | 0.1 | 0.05 | 0.1 | 0.05 | 0.1 | 0.05 |
| LENS, | XGBoost | 0.5 | 0.40 | 0.6 | 0.45 | 0.5 | 0.50 |
| LENS, | XGBoost | 1.9 | 0.95 | 10.1 | 1.00 | 20.2 | 1.00 |
| LENS, | XGBoost | 9.4 | 1.00 | 92.4 | 1.00 | 228.4 | 1.00 |
| LENS (default, ) | MLP | 0.2 | 0.20 | 0.1 | 0.10 | 0.1 | 0.15 |
| Method | Model | F1 | F2 | F3 | F4 | F5 | F6 | F7 | F8 | F9 | F10 | mean |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LENS (default, ) | XGBoost | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| LENS (default, ) | MLP | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| DIAMOND, | XGBoost | 0.11 | 0.13 | 0.11 | 0.09 | 0.11 | 0.03 | 0.05 | 0.03 | 0.02 | 0.07 | 0.07 |
| DIAMOND, | MLP | 0.16 | 0.18 | 0.14 | 0.13 | 0.16 | 0.05 | 0.03 | 0.02 | 0.02 | 0.07 | 0.10 |
| , normal refit null | XGBoost | 0.76 | 0.78 | 0.82 | 0.77 | 0.70 | 0.83 | 0.31 | 0.70 | 0.72 | 0.83 | 0.72 |
| iLOCO-Split, | XGBoost | 0.72 | 0.64 | 0.76 | 0.76 | 0.49 | 0.70 | 0.63 | 0.61 | 0.62 | 0.77 | 0.67 |
| Method | Model | F1 | F2 | F3 | F4 | F5 | F6 | F7 | F8 | F9 | F10 | mean |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LENS (default, ) | XGBoost | 0.54 | 0.55 | 1.00 | 1.00 | 0.88 | 0.25 | 0.21 | 0.67 | 0.13 | 1.00 | 0.62 |
| LENS (default, ) | MLP | 0.64 | 0.64 | 1.00 | 1.00 | 0.88 | 0.25 | 0.21 | 0.75 | 0.27 | 1.00 | 0.66 |
| DIAMOND, | XGBoost | 0.29 | 0.29 | 0.40 | 0.35 | 0.76 | 0.12 | 0.14 | 0.26 | 0.06 | 0.47 | 0.31 |
| DIAMOND, | MLP | 0.29 | 0.28 | 0.16 | 0.20 | 0.75 | 0.12 | 0.15 | 0.28 | 0.10 | 0.59 | 0.29 |
| , normal refit null | XGBoost | 0.94 | 0.93 | 1.00 | 1.00 | 0.88 | 1.00 | 0.84 | 1.00 | 1.00 | 1.00 | 0.96 |
| iLOCO-Split, | XGBoost | 0.57 | 0.55 | 0.94 | 0.95 | 0.76 | 0.81 | 0.02 | 0.19 | 0.56 | 0.64 | 0.60 |
| Method | Model | F1 | F2 | F3 | F4 | F5 | F6 | F7 | F8 | F9 | F10 | mean |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LENS (default, ) | XGBoost | 6.0 | 6.0 | 9.0 | 10.0 | 7.0 | 2.0 | 3.0 | 8.0 | 2.0 | 6.0 | 5.9 |
| LENS (default, ) | MLP | 7.0 | 7.0 | 9.0 | 10.0 | 7.0 | 2.0 | 3.0 | 9.0 | 4.0 | 6.0 | 6.4 |
| DIAMOND, | XGBoost | 3.6 | 3.7 | 4.5 | 4.2 | 7.0 | 1.0 | 2.1 | 3.3 | 0.9 | 3.2 | 3.4 |
| DIAMOND, | MLP | 4.0 | 3.9 | 2.4 | 3.1 | 7.3 | 1.1 | 2.2 | 3.5 | 1.6 | 4.0 | 3.3 |
| , normal refit null | XGBoost | 43.2 | 46.5 | 50.3 | 45.2 | 24.5 | 48.6 | 17.5 | 41.1 | 56.0 | 36.2 | 40.9 |
| iLOCO-Split, | XGBoost | 34.4 | 35.8 | 67.7 | 52.5 | 19.4 | 39.1 | 3.4 | 18.1 | 27.7 | 38.1 | 33.6 |
| Additive | Additive-corr | Sparse | Pair-corr | F1–F10 ( ) | |||||||
| Rule | any | any | size | FDP | power | size | FDP | power | size | FDP | power |
| LENS (default) | 0.05 | 0.05 | 3.0 | 0.01 | 1.00 | 3.0 | 0.67 | 1.00 | |||
| above the boundary, no test | 0.05 | 0.05 | 3.0 | 0.01 | 1.00 | 3.0 | 0.67 | 1.00 | |||
| Top- by energy | 0.05 | 0.05 | 3.0 | 0.01 | 1.00 | 3.0 | 0.67 | 1.00 | |||
| Both fits above the floor, no comparison | 1.00 | 1.00 | 4.3 | 0.27 | 1.00 | 4.1 | 0.76 | 1.00 | |||
| freMTPL2 (rows per model) | Credit | |||||||||
| 20,000 | 50,000 | 184,420 | HELOC | UCI | ||||||
| Rule | size | loss | size | loss | size | loss | size | loss | size | loss |
| Additive | – | 0.31587 | – | 0.31447 | – | 0.31285 | – | 0.55084 | – | 0.43118 |
| Black box | – | 0.31747 | – | 0.31002 | – | 0.30536 | – | 0.56082 | – | 0.43097 |
| LENS (default) | 6.6 | 0.31318 | 9.0 | 0.31168 | 12.0 | 0.30950 | 2.0 | 0.55076 | 5.2 | 0.43028 |
| LENS, | 6.6 | 0.31318 | 9.0 | 0.31168 | 13.0 | 0.30945 | 23.2 | 0.55408 | 35.2 | 0.43177 |
Feature Interaction Modeling
Feature interaction modeling captures how combinations of inputs jointly shape predictions beyond their individual effects.
See all 77 papers