CRAFT-GBM: Selecting Best Features for Gradient-Boosted Machine Learning
Organizations: Massachusetts Institute of Technology (MIT) · National University of Singapore (NUS) · Asian Institute of Digital Finance (AIDF)
Abstract
Feature subsampling is used to accelerate gradient-boosted trees, yet standard approaches fix the feature count in advance and largely ignore the split-gain evidence accumulated during training. We introduce CRAFT-GBM (Confidence-Ranked Adaptive Feature Tracking), a native LightGBM extension that uses this evidence to adapt which---and, by default, how many---features are searched each boosting round. CRAFT-GBM transforms root-split gains into bounded observations and constructs joint time-uniform bounds valid under adaptive, non-stationary training histories. In its default endogenous mode, after initialization, CRAFT-GBM retains exactly the features that cannot yet be ruled out as best, letting the selected feature-set size emerge from training history rather than a pre-specified fraction. With probability at least , every feature with the largest expected gain is retained jointly across all rounds after initialization. When a feature fraction is specified instead, CRAFT-GBM ranks features by their interval point estimates and provides an approximate top- guarantee for feature budget . We implement CRAFT-GBM within LightGBM while preserving its native feature-bundling representation, and evaluate it across 14 public and synthetic dataset families and four noise-augmented settings. In the fully endogenous setting, CRAFT-GBM reduces per-fit training time by 39.74% with 1.04% lower test loss in the geometric mean across all 18 RQ1 experiments. Under pre-specified feature fraction , Exogenous CRAFT-GBM also yields aggregate training-time reductions up to 13.52 percentage points larger than random feature subsampling, with no increase in aggregate test loss relative to full-feature LightGBM. Accumulated training evidence can thus support adaptive feature search with statistical guarantees while retaining the practical advantages of native gradient boosting.
Figures & tables
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Quantity | Value | Role and limitation |
|---|---|---|
| Enable selector | true for CRAFT -GBM | Set adaptive_feature_selection=true . The native default is false . |
| Feature count | Number of features retained by native preprocessing, fixed before covered gains. | |
| Joint-confidence error level | (default) | Joint-confidence error level over the fixed features for all boosting rounds, fixed at the beginning of training. |
| Per-tail error level | Derived from the fixed feature count and joint-confidence error level , not from observed gains. | |
| Boosting-round endpoint | endogenous or protocol-fixed | RQ1 endogenous fits use training-loss patience without an external cap. Fixed-horizon fits use their prescribed endpoint. Neither determines the feature-confidence boundary . |
| feature_fraction | (default) or | With the selector enabled, invokes the endogenous rule for the selection size , while invokes Exogenous as defined in Equation ( 13 ). |
| Dataset | Task | Training rows | Test rows | Outer split |
|---|---|---|---|---|
| HIGGS | C | 10,500,000 | 500,000 | First/last rows |
| Epsilon | C | 400,000 | 100,000 | Official |
| Bosch | C | 946,997 | 236,750 | Stratified |
| YearMSD | R | 463,715 | 51,630 | First/last rows |
| APS Failure | C | 60,000 | 16,000 | Official |
| KDD 2009 | C | 40,000 | 10,000 | Stratified |
| Model | time (%) | loss (%) |
|---|---|---|
| Endogenous CRAFT-GBM | -39.74 | -1.04 |
| LightGBM-Fixed | -30.91 | -1.18 |
| XGBoost | 148.74 | 4.57 |
| CatBoost | 153.23 | 5.56 |
| Dataset | Pre-run rounds | Pre-run fraction (%) | |
|---|---|---|---|
| HIGGS | 28 | 188.0 | 59.65 |
| Epsilon | 2000 | 1128.0 | 16.40 |
| Bosch | 968 | 1339.0 | 31.64 |
| YearMSD | 90 | 270.0 | 69.77 |
| APS Failure | 170 | 364.0 | 59.90 |
| KDD 2009 | 212 | 517.0 | 22.89 |
| Model | Mean rounds | Training (s) | Test loss |
|---|---|---|---|
| HIGGS — Binary log loss (nats/observation) | |||
| LightGBM-Full | 188.0 | ||
| Endogenous CRAFT-GBM | 199.4 | ||
| LightGBM-Fixed | 188.0 | ||
| XGBoost | 188.0 | ||
| CatBoost | 188.0 | ||
| Model | time (%) | Interval | loss (%) | Interval |
|---|---|---|---|---|
| HIGGS | ||||
| Endogenous CRAFT-GBM | -3.59 | [-7.27, 0.23] | 0.39 | [0.14, 0.64] |
| LightGBM-Fixed | -7.09 | [-8.05, -6.11] | 0.28 | [0.20, 0.35] |
| XGBoost | 42.44 | [41.36, 43.53] | 0.39 | [0.27, 0.51] |
| CatBoost | 48.49 | [48.01, 48.97] | 3.44 | [3.39, 3.50] |
| Epsilon | ||||
| Model | Oracle CE / function MSE | Bernoulli KL |
|---|---|---|
| Sparse 1 — Oracle cross-entropy and Bernoulli KL (nats/observation) | ||
| LightGBM-Full | ||
| Endogenous CRAFT-GBM | ||
| LightGBM-Fixed | ||
| XGBoost | ||
| CatBoost | ||
| Model | time (%) | loss (%) | |
|---|---|---|---|
| Exogenous CRAFT-GBM | 0.1 | -62.74 [-73.74, -47.52] | -0.04 [-2.83, 2.70] |
| Exogenous CRAFT-GBM | 0.3 | -44.20 [-54.21, -32.29] | -1.16 [-2.76, 0.29] |
| Exogenous CRAFT-GBM | 0.5 | -28.02 [-36.05, -19.37] | -0.27 [-1.77, 1.71] |
| Exogenous CRAFT-GBM | 0.7 | -11.77 [-17.23, -6.25] | -0.38 [-1.48, 0.96] |
| Exogenous CRAFT-GBM | 0.9 | -1.81 [-4.02, 0.51] | -0.26 [-0.49, -0.05] |
| LightGBM-Fixed | 0.1 | -49.22 [-60.91, -35.28] | 4.60 [-3.04, 17.28] |
| Dataset / identity | Validation loss | Boundary | Tuning (s) | ||
|---|---|---|---|---|---|
| HIGGS | 1955 | 0.152296 | 0.48986 | none | 4504.9 |
| Epsilon | 1842 | 0.066851 | 0.27077 | none | 3556.7 |
| Bosch | 1780 | 0.013433 | 0.03087 | none | 2448.3 |
| YearMSD | 1865 | 0.065079 | 8.72367 | none | 436.4 |
| APS Failure | 571 | 0.023038 | 0.01752 | none | 146.5 |
| KDD 2009 | 93 | 0.022992 | 0.07730 | none | 116.2 |
| Model | Mean rounds | Training (s) | Test loss | |
|---|---|---|---|---|
| HIGGS — Binary log loss (nats/observation) | ||||
| LightGBM-Full | — | 1955.0 | ||
| LightGBM-Full (CRAFT-GBM ref.) | — | 1955.0 | ||
| LightGBM-Full (ablation ref.) | — | 1955.0 | ||
| Exogenous CRAFT-GBM | 0.1 | 1955.0 | ||
| Exogenous CRAFT-GBM | 0.3 | 1955.0 | ||
| Model | time (%) | Interval | loss (%) | Interval | |
|---|---|---|---|---|---|
| HIGGS | |||||
| Exogenous CRAFT-GBM | 0.1 | -22.00 | [-22.72, -21.27] | 9.66 | [8.94, 10.38] |
| Exogenous CRAFT-GBM | 0.3 | -17.42 | [-17.73, -17.11] | 4.07 | [3.53, 4.62] |
| Exogenous CRAFT-GBM | 0.5 | -13.77 | [-14.36, -13.18] | 2.60 | [2.48, 2.72] |
| Exogenous CRAFT-GBM | 0.7 | -5.57 | [-5.98, -5.17] | 0.63 | [0.49, 0.76] |
| Exogenous CRAFT-GBM | 0.9 | -2.83 | [-3.09, -2.57] | 0.49 | [0.34, 0.65] |
| Model | Oracle CE / function MSE | Bernoulli KL | |
|---|---|---|---|
| Sparse 1 — Oracle cross-entropy and Bernoulli KL (nats/observation) | |||
| LightGBM-Full (CRAFT-GBM ref.) | — | ||
| LightGBM-Full (ablation ref.) | — | ||
| LightGBM-Full | — | ||
| Exogenous CRAFT-GBM | 0.1 | ||
| Exogenous CRAFT-GBM | 0.3 | ||
Gradient-Boosted Decision Trees
Gradient-boosted decision trees combine shallow trees sequentially to make accurate predictions on structured data.
See all 62 papers