cs.LGNov 9, 2026

CRAFT-GBM: Selecting Best Features for Gradient-Boosted Machine Learning

Authors: Basil Kuok, Yi Zhou, Yao Xuan, Shuping Li, Ke-Wei Huang

Organizations: Massachusetts Institute of Technology (MIT) · National University of Singapore (NUS) · Asian Institute of Digital Finance (AIDF)

Abstract

Feature subsampling is used to accelerate gradient-boosted trees, yet standard approaches fix the feature count in advance and largely ignore the split-gain evidence accumulated during training. We introduce CRAFT-GBM (Confidence-Ranked Adaptive Feature Tracking), a native LightGBM extension that uses this evidence to adapt which---and, by default, how many---features are searched each boosting round. CRAFT-GBM transforms root-split gains into bounded observations and constructs joint time-uniform bounds valid under adaptive, non-stationary training histories. In its default endogenous mode, after initialization, CRAFT-GBM retains exactly the features that cannot yet be ruled out as best, letting the selected feature-set size KtK_t emerge from training history rather than a pre-specified fraction. With probability at least 1−δ1-δ, every feature with the largest expected gain is retained jointly across all rounds after initialization. When a feature fraction ff is specified instead, CRAFT-GBM ranks features by their interval point estimates and provides an approximate top-KK guarantee for feature budget KfK_f. We implement CRAFT-GBM within LightGBM while preserving its native feature-bundling representation, and evaluate it across 14 public and synthetic dataset families and four noise-augmented settings. In the fully endogenous setting, CRAFT-GBM reduces per-fit training time by 39.74% with 1.04% lower test loss in the geometric mean across all 18 RQ1 experiments. Under pre-specified feature fraction ff, Exogenous CRAFT-GBM also yields aggregate training-time reductions up to 13.52 percentage points larger than random feature subsampling, with no increase in aggregate test loss relative to full-feature LightGBM. Accumulated training evidence can thus support adaptive feature search with statistical guarantees while retaining the practical advantages of native gradient boosting.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Understanding Randomization in Greedy Model Search

    Jun 18, 2025Xin Chen, Jason M. Klusowski, Yan Shuo Tan +1Feature SelectionEnsemble Learning

  2. Feature Bagging Provides Stability

    Jul 29, 2026Yuheng Ma, Qiang SunEnsemble LearningAlgorithmic Stability