When Do LLMs Help with Feature Selection? A Systematic Empirical Study
Authors: Fateme Memar, Dongjie Wang, Arvin Agah
Organizations: Department of Electrical Engineering and Computer Science, University of Kansas, Lawrence, KS 66045, USA · School of Information Science and Technology, Northeast Normal University, China
Large language models (LLMs) can leverage semantic feature names alongside summary statistics for tabular feature selection, yet it remains unclear when such information improves subset selection relative to established statistical methods. We investigate this question within an 18-dataset framework comprising 10 datasets for Group Relative Policy Optimization (GRPO) training, 2 for checkpoint validation, and 6 held-out evaluation datasets. We compare five classical baselines, frozen Qwen2.5-7B and Llama-3.3-70B selectors, and two GRPO-adapted Qwen2.5-7B models, reporting classification ROC-AUC and regression R^2 separately and using dataset-wise ranks for cross-task comparison. Across five seeded evaluation partitions, classical selectors generally achieve higher predictive performance than the tested 7B LLM/RL selectors, including under a pre-specified ~33% feature budget with strict train-only classical fitting, while the 7B selectors retain compact subsets. A cardinality-matched semantic sensitivity experiment shows that, for Zero-shot Qwen-7B, informative feature names improve regression performance over anonymized and shuffled alternatives (+0.1964 and +0.1606 mean R^2), whereas aggregate classification differences are smaller (+0.0159 and +0.0084 ROC-AUC) and dataset-dependent. These results characterize when semantic information benefits LLM-based feature selection and show that classical selectors remain strong baselines for predictive tabular feature selection.
Figures & tables
Fig. 1: Overview of the experimental framework. The 18 datasets are partitioned into 10 for GRPO training, 2 for checkpoint validation, and 6 held-out evaluation datasets. Classical, frozen-LLM, and GRPO-adapted selectors are compared using task-separated ROC-AUC and R2 , with additional cardinality-matched semantic and train-only matched-budget controls.
Fig. 2: RL-GRPO v1 training and validation rewards across checkpoints 100–500; Checkpoint-100 achieves the highest mean validation reward ( 0.8282 ).
Classification (ROC-AUC)
Regression ( R2 )
Task-Separated Summary
Method
Category
Adult
Sonar
Dropout
Ames
Park.
Perform.
Clf AUC
Reg R2
Rank ↓
Feat%
No Selection
Classical
0.922±.003
0.912±.042
0.894±.008
0.915±.009
0.884±.005
0.305±.056
0.909
0.701
2.67
100%
mRMR [ 3 ]
Classical
0.882±.004
0.899±.067
0.886±.009
0.911±.010
0.896±.006
0.269±.043
0.889
0.692
3.67
49%
LASSO [ 4 ]
Classical
0.922±.003
0.914±.036
0.893±.009
0.911±.008
0.671±.013
0.278±.054
0.910
0.620
3.25
67%
Boruta [ 5 ]
Classical
0.903±.003
0.913±.053
0.881±.009
0.907±.010
0.884±.005
0.195±.057
0.899
0.662
5.08
38%
XGB Imp. [ 6 ]
Classical
0.912±.002
0.887±.026
0.891±.008
0.918±.006
0.890±.015
0.258±.105
0.897
0.689
3.50
49%
TABLE I: Predictive performance across the 6 held-out datasets over N=5 seeded evaluation partitions (mean ± SD). Classification reports ROC-AUC and regression reports R2 ; summary columns give task-specific means, 6-dataset mean rank ( 1=best ), and mean feature retention. Best predictive-performance and rank values are bold .
Fig. 3: Per-dataset mean predictive performance across the 10 methods over N=5 seeded evaluation partitions.
Fig. 4: Performance–sparsity trade-off using six-dataset mean rank and mean feature retention; lower values are preferable on both axes.
Fig. 5: Mean feature retention and syntax parse failure rates across the primary benchmark methods.
Method
Mean Feat%
Clf AUC (3)
Reg R2 (3)
mRMR Top- k (Train-Only)
32.7%
0.872±.016
0.663±.033
XGB Imp. Top- k (Train-Only)
32.7%
0.890±.017
0.646±.028
LASSO Top- k (Train-Only)
32.7%
0.877±.018
0.631±.034
RL-GRPO v1 (Ours)
33.4%
0.799±.019
0.596±.015
TABLE II: Matched-budget train-only classical control near RL-GRPO v1’s 33.4% retention budget over N=5 partitions (mean ± SD).
Fig. 6: Cardinality-matched semantic sensitivity of Zero-shot Qwen-7B across six held-out datasets over N=5 seeded evaluation partitions. Original, Anonymized, and Shuffled conditions use identical per-completion cardinality ( ΔRetention=0% ).
We introduce an iterative framework that automates the extraction of interpretable, schema-bound categorical features from unstructured text for tabular prediction models. To navigate the feature space, a generator LLM proposes semantic definitions, a separate extractor LLM materializes the features, and a downstream tabular model evaluates their predictive performance. We optimize this search by translating explicit model errors, such as AUC ranking inversions, into natural-language feedback, steering the LLM to resolve specific predictive failures. Evaluated across three public datasets, this error-driven loop accelerates feature discovery by up to 3× compared to unguided search. Empirically, the generated features demonstrate strong multi-view complementarity, strictly outperforming any subset when combined with TF-IDF and dense embeddings. Finally, the framework guarantees instance-level interpretability: the discovered features dominate SHAP importance rankings and provide a fully transparent, semantic audit trail for every prediction.
Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23 datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.
Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2
Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23 datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.
Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2