When Do LLMs Help with Feature Selection? A Systematic Empirical Study

Fateme Memar; Dongjie Wang; Arvin Agah

cs.LGOct 20, 2026

When Do LLMs Help with Feature Selection? A Systematic Empirical Study

Authors: Fateme Memar, Dongjie Wang, Arvin Agah

Organizations: Department of Electrical Engineering and Computer Science, University of Kansas, Lawrence, KS 66045, USA · School of Information Science and Technology, Northeast Normal University, China

Abstract

Large language models (LLMs) can leverage semantic feature names alongside summary statistics for tabular feature selection, yet it remains unclear when such information improves subset selection relative to established statistical methods. We investigate this question within an 18-dataset framework comprising 10 datasets for Group Relative Policy Optimization (GRPO) training, 2 for checkpoint validation, and 6 held-out evaluation datasets. We compare five classical baselines, frozen Qwen2.5-7B and Llama-3.3-70B selectors, and two GRPO-adapted Qwen2.5-7B models, reporting classification ROC-AUC and regression R^2 separately and using dataset-wise ranks for cross-task comparison. Across five seeded evaluation partitions, classical selectors generally achieve higher predictive performance than the tested 7B LLM/RL selectors, including under a pre-specified ~33% feature budget with strict train-only classical fitting, while the 7B selectors retain compact subsets. A cardinality-matched semantic sensitivity experiment shows that, for Zero-shot Qwen-7B, informative feature names improve regression performance over anonymized and shuffled alternatives (+0.1964 and +0.1606 mean R^2), whereas aggregate classification differences are smaller (+0.0159 and +0.0084 ROC-AUC) and dataset-dependent. These results characterize when semantic information benefits LLM-based feature selection and show that classical selectors remain strong baselines for predictive tabular feature selection.

Figures & tables

Fig. 1: Overview of the experimental framework. The 18 datasets are partitioned into 10 for GRPO training, 2 for checkpoint validation, and 6 held-out evaluation datasets. Classical, frozen-LLM, and GRPO-adapted selectors are compared using task-separated ROC-AUC and $R^{2}$ , with additional cardinality-matched semantic and train-only matched-budget controls.

Fig. 2: RL-GRPO v1 training and validation rewards across checkpoints 100–500; Checkpoint-100 achieves the highest mean validation reward (

0.8282

Method	Category	Adult	Sonar	Dropout	Ames	Park.	Perform.	Clf AUC	Reg $R^{2}$	Rank $\downarrow$	Feat%
		Classification (ROC-AUC)			Regression ( $R^{2}$ )			Task-Separated Summary
No Selection	Classical	$\mathbf{0.922{\pm}.003}$	$0.912{\pm}.042$	$\mathbf{0.894{\pm}.008}$	$0.915{\pm}.009$	$0.884{\pm}.005$	$\mathbf{0.305{\pm}.056}$	$0.909$	$\mathbf{0.701}$	$\mathbf{2.67}$	$100\%$
mRMR [ 3 ]	Classical	$0.882{\pm}.004$	$0.899{\pm}.067$	$0.886{\pm}.009$	$0.911{\pm}.010$	$\mathbf{0.896{\pm}.006}$	$0.269{\pm}.043$	$0.889$	$0.692$	$3.67$	$49\%$
LASSO [ 4 ]	Classical	$\mathbf{0.922{\pm}.003}$	$\mathbf{0.914{\pm}.036}$	$0.893{\pm}.009$	$0.911{\pm}.008$	$0.671{\pm}.013$	$0.278{\pm}.054$	$\mathbf{0.910}$	$0.620$	$3.25$	$67\%$
Boruta [ 5 ]	Classical	$0.903{\pm}.003$	$0.913{\pm}.053$	$0.881{\pm}.009$	$0.907{\pm}.010$	$0.884{\pm}.005$	$0.195{\pm}.057$	$0.899$	$0.662$	$5.08$	$38\%$
XGB Imp. [ 6 ]	Classical	$0.912{\pm}.002$	$0.887{\pm}.026$	$0.891{\pm}.008$	$\mathbf{0.918{\pm}.006}$	$0.890{\pm}.015$	$0.258{\pm}.105$	$0.897$	$0.689$	$3.50$	$49\%$

TABLE I: Predictive performance across the 6 held-out datasets over

N=5

seeded evaluation partitions (mean

\pm

SD). Classification reports ROC-AUC and regression reports

R^{2}

; summary columns give task-specific means, 6-dataset mean rank (

1=\text{best}

), and mean feature retention. Best predictive-performance and rank values are bold .

Fig. 3: Per-dataset mean predictive performance across the 10 methods over $N=5$ seeded evaluation partitions.

Fig. 4: Performance–sparsity trade-off using six-dataset mean rank and mean feature retention; lower values are preferable on both axes.

Fig. 5: Mean feature retention and syntax parse failure rates across the primary benchmark methods.

Method	Mean Feat%	Clf AUC (3)	Reg $R^{2}$ (3)
mRMR Top- $k$ (Train-Only)	$32.7\%$	$0.872\pm.016$	$\mathbf{0.663\pm.033}$
XGB Imp. Top- $k$ (Train-Only)	$32.7\%$	$\mathbf{0.890\pm.017}$	$0.646\pm.028$
LASSO Top- $k$ (Train-Only)	$32.7\%$	$0.877\pm.018$	$0.631\pm.034$
RL-GRPO v1 (Ours)	$33.4\%$	$0.799\pm.019$	$0.596\pm.015$

TABLE II: Matched-budget train-only classical control near RL-GRPO v1’s

33.4\%

retention budget over

N=5

partitions (mean

\pm

SD).

Fig. 6: Cardinality-matched semantic sensitivity of Zero-shot Qwen-7B across six held-out datasets over

N=5

seeded evaluation partitions. Original, Anonymized, and Shuffled conditions use identical per-completion cardinality (

\Delta\mathrm{Retention}=0\%

Feature Selection

Feature selection identifies compact, informative variables or input dimensions to improve prediction, efficiency, interpretability, and statistical inference.

See all 194 papers

LLMs as Feature Engineers for Text-and-Tabular Prediction
Sep 18, 2026Merwan Barlier, Blaz SkrljAutomated Feature Engineering Large Language Model-Based Feature Engineering
Teads
Large Language Model Selection with Limited Annotations
Oct 10, 2025Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2Model Selection LLM Evaluation
TU Delft · ETH Zurich
Large Language Model Selection with Limited Annotations
May 24, 2026Yavuz Durmazkeser, Patrik Okanovic, Andreas Kirsch +2Model Selection LLM Evaluation
TU Delft · ETH Zurich

Sep 18, 2026J/K move · Enter open · S save

LLMs as Feature Engineers for Text-and-Tabular Prediction

Merwan Barlier, Blaz Skrlj

Teads

Automated Feature Engineering Large Language Model-Based Feature Engineering Tabular Prediction LLM Information Extraction

Open paper AskPDF arXiv

We introduce an iterative framework that automates the extraction of interpretable, schema-bound categorical features from unstructured text for tabular prediction models. To navigate the feature space, a generator LLM proposes semantic definitions, a separate extractor LLM materializes the features, and a downstream tabular model evaluates their predictive performance. We optimize this search by translating explicit model errors, such as AUC ranking inversions, into natural-language feedback, steering the LLM to resolve specific predictive failures. Evaluated across three public datasets, this error-driven loop accelerates feature discovery by up to

3\times

compared to unguided search. Empirically, the generated features demonstrate strong multi-view complementarity, strictly outperforming any subset when combined with TF-IDF and dense embeddings. Finally, the framework guarantees instance-level interpretability: the discovered features dominate SHAP importance rankings and provide a fully transparent, semantic audit trail for every prediction.

Abstract

Feature Selection

Explore similar work

LLMs as Feature Engineers for Text-and-Tabular Prediction

Large Language Model Selection with Limited Annotations

Large Language Model Selection with Limited Annotations