LLMs as Feature Engineers for Text-and-Tabular Prediction

Merwan Barlier; Blaz Skrlj

cs.LGSep 18, 2026

LLMs as Feature Engineers for Text-and-Tabular Prediction

Authors: Merwan Barlier, Blaz Skrlj

Organizations: Teads

Abstract

We introduce an iterative framework that automates the extraction of interpretable, schema-bound categorical features from unstructured text for tabular prediction models. To navigate the feature space, a generator LLM proposes semantic definitions, a separate extractor LLM materializes the features, and a downstream tabular model evaluates their predictive performance. We optimize this search by translating explicit model errors, such as AUC ranking inversions, into natural-language feedback, steering the LLM to resolve specific predictive failures. Evaluated across three public datasets, this error-driven loop accelerates feature discovery by up to $3\times$ compared to unguided search. Empirically, the generated features demonstrate strong multi-view complementarity, strictly outperforming any subset when combined with TF-IDF and dense embeddings. Finally, the framework guarantees instance-level interpretability: the discovered features dominate SHAP importance rankings and provide a fully transparent, semantic audit trail for every prediction.

Figures & tables

Method	Input Modality	Search Space	Feedback Signal	Live Inference
CAAFE ( Hollmann et al., 2023 )	Tabular only	Python code transformations	Zero-shot (No iterative feedback loop)	Fast (LLM-free)
OPRO ( Yang et al., 2024 )	Unstructured text	Free-form text prompts	Scalar metrics (Past fitness scores)	Slow (LLM call)
Ours	Raw text $\to$ Tabular	Schema-bound categorical definitions	Textual gradients (error-based)	Fast (LLM-free)

Table 1: Conceptual comparison of LLM-driven optimization and feature engineering frameworks. Unlike prior work, our approach bridges unstructured text and tabular prediction by iteratively optimizing schema-bound categories using explicit model failures, all while maintaining LLM-free real-time inference.

Panel of Figure 1 — Figure 1: Agentic recommender workflow. An architect proposes features, an executor evaluates them, and feedback drives the search loop.

Method	Kickstarter	Amazon Books	Stack Overflow
Tabular only	$0.7444$	$0.7611$	$0.8544$
+ TF-IDF SVD-50	$0.7528$	$0.8248$	$0.9468$
+ MiniLM PCA-50	$0.7572$	$0.8201$	$\mathbf{0.9510}$
+ 10 LLM features (ours)	$\mathbf{0.7639}$	$\mathbf{0.8366}$	$0.9320$
+ LLM + TF-IDF	$0.7662$	$0.8445$	$0.9599$
+ LLM + Embeddings	$0.7680$	$0.8481$	$0.9612$

Table 2: Multi-view comparison: 5-fold cross-validated AUC for each combination of representations on each dataset. Bold values mark the best single-view text representation (top group) and the best overall combination (bottom row). LLM features are the strongest single view on Kickstarter and Amazon, while MiniLM is strongest on Stack Overflow — yet adding LLM features always improves on top of either alternative, and all three combined is strictly best across datasets.

Dataset	Method	Baseline	Reaches
	$\Delta$ AUC	$\Delta$ AUC	at iter
Kickstarter	$\mathbf{+0.0225}$	$+0.0212$	4
Amazon Books	$\mathbf{+0.0731}$	$+0.0679$	3
Stack Overflow	$\mathbf{+0.0761}$	$+0.0752$	9

Table 3: Method vs. no-feedback baseline. Both variants run for 15 iterations with identical generator/extractor models and per-iteration budgets. Our method strictly outperforms the baseline’s final

\Delta

AUC across all three datasets. More importantly, the “Reaches at iter” column demonstrates that loss-aligned feedback achieves the baseline’s absolute peak performance in a fraction of the time, delivering up to a 5x reduction in LLM inference costs for the same downstream performance.

Kickstarter		Amazon Books		Stack Overflow
sem_specificity_of_core_noun	2.25	sem_content_materiality_focus	0.59	struct_title_orthography_quality	1.10
sem_target_audience	0.24	sem_negative_target_location	0.45	primary_tag	0.67
country	0.17	review_time	0.35	creation_year	0.62
launch_year	0.15	sem_inspection_depth	0.30	tone_formality_vs_fragmentation	0.40
category	0.15	sem_review_depth	0.24	n_tags	0.37
goal_usd	0.09	sem_bias_or_agenda_signal	0.19	struct_body_formatting_stability	0.32

Table 4: Top-10 features by mean

|

SHAP

|

across the test set for each dataset. LLM-generated features are highlighted (shaded cells). The top feature on every dataset is LLM-generated, and the LLM-feature dominance strengthens as the text content of the task gets richer.

Automated Feature Engineering

Automated Feature Engineering creates, transforms, and selects predictive inputs with less manual design across data types and applications.

See all 54 papers

MedFeat: Model-Aware and Explainability-Driven Feature Engineering with LLMs for Tabular Prediction
Feb 10, 2026Zizheng Zhang, Yiming Li, Justin Xu +6Interpretable ML Clinical Prediction
Microsoft Research · University of Oxford
TabSHAP
Apr 22, 2026Aryan Chaudhary, Prateek Agarwal, Tejasvi AlladiFeature Attribution SHAP Feature Attribution
Department of Computer Science Birla Institute of Technology and Science India
Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMs
Jul 25, 2026Sha Li, Naren RamakrishnanEvolutionary Optimization Large Language Model-Guided Optimization
Department of Computer Science Virginia Tech Blacksburg, VA 24060 · Department of Computer Science Virginia Tech Alexandria, VA 22305

Feb 10, 2026J/K move · Enter open · S save

MedFeat: Model-Aware and Explainability-Driven Feature Engineering with LLMs for Tabular Prediction

Zizheng Zhang, Yiming Li, Justin Xu, Jinyu Wang+5

Microsoft Research · University of Oxford

Interpretable ML Clinical Prediction Automated Feature Engineering Tabular Prediction Large Language Model-Based Feature Engineering

Open paper AskPDF arXiv

In clinical tabular prediction, classical machine learning models with feature engineering often outperform neural methods. LLMs are increasingly used to automate this process, acting as domain experts that propose diverse feature transformations to boost downstream performance. However, the feature generation process of existing LLM-based methods is agnostic to the downstream learner: the LLM receives no signal about which features currently drive predictions or where the model's representational capacity falls short, so proposals are neither targeted to promising regions of the feature space nor tailored to the learner's inductive bias. This shortcoming is amplified in healthcare data, which simultaneously exhibits class imbalance, heterogeneous feature spaces, and strict interpretability requirements. In this paper, we propose MedFeat, the first feature engineering framework inspired by the workflow of machine learning practitioners, leveraging model-awareness and feature importance signals to iteratively guide feature discovery for clinical tabular learning. We evaluate MedFeat on a broad range of challenging real-world clinical tasks and show that it statistically significantly outperforms state-of-the-art baselines, with an average F1 improvement of more than 10% over the baseline across models with distinct inductive biases.

Abstract

Automated Feature Engineering

Explore similar work

MedFeat: Model-Aware and Explainability-Driven Feature Engineering with LLMs for Tabular Prediction

TabSHAP

Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMs