FAMOSE: A ReAct Approach to Automated Feature Discovery

Keith Burghardt; Jienan Liu; Sadman Sakib; Yuning Hao; Bo Li

cs.LGFeb 19, 2026

FAMOSE: A ReAct Approach to Automated Feature Discovery

Authors: Keith Burghardt, Jienan Liu, Sadman Sakib, Yuning Hao, Bo Li

Organizations: Amazon.com, Inc., 410 Terry Ave N, Seattle, WA 98109, United States.

Abstract

Feature engineering remains a critical yet challenging bottleneck in machine learning, particularly for tabular data, as identifying optimal features from an exponentially large feature space traditionally demands substantial domain expertise. To address this challenge, we introduce FAMOSE (Feature AugMentation and Optimal Selection agEnt), a novel framework that leverages the ReAct paradigm to autonomously explore, generate, and refine features while integrating feature selection and evaluation tools within an agent architecture. To our knowledge, FAMOSE represents the first application of an agentic ReAct framework to automated feature engineering, especially for both regression and classification tasks. Extensive experiments demonstrate that FAMOSE is at or near the state-of-the-art on classification tasks (especially tasks with more than 10K instances, where ROC-AUC increases 0.23% on average), and achieves the state-of-the-art for regression tasks by reducing RMSE by 2.0% on average, while remaining more robust to errors than other algorithms. We hypothesize that FAMOSE's strong performance is because ReAct allows the LLM context window to record (via iterative feature discovery and evaluation steps) what features did or did not work. This is similar to a few-shot prompt and guides the LLM to invent better, more innovative features. Our work offers evidence that AI agents are remarkably effective in solving problems that require highly inventive solutions, such as feature engineering.

Figures & tables

Figure 1: Types of AutoML. In traditional AutoML, feature engineering would include feature discovery, such as within OpenFE ( Zhang et al., 2023 ) , or iterative feature modification, such as the method of Piramuthu et al. ( Piramuthu & Sikora, 2009 ) . Although LLM methods, such as CAAFE, often excel at feature generation, iterative feature modification has not been explored as often before now. FAMOSE offers a way to learn through trial-and-error which features work and which ones do not until a better feature is developed.

Figure 2: An illustration of FAMOSE applied to the balance-scale task ( Siegler, 1976 ) . FAMOSE first invents features and observes their performance, in this case using the difference in arm weights to see whether a scale is balanced, tilting right, or tilting left. If performance is insufficient, FAMOSE then thinks about how to create better features, ultimately discovering the feature, moment, that exactly predicts whether a scale is balanced. Because the performance cannot be improved further in this task, the feature discovery step eventually stops. The feature selection step then selects this feature using mRMR ( Ding & Peng, 2005 ) and removes the four extraneous features in the task dataset: $W_{1}$ , $W_{2}$ , $L_{1}$ , and $L_{2}$ , thus instead of an error-prone model with four features, we discover a perfect predictor with only one feature.

Task	Baseline	OpenFE	AutoFeat	FAMOSE
bike	40.3 $\pm$ 1.03	92.09 $\pm$ 6.35	41.47 $\pm$ 1.01	40.05 $\pm$ 0.99
crab	2.32 $\pm$ 0.13	2.26 $\pm$ 0.15	–	2.34 $\pm$ 0.08
cybersecurity_attacks	0.82 $\pm$ 0.0	–	–	0.82 $\pm$ 0.0
housing	409.58 $\pm$ 10.54	432.87 $\pm$ 10.22	403.96 $\pm$ 9.61	408.56 $\pm$ 26.34
insurance	192.89 $\pm$ 13.91	190.23 $\pm$ 11.18	187.6 $\pm$ 12.48	191.64 $\pm$ 11.16
forest-fires	92.7 $\pm$ 5.34	88.52 $\pm$ 5.91	93.49 $\pm$ 5.42	79.49 $\pm$ 5.87

Table 1: RMSE comparison of FAMOSE to SOTA feature engineering algorithms across 5 folds: OpenFE ( Zhang et al., 2023 ) and AutoFeat ( Horn et al., 2019 ) . The “% Reduction” is the mean percent reduction of RMSE across all folds and tasks. This is calculated by replacing any algorithms that could not run with the baseline performance. Bold indicates best model.

Task	Baseline	OpenFE	AutoFeat	FeatLLM	CAAFE	FAMOSE
balance-scale	0.914 $\pm$ 0.01	0.999 $\pm$ 0.001	0.977 $\pm$ 0.025	1.0 $\pm$ 0.0	0.965 $\pm$ 0.022	1.0 $\pm$ 0.0
blood	0.698 $\pm$ 0.03	0.681 $\pm$ 0.048	0.677 $\pm$ 0.033	0.426 $\pm$ 0.063	0.697 $\pm$ 0.03	0.704 $\pm$ 0.027
breast-w	0.989 $\pm$ 0.007	0.993 $\pm$ 0.004	0.993 $\pm$ 0.006	0.994 $\pm$ 0.004	0.99 $\pm$ 0.01	0.989 $\pm$ 0.008
car	0.999 $\pm$ 0.0	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0	0.86 $\pm$ 0.019	0.999 $\pm$ 0.0	0.997 $\pm$ 0.003
cmc	0.704 $\pm$ 0.019	0.693 $\pm$ 0.013	0.683 $\pm$ 0.019	0.709 $\pm$ 0.031	0.7 $\pm$ 0.02	0.696 $\pm$ 0.023
communities	0.866 $\pm$ 0.008	–	–	0.808 $\pm$ 0.011	0.864 $\pm$ 0.007	0.861 $\pm$ 0.007

Table 2: ROC-AUC comparison of FAMOSE to SOTA feature engineering algorithms across 5 folds: AutoFeat ( Horn et al., 2019 ) , OpenFE ( Zhang et al., 2023 ) , FeatLLM ( Han et al., 2024 ) , and CAAFE ( Hollmann et al., 2023 ) . Algorithms that could not run due to errors are indicated by “–”. The “% Improvement” is the mean percent increase in ROC-AUC across folds and tasks. This is calculated by replacing any algorithms that could not run with the baseline performance. Bold indicates best model.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Task	# Datapoints	# Features	Imbalance	Target Variance
adult	48842	15	0.24	–
arrhythmia	452	280	0.0	–
balance-scale	625	5	0.08	–
bank_fraud_base	1000000	32	0.01	–
bank_marketing	45211	17	0.12	–
blood	748	5	0.24	–

Appendix

Table S1 : Data statistics: number of datapoints, number of features, data imbalance (if a classification task) or target variance (if a regression task).

Task Type	Task	Prediction Task
Classification	adult ( Becker & Kohavi, 1996 )	Earn more than $50,000 per year (yes/no)
	arrhythmia ( Guvenir et al., 1997 )	Presence of cardiac arrhythmia (16 classes)
	balance-scale ( Siegler, 1976 )	Balance direction: left, right, or balanced
	breast-w ( Wolberg, 1990 )	Breast mass malignant (M) or benign (B)
	blood ( Yeh, 2008 )	Blood donation occurred (yes/no)
	car ( Bohanec, 1988 )	Car acceptability rating (4 classes)

Appendix

Table S2 : Tasks used to evaluate FAMOSE.

age	“the age of an individual”
workclass	“a general term to represent the employment status of an individual”
fnlwgt	“the number of units in the target population that the responding unit represents”
education	“the highest level of education achieved by an individual”
educational-num	“the highest level of education achieved in numerical form”
marital-status	“marital status of an individual”
occupation	“the general type of occupation of an individual”

Appendix

Table S3 : Example feature descriptions for the adult task.

age	“age in years”
job	“type of job (categorical: ’admin.’, ’unknown’, ’unemployed’, ’management’, ’housemaid’, ’entrepreneur’, ’student’, ’blue-collar’, ’self-employed,retired’, ’technician’, ’services’)”
marital	“marital status (categorical: ’married’,’divorced’,’single’; note: ’divorced’ means divorced or widowed)”
education	“(categorical: ’unknown’,’secondary’,’primary’,’tertiary’)”
default	“has credit in default? (binary: ’yes’,’no’)”
balance	“average yearly balance, in euros (numeric)”
housing	“has housing loan? (binary: ’yes’,’no’)”

Appendix

Table S4 : Example feature descriptions for the bank-marketing task.

Figure S1 : Number of functions used in CAAFE and FAMOSE code across all task folds (5

\times

number of tasks).

Figure S2 : Frequency each function used in CAAFE code across all task folds (5

\times

number of tasks).

Figure S3 : Frequency each function used in FAMOSE code across all task folds (5

\times

number of tasks).

Figure S4 : Percentage of task folds (5

\times

number of tasks) where FAMOSE or CAAFE features were added.

	Baseline	FAMOSE (Claude)	FAMOSE (Deepseek)
adult	0.929 $\pm$ 0.002	0.929 $\pm$ 0.002	0.929 $\pm$ 0.002
balance-scale	0.914 $\pm$ 0.01	1.0 $\pm$ 0.0	1.0 $\pm$ 0.0
bank_fraud_base	0.886 $\pm$ 0.004	0.886 $\pm$ 0.003	0.885 $\pm$ 0.004
bank_marketing	0.934 $\pm$ 0.005	0.928 $\pm$ 0.013	0.933 $\pm$ 0.004
blood	0.698 $\pm$ 0.03	0.704 $\pm$ 0.027	0.687 $\pm$ 0.009
breast-w	0.989 $\pm$ 0.007	0.989 $\pm$ 0.008	0.986 $\pm$ 0.008

Appendix

Table S5 : FAMOSE classifier performance (ROC-AUC) for Claude and Deepseek. We notice that the algorithm works well for both closed-source (Claude 3.5 Sonnet V2) and open-source (Deepseek-R1) LLMs.

	Baseline	FAMOSE (Sonnet 3.5 V2)	FAMOSE (Deepseek)
bike	40.3 $\pm$ 1.03	40.05 $\pm$ 0.99	39.79 $\pm$ 0.73
crab	2.32 $\pm$ 0.13	2.34 $\pm$ 0.08	2.31 $\pm$ 0.16
housing	409.58 $\pm$ 10.54	408.56 $\pm$ 26.34	396.44 $\pm$ 19.22
insurance	192.89 $\pm$ 13.91	191.64 $\pm$ 11.16	190.16 $\pm$ 11.96
forest-fires	92.7 $\pm$ 5.34	79.49 $\pm$ 5.87	79.47 $\pm$ 4.6
wine-quality	0.64 $\pm$ 0.02	0.64 $\pm$ 0.01	0.64 $\pm$ 0.02

Appendix

Table S6 : FAMOSE regression performance (RMSE) for Claude and Deepseek. Bold are best-performing models. We notice that the algorithm works well for both closed-source (Claude 3.5 Sonnet V2) and open-source (Deepseek-R1) LLMs, with a slight edge for Deepseek-R1.

	Autogluon		Random Forest
	Baseline	FAMOSE	Baseline	FAMOSE
adult	0.929 $\pm$ 0.003	0.929 $\pm$ 0.002	0.903 $\pm$ 0.002	0.901 $\pm$ 0.005
balance-scale	0.995 $\pm$ 0.004	1.0 $\pm$ 0.0	0.806 $\pm$ 0.009	1.0 $\pm$ 0.0
bank_marketing	0.938 $\pm$ 0.004	0.937 $\pm$ 0.006	0.929 $\pm$ 0.004	0.928 $\pm$ 0.003
bank_fraud_base	0.877 $\pm$ 0.01	0.884 $\pm$ 0.006	0.834 $\pm$ 0.003	0.835 $\pm$ 0.004
blood	0.757 $\pm$ 0.035	0.747 $\pm$ 0.029	0.686 $\pm$ 0.042	0.706 $\pm$ 0.03

Appendix

Table S7 : Robustness of classifier performance (ROC-AUC) for FAMOSE XGBoost features applied to different models (Random forest and Autogluon). Bold indicates the best model.

	Autogluon		Random Forest
	Baseline	FAMOSE	Baseline	FAMOSE
bike	35.37 $\pm$ 0.92	35.41 $\pm$ 0.96	42.02 $\pm$ 1.53	41.97 $\pm$ 1.23
crab	2.13 $\pm$ 0.11	2.1 $\pm$ 0.13	2.19 $\pm$ 0.09	2.18 $\pm$ 0.09
housing	384.31 $\pm$ 15.11	381.08 $\pm$ 18.47	418.38 $\pm$ 11.23	416.28 $\pm$ 14.05
insurance	166.59 $\pm$ 13.31	174.56 $\pm$ 16.7	178.82 $\pm$ 8.87	183.44 $\pm$ 11.0
forest-fires	80.89 $\pm$ 6.11	79.16 $\pm$ 5.59	83.13 $\pm$ 5.04	79.22 $\pm$ 5.68

Appendix

Table S8 : Robustness of classifier performance (RMSE) for FAMOSE XGBoost features applied to different models (Random forest and Autogluon). Bold indicates the best model.

	No goal in prompt	Only feature selection	No feature selection
bike	40.19 $\pm$ 0.89	40.3 $\pm$ 1.03	40.43 $\pm$ 1.54
crab	2.34 $\pm$ 0.14	2.34 $\pm$ 0.12	2.34 $\pm$ 0.09
cybersecurity_attacks	0.82 $\pm$ 0.0	0.82 $\pm$ 0.0	0.83 $\pm$ 0.0
forest-fires	79.87 $\pm$ 4.98	79.54 $\pm$ 5.89	92.93 $\pm$ 5.24
housing	409.03 $\pm$ 26.18	409.58 $\pm$ 10.54	418.51 $\pm$ 38.84
insurance	186.53 $\pm$ 12.52	184.43 $\pm$ 16.86	193.95 $\pm$ 13.82

Appendix

Table S9 : Ablation for regression tasks. We see a slight improvement in the RMSE reduction when we only select features, although the difference is not significant (Wilcoxon signed rank test p-value

=0.3

), but the difference between FAMOSE and no feature selection is significant (Wilcoxon signed rank test p-value

=0.007

). Overall, our results show FAMOSE performs best when we have a goal and feature selection.

	No goal in prompt	Only feature selection	No feature selection
adult	0.928 $\pm$ 0.002	0.929 $\pm$ 0.002	0.929 $\pm$ 0.002
balance-scale	1.0 $\pm$ 0.0	0.914 $\pm$ 0.01	1.0 $\pm$ 0.0
bank_fraud_base	0.885 $\pm$ 0.004	0.885 $\pm$ 0.005	0.881 $\pm$ 0.01
bank_marketing	0.928 $\pm$ 0.005	0.933 $\pm$ 0.004	0.929 $\pm$ 0.011
blood	0.678 $\pm$ 0.038	0.688 $\pm$ 0.024	0.679 $\pm$ 0.024
breast-w	0.99 $\pm$ 0.008	0.989 $\pm$ 0.006	0.985 $\pm$ 0.011

Appendix

Table S10 : Ablation for classifier tasks. Due to the small changes in ROC-AUC, the results are not statistically significant, except for only feature selection, which is significantly worse than the baseline and FAMOSE (Wilcoxon signed rank test p-value

<0.04

), which suggests that we need to go beyond feature selection along to achieve high performance. Overall, our results show FAMOSE performs best when we have a goal and feature selection.

Automated Feature Engineering

Automated Feature Engineering creates, transforms, and selects predictive inputs with less manual design across data types and applications.

See all 54 papers

Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data
Apr 22, 2026Fengxian Dong, Zhi Zheng, Xiao Han +5Tabular ML Multi-Agent LLMs
University of Science and Technology of China · Zhejiang University of Technology · Meituan
Causally-Guided Automated Feature Engineering with Multi-Agent Reinforcement Learning
Feb 18, 2026Arun Vignesh Malarkkan, Wangyang Ying, Yanjie FuAutomated Feature Engineering Distribution Shift Robustness
School of Computing, Arizona State University
Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMs
Jul 25, 2026Sha Li, Naren RamakrishnanEvolutionary Optimization Large Language Model-Guided Optimization
Department of Computer Science Virginia Tech Blacksburg, VA 24060 · Department of Computer Science Virginia Tech Alexandria, VA 22305

Apr 22, 2026J/K move · Enter open · S save

Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data

Fengxian Dong, Zhi Zheng, Xiao Han, Wei Chen+4

University of Science and Technology of China · Zhejiang University of Technology · Meituan

Tabular ML Multi-Agent LLMs Automated Feature Engineering Memory-Augmented Language Model Agents Large Language Model-Based Feature Engineering

Open paper AskPDF arXiv

Automated feature generation extracts informative features from raw tabular data without manual intervention and is crucial for accurate, generalizable machine learning. Traditional methods rely on predefined operator libraries and cannot leverage task semantics, limiting their ability to produce diverse, high-value features for complex tasks. Recent Large Language Model (LLM)-based approaches introduce richer semantic signals, but still suffer from a restricted feature space due to fixed generation patterns and from the absence of feedback from the learning objective. To address these challenges, we propose a Memory-Augmented LLM-based Multi-Agent System (MALMAS) for automated feature generation. MALMAS decomposes the generation process into agents with distinct responsibilities, and a Router Agent activates an appropriate subset of agents per iteration, further broadening exploration of the feature space. We further integrate a memory module comprising procedural memory, feedback memory, and conceptual memory, enabling iterative refinement that adaptively guides subsequent feature generation and improves feature quality and diversity. Extensive experiments on multiple public datasets against state-of-the-art baselines demonstrate the effectiveness of our approach. The code is available at https://github.com/fxdong24/MALMAS

Abstract

Automated Feature Engineering

Explore similar work

Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data

Causally-Guided Automated Feature Engineering with Multi-Agent Reinforcement Learning

Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMs