SIGMA: SHAP-Guided Implicit-Trajectory Generation for Metadata-Free LLM-Based AutoFE

Xuan Zheng; Kento Uchida; Shinichi Shirakawa

cs.LGAug 18, 2026

SIGMA: SHAP-Guided Implicit-Trajectory Generation for Metadata-Free LLM-Based AutoFE

Authors: Xuan Zheng, Kento Uchida, Shinichi Shirakawa

Organizations: Yokohama National University

Abstract

Recent research has leveraged Large Language Models (LLMs) to enhance Automated Feature Engineering (AutoFE) through semantic descriptions and trajectory-based prompting. However, there exist two challenges that limit their applicability and scalability in long-horizon optimization: (1) semantic metadata is unavailable in many practical settings, and (2) trajectory accumulation increases the risk of exceeding the context window, while without it, the generation process can become unstable, leading to becoming stuck in the local optima and a high duplicate rate of generated features. To this end, we propose a SHAP-enhanced Implicit-trajectory Generation for Metadata-free AutoFE (SIGMA), a scalable constant-context optimization framework. SIGMA leverages SHAP values to provide task-aware signals for guiding group feature generation instead of semantic information. In addition, we adopt an EXposed-feature Implicit Trajectory (EXIT) approach, where the exposed features in the prompt implicitly represent the trajectory. Empirical results demonstrate that SIGMA achieves performance comparable to the state-of-the-art (SOTA) LLM baselines with a nearly constant prompt length. Notably, EXIT significantly reduces the duplicate ratio of generated features from 37.2% to 6.8%. At the same time, SIGMA matches traditional SOTA performance with only 5.4 features on average, demonstrating substantial efficiency gains in feature utilization.

Figures & tables

Dataset	Baseline (w.o. AutoFE)	CAAFE	OCTree	SIGMA (ours)
eucalyptus	64.94 $\pm$ 1.70	65.27 $\pm$ 2.04	65.12 $\pm$ 1.24	66.39 $\pm$ 2.29
diabetes	73.55 $\pm$ 5.40	73.58 $\pm$ 3.62	73.62 $\pm$ 4.37	74.82 $\pm$ 3.16
credit-g	74.28 $\pm$ 3.11	73.46 $\pm$ 2.90	74.56 $\pm$ 1.73	74.84 $\pm$ 2.19
pc1	92.71 $\pm$ 1.21	92.64 $\pm$ 0.95	92.83 $\pm$ 1.42	92.22 $\pm$ 1.06
cmc	51.22 $\pm$ 2.60	51.76 $\pm$ 2.01	49.97 $\pm$ 1.95	51.46 $\pm$ 2.50
wine	80.53 $\pm$ 0.39	80.07 $\pm$ 1.66	80.15 $\pm$ 1.36	79.13 $\pm$ 0.89

Table 1: F1-score comparison of LLM-based AutoFE using Qwen3-4B-Instruct-2507. The best results are highlighted in bold , and the second-best results are underlined .

Figure 2: Efficiency analysis of methods: (a) the average prompt token trends of LLM-based AutoFE methods. Since SIGMA generates two features at each iteration, the final step is 25. (b) the feature-efficiency comparison with traditional AutoFE.

Dataset	Baseline (w.o. AutoFE)	AutoFeat	DFS	OpenFE	SIGMA (Ours)
eucalyptus	64.94 $\pm$ 1.70	65.72 $\pm$ 2.93	66.71 $\pm$ 1.71	65.67 $\pm$ 0.41	66.39 $\pm$ 2.29
diabetes	73.55 $\pm$ 5.40	74.87 $\pm$ 2.34	74.32 $\pm$ 2.95	75.93 $\pm$ 2.82	74.82 $\pm$ 3.16
credit-g	74.28 $\pm$ 3.11	73.97 $\pm$ 2.33	74.18 $\pm$ 3.05	73.79 $\pm$ 2.24	74.84 $\pm$ 2.19
pc1	92.71 $\pm$ 1.21	92.94 $\pm$ 0.66	92.65 $\pm$ 1.29	92.39 $\pm$ 0.38	92.22 $\pm$ 1.06
cmc	51.22 $\pm$ 2.60	50.51 $\pm$ 2.75	52.32 $\pm$ 2.96	52.60 $\pm$ 2.30	51.46 $\pm$ 2.50
wine	80.53 $\pm$ 0.39	78.42 $\pm$ 1.46	79.87 $\pm$ 1.52	78.83 $\pm$ 0.32	79.13 $\pm$ 0.89

Table 2: F1-score results of traditional AutoFE methods under the same feature budget. Bold and underlined indicate the best and second-best results, respectively.

Figure 3: The topology of generated features on the jungle_chess and compass datasets: (a) and (b) are the jungle_chess and compass datasets, respectively.

Panel of Figure 4 — Figure 4: Ablation results of SIGMA. “ w/o predefine operations ” represents that EXIT is still used, but operations are not provided. “ w/o track operations ” represents that EXIT is still used and operations are provided, but LLMs are not forbidden from using the top-2 frequent operations. “ w/o EXIT ” represents that operation predefinition and tracking are kept the same as SIGMA, but without the EXIT. (a) depicts the acceptance rate of generated features over all experiments. (b) and (c) show the code error rate and feature duplication rate, respectively.

Dataset Size	Mean F1-Score			Mean Rank			vs Qwen3-4B
Dataset Size	Qwen3 4B	Qwen3 Coder	Llama 70B	Qwen3 4B	Qwen3 Coder	Llama 70B	Qwen3 Coder	Llama 70B
Small ( $\leq$ 2000)	71.95%	71.34%	71.27%	1.60	2.2	2.2	-0.61%	-0.67%
Large ( $>$ 2000)	83.37%	83.51%	83.30%	2.36	1.73	1.91	+0.14%	-0.07%
All	79.80%	79.71%	79.54%	2.12	1.88	2.00	-0.10%	-0.26%

Table 3: SIGMA performance with different LLMs by dataset size. LLama-70B represents Llama-3.1-70B. The best results are highlighted in bold .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Metric	Baseline (w.o. AutoFE)	CAAFE	OCTree	SIGMA
Average ACC	79.22	79.91 $\pm$ 0.18	79.10 $\pm$ 0.14	79.98 $\pm$ 0.24
Avg ACC Rank	2.50	2.38	3.12	2.00
Average AUC	87.01	87.43 $\pm$ 0.12	86.77 $\pm$ 0.10	87.41 $\pm$ 0.04
Avg AUC Rank	2.44	2.12	3.38	2.00

Appendix

Table 4: Overall accuracy (ACC) and AUC-ROC comparison of LLM-based AutoFE.

Dataset	Baseline (w.o. AutoFE)	AutoFeat	DFS	OpenFE	SIGMA
Average ACC	79.22	79.36	79.39	80.18	79.98 $\pm$ 0.24
Avg ACC Rank	3.38	3.06	3.44	2.38	2.69
Average AUC	87.01	87.30	87.09	87.60	87.41 $\pm$ 0.04
Avg AUC Rank	3.38	2.81	3.44	2.50	2.81

Appendix

Table 5: Overall accuracy (ACC) and AUC-ROC comparison of traditional methods under a feature budget of 20.

Dataset	$C$	$F$	$N$	Llama3.1-70B	Qwen3-4B	Qwen3-Coder-Next
eucalyptus	5	19	736	65.15 $\pm$ 2.35	66.39 $\pm$ 2.29	65.43 $\pm$ 2.45
diabetes	2	8	768	73.50 $\pm$ 3.72	74.82 $\pm$ 3.16	73.97 $\pm$ 3.92
credit-g	2	20	1,000	73.33 $\pm$ 2.70	74.84 $\pm$ 2.19	73.84 $\pm$ 2.00
pc1	2	21	1,109	92.70 $\pm$ 1.04	92.22 $\pm$ 1.06	92.32 $\pm$ 1.06
cmc	3	9	1,473	51.66 $\pm$ 2.65	51.46 $\pm$ 2.50	51.14 $\pm$ 3.16
wine	2	11	2,554	79.76 $\pm$ 1.36	79.13 $\pm$ 0.89	79.31 $\pm$ 1.24

Appendix

Table 6: Impact of LLMs on each dataset of F1-score.

C

denotes the number of classes,

F

denotes the number of features, and

N

denotes the number of samples.

Large Language Model-Based Feature Engineering

See all 18 papers

Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMs
Jul 25, 2026Sha Li, Naren RamakrishnanEvolutionary Optimization Large Language Model-Guided Optimization
Department of Computer Science Virginia Tech Blacksburg, VA 24060 · Department of Computer Science Virginia Tech Alexandria, VA 22305
SymboLLM-FE: LLM-Accelerated Symbolic Regression for Automated Feature Engineering on Tabular Data
Aug 28, 2026Zi-Jian Cheng, Zi-Yi Jia, Zhi Zhou +2Large Language Model-Based Feature Engineering Automated Feature Engineering
National Key Laboratory for Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · School of Artificial Intelligence, Nanjing University
Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data
Apr 22, 2026Fengxian Dong, Zhi Zheng, Xiao Han +5Tabular ML Multi-Agent LLMs
University of Science and Technology of China · Zhejiang University of Technology · Meituan

Jul 25, 2026J/K move · Enter open · S save

Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMs

Sha Li, Naren Ramakrishnan

Department of Computer Science Virginia Tech Blacksburg, VA 24060 · Department of Computer Science Virginia Tech Alexandria, VA 22305

Evolutionary Optimization Large Language Model-Guided Optimization LLM-Based Program Synthesis Tabular ML Automated Feature Engineering Large Language Model-Based Feature Engineering

Open paper AskPDF arXiv

Automatic feature engineering (AutoFE) for tabular data requires discovering informative transformations from a large program space. Existing approaches suffer from three limitations: classical methods rely on fixed operator libraries with limited expressivity, LLM-based methods generate proposals from static prompts without retaining search experience, and evolutionary methods use fixed migration policies that ignore task-specific cross-family transfer utility. We introduce TOPOFE, a framework that formulates AutoFE as graph-structured multi-island evolutionary program search. The transformation space is partitioned into semantically coherent families, each explored by an island through LLM-guided mutation and crossover. Each island maintains a Prompt Adaptation Memory that accumulates accept/reject feedback to steer proposals toward productive regions without parameter updates. To coordinate global exploration, TOPOFE dynamically learns a directed topology graph whose edge weights encode transfer utility between transformation families. Cross-island transfer is triggered by adaptive saturation detection and performed through LLM-mediated hybrid synthesis, enabling discovery of compositional feature programs that cannot emerge from isolated local search. Experiments on 29 tabular datasets show that TOPOFE consistently outperforms most state-of-the-art AutoFE methods on classification and regression tasks. Beyond predictive performance, TOPOFE produces feature sets with lower redundancy and higher representational coverage, while the learned topology graph acquires meaningful task-specific transfer structure correlated with downstream gains. The discovered feature programs transfer reliably across diverse predictors and LLM backbones, demonstrating that improvements arise from TOPOFE's structured search and adaptive coordination rather than backbone-specific generation capability.

Abstract

Large Language Model-Based Feature Engineering

Explore similar work

Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMs

SymboLLM-FE: LLM-Accelerated Symbolic Regression for Automated Feature Engineering on Tabular Data

Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data