Long Text to Predictive Features: LLM-Guided Blockwise Feature Engineering via Executable Program Search

Ziming Dai; Dabiao Ma; Ziheng Guo; Jack Dong; Zimu Zhou

cs.LGOct 8, 2026

Long Text to Predictive Features: LLM-Guided Blockwise Feature Engineering via Executable Program Search

Authors: Ziming Dai, Dabiao Ma, Ziheng Guo, Jack Dong, Zimu Zhou

Organizations: City University of Hong Kong Hong Kong, China

Abstract

Industrial risk-control systems typically rely on structured-data models for efficient prediction, yet substantial valuable information remains embedded in unstructured long text. Extracting this information through manual feature engineering is labor-intensive, while requiring a large language model (LLM) to process every real-time input may not meet practical deployment requirements. To address this challenge, we propose LLM-BlockFE, an LLM-guided offline feature construction framework that converts long text into executable feature programs, thereby avoiding LLM calls during online inference. LLM-BlockFE constructs feature programs by incrementally appending immutable code blocks and evaluates candidate features using a downstream model. To address the tendency of conventional greedy search to become trapped in suboptimal solutions, our method introduces a block-level rollback mechanism based on depth-calibrated credit allocation and advances multiple independent search trajectories in an interleaved manner, reducing redundant exploration by sharing fixed descriptions of each trajectory's exploration direction. After the search, the resulting programs are frozen and deployed to extract structured features for downstream prediction models. Across two public and two private datasets, LLM-BlockFE achieves absolute AUC improvements of 0.0069 to 0.0358 over the strongest baseline on each dataset in the full-dataset comparison. Post-launch monitoring across five deployed financial risk-control applications shows absolute KS improvements of 0.02 to 1.56 percentage points over the existing manually designed strategy.

Figures & tables

Figure 1. Overview of LLM-BlockFE, which searches for text feature programs offline and executes the selected programs for online prediction.

Figure 2. Illustration of the Seed program and LLM-generated candidate program.

Dataset	Base-XGB	TextStats +XGB	TF-IDF +LR	TF-IDF +XGB	TF-IDF+SVD +XGB	Embedding +XGB	OpenFE	CAAFE	OCTree	LLM-FE	Ours
Remark	0.6265	0.6146	0.6058	0.6323	0.6261	0.5685	0.6439	0.6314	0.6435	0.6446	0.6804
Report	0.5323	0.5245	0.5232	0.5314	0.5227	0.5125	0.5250	0.5336	0.5159	0.5324	0.5545
CFPB-Timely	0.8571	0.8418	0.8699	0.8663	0.8707	0.8717	0.8541	0.8684	0.8701	0.8729	0.8881
GossipCop	0.9145	0.9224	0.8603	0.9380	0.9403	0.9330	0.9206	0.9257	0.9206	0.9242	0.9472

Table 1. AUC comparison across datasets. Best results are highlighted in bold, and second-best results are underlined.

Figure 3. Inference latency and predictive performance across methods.

Figure 4. Online performance of LLM-BlockFE across five production risk-control applications. The figure compares monthly KS values for LLM-BlockFE and the existing manually designed strategy on the same online samples from September to November 2025, covering 2,694,342 real-world users.

Setting	AUC	AUC Drop
Base-XGB	0.6190	0.0570
w/o Append-only Constraint	0.6357	0.0403
w/o Bayesian Optimization	0.6383	0.0377
w/o Downstream AUC Gate	0.6268	0.0492
w/o Backtracking	0.6470	0.0290
w/o Credit Guidance	0.6573	0.0187

Table 2. Ablation study on the Remark dataset. AUC Drop denotes the absolute performance decrease relative to the full method.

Figure 5. Generalization of LLM-BlockFE across downstream models and generator LLMs on the Remark dataset.

Dataset	Mean $\|\mathrm{SHAP}\|$	Contribution Ratio
Remark	0.3900	41.4%
Report	0.0287	61.1%
CFPB-Timely	0.0235	9.5%
GossipCop	0.6477	19.4%

Table 3. SHAP-based contribution of the features generated by LLM-BlockFE across datasets.

Figure 6. Case study of a deployed feature program on a real customer-service dialogue.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Structured Features	Unstructured Long Text	Samples	Source
CFPB-Timely	7	1	200,000	Kaggle
GossipCop	7	1	6,344	Kaggle
Remark	32	1	100,000	Qfin
Report	5	1	100,000	Qfin

Appendix

Table 4. Dataset statistics.

Symbol	Description	Value
$W$	Window length	12
$H$	Prediction horizon (attempts)	3
$\kappa$	Prior strength	4
$\rho$	Rollback threshold	0.5
$p_{0}$	Default success rate	0.1

Appendix

Table 5. Hyperparameter settings.

Dataset	Base-XGB	TextStats +XGB	TF-IDF +LR	TF-IDF +XGB	TF-IDF+SVD +XGB	Embedding +XGB	OpenFE	CAAFE	OCTree	LLM-FE	Ours
Remark	$\pm 0.0013$	$\pm 0.0026$	$\pm 0.0000$	$\pm 0.0013$	$\pm 0.0038$	$\pm 0.0031$	$\pm 0.0028$	$\pm 0.0071$	$\pm 0.0020$	$\pm 0.0065$	$\pm 0.0046$
Report	$\pm 0.0065$	$\pm 0.0046$	$\pm 0.0000$	$\pm 0.0043$	$\pm 0.0037$	$\pm 0.0069$	$\pm 0.0035$	$\pm 0.0144$	$\pm 0.0101$	$\pm 0.0024$	$\pm 0.0088$
CFPB-Timely	$\pm 0.0049$	$\pm 0.0023$	$\pm 0.0000$	$\pm 0.0012$	$\pm 0.0007$	$\pm 0.0016$	$\pm 0.0050$	$\pm 0.0063$	$\pm 0.0034$	$\pm 0.0051$	$\pm 0.0062$
GossipCop	$\pm 0.0008$	$\pm 0.0013$	$\pm 0.0000$	$\pm 0.0017$	$\pm 0.0009$	$\pm 0.0018$	$\pm 0.0014$	$\pm 0.0046$	$\pm 0.0019$	$\pm 0.0046$	$\pm 0.0063$

Appendix

Table 6. The standard deviation of AUC values obtained from five runs on different datasets.

Explore similar work

CoFEE: Reasoning Control for LLM-Based Feature Discovery
Apr 23, 2026Maximilian Westermann, Ben Griffin, Aaron Ontoyin Yin +6LLM Prompting Automated Feature Engineering
University of Oxford · Vela Research · Amazon
Eureka: Intelligent Feature Engineering for Enterprise AI Cloud Resource Demand Prediction
May 24, 2026Hangxuan Li, Renjun Jia, Xuezhang Wu +3LLM-Based Program Synthesis Cloud Computing
Alibaba Cloud Computing Co. Ltd, Hangzhou, China · School of Computer Science, Fudan University, Shanghai, China · School of Computer Science and Technology, Tongji University, Shanghai, China +1
Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMs
Jul 25, 2026Sha Li, Naren RamakrishnanEvolutionary Optimization Large Language Model-Guided Optimization
Department of Computer Science Virginia Tech Blacksburg, VA 24060 · Department of Computer Science Virginia Tech Alexandria, VA 22305

Apr 23, 2026, cs.AIJ/K move · Enter open · S save

CoFEE: Reasoning Control for LLM-Based Feature Discovery

Maximilian Westermann, Ben Griffin, Aaron Ontoyin Yin, Zakari Salifu+5

University of Oxford · Vela Research · Amazon

LLM Prompting Automated Feature Engineering LLM Reasoning

Open paper AskPDF arXiv

Feature discovery from complex unstructured data is fundamentally a reasoning problem: it requires identifying abstractions that are predictive of a target outcome while avoiding leakage, proxies, and post-outcome signals. With the introduction of ever-improving Large Language Models (LLMs), our method provides a structured method for addressing this challenge. LLMs are well suited for this task by being able to process large amounts of information, but unconstrained feature generation can lead to weak features. In this work, we study reasoning control in LLMs by inducing cognitive behaviors for improving feature discovery. We introduce CoFEE (Cognitive Feature Engineering Engine), a reasoning control framework that enforces cognitive behaviors in how the LLM reasons during feature discovery. From a machine learning perspective, these cognitive behaviors act as structured inductive biases over the space of candidate features generated by the model. These behaviors have been exploited with success in ML models, and include backward chaining from outcomes, subgoal decomposition, verification against observability and leakage criteria, and explicit backtracking of rejected reasoning paths. In a controlled comparison, we show that enforcing cognitive behaviors yields features with higher empirical predictability than those under unconstrained vanilla LLM prompts. CoFEE achieves an average Success Rate Score that is 15.2% higher than the vanilla approach, while generating 29% fewer features and reducing costs by 53.3%. Using held-out feature evaluation, we assess whether cognitively induced features generalize beyond the data used for discovery. Our results indicate that, in our evaluated setting, reasoning control is associated with improvements in quality and efficiency of LLM-based feature discovery.

Abstract

Explore similar work

CoFEE: Reasoning Control for LLM-Based Feature Discovery

Eureka: Intelligent Feature Engineering for Enterprise AI Cloud Resource Demand Prediction

Adaptive Graph-of-Islands Evolution for Automatic Feature Engineering with LLMs