Industrial financial systems operate on temporal event sequences such as transactions, user actions, and system logs. While recent research emphasizes representation learning and large language models, production systems continue to rely heavily on handcrafted statistical features due to their interpretability, robustness under limited supervision, and strict latency constraints. This creates a persistent disconnect between learned embeddings and feature-based pipelines. We introduce Embedding-Aware Feature Discovery (EAFD), a unified framework that bridges this gap by coupling pretrained event-sequence embeddings with a self-reflective LLM-driven feature generation agent. EAFD iteratively discovers, evaluates, and refines features directly from raw event sequences using two complementary criteria: alignment, which explains information already encoded in embeddings, and complementarity, which identifies predictive signals missing from them. Across both open-source and industrial transaction benchmarks, EAFD consistently outperforms embedding-only and feature-based baselines, achieving relative gains of up to +5.8% over state-of-the-art pretrained embeddings, resulting in new state-of-the-art performance across event-sequence datasets.
Figures & tables
Figure 1: Blind spots of embeddings. Coefficient of determination ( R2 ) of a feature reconstruction based on embedding representations CoLES, NTP, and LLM4ES on the Rosbank dataset. Values highlight systematic representational blind spots across embeddings.
Figure 2: Overview of Embedding-Aware Feature Discovery (EAFD). (a) Latent embedding pipeline mapping event sequences to continuous user representations. (b) EAFD agent loop: an LLM-based generator proposes interpretable features from raw sequences, evaluated by embedding–feature alignment and downstream utility, with reflective updates guiding iteration. (c) Feature outcomes: aligned features recover information encoded in the embedding, while complementary features capture predictive factors missing from it. (d) Embedding–feature space decomposition into latent, interpretable, and blind-spot regions. (e) Deployment and refinement: discovered features improve downstream tasks and support targeted encoder refinement (e.g., coverage, robustness, privacy).
Method
Age
Gender
Rosbank
DataFusion
Acc
Δ %
AUC
Δ %
AUC
Δ %
AUC
Δ %
CoLES
0.645
–
0.888
–
0.835
–
0.738
–
NTP
0.540
–
0.849
–
0.798
–
0.648
–
LLM4ES
0.651
–
–
–
0.849
–
–
–
Agg Features + CoLES
0.649
+0.62
0.889
+0.11
0.832
-0.36
0.740
+0.27
Featuretools + CoLES
0.644
-0.16
0.889
+0.11
0.863
+3.35
0.760
+2.98
Table 1: Performance comparison of EAFD against baseline embeddings and feature-based agents. Relative improvement ( Δ %) is reported w.r.t. the corresponding backbone embedding. Standard deviations are approximately: Age ±0.004 , Gender ±0.007 , Rosbank ±0.005 , DataFusion ±0.008 .
Figure 3: EAFD iteration dynamics of Rosbank validation ROC-AUC and feature composition across EAFD iterations.
Method
Private Dataset
Age
Gender
Regression
CoLES
0.743
0.898
11373
NTP
0.736
0.895
10839
LLM4ES
0.692
0.789
11462
EAFD (CoLES)
0.756 (+1.75%)
0.901 (+0.33%)
10933 (-3.87%)
EAFD (NTP)
0.739 (+0.41%)
0.897 (+0.23%)
10671 (-1.55%)
Table 2: Performance comparison on the Private Dataset across classification and regression tasks. Percentages denote relative change with respect to the corresponding base embedding; for regression, negative values indicate MAE reduction (improvement).
Figure 4: Task-adaptive distribution of discovered feature types. Feature category proportions generated by EAFD for age, gender, and regression tasks.
Category
Base Encoder
Enhanced Encoder
Amount ( R2 )
0.451
0.484 (+7.39%)
Category ( R2 )
0.816
0.687 (-15.85%)
Time ( R2 )
0.327
0.608 (+85.68%)
Other features ( R2 )
0.897
0.870 (-3.07%)
Downstream (AUC)
0.835
0.845 (+1.20%)
Table 3: Reconstruction Score R2 of feature classes from embedding representations and downstream performance on the Rosbank dataset between Base and Enhanced Encoder.
Figure 5: Selective feature erasure in embeddings. Top panels show R2 reconstruction performance before and after erasing mcc (a) and trx_amount (b) feature categories, while the bottom panels report the corresponding relative performance change ( ΔR2 ).
Llama3.1-8B
Llama3.3-70B
gpt-oss-20B
gpt-oss-120B
0.835
0.856
0.859
0.872
Table 4: EAFD performance on the Rosbank (AUC) for different LLM generator backbones.
Method
NTP
CoLES
Embeddings
0.648
0.738
Featuretools
0.728 ( +12.3% )
0.760 ( +3.0% )
OpenFE
0.730 ( +12.7% )
0.753 ( +2.0% )
LightAutoML
0.731 ( +12.8% )
0.760 ( +3.0% )
LLMFE
0.728 ( +12.3% )
0.760 ( +3.0% )
CAAFE
0.750 ( +15.7% )
0.762 ( +3.3% )
Table 5: Comparison of automated feature engineering methods across different embeddings.
Large-scale digital platforms generate billions of timestamped user-item interactions (events) that are crucial for predicting user attributes in, e.g., fraud prevention and recommendations. While self-supervised learning (SSL) effectively models the temporal order of events, it typically overlooks the global structure of the user-item interaction graph. To bridge this gap, we propose three model-agnostic strategies for integrating this structural information into contrastive SSL: enriching event embeddings, aligning client representations with graph embeddings, and adding a structural pretext task. Experiments on four financial and e-commerce datasets demonstrate that our approach consistently improves the accuracy (up to a 2.3% AUC) and reveals that graph density is a key factor in selecting the optimal integration strategy.
Harry Proshian, Nikita Severin, Sergey Nikolenko +5
Steklov Institute of Mathematics Saint Petersburg, Russia · Independent Researcher Belgrade, Serbia · ISP RAS Steklov Institute of Mathematics Saint Petersburg, Russia +4
Modern financial systems generate vast quantities of transactional and event-level data that encode rich economic signals. This paper presents PRAGMA, a family of foundation models for banking event sequences. Our approach pre-trains a Transformer-based architecture with masked modelling on a large-scale, heterogeneous banking event corpus using a self-supervised objective tailored to the discrete, variable-length nature of financial records. The resulting model supports a wide range of downstream tasks such as credit scoring, fraud detection, and lifetime value prediction: strong performance can be achieved by training a simple linear model on top of the extracted embeddings and can be further improved with lightweight fine-tuning. Through extensive evaluation on downstream tasks, we demonstrate that PRAGMA achieves superior performance across multiple domains directly from raw event sequences, providing a general-purpose representation layer for financial applications.
Maxim Ostroukhov, Ruslan Mikhailov, Vladimir Iashin +11
Predictive modeling is a core component of modern financial services, where a wide range of tasks are traditionally addressed using separate models trained on manually engineered tabular features. This task-specific approach limits reuse and makes it difficult to fully exploit heterogeneous data sources such as transaction histories and digital interaction signals. In this paper, we present an approach based on pretraining a foundation transformer model on multimodal sequences of user events. Events from multiple data sources are unified into a single chronological sequence, enabling early fusion of heterogeneous modalities and learning of general-purpose representations via a next-event prediction objective. These representations are combined with existing engineered user features, on top of which lightweight neural models are trained for multiple downstream tasks. The proposed system outperforms traditional task-specific models while reducing development overhead. The approach was deployed in production at one of the biggest banks in Eastern Europe, resulting in measurable improvements in business metrics.
Nikita Rusakov, Vladislav Meshkov, Konstantin Zorin +4