Embedding-Aware Feature Discovery: Bridging Latent Representations and Interpretable Features in Event Sequences

Artem Sakhno; Ivan Sergeev; Alexey Shestov; Omar Zoloev; Elizaveta Kovtun; Gleb Gusev; Andrey Savchenko; Maksim Makarenko

cs.LGMar 16, 2026

Embedding-Aware Feature Discovery: Bridging Latent Representations and Interpretable Features in Event Sequences

Authors: Artem Sakhno, Ivan Sergeev, Alexey Shestov, Omar Zoloev, Elizaveta Kovtun, Gleb Gusev, Andrey Savchenko, Maksim Makarenko

Organizations: Sber AI Lab · ISP RAS Research Center for Trusted AI

Abstract

Industrial financial systems operate on temporal event sequences such as transactions, user actions, and system logs. While recent research emphasizes representation learning and large language models, production systems continue to rely heavily on handcrafted statistical features due to their interpretability, robustness under limited supervision, and strict latency constraints. This creates a persistent disconnect between learned embeddings and feature-based pipelines. We introduce Embedding-Aware Feature Discovery (EAFD), a unified framework that bridges this gap by coupling pretrained event-sequence embeddings with a self-reflective LLM-driven feature generation agent. EAFD iteratively discovers, evaluates, and refines features directly from raw event sequences using two complementary criteria: alignment, which explains information already encoded in embeddings, and complementarity, which identifies predictive signals missing from them. Across both open-source and industrial transaction benchmarks, EAFD consistently outperforms embedding-only and feature-based baselines, achieving relative gains of up to $+5.8\%$ over state-of-the-art pretrained embeddings, resulting in new state-of-the-art performance across event-sequence datasets.

Figures & tables

Figure 1: Blind spots of embeddings. Coefficient of determination ( $R^{2}$ ) of a feature reconstruction based on embedding representations CoLES, NTP, and LLM4ES on the Rosbank dataset. Values highlight systematic representational blind spots across embeddings.

Figure 2: Overview of Embedding-Aware Feature Discovery (EAFD). (a) Latent embedding pipeline mapping event sequences to continuous user representations. (b) EAFD agent loop: an LLM-based generator proposes interpretable features from raw sequences, evaluated by embedding–feature alignment and downstream utility, with reflective updates guiding iteration. (c) Feature outcomes: aligned features recover information encoded in the embedding, while complementary features capture predictive factors missing from it. (d) Embedding–feature space decomposition into latent, interpretable, and blind-spot regions. (e) Deployment and refinement: discovered features improve downstream tasks and support targeted encoder refinement (e.g., coverage, robustness, privacy).

Method	Age		Gender		Rosbank		DataFusion
	Acc	$\Delta$ %	AUC	$\Delta$ %	AUC	$\Delta$ %	AUC	$\Delta$ %
CoLES	0.645	–	0.888	–	0.835	–	0.738	–
NTP	0.540	–	0.849	–	0.798	–	0.648	–
LLM4ES	0.651	–	–	–	0.849	–	–	–
Agg Features + CoLES	0.649	+0.62	0.889	+0.11	0.832	-0.36	0.740	+0.27
Featuretools + CoLES	0.644	-0.16	0.889	+0.11	0.863	+3.35	0.760	+2.98

Table 1: Performance comparison of EAFD against baseline embeddings and feature-based agents. Relative improvement (

\Delta

%) is reported w.r.t. the corresponding backbone embedding. Standard deviations are approximately: Age

\pm 0.004

, Gender

\pm 0.007

, Rosbank

\pm 0.005

, DataFusion

\pm 0.008

Figure 3: EAFD iteration dynamics of Rosbank validation ROC-AUC and feature composition across EAFD iterations.

Method	Private Dataset
	Age	Gender	Regression
CoLES	0.743	0.898	11373
NTP	0.736	0.895	10839
LLM4ES	0.692	0.789	11462
EAFD (CoLES)	0.756 (+1.75%)	0.901 (+0.33%)	10933 (-3.87%)
EAFD (NTP)	0.739 (+0.41%)	0.897 (+0.23%)	10671 (-1.55%)

Table 2: Performance comparison on the Private Dataset across classification and regression tasks. Percentages denote relative change with respect to the corresponding base embedding; for regression, negative values indicate MAE reduction (improvement).

Figure 4: Task-adaptive distribution of discovered feature types. Feature category proportions generated by EAFD for age, gender, and regression tasks.

Category	Base Encoder	Enhanced Encoder
Amount ( $R^{2}$ )	0.451	0.484 (+7.39%)
Category ( $R^{2}$ )	0.816	0.687 (-15.85%)
Time ( $R^{2}$ )	0.327	0.608 (+85.68%)
Other features ( $R^{2}$ )	0.897	0.870 (-3.07%)
Downstream (AUC)	0.835	0.845 (+1.20%)

Table 3: Reconstruction Score

R^{2}

of feature classes from embedding representations and downstream performance on the Rosbank dataset between Base and Enhanced Encoder.

Figure 5: Selective feature erasure in embeddings. Top panels show $R^{2}$ reconstruction performance before and after erasing mcc (a) and trx_amount (b) feature categories, while the bottom panels report the corresponding relative performance change ( $\Delta R^{2}$ ).

Llama3.1-8B	Llama3.3-70B	gpt-oss-20B	gpt-oss-120B
0.835	0.856	0.859	0.872

Table 4: EAFD performance on the Rosbank (AUC) for different LLM generator backbones.

Method	NTP	CoLES
Embeddings	0.648	0.738
Featuretools	0.728 ( +12.3% )	0.760 ( +3.0% )
OpenFE	0.730 ( +12.7% )	0.753 ( +2.0% )
LightAutoML	0.731 ( +12.8% )	0.760 ( +3.0% )
LLMFE	0.728 ( +12.3% )	0.760 ( +3.0% )
CAAFE	0.750 ( +15.7% )	0.762 ( +3.3% )

Table 5: Comparison of automated feature engineering methods across different embeddings.

Large Language Model-Based Feature Engineering

See all 18 papers

Beyond Isolated Clients: Integrating Graph-Based Embeddings into Event Sequence Models
Apr 10, 2026Harry Proshian, Nikita Severin, Sergey Nikolenko +5Self-Supervised Learning Contrastive Learning
Steklov Institute of Mathematics Saint Petersburg, Russia · Independent Researcher Belgrade, Serbia · ISP RAS Steklov Institute of Mathematics Saint Petersburg, Russia +4
PRAGMA: Revolut Foundation Model
Apr 9, 2026Maxim Ostroukhov, Ruslan Mikhailov, Vladimir Iashin +11Financial Services Sequence Modeling
Revolut Research · NVIDIA
A Foundation Model for Multimodal Event Sequences in Financial Applications
Jul 10, 2026Nikita Rusakov, Vladislav Meshkov, Konstantin Zorin +4Financial Services Multimodal Pretraining
Sber Moscow, Russian Federation · Sber AI Lab Moscow, Russian Federation

Apr 10, 2026J/K move · Enter open · S save

Beyond Isolated Clients: Integrating Graph-Based Embeddings into Event Sequence Models

Harry Proshian, Nikita Severin, Sergey Nikolenko, Kireev Ivan+4

Steklov Institute of Mathematics Saint Petersburg, Russia · Independent Researcher Belgrade, Serbia · ISP RAS Steklov Institute of Mathematics Saint Petersburg, Russia+4

Self-Supervised Learning Contrastive Learning Sequence Modeling Graph Representation Learning

Open paper AskPDF arXiv

Large-scale digital platforms generate billions of timestamped user-item interactions (events) that are crucial for predicting user attributes in, e.g., fraud prevention and recommendations. While self-supervised learning (SSL) effectively models the temporal order of events, it typically overlooks the global structure of the user-item interaction graph. To bridge this gap, we propose three model-agnostic strategies for integrating this structural information into contrastive SSL: enriching event embeddings, aligning client representations with graph embeddings, and adding a structural pretext task. Experiments on four financial and e-commerce datasets demonstrate that our approach consistently improves the accuracy (up to a 2.3% AUC) and reveals that graph density is a key factor in selecting the optimal integration strategy.

Abstract

Large Language Model-Based Feature Engineering

Explore similar work

Beyond Isolated Clients: Integrating Graph-Based Embeddings into Event Sequence Models

PRAGMA: Revolut Foundation Model

A Foundation Model for Multimodal Event Sequences in Financial Applications