Transaction event sequences support diverse prediction tasks in online commerce and digital financial services. The same transaction records can yield prediction targets with different semantics, including values, changes, and relations to past events. These targets impose different constraints on history representations, but their predictability alone does not establish the downstream utility of a target set. Moreover, target-wise prediction losses do not explicitly model conditional co-occurrence among targets. To address these challenges, we propose TAPES, a task-aligned joint predictive pretraining framework for transaction sequences. TAPES constructs a finite collection of candidate target sets from admissible field-operator combinations. It selects a target set according to the downstream utility of representations learned through short joint pretraining trials under matched budgets. It then reinitializes the encoder and pretrains it on the selected targets through a joint likelihood parameterized by a shared discrete latent variable, before downstream supervised adaptation. Experiments in a real-world transaction-risk setting and on multiple public financial benchmarks spanning diverse downstream tasks demonstrate that TAPES outperforms representative baselines. Ablation studies further validate the contributions of predictive target selection and joint modeling.
Figures & tables
Figure 1. Motivation for TAPES. (a) Different transformations of the same transaction field yield prediction targets with distinct semantics, illustrated by amount level, adjacent change, and historical deviation. (b) Two historical contexts share the same target-wise conditional marginals but exhibit different joint outcome patterns, motivating joint prediction. Event values, target combinations, and distributions are illustrative; bin denotes target-specific discretization.
Figure 2. Overview of Task-Aligned Joint Predictive Pretraining for Transaction Event Sequences (TAPES). (1) Stage 1: Task-guided target-set selection. Field–operator combinations capture predictive semantics beyond raw field values, and short pretraining trials identify a target set useful for the downstream task. (2) Stage 2: Shared-latent joint pretraining. The selected targets are jointly modeled through a shared discrete latent variable to learn representations of transaction histories. Candidate sets and probabilities are illustrative.
Method
AlfaBattle
Churn
Taobao
MBD-mini
Retail X5
Avg.Rank ↓
AUROC ↑
AP ↑
AUROC ↑
AP ↑
AUROC ↑
AP ↑
mAUROC ↑
mAP ↑
Macro F1 ↑
Acc. ↑
AUROC ↑
Tree ensembles on aggregated features
CatBoost
76.64
12.80
81.60
84.39
70.66
75.61
77.60
1.81
49.00
51.44
77.17
5.40
XGBoost
76.95
12.87
81.82
84.05
70.65
75.30
79.33
1.75
49.86
51.85
77.53
4.00
Supervised sequence models
GRU
74.33
10.27
81.48
84.52
68.08
72.85
78.05
1.31
53.79
54.80
80.17
6.00
Table 1. Development results for the evaluated configurations on five public benchmarks (%). PANTHER uses the SPRM adaptation specified in Appendix A.2 . Bold and underline indicate the best and second-best scores in each column. Avg. Rank is the mean rank over the five primary metrics.
Backbone
d
L
AlfaBattle
Retail X5
MBD
Churn
Taobao
GRU
512
2
74.59
54.34
78.05
81.48
68.08
+ TAPES(Ours)
76.05 (+1.96%)
53.85 (-0.90%)
75.93 (-2.72%)
81.56 (+0.10%)
71.03 (+4.33%)
LSTM
512
2
74.78
52.97
77.15
80.90
67.15
+ TAPES(Ours)
75.51 (+0.98%)
53.37 (+0.76%)
77.41 (+0.34%)
81.78 (+1.09%)
70.36 (+4.78%)
Transformer
256
4
75.57
53.60
79.27
80.90
69.85
+ TAPES(Ours)
78.09 (+3.33%)
54.30 (+1.31%)
79.95 (+0.86%)
82.99 (+2.58%)
70.55 (+1.00%)
Table 2. Development performance across encoder configurations (%). Parentheses show relative percentage changes from the displayed supervised score. Retail uses Macro-F1, MBD-mini uses macro-AUROC, and the remaining datasets use AUROC. d and L denote hidden dimension and depth. Bold marks the higher score within each two-row group.
Figure 3. Development performance at different labeled-data fractions for downstream fine-tuning on AlfaBattle, Retail X5, MBD-mini, and Rosbank Churn. Curves show means over fine-tuning seeds; shaded regions indicate ± one sample standard deviation.
Method
AUROC (%)
Passed-CB AP (%)
Relative AddCB change (%)
Scratch
90.88
5.80
0.00
CoLES
89.66
4.35
-0.93
TabBERT
90.83
4.95
-0.93
MLEM
90.13
5.47
+0.28
TAPES (Ours)
92.04
4.48
+0.30
Table 3. Validation results on TransRisk after 25k fine-tuning updates. Bold indicates the best value.
Figure 4. Industrial fine-tuning evaluations on TransRisk. (a) AUROC of TAPES and the supervised control without history pretraining after 25k fine-tuning updates. (b) Comparison with baseline using 24k pretraining updates and 25k fine-tuning updates. (c) Relative AddCB recall change at a 0.5% intervention budget, with the business baseline selected at 10k updates as reference. Stars identify checkpoints selected by the validation business criterion.
Figure 5. Frozen-representation evaluation on TransRisk. Bars show Recall@top1% for TAPES and the business baseline (left axis). The dashed curve shows TAPES’s relative recall gain over an architecture-matched random encoder (right axis).
Ablation Variant
S
J
AlfaBattle
Retail X5
AUROC ↑
AP ↑
Macro-F1 ↑
Acc. ↑
AUROC ↑
w/o Pretraining
–
–
75.25
10.80
53.18
54.17
79.66
w/o Both
×
×
76.98 (+1.73)
12.01 (+1.21)
53.32 (+0.14)
54.15 (-0.02)
79.78 (+0.12)
w/o Selection
×
✓
77.09 (+1.84)
12.18 (+1.38)
53.40 (+0.22)
54.25 (+0.08)
79.82 (+0.16)
w/o Joint Prediction
✓
×
77.12 (+1.87)
12.35 (+1.55)
53.79 (+0.61)
54.47 (+0.30)
79.90 (+0.24)
TAPES (Ours)
✓
✓
77.38 (+2.13)
12.52 (+1.72)
54.08 (+0.90)
54.60 (+0.43)
79.90 (+0.24)
Table 4. Component ablation under full fine-tuning. S : target-set selection; J : joint likelihood. All metrics within each configuration–dataset pair come from the same checkpoint. Parenthesized values indicate changes in percentage points relative to w/o Pretraining. Green and red indicate increases and decreases, respectively. AP denotes average precision; Retail AUROC is macro one-vs-rest AUROC. Bold marks the best scores at the reported precision.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Item
Value
Training examples
24,698,621
Validation examples
4,825,198
Total examples
29,523,819
Categorical attributes per historical event
11
Maximum historical events
300
Current-transaction feature dimension
652
Appendix
Table 5. TransRisk data used in the experiments.
Dataset
Task
# Train sequences
# Train events
Primary metric
Alpha
Credit default
674,667
106,567,621
AUROC
Retail X5
Age classification
222,822
25,324,802
Macro-F1
MBD-mini
Product purchase, four labels
499,930
10,917,772
Macro-AUROC
Rosbank Churn
Churn prediction
3,475
333,261
AUROC
Taobao
Future purchase
12,837
3,220,892
AUROC
Appendix
Table 6. Dataset summary. Public sequence and event counts refer to the full downstream training sets after eligibility filtering and history truncation. Event counts exclude padding; validation and test partitions are not included in these counts.
Setting
Configuration
Optimizer
AdamW
Weight decay
0.01
Gradient-norm clipping
1.0
Effective batch size
At most 128 ; final batch retained
Task-head learning rate
10−3
Maximum/minimum epochs
10/3
Appendix
Table 7. Reference configuration for standard sequence-classification adaptation.
Figure 6. Effect of the Stage 2 pretraining budget on Rosbank Churn with the CT256 × 4 backbone. (a) Training negative log-likelihood per target over 60 epochs. (b) Stage 2 optimizer updates (bars, left axis) and development AUROC after fine-tuning (line, right axis) at 5, 10, 20, 40, and 60 epochs. The Stage 1-selected target set, fitted transformations, and fine-tuning protocol are fixed across budgets. Each score uses the fine-tuning checkpoint selected by early-stopping AUROC.
Predictive modeling is a core component of modern financial services, where a wide range of tasks are traditionally addressed using separate models trained on manually engineered tabular features. This task-specific approach limits reuse and makes it difficult to fully exploit heterogeneous data sources such as transaction histories and digital interaction signals. In this paper, we present an approach based on pretraining a foundation transformer model on multimodal sequences of user events. Events from multiple data sources are unified into a single chronological sequence, enabling early fusion of heterogeneous modalities and learning of general-purpose representations via a next-event prediction objective. These representations are combined with existing engineered user features, on top of which lightweight neural models are trained for multiple downstream tasks. The proposed system outperforms traditional task-specific models while reducing development overhead. The approach was deployed in production at one of the biggest banks in Eastern Europe, resulting in measurable improvements in business metrics.
Nikita Rusakov, Vladislav Meshkov, Konstantin Zorin +4
Modern financial systems generate vast quantities of transactional and event-level data that encode rich economic signals. This paper presents PRAGMA, a family of foundation models for banking event sequences. Our approach pre-trains a Transformer-based architecture with masked modelling on a large-scale, heterogeneous banking event corpus using a self-supervised objective tailored to the discrete, variable-length nature of financial records. The resulting model supports a wide range of downstream tasks such as credit scoring, fraud detection, and lifetime value prediction: strong performance can be achieved by training a simple linear model on top of the extracted embeddings and can be further improved with lightweight fine-tuning. Through extensive evaluation on downstream tasks, we demonstrate that PRAGMA achieves superior performance across multiple domains directly from raw event sequences, providing a general-purpose representation layer for financial applications.
Maxim Ostroukhov, Ruslan Mikhailov, Vladimir Iashin +11
A model pretrained to predict a customer's next event can estimate the probability of future events without training on their labels and provide a reusable representation for supervised downstream banking tasks. We pretrain EventGPT, a hierarchical causal transformer, by next-token maximum likelihood estimation on the public Multimodal Banking Dataset (MBD-mini: 100,224 clients, 38.8 million transactions and 66.3 million geolocation pings). Keeping one state per event, EventGPT trains in about two hours on one L4 GPU at windows up to 8,192 tokens. From its next-event distribution, with no labels, EventGPT scores five targets of the form "the client makes a transaction of a given kind for the first time next month" at ROC-AUC 0.79 to 0.92. Gradient boosting on 114 engineered features needs 10,000 to 300,000 labeled client-months to match it on four targets and stays below it on the fifth. For product prediction, frozen EventGPT embeddings score below the engineered features (mean ROC-AUC 0.787 against 0.808 over five test folds), and a rank-averaged ensemble of EventGPT, profile fields and token counts with the engineered features adds 0.014 to 0.017, of which 0.007 to 0.016 is EventGPT's own. The code and pretrained weights are released.