EventGPT: Next-Event Pretraining on Banking Customer Histories

Denis Burakov

stat.MLNov 2, 2026

EventGPT: Next-Event Pretraining on Banking Customer Histories

Authors: Denis Burakov

Abstract

A model pretrained to predict a customer's next event can estimate the probability of future events without training on their labels and provide a reusable representation for supervised downstream banking tasks. We pretrain EventGPT, a hierarchical causal transformer, by next-token maximum likelihood estimation on the public Multimodal Banking Dataset (MBD-mini: 100,224 clients, 38.8 million transactions and 66.3 million geolocation pings). Keeping one state per event, EventGPT trains in about two hours on one L4 GPU at windows up to 8,192 tokens. From its next-event distribution, with no labels, EventGPT scores five targets of the form "the client makes a transaction of a given kind for the first time next month" at ROC-AUC 0.79 to 0.92. Gradient boosting on 114 engineered features needs 10,000 to 300,000 labeled client-months to match it on four targets and stays below it on the fifth. For product prediction, frozen EventGPT embeddings score below the engineered features (mean ROC-AUC 0.787 against 0.808 over five test folds), and a rank-averaged ensemble of EventGPT, profile fields and token counts with the engineered features adds 0.014 to 0.017, of which 0.007 to 0.016 is EventGPT's own. The code and pretrained weights are released.

Figures & tables

Figure 1: Event-sequence models for banking by year of publication. Color marks the pretraining objective; TabBERT and TabGPT are a masked and a causal model from one paper, and MBD is the dataset used here.

Figure 2: The three stages of EventGPT on three MBD events. Token layers read each event on its own and mean-pool it into one vector; causal event layers with temporal RoPE model the event vectors; token layers decode the next event given the hidden state of the previous one.

Figure 3: The layers inside each stage. The event encoder attends within one event, without positions, and mean-pools the event’s tokens into

e_{i}

. The history decoder attends causally over the event vectors, with temporal RoPE on queries and keys, and gives the hidden state

h_{i}

. The token decoder adds

h_{i}

to every token embedding of event

i+1

and attends causally within that event; the output layer shares its weights with the token embedding.

Figure 4: Attention at the two levels. Within an event every token attends to every other; across events attention is causal and confined to one customer. The tokens of an event reach earlier events only through the hidden state of the previous event.

Figure 5: Ordinal positions against temporal RoPE, Eq. ( 1 ). With ordinal positions every event is one step from the next; with temporal RoPE the distance between two events is set by the ratio of their elapsed times.

Figure 6: What the token decoder reads. Each token of event 3 attends to the earlier tokens of its own event and to

h_{2}

, the history decoder’s hidden state after event 2. It has no path to the tokens inside earlier events, which reach it only through the event vectors and

h_{2}

Figure 7: A training window drawn at a random offset and the aligned window over the same events, on a client from fold 1, shortened to 110 tokens. The random window starts inside an event and holds no [USR] or profile event; the aligned window puts both back in front, and the gap token of its first event counts from the client’s first event.

modality	events	clients	span	per client (p50 / p90 / p99)
transactions	38.8M	98,721	Jan 2021 – Dec 2022	170 / 1,007 / 2,243
geolocation pings	66.3M	72,573	Dec 2021 – Dec 2022	507 / 2,293 / 5,580
dialogues	0.22M	46,320	Feb 2021 – Dec 2022	3 / 10 / 27

Table 1: MBD-mini modalities. Events per client are the 50th, 90th and 99th percentiles.

Figure 8: Inference from one history. The hidden state

h

at the as-of date is the embedding read by a supervised head. From

h

the model gives the distribution of the next event, which the one-step score reads for the target type; the widths are the probabilities for the client of Figure 9 , whose next event was a 50/19. Sampling events one after another continues each branch to a horizon, here the end of the label month; 16 sampled futures give the monthly probability of the target.

method	transactions	transactions + geo
aggregation	0.783
CoLES	0.773	0.775
TabBERT	0.762	0.764
TabGPT	0.802	0.800
supervised RNN	0.819	0.819

Table 2: Mean ROC-AUC over the four product targets on full MBD, as reported by [ 12 ] (late fusion for the two-modality column).

		Markov			Markov		EventGPT $-$
target	pos.	last event	last ten	activity	$\times$ events	EventGPT	Markov $\times$ events
first 48/31	59	0.671	0.835	0.848	0.882	0.906 (0.022)	$+0.024$ (0.025)
first 23/49	195	0.667	0.833	0.811	0.860	0.916 (0.011)	$+0.057$ (0.016)
first 26/28	252	0.577	0.743	0.787	0.833	0.823 (0.013)	$-0.010$ (0.014)
first 48/10	290	0.608	0.718	0.759	0.791	0.777 (0.014)	$-0.013$ (0.013)
first 50/19	770	0.667	0.802	0.773	0.811	0.906 (0.006)	$+0.095$ (0.007)

Table 3: ROC-AUC on the five event targets from event statistics without labels, on the rows of Table 4 with the rule for earlier occurrences. Markov: the transition probability to the target kind from the last event, and its mean over the last ten events, fitted on the pretraining folds. Activity: transactions in the 30 days before the as-of date. Markov

\times

events: the mean transition probability times the number of events in those 30 days. EventGPT is zero-shot on aligned windows, seed 42. The last column is EventGPT minus Markov

\times

events, with the paired standard error clustered by client.

		EventGPT zero-shot			flat decoder	supervised	zero-shot	engineered
target	pos.	seed 42	seed 43	aligned	zero-shot	head	$-$ head ( $p$ )	features
first 48/31	59	0.916 (0.019)	0.902	0.913	0.909	0.891 (0.026)	$+0.025$ (0.19)	0.920
first 23/49	195	0.918 (0.011)	0.918	0.909	0.905	0.930 (0.009)	$-0.011$ (0.16)	0.920
first 26/28	252	0.829 (0.014)	0.827	0.826	0.799	0.875 (0.010)	$-0.046$ ( $<0.001$ )	0.868
first 48/10	290	0.791 (0.014)	0.784	0.780	0.784	0.836 (0.012)	$-0.045$ ( $<0.001$ )	0.835
first 50/19	770	0.906 (0.006)	0.904	0.905	0.915	0.896 (0.006)	$+0.010$ (0.07)	0.937

Table 4: ROC-AUC on the five event targets. Zero-shot scores use no labels and set the score of clients who already had the kind to zero (Section 6.3 ); the flat decoder column is the mean of two seeds, as is the aligned column. The supervised head is the logistic head on the seed-42 embedding (random windows), on the same rows. In brackets the standard error, clustered by client; the seed-43 and aligned columns have standard errors within 0.005 of seed 42. The difference column pairs the seed-42 zero-shot score with the head. Its

p

comes from the standard error of the difference, which is smaller than either score’s own because both scores rank the same clients (Section 6.2 ). The engineered-feature column is the full test fold, so it is comparable in expectation but not row for row. The supervised head and the engineered features are scored without the rule for earlier occurrences.

Figure 9: One test client at the as-of date 1 December 2022, drawn with a fixed seed among the 770 test rows positive for first 50/19. Top row: the client’s last ten events and its label month. Below: three of 64 months sampled from EventGPT (aligned windows, seed 42) from the same history. Right: the model’s distribution over the type of the next event, with the type that came next, and the probability of a first 50/19 in the month, from 64 sampled months and from one step, against the share of test rows that are positive.

target	positives	one-step	scaled	month rollout	rollout $-$ one-step
first 48/31	59	0.912	0.927	0.922	0.010 (0.007)
first 23/49	195	0.908	0.928	0.926	0.018 (0.005)
first 26/28	252	0.826	0.859	0.861	0.036 (0.005)
first 48/10	290	0.781	0.812	0.813	0.032 (0.004)
first 50/19	770	0.907	0.907	0.919	0.012 (0.002)

Table 5: Event targets scored from 16 sampled months per row against the one-step zero-shot score and the scaled score, EventGPT on aligned windows, mean of seeds 42 and 43. ROC-AUC on each target’s positives and 10,000 uniform test rows, with the rule for earlier occurrences; differences with paired standard errors clustered by client.

Figure 10: ROC-AUC on the test fold of CatBoost on the engineered features, fitted on

n

client-months drawn at random from the pretraining folds (mean over draws), against EventGPT’s zero-shot score (mean of two seeds, no labels). Both apply the rule for earlier occurrences. Values are those of Table 10 .

	zero-shot	engineered features, labeled positives $n$
target	(no labels)	5	10	20	50	100	200	400
first 48/31	0.909	0.865 (0.025)	0.833 (0.101)	0.893 (0.018)	0.911 (0.009)	0.914 (0.006)
first 23/49	0.918	0.839 (0.023)	0.857 (0.015)	0.854 (0.018)	0.882 (0.018)	0.897 (0.024)	0.910 (0.008)	0.927 (0.002)
first 26/28	0.828	0.800 (0.040)	0.763 (0.057)	0.813 (0.039)	0.826 (0.023)	0.844 (0.017)	0.860 (0.011)	0.874 (0.003)
first 48/10	0.787	0.678 (0.089)	0.721 (0.030)	0.746 (0.047)	0.777 (0.018)	0.797 (0.026)	0.817 (0.005)	0.836 (0.009)
first 50/19	0.905	0.815 (0.031)	0.847 (0.022)	0.875 (0.021)	0.897 (0.007)	0.921 (0.006)	0.927 (0.002)	0.926 (0.010)

Table 6: Engineered features fitted on

n

positives of the target and 10,000 negatives drawn at random from the pretraining folds, against EventGPT zero-shot (as in Table 10 ). ROC-AUC on the test fold with the rule for earlier occurrences, mean over five draws; in brackets the standard deviation over draws. The pretraining folds hold fewer than 200 positives of 48/31.

	engineered	token counts		flat decoder		EventGPT, last
target	features	full history	2,048 tokens	last	recency	random	aligned
target_1	0.787	0.781	0.757	0.692	0.738	0.740	0.756
target_2	0.843	0.849	0.792	0.725	0.823	0.829	0.850
target_3	0.816	0.809	0.788	0.729	0.783	0.776	0.789
target_4	0.818	0.774	0.770	0.709	0.765	0.753	0.752
mean	0.816	0.803	0.777	0.714	0.777	0.774	0.787

Table 7: ROC-AUC on the test fold. Model columns are the mean of two seeds, with a logistic head on the frozen embedding; EventGPT is trained on windows at random offsets or on aligned windows (Section D ). Token counts and engineered features are fitted with CatBoost. Churn leaves out January 2023 (Section 6 ). The engineered features are fitted on the training rows drawn once January 2023 had left the churn label, the other columns on rows drawn before, which also held the 20,000 January churn positives. Every column scores the event targets without the rule for earlier occurrences. PR-AUC for the official targets is in Table 11 .

	seed 42			seed 43
model	mean	difference [95% int.]	$p$	mean	difference [95% int.]	$p$
EventGPT
hidden state	0.786	$-0.030$ [ $-0.038$ , $-0.022$ ]	$<0.001$	0.788	$-0.028$ [ $-0.037$ , $-0.020$ ]	$<0.001$
+ profile fields, late fusion	0.800	$-0.016$ [ $-0.025$ , $-0.007$ ]	$<0.001$	0.801	$-0.015$ [ $-0.026$ , $-0.005$ ]	0.004
EventGPT with early fusion of the profile fields
hidden state	0.784	$-0.032$ [ $-0.041$ , $-0.022$ ]	$<0.001$	0.793	$-0.023$ [ $-0.032$ , $-0.015$ ]	$<0.001$

Table 8: The gap to the engineered features, aligned windows with query-key normalization, two pretraining seeds. Official mean ROC-AUC with its difference from the engineered features (0.816), a 95% interval and

p

from the paired standard error, clustered by client. The last two rows are rank-averaged ensembles: they average, with equal weights, the ranks given by the embedding with late-fused profile fields and token counts and by CatBoost on token counts over the full history or on the engineered features. The timing and token-count rows each add their features to the hidden state and the late-fused profile fields, not to the row above. Fitted on these training rows, the token counts reach 0.783 against 0.803 in Table 7 , most of it on target_2 (0.780 against 0.849).

test fold	features	hidden state	+ profile, counts	ensemble	engineered
	engineered	EventGPT			ensemble $-$
0	0.816	0.789	0.808	0.830	$+0.014$
1	0.804	0.790	0.802	0.819	$+0.016$
2	0.810	0.782	0.805	0.826	$+0.016$
3	0.813	0.794	0.809	0.830	$+0.017$
4	0.798	0.780	0.796	0.814	$+0.015$

Table 9: Official mean ROC-AUC with each fold as the test fold, EventGPT with early fusion of the profile fields as in Table 8 , mean of two pretraining seeds. Engineered features and token counts are refitted on each split. The last row gives the mean over folds and, in parentheses, the standard deviation. The hidden-state column is the early-fusion model; its fold-0 value (0.789) differs from the aligned column of Table 7 (0.787), which is trained without early fusion.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 11: Delta scoring with a now event. Client A has a new event since yesterday’s run. Client B has none, but its silence grew from three to four days and its now token moved from dt=83 to dt=85 . Client C has been silent for 45 days, and one more day leaves its now token at dt=104 . The job rescores A and B and keeps C’s scores from yesterday.

	zero-shot	engineered features, labeled rows $n$						all
target	(no labels)	1,000	3,000	10,000	30,000	100,000	300,000	labels
first 48/31	0.909	0.573 (0)	0.650 (0)	0.718 (2)	0.871 (6)	0.894 (19)	0.912 (66)	0.921
first 23/49	0.918	0.784 (2)	0.809 (2)	0.866 (7)	0.856 (25)	0.896 (81)	0.902 (263)	0.930
first 26/28	0.828	0.735 (2)	0.739 (4)	0.776 (11)	0.817 (34)	0.858 (128)	0.849 (355)	0.880
first 48/10	0.787	0.739 (2)	0.668 (5)	0.695 (11)	0.763 (34)	0.771 (134)	0.826 (394)	0.841
first 50/19	0.905	0.746 (2)	0.821 (8)	0.854 (34)	0.921 (95)	0.929 (342)	0.932 (992)	0.937

Appendix

Table 10: Engineered features fitted on

n

labeled client-months drawn at random, against EventGPT zero-shot (mean of the two seeds of Table 4 , no labels). ROC-AUC on the test fold, mean over draws (five up to 30,000 rows, three at 100,000, two at 300,000), with the rule for earlier occurrences applied to both; in brackets the mean number of positives in a draw. The last column is the engineered-feature baseline of Table 4 , fitted on every positive of the pretraining folds and 60,000 negatives, under the same rule.

target	prevalence	features	hidden state	+ profile fields, token counts	ensemble
		engineered	EventGPT
target_1	0.0044	0.023	0.014	0.016	0.023
target_2	0.0005	0.014	0.008	0.017	0.021
target_3	0.0038	0.021	0.014	0.019	0.022
target_4	0.0026	0.017	0.011	0.018	0.019
mean	0.0028	0.019	0.012	0.017	0.021

Appendix

Table 11: PR-AUC on the official targets, mean of two seeds, and the share of positives in the top 1% of test rows. A random ranking scores the prevalence and 0.010. Late fusion and the ensemble are those of Table 8 .

	EventGPT		flat decoder
	seed 42	seed 43	seed 42	seed 43
throughput (tokens per second)	117,300	117,600	47,100	55,600
validation loss	1.023	1.020	0.944	0.947
top-1 accuracy, geohash_5	0.582	0.567	0.686	0.680
top-1 accuracy, event name	0.573	0.574	0.579	0.580
official mean ROC-AUC	0.774	0.775	0.786	0.768

Appendix

Table 12: EventGPT and the flat decoder, both on windows at random offsets, per pretraining seed. Throughput on one NVIDIA L4; validation loss and top-1 accuracy of the next token at the end of pretraining; the official mean uses the last hidden state for EventGPT and the recency-weighted embedding for the flat decoder. The flat decoder attends to every earlier token, so it can copy fields of recent events, and most of its lower loss is on the geolocation cell; on the event name the two are level. Its official mean is 0.003 above EventGPT’s, less than the 0.018 between its own two seeds.

element	variant	seeds	mean ROC-AUC
architecture	hierarchy (EventGPT), last state	2	0.774 (0.774, 0.775)
	flat decoder, last token	2	0.714 (0.713, 0.714)
	flat decoder, recency over tokens	2	0.777 (0.768, 0.786)
positions §	temporal RoPE	2	0.781 (0.777, 0.786)
	ordinal	2	0.778 (0.775, 0.780)
modalities	transactions + geolocation	2	0.774 (0.774, 0.775)

Appendix

Table 13: Mean ROC-AUC over the official targets, logistic head. Rows with two seeds give the mean and, in brackets, the two seeds’ scores; the row marked ‡ uses transactions only; the row marked § trains at learning rate

3\times 10^{-4}

. Every row but the windows and width rows uses windows at random offsets; the width rows add QK normalization. The pretraining-period rows compare with transactions only over the full period (0.793), and every other row with the first.

	EventGPT		flat decoder
window	hours	tokens/s	hours	tokens/s
2,048	1.5	118k	3.9	47k
4,096	1.5	125k	6.2	29k
8,192	1.4	126k

Appendix

Table 14: Training time on one NVIDIA L4 GPU for 20,000 steps of 32,768 tokens (655 million tokens), windows at random offsets, one run per cell, without query-key normalization. Throughput is in thousands of tokens per second, from the training log; for runs resumed after a preemption it is measured within each uninterrupted segment. Query-key normalization slows EventGPT to about 88,000 tokens per second (2.1 hours at 2,048 tokens, aligned windows).

	token counts	EventGPT		flat decoder
window	(CatBoost)	random	aligned	recency
2,048	0.777	0.774 (0.934)	0.788 (0.933)	0.786 (0.900)
4,096	0.782	0.772 (0.936)		0.785 (0.896)
8,192	0.801	0.772 (0.935)	0.780 (0.935)

Appendix

Table 15: Mean ROC-AUC over the official targets (churn without January 2023 in parentheses) against the window, seed 42 for the models. The aligned column is trained without query-key normalization (0.786 with it at 2,048 tokens). Token counts at full history reach 0.803.

Figure 12: Share of rows positive for each label by month, on a log scale, over all five folds (the test fold in full, the training rows weighted back to their share). Product targets are solid, event targets dashed and churn blue. January 2023 follows the last transaction, so the event targets and churn end in December.

Next-Event Prediction

See all 22 papers

Apr 9, 2026

PRAGMA: Revolut Foundation Model

Modern financial systems generate vast quantities of transactional and event-level data that encode rich economic signals. This paper presents PRAGMA, a family of foundation models for banking event sequences. Our approach pre-trains a Transformer-based architecture with masked modelling on a large-scale, heterogeneous banking event corpus using a self-supervised objective tailored to the discrete, variable-length nature of financial records. The resulting model supports a wide range of downstream tasks such as credit scoring, fraud detection, and lifetime value prediction: strong performance can be achieved by training a simple linear model on top of the extracted embeddings and can be further improved with lightweight fine-tuning. Through extensive evaluation on downstream tasks, we demonstrate that PRAGMA achieves superior performance across multiple domains directly from raw event sequences, providing a general-purpose representation layer for financial applications.

Maxim Ostroukhov, Ruslan Mikhailov, Vladimir Iashin +11

Revolut Research · NVIDIA

Financial Services Sequence Modeling Generative Modeling

Ask PDF

Jul 10, 2026

A Foundation Model for Multimodal Event Sequences in Financial Applications

Predictive modeling is a core component of modern financial services, where a wide range of tasks are traditionally addressed using separate models trained on manually engineered tabular features. This task-specific approach limits reuse and makes it difficult to fully exploit heterogeneous data sources such as transaction histories and digital interaction signals. In this paper, we present an approach based on pretraining a foundation transformer model on multimodal sequences of user events. Events from multiple data sources are unified into a single chronological sequence, enabling early fusion of heterogeneous modalities and learning of general-purpose representations via a next-event prediction objective. These representations are combined with existing engineered user features, on top of which lightweight neural models are trained for multiple downstream tasks. The proposed system outperforms traditional task-specific models while reducing development overhead. The approach was deployed in production at one of the biggest banks in Eastern Europe, resulting in measurable improvements in business metrics.

Nikita Rusakov, Vladislav Meshkov, Konstantin Zorin +4

Sber Moscow, Russian Federation · Sber AI Lab Moscow, Russian Federation

Financial Services Multimodal Pretraining Multimodal Foundation Models Multimodal Fusion Next-Event Prediction Sequence Modeling+1

Ask PDF

Jun 19, 2026

SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training

Many machine learning applications rely on heterogeneous event streams to make predictions, either causally as events arrive or bidirectionally over complete sequences. We propose SOHET (Sequence Of Heterogeneous Events Transformer), a hierarchical architecture combining event-type-specific tabular encoders with temporal and type embeddings, processed by a causal or bidirectional transformer. We introduce three self-supervised pre-training objectives for the causal setting. On a proprietary large-scale real-world Booking.com fraud detection task with 17 event types, SOHET outperforms FlexTPP, NAPPT, and CIPPT by 5.8%. Pre-training yields an additional 2.6% gain and 2.4% faster convergence. On the EBES benchmark, bidirectional SOHET matches or exceeds the published best on 6 out of 8 tasks.

Kees Jan de Vries, Mustafa Radha, Mathijs de Jong

Booking.com, Amsterdam, The Netherlands

Transformer Self-Supervised Pre-Training Sequence Modeling

Ask PDF

Abstract

Next-Event Prediction

Explore similar work

PRAGMA: Revolut Foundation Model

A Foundation Model for Multimodal Event Sequences in Financial Applications

SOHET: Sequence Of Heterogeneous Events Transformer with Self-Supervised Pre-Training