PRAGMA: Revolut Foundation Model

Maxim Ostroukhov; Ruslan Mikhailov; Vladimir Iashin; Artem Sokolov; Andrei Akshonov; Vitaly Protasov; Andrey Goncharov; Dmitrii Beloborodov

cs.LGApr 9, 2026

PRAGMA: Revolut Foundation Model

Authors: Maxim Ostroukhov, Ruslan Mikhailov, Vladimir Iashin, Artem Sokolov, Andrei Akshonov, Vitaly Protasov, Andrey Goncharov, Dmitrii Beloborodov,

Organizations: Revolut Research · NVIDIA

Abstract

Modern financial systems generate vast quantities of transactional and event-level data that encode rich economic signals. This paper presents PRAGMA, a family of foundation models for banking event sequences. Our approach pre-trains a Transformer-based architecture with masked modelling on a large-scale, heterogeneous banking event corpus using a self-supervised objective tailored to the discrete, variable-length nature of financial records. The resulting model supports a wide range of downstream tasks such as credit scoring, fraud detection, and lifetime value prediction: strong performance can be achieved by training a simple linear model on top of the extracted embeddings and can be further improved with lightweight fine-tuning. Through extensive evaluation on downstream tasks, we demonstrate that PRAGMA achieves superior performance across multiple domains directly from raw event sequences, providing a general-purpose representation layer for financial applications.

Figures & tables

Figure 1: A single architecture from 10M to 1B parameters that outperforms task-specific models across tasks.

Figure 2: Event timeline overview . After account creation, users generate a sequence of platform interactions over time, spanning transactions, in-app navigation, and communications. We aggregate the event history up until a designated evaluation point. Alongside these sequential events, we capture contextual attributes that describe the record’s state at that point, e.g., membership plan or service region. Both events and attributes share a uniform representation: a timestamp and a set of key–value pairs (e.g., Type: card_payment , Channel: email ). All values shown are synthetic; the figure is for illustration purposes only.

Figure 3: Tokenisation overview . A raw event record is decomposed into a temporal coordinate, semantic types (keys), and values. Keys are always represented by one token, while values use type-specific tokenisation: numerical values are bucketised by percentile, categorical values map to a single token, and textual values are split into subword tokens. Some keys therefore expand to multiple value tokens, e.g., Description

\rightarrow

met , al , plan . Time is encoded both as log-seconds to the last event and as calendar and time features derived from the timestamp. Profile state is encoded similarly to an event record.

Figure 4: PRAGMA backbone overview . Each user record is represented as an ordered event history and profile state, where every field is decomposed into a semantic type (key), one or more values, and a temporal coordinate. Keys and values are embedded from a shared lookup table, and value tokens receive within-field positional embeddings. A Profile State Encoder maps profile state

x_{a}

, with time since life-long events

t_{a}

encoded via RoPE, into a [USR] embedding

z_{a}

, while an Event Encoder independently maps the tokens of each event

x_{e}

into a [EVT] embedding

z_{e}^{\prime}

and adds calendar features

z_{t}

. A History Encoder then contextualises the sequence

z=[z_{a}:z_{e}]

with time to the last event

t_{e}

encoded via RoPE, producing a representation for a user record

z_{h}

Model	Params	$d_{\mathrm{model}}$	$d_{\mathrm{ffn}}$	Profile	Event	History	Heads
		Width		Depth
PRAGMA-S	10 M	192	768	1	5	2	3
PRAGMA-M	100 M	512	2048	3	16	6	8
PRAGMA-L	1 B	1024	4096	9	45	18	16

Table 1: PRAGMA model family . PRAGMA scales across three variants (10 M, 100 M, 1 B parameters) by jointly increasing model width (

d_{\mathrm{model}}

d_{\mathrm{ffn}}

), depth of the profile-state, event, and history encoders, and the number of attention heads.

			PRAGMA			S		M		L		PRAGMA-M
Task	Metric	Baseline	S	M	L	Emb.	LoRA	Emb.	LoRA	Emb.	LoRA	Scratch	LoRA
Credit scoring	PR-AUC	–	+70.3 %	+98.0 %	+130.2 %	–	+18.0 %	–	+20.4 %	–	+10.3 %	–	+13.0 %
	ROC-AUC	–	+6.2 %	+10.1 %	+12.4 %	–	+0.2 %	–	+2.4 %	–	+1.5 %	–	+1.6 %
Comm. engagement	PR-AUC	–	+76.6 %	+76.7 %	+79.4 %	–	+72.9 %	–	+49.7 %	–	+54.1 %	–	+18.6 %
	ROC-AUC	–	+19.6 %	+17.4 %	+20.4 %	–	+16.9 %	–	+11.2 %	–	+13.5 %	–	+5.0 %
External fraud	Precision	–	+0.3 %	+12.3 %	+16.7 %	–	+30.8 %	–	+29.8 %	–	+23.8 %	–	+8.0 %

Table 2: Left: LoRA-adapted PRAGMA backbones are compared against task-specific baselines across scales (S/M/L). Middle: Within each model size (S/M/L), LoRA is compared against embedding-only probes on the same backbone. Right: At PRAGMA-M, LoRA fine-tuning of the pre-trained backbone is compared against training PRAGMA-M from scratch. All values are relative deltas

(\text{variant}/\text{ref}-1)\%

		PRAGMA-S
Task	Metric	Event-only (ref.)	Full
External fraud	Precision	–	+46.8 %
	Recall	–	+85.6 %
Credit scoring	PR-AUC	–	+31.8 %
	ROC-AUC	–	+4.9 %
Product rec.	mAP	–	+3.5 %

Table 3: Profile state contributes substantially to tasks where static user characteristics are discriminative. The relative performance is computed as (

\text{Full}/\text{Event-only}-1

Task	Metric	Baseline (ref.)	PRAGMA
Comm. engagement (uplift)	AUUC	–	+163.7 %
	SNIPS	–	+10.8 %

Table 4: Performance comparison of PRAGMA-L against the internal uplift baseline using the same meta-learner framework. The relative performance is computed as (

\text{PRAGMA-L}/\text{Baseline}-1

Figure 5: Comparison of text embeddings with PRAGMA (left) and a version using pre-trained Nemotron-1B-v2 (right) . Instead of our custom-trained BPE tokeniser and a trainable embedding lookup table, a pre-trained “frozen” Nemotron maps an entire text value to a single text embedding vector, which is then projected into the Transformer’s base dimension with a trainable projection.

		PRAGMA-M
Task	Metric	ref.	+Nemotron
Credit scoring	PR-AUC	–	+16.1 %
	ROC-AUC	–	+2.8 %
Recurrent txns	$F_{\text{1}}$	–	+0.1 %
Lifetime value	PR-AUC	–	+0.8 %
	ROC-AUC	–	+0.6 %

Table 5: Impact of pre-trained text embeddings on downstream tasks is concentrated in text-heavy domains. The performance is estimated relative to a LoRA-tuned PRAGMA-M.

Task	Metric	Baseline (ref.)	PRAGMA
Anti-money laundering	$F_{\text{0.5}}$	–	$-$ 47.1 %

Table 6: Performance comparison of PRAGMA against baseline for Anti-Money Laundering. The relative performance is computed as (

\text{PRAGMA}/\text{Baseline}-1

) using a linear probe on PRAGMA-L embeddings.

Financial Services

Financial services applies AI to banking, lending, payments, insurance, investment, and financial risk management.

See all 104 papers

A Foundation Model for Multimodal Event Sequences in Financial Applications
Jul 10, 2026Nikita Rusakov, Vladislav Meshkov, Konstantin Zorin +4Financial Services Multimodal Pretraining
Sber Moscow, Russian Federation · Sber AI Lab Moscow, Russian Federation
Toward Structured Financial Foundation Models
Oct 19, 2026Natraj Raman, Saumitra Mishra, Vamsi Potluru +1Finance Financial Services
JPMorgan AI Research
MINT: A Universal Zero-Shot Predictor for Transaction Data
Aug 14, 2026Parameswaran Kamalaruban, Viktor Drobnyi, Maeve Madigan +3Zero-Shot Learning Multimodal Foundation Models
Risk and Security AI Lab, Visa Inc. United Kingdom

Jul 10, 2026J/K move · Enter open · S save

A Foundation Model for Multimodal Event Sequences in Financial Applications

Nikita Rusakov, Vladislav Meshkov, Konstantin Zorin, Gleb Zaripov+3

Sber Moscow, Russian Federation · Sber AI Lab Moscow, Russian Federation

Financial Services Multimodal Pretraining Multimodal Foundation Models Multimodal Fusion Next-Event Prediction Sequence Modeling+1

Open paper AskPDF arXiv

Predictive modeling is a core component of modern financial services, where a wide range of tasks are traditionally addressed using separate models trained on manually engineered tabular features. This task-specific approach limits reuse and makes it difficult to fully exploit heterogeneous data sources such as transaction histories and digital interaction signals. In this paper, we present an approach based on pretraining a foundation transformer model on multimodal sequences of user events. Events from multiple data sources are unified into a single chronological sequence, enabling early fusion of heterogeneous modalities and learning of general-purpose representations via a next-event prediction objective. These representations are combined with existing engineered user features, on top of which lightweight neural models are trained for multiple downstream tasks. The proposed system outperforms traditional task-specific models while reducing development overhead. The approach was deployed in production at one of the biggest banks in Eastern Europe, resulting in measurable improvements in business metrics.

Abstract

Financial Services

Explore similar work

A Foundation Model for Multimodal Event Sequences in Financial Applications

Toward Structured Financial Foundation Models

MINT: A Universal Zero-Shot Predictor for Transaction Data