Banks analyse sequential financial transaction data to perform many tasks, including fraud prevention, credit risk assessment and offer personalization. To improve the predictive accuracy of these tasks, Payments Foundation Models encode transaction sequence data as rich contextual embeddings, which can then be provided to task-specific models as features. However, these Foundation Models are not designed for flexible zero-shot reasoning across novel downstream prediction tasks, limiting their adaptability and utility. Existing LLM-based approaches to zero-shot prediction often fail to fully exploit the predictive signal within transaction data, while relying on costly text serialization or task-specific architectures that scale poorly. To address these limitations, we present the Multimodal Instruction Network for Transactions (MINT), a framework that connects a pretrained transaction sequence encoder to a decoder-only LLM through lightweight embedding injection, transaction-language alignment, and instruction tuning. We find that MINT achieves state-of-the-art predictive question-answering performance in both in-distribution and out-of-distribution questions, while substantially reducing input tokens, latency, and memory consumption compared to text-serialization baselines. Through comprehensive analyses of representations, alignment strategies, training data, and history length, we establish that compact transaction embeddings are a superior approach to transaction representation than text serialization for multimodal reasoning and zero-shot prediction tasks.
Figures & tables
Figure 1. Overview of MINT. Transaction histories are encoded into compact transaction embeddings, enabling predictive question answering and financial reasoning through a natural-language interface. Example applications include forecasting future customer behavior, such as transaction volume and average transaction amount over the next 30 days, alongside summarization, fraud-risk assessment, and financial analysis.
Figure 2. Accuracy-efficiency trade-off on predictive QA. The x-axis shows input token count and the y-axis shows accuracy; filled and unfilled markers correspond to ID and OOD evaluation, respectively. MINT occupies a more favorable Pareto region than competing methods, with MINT ( h=1 ) providing the strongest predictive QA performance while balancing the performance and inference efficiency. MINT also shows the least degradation from ID to OOD among top performing methods.
Figure 3. MINT architecture: frozen transaction sequence encoder produces per-transaction embeddings, a trainable connector projects them into the LLM hidden space, and a decoder-only LLM adapted via LoRA generates text outputs conditioned on injected embeddings.
Predictive QA
Extractive QA
Inference Metrics
Model
ID
OOD
ID
OOD
TTFT ms ↓
Tok./sec ↑
VRAM (GB) ↓
Input tok. ↓
Prior sampling
0.724
0.622 †
0.635
0.651 †
–
–
–
–
Task-specific classifier
0.752
0.687 †
0.756
0.741 †
–
–
–
–
Contrastive alignment
0.461
0.509
0.418
0.225
–
–
–
–
LLM SFT ( h=1 )
0.794
0.704
0.850
0.536
164.91
57.08
9.02
348.50
LLM SFT ( h=5 )
0.811
0.750
0.893
0.605
683.96
47.11
11.20
1370.12
Table 1. Overall accuracy on extractive and predictive QA tasks, together with inference efficiency metrics (TTFT, decode throughput, peak VRAM, and input token count) measured on Predictive QA. Results marked with † are trained and evaluated on the same question set and are included for reference only.
Predictive QA
Extractive QA
Model
ID
OOD
ID
OOD
Transactions in the context prompt
with timestamp
0.832
0.696
0.861
0.600
without timestamp
0.826
0.805
0.824
0.607
Decoding strategy
deterministic decoding
0.832
0.696
0.861
0.600
Table 2. Ablation study of MINT. We analyze the effects of input representation, training data composition, decoding strategy, transaction embedding design, and model capacity on extractive and predictive QA performance (overall accuracy) in both ID and OOD settings.
Predictive QA
Extractive QA
Model
ID
OOD
ID
OOD
Qwen3 0.6B
LTM 20M
0.832
0.675
0.858
0.362
LTM 120M
0.831
0.632
0.861
0.592
LTM 720M
0.835
0.640
0.859
0.545
Qwen3 1.7B
Table 3. Impact of LLM-LTM pairing on MINT performance (overall accuracy). Results are reported for extractive and predictive QA under both ID and OOD evaluation settings across different Qwen3 backbone sizes and pretrained LTMs.
Zero-shot learning lets models recognize classes or solve tasks absent from task-specific training examples by using semantic descriptions or pretrained knowledge.
Modern financial systems generate vast quantities of transactional and event-level data that encode rich economic signals. This paper presents PRAGMA, a family of foundation models for banking event sequences. Our approach pre-trains a Transformer-based architecture with masked modelling on a large-scale, heterogeneous banking event corpus using a self-supervised objective tailored to the discrete, variable-length nature of financial records. The resulting model supports a wide range of downstream tasks such as credit scoring, fraud detection, and lifetime value prediction: strong performance can be achieved by training a simple linear model on top of the extracted embeddings and can be further improved with lightweight fine-tuning. Through extensive evaluation on downstream tasks, we demonstrate that PRAGMA achieves superior performance across multiple domains directly from raw event sequences, providing a general-purpose representation layer for financial applications.
Maxim Ostroukhov, Ruslan Mikhailov, Vladimir Iashin +11
Predictive modeling is a core component of modern financial services, where a wide range of tasks are traditionally addressed using separate models trained on manually engineered tabular features. This task-specific approach limits reuse and makes it difficult to fully exploit heterogeneous data sources such as transaction histories and digital interaction signals. In this paper, we present an approach based on pretraining a foundation transformer model on multimodal sequences of user events. Events from multiple data sources are unified into a single chronological sequence, enabling early fusion of heterogeneous modalities and learning of general-purpose representations via a next-event prediction objective. These representations are combined with existing engineered user features, on top of which lightweight neural models are trained for multiple downstream tasks. The proposed system outperforms traditional task-specific models while reducing development overhead. The approach was deployed in production at one of the biggest banks in Eastern Europe, resulting in measurable improvements in business metrics.
Nikita Rusakov, Vladislav Meshkov, Konstantin Zorin +4
Merchant information extraction turns noisy financial transaction descriptors into structured fields at production scale. Our deployed LoRA-fine-tuned LLaMA3.1-8B reaches 96.95% F1, but its memory and throughput motivate smaller replacements. We evaluate 23 retained fine-tuning runs plus a separately trained production reference, spanning Gemma3 (270M--4B), Qwen3.5 (0.8B--4B), Aya3.35B, and LLaMA3.1-8B across LoRA ranks, prompts, training templates, and serving environments. A rank-8 LLaMA fine-tune reaches 96.75% F1, only 0.20 points below the rank-32 production reference. Qwen3.54B with JSON-Only prompting reaches 96.60% F1 and strict record-level exact match of 91.67%, with a 3.8× lower inverse-throughput time estimate than the rank-8 8B model. Qwen3.5~0.8B reaches 94.75% F1, and Qwen Think and Nothink templates differ by less than 0.004 F1. Across 14 Databricks endpoints, mean F1 change from local evaluation is −0.0081; Aya is the only family with a 2.7--5.1 point decline. These results show that compact fine-tuned models can preserve most extraction accuracy, but model selection must account for prompt choice, throughput, and serving-stack behavior.
Donghao Huang, Tomas Drietomsky, Benjamin Barrett +1
Research and Development, Mastercard, Arlington, VA, USA · School of Computing and Information Systems, Singapore Management University, Singapore · Economic Intelligence, Mastercard, Prague, Czech Republic