Foundation models promise reusable capabilities and step-change performance gains across tasks. In finance, large-scale pretraining of domain-specialized foundation models can advance both goals. However, prevailing architectures and objectives do not by themselves address the combined demands of financial systems: partial observability, interacting entities, asynchronous events, changing regimes, action-dependent outcomes, and rare states with disproportionate consequences. Moreover, independently developed specialists may learn incompatible representations, while multi-domain monolithic alternatives couple data access, model updates, and governance. We therefore propose a structured-data-first research agenda centered on modular, domain-specialized financial foundation models connected by a representation fabric for reuse and composition across domains and tasks. These models should maintain persistent financial state across aggregation levels and time horizons, capture its dynamics, and simulate action-conditioned futures, while remaining calibrated across regimes and for tail outcomes. They must also learn efficiently under data and access constraints and provide faithful, traceable explanations. We identify research challenges, directions, and capability-oriented evaluation criteria.
Figures & tables
Figure 1: From isolated models to a structured financial foundation-model paradigm . Top : Diverse financial domains and modalities connect through shared abstractions and support a broad range of tasks. Left : Task-centric models learn specialized mappings for individual financial problems. Middle : Modality-specific FMs share pretrained representations across downstream tasks. Right : Specialist financial FMs produce interoperable, multi-resolution representations that a representation fabric can compose across model boundaries, enabling task heads to combine state from multiple FMs.
Financial property
Assumption challenged
Required capability
Interacting entities
Independent observations
Relational state representation
Multi-scale event streams
Regular sequences
Event-aware, multi-scale representation
Regime dependence
Stable distributions
Adaptive dynamics
Actions affect outcomes
Passive prediction
Action-conditioned simulation
Consequential tails
Average-case objectives
Tail-aware learning
Partial observability
Complete context
Uncertainty-aware state inference
Table 1: Structural and institutional mismatches and the capabilities they motivate.
Figure 2: Illustrative (hypothetical) capability view for structured financial foundation models. Specialist FMs independently construct financial state representations, model their regime-conditioned dynamics, and simulate action-conditioned futures. In the cross-model example, the Credit FM augments its native borrower state with sector state from the Market FM and customer state from the Transaction FM , exchanged through the representation fabric. Task heads consume representations either directly from specialists ( Fraud Detection ) or after adaptation and composition through the fabric ( Credit Limit Decisioning ). Uncertainty is preserved and propagated throughout, while a provenance-aware trace links a decision to its evidence, state changes, and faithfulness checks.
Figure 3: Seven interconnected research bets. Each card contrasts prevailing practice with the proposed shift. Top pane traces the operational flow. Bottom pane shows capabilities that act across it.
Capability
Test design
Evidence of progress
Task quality
Compare models under the same task-specific protocol
Meets task-specific quality and risk tolerances
Transfer and adaptation
Hold out entities, tasks, schemas, or regimes
Required quality with less data or retraining
Dynamics, actions & tails
Test shifts, interventions, and stress chronologically
Modern financial systems generate vast quantities of transactional and event-level data that encode rich economic signals. This paper presents PRAGMA, a family of foundation models for banking event sequences. Our approach pre-trains a Transformer-based architecture with masked modelling on a large-scale, heterogeneous banking event corpus using a self-supervised objective tailored to the discrete, variable-length nature of financial records. The resulting model supports a wide range of downstream tasks such as credit scoring, fraud detection, and lifetime value prediction: strong performance can be achieved by training a simple linear model on top of the extracted embeddings and can be further improved with lightweight fine-tuning. Through extensive evaluation on downstream tasks, we demonstrate that PRAGMA achieves superior performance across multiple domains directly from raw event sequences, providing a general-purpose representation layer for financial applications.
Maxim Ostroukhov, Ruslan Mikhailov, Vladimir Iashin +11
Staged post-training of multimodal foundation models chains vision-language alignment, reasoning distillation, and domain tuning, yet how each phase trades capabilities against the others remains under-characterised. Tracking an 8.4B-parameter model from an un-fine-tuned baseline through three training phases on Japanese financial documents (evaluated on five text-input benchmarks), we find the phases interact non-additively: math-only distillation overwrites the financial knowledge acquired earlier even as procedural math skill survives, and the domain phase repairs more than it installs. A skip-phase ablation is best for domain knowledge but forfeits numerical-reasoning gains, so per-phase contributions are not decomposable and the best checkpoint is task-dependent, with implications for checkpoint retention and scheduling rather than added stages.
Predictive modeling is a core component of modern financial services, where a wide range of tasks are traditionally addressed using separate models trained on manually engineered tabular features. This task-specific approach limits reuse and makes it difficult to fully exploit heterogeneous data sources such as transaction histories and digital interaction signals. In this paper, we present an approach based on pretraining a foundation transformer model on multimodal sequences of user events. Events from multiple data sources are unified into a single chronological sequence, enabling early fusion of heterogeneous modalities and learning of general-purpose representations via a next-event prediction objective. These representations are combined with existing engineered user features, on top of which lightweight neural models are trained for multiple downstream tasks. The proposed system outperforms traditional task-specific models while reducing development overhead. The approach was deployed in production at one of the biggest banks in Eastern Europe, resulting in measurable improvements in business metrics.
Nikita Rusakov, Vladislav Meshkov, Konstantin Zorin +4