cs.LGNov 13, 2026

How Much Do Foundation Models Add to Gradient Boosting on Financial Event Sequences?

Authors: Carlos Eduardo Martins Relvas, Gustavo di Giovanni Bernardo

Organizations: WeCogno

Abstract

Foundation models pretrained on raw transaction sequences are proposed for credit, fraud and churn modeling, with reported gains over gradient boosting, but the data and baselines behind these gains are rarely public. We measure how much the pretraining and adaptation recipes of these models add to a baseline of point-in-time window aggregates and LightGBM, and we propose TBS (Temporal Boosting-Sequence ensemble), which averages the ranks of a boosting model enriched with label-free event-transition features and of a pretrained sequence network fine-tuned jointly with the aggregates. On a synthetic credit-card dataset, the public MBD-mini banking dataset and two proprietary datasets of trade credit and B2B churn, we compare 27 methods at three label budgets with five seeds and paired bootstrap intervals, using sequence models of about 0.4 million parameters pretrained per dataset. Frozen self-supervised embeddings rarely improve the baseline, and a fine-tuned sequence network alone clears it in 9 of 21 target-budget cells. TBS clears it in 19 of 21 cells (18 after FDR adjustment), with a median gain of +1.8 AUC points, and the controls suggest that combining boosting with the pretrained network is the main source of the observed gain. Gains shrink as labels grow and with rich tabular data: with both, the gain on the proprietary datasets is +0.7 and +0.8 points, the latter with an interval including zero. We also re-examine two benchmarks. On IBM TabFormer, the test follows a change in the fraud pattern and the published baseline has no history; with the training period the vendor states, our history-aware LightGBM and our own foundation model stay below the published scores of a commercial model. On IBM AML, a simulation artifact lets a date rule reach F1 0.482, and a point-in-time LightGBM reaches the F1 of graph neural networks whose test graph includes later transactions.

Figures & tables

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

Apr 9, 2026

PRAGMA: Revolut Foundation Model

Modern financial systems generate vast quantities of transactional and event-level data that encode rich economic signals. This paper presents PRAGMA, a family of foundation models for banking event sequences. Our approach pre-trains a Transformer-based architecture with masked modelling on a large-scale, heterogeneous banking event corpus using a self-supervised objective tailored to the discrete, variable-length nature of financial records. The resulting model supports a wide range of downstream tasks such as credit scoring, fraud detection, and lifetime value prediction: strong performance can be achieved by training a simple linear model on top of the extracted embeddings and can be further improved with lightweight fine-tuning. Through extensive evaluation on downstream tasks, we demonstrate that PRAGMA achieves superior performance across multiple domains directly from raw event sequences, providing a general-purpose representation layer for financial applications.
Nov 6, 2026

Tabular Foundation Models vs Tuned Gradient Boosting for Telecom Churn: A Profit- and Calibration-Aware Evaluation

A churn model is useful through the customers it selects for a retention offer: the better model is the one whose selection makes the campaign more profitable. Tabular foundation models (TFMs), pretrained networks that predict from a table of labelled examples without being trained on it, were recently reported to decisively outperform tree ensembles for churn. That comparison, however, used untuned gradient-boosted trees (GBDTs) and ranking metrics only. We ask whether the advantage holds against tuned GBDTs when models are judged by expected campaign profit and by the reliability of their probabilities. Four TFMs (TabPFN-2.5, TabPFN-3.5, TabPFN-3.5-Fast, TabICL v2) face LightGBM, XGBoost and CatBoost tuned with Optuna on validation AUC, on five public telecom datasets, three from real operators, with 10 repeated splits each and 20 pre-declared comparisons tested with a corrected t-test and Holm correction. TabPFN-3.5 earns significantly more expected maximum profit (EMP) than each tuned GBDT on two datasets, Cell2Cell (1.9%) and Iranian (7.6%). On Orange it is ahead in every split (14.3%), but not significantly after correction, and no TFM differs significantly from the GBDTs on the two fictional datasets. Across 27 assumed combinations of customer value, incentive cost and acceptance rate, TabPFN-3.5 stays ahead on all three real-operator datasets, though the size of its gain varies widely. Retuning the GBDTs on EMP moved no gap by more than 0.014 per customer. Ranking and probability reliability diverge: raw TabICL v2 underpredicts churn on Orange, and a threshold policy built on its probabilities loses 15.7% of attainable profit until it is recalibrated. Because the splits resample the same customers, the results describe these datasets, not telecom operators in general. The non-commercial licence of the TabPFN weights, rather than scoring cost, restricts production use.
Jul 10, 2026

A Foundation Model for Multimodal Event Sequences in Financial Applications

Predictive modeling is a core component of modern financial services, where a wide range of tasks are traditionally addressed using separate models trained on manually engineered tabular features. This task-specific approach limits reuse and makes it difficult to fully exploit heterogeneous data sources such as transaction histories and digital interaction signals. In this paper, we present an approach based on pretraining a foundation transformer model on multimodal sequences of user events. Events from multiple data sources are unified into a single chronological sequence, enabling early fusion of heterogeneous modalities and learning of general-purpose representations via a next-event prediction objective. These representations are combined with existing engineered user features, on top of which lightweight neural models are trained for multiple downstream tasks. The proposed system outperforms traditional task-specific models while reducing development overhead. The approach was deployed in production at one of the biggest banks in Eastern Europe, resulting in measurable improvements in business metrics.