How Much Do Foundation Models Add to Gradient Boosting on Financial Event Sequences?
Organizations: WeCogno
Abstract
Foundation models pretrained on raw transaction sequences are proposed for credit, fraud and churn modeling, with reported gains over gradient boosting, but the data and baselines behind these gains are rarely public. We measure how much the pretraining and adaptation recipes of these models add to a baseline of point-in-time window aggregates and LightGBM, and we propose TBS (Temporal Boosting-Sequence ensemble), which averages the ranks of a boosting model enriched with label-free event-transition features and of a pretrained sequence network fine-tuned jointly with the aggregates. On a synthetic credit-card dataset, the public MBD-mini banking dataset and two proprietary datasets of trade credit and B2B churn, we compare 27 methods at three label budgets with five seeds and paired bootstrap intervals, using sequence models of about 0.4 million parameters pretrained per dataset. Frozen self-supervised embeddings rarely improve the baseline, and a fine-tuned sequence network alone clears it in 9 of 21 target-budget cells. TBS clears it in 19 of 21 cells (18 after FDR adjustment), with a median gain of +1.8 AUC points, and the controls suggest that combining boosting with the pretrained network is the main source of the observed gain. Gains shrink as labels grow and with rich tabular data: with both, the gain on the proprietary datasets is +0.7 and +0.8 points, the latter with an interval including zero. We also re-examine two benchmarks. On IBM TabFormer, the test follows a change in the fraud pattern and the published baseline has no history; with the training period the vendor states, our history-aware LightGBM and our own foundation model stay below the published scores of a commercial model. On IBM AML, a simulation artifact lets a date rule reach F1 0.482, and a point-in-time LightGBM reaches the F1 of graph neural networks whose test graph includes later transactions.
Figures & tables
| Raw attribute | Raw value | Token field | Contribution to |
|---|---|---|---|
| Merchant group | grocery | merchant group = grocery | row of |
| Channel | online | channel = online | row of |
| Amount | 120.00 | amount band 21 of 32 | row of |
| Gap to previous event | 2 days | gap band 11 of 16 | row of |
| Time to the as-of date | 5 days | recency band 4 of 16 | row of |
| Timestamp | 14:32 | hour = 14 | row of |
| Study | Credit | Public data | SSL | Fine- tuned | Temporal pretraining | GBDT | Combination with GBDT | Budgets | Uncertainty |
|---|---|---|---|---|---|---|---|---|---|
| TabBERT (2021) | no | yes | yes | no | n.r. | no | no | no | no |
| E.T.-RNN (2019) | yes | no | no | – | – | yes | no | yes | yes |
| CoLES (2022) | yes | yes | yes | yes | n.r. | yes | feature | yes | yes |
| EBES (2025) | no | yes | yes | yes | n.r. | no | no | no | yes |
| MBD (2025) | yes | yes | yes | no | n.r. | yes | feature | no | yes |
| nuFormer (2026) | yes | no | yes | yes | n.r. | yes | feature | yes | no |
| Group | Methods | Role |
|---|---|---|
| Baseline | Aggregates + GBDT (Agg) | reference |
| Frozen representation + GBDT | AE, Transformer, masked numeric; AE, Transformer, + one-hot; AE, MLP, whole sequence; AE, MLP, masked events; MLM (TabBERT-style); Nep (nuFormer-style); CoLES; Mlm-Cl | alternatives (RQ1) |
| End-to-end | GRU; JF-rand (no pretraining); JF (pretrained Nep + joint fusion); TabPFN v2 | alternatives and ablation (RQ2) |
| Expert features | Expert features alone; Agg + expert features | reference |
| Transition features | Agg + Trans; Agg + Trans + Mlm-Cl | components of TBS |
| TBS and hybrids | TBS; TBS-R (boosting residual); TBS-R, no pretraining; TBS-R, no Trans in offset; TBS-R, Nep encoder; JF + Trans | proposed method, variants and ablations (RQ3) |
| Comparison | What it isolates | Hypothesis |
|---|---|---|
| JF vs. JF-rand | pretraining of the fine-tuned network | H2 |
| JF vs. frozen Nep + GBDT | fine-tuning, together with a change of predictor | H2 |
| TBS vs. Agg + Trans | adding the pretrained network to the boosting model | H3 |
| TBS vs. JF | adding the boosting model to the network | H3 |
| TBS vs. ensemble of Agg + Trans and JF-rand | pretraining, within the combination | H3 |
| TBS vs. ensemble of Agg and JF | transition features, within the combination | H3 |
| Dataset | Entities | Events | Train labels | Test | Test positives | Targets |
|---|---|---|---|---|---|---|
| Synthetic | 48,000 | 6.8 million | 17,879 | 6,121 | 610, 601, 618 | Default, Seq. trigger, Churn |
| MBD-mini | 54,000 | 10.7 million | 22,000 | 8,000 | 416, 153 | Any product, Product 1 |
| Client A | 56,000 | 1.4 million | 24,000 | 8,000 | 563 | Late payment |
| Client B | 35,292 | 18.8 million | 15,376 | 10,000 | 600 | Churn (6m) |
| Dataset | Training as-of dates | Label | Last training label known | Test as-of dates |
|---|---|---|---|---|
| Synthetic | 2025-01-01 to 2025-09-01 | drawn at generation | n/a | 2025-10-01 to 2025-12-01 |
| MBD-mini | 2022-04-01 to 2022-10-01 | purchase in the month that starts at the as-of date | 2022-11-01 | 2022-11-01 to 2023-01-01 |
| Client A | 2019-03-01 to 2019-05-31 | known at payment, or at due date 8 days | 2019-06-15 | 2019-06-15 to 2019-07-08 |
| Client B | 2023-06-01 to 2025-03-01 | net cancellation recorded in month | 2025-10-01 | 2025-10-01 to 2025-12-01 |
| Synthetic | MBD-mini | Client A | Client B | ||||
| Method | Default | Seq. trigger | Churn | Any product | Product 1 | Late payment | Churn (6m) |
| Agg AUC | 0.654 | 0.733 | 0.653 | 0.741 | 0.719 | 0.846 | 0.609 |
| Frozen representation + GBDT | |||||||
| AE, Transformer, masked numeric | +0.5 | +0.0 | 0.4 | 0.1 | +0.5 | 0.2 | +0.1 |
| AE, Transformer, + one-hot | +0.3 | +0.9 | 0.4 | 0.1 | +0.5 | +0.0 | +0.5 |
| AE, MLP, whole sequence | 0.2 | 0.2 | 0.6 | 0.2 | +1.4 | 0.2 | +0.2 |
| Above zero | |||||||
| Method | Cells | Nominal | BH | Seeds res. | Both | Below | Median |
| AE, Transformer, masked numeric | 21 | 0 | 0 | 0 | 0 | 0 | +0.1 |
| AE, Transformer, + one-hot | 21 | 1 | 1 | 1 | 1 | 0 | +0.2 |
| AE, MLP, whole sequence | 21 | 0 | 0 | 0 | 0 | 0 | 0.2 |
| AE, MLP, masked events | 21 | 0 | 0 | 0 | 0 | 4 | 0.1 |
| MLM (TabBERT-style) | 21 | 7 | 4 | 3 | 3 | 0 | +0.6 |
| Above zero | |||||||
| Comparison | Cells | Nominal | BH | Seeds res. | Both | Below | Median |
| JF vs. JF-rand (pretraining) | 21 | 10 | 8 | 6 | 6 | 1 | +0.8 |
| JF vs. frozen Nep in boosting | 21 | 6 | 5 | 3 | 3 | 0 | +0.5 |
| TBS vs. Agg + Trans | 21 | 13 | 13 | 13 | 13 | 0 | +1.2 |
| TBS vs. JF | 21 | 8 | 7 | 3 | 3 | 0 | +0.5 |
| TBS-R vs. Agg + Trans | 21 | 12 | 10 | 9 | 9 | 1 | +0.7 |
| Strict | Calendar | Through | ||||
|---|---|---|---|---|---|---|
| Features | AUPRC | F1 | AUPRC | F1 | AUPRC | F1 |
| 13 raw columns | 0.022 | 0.017 | 0.135 | 0.103 | 0.862 | 0.580 |
| Window summary (1/7/30 days) | – | – | 0.605 | 0.545 | 0.957 | 0.879 |
| Card history (72 features) | 0.060 | 0.060 | 0.809 | 0.656 | 0.980 | 0.925 |
| FM, first version, alone | – | – | 0.861 | 0.742 | 0.977 | 0.941 |
| Window summary + FM score (v1) | – | – | 0.896 | 0.826 | 0.983 | 0.941 |
| Use | Period |
|---|---|
| Pretraining and model training | up to December 31, 2017 |
| Early stopping | January 1 to June 30, 2018 |
| CatBoost hyperparameter search and error analysis | January 1 to December 31, 2018 |
| Selection among the 18 candidates | July 1, 2018 to the start of the test |
| Test, last 10% of transactions | from October 12, 2018 |
| Test, 2019–2020 | from January 1, 2019 |
| HI-Small | LI-Small | |||||
|---|---|---|---|---|---|---|
| F1 test | AUPRC act. | F1 act. | F1 test | AUPRC act. | F1 act. | |
| Published: LightGBM+GFs | 0.629 0.003 | – | – | 0.208 0.015 | – | – |
| Published: XGBoost+GFs | 0.632 0.002 | – | – | 0.273 0.003 | – | – |
| Published: Multi-GIN+EU | 0.648 0.012 | – | – | 0.269 0.066 | – | – |
| Published: Multi-PNA+EU | 0.682 0.027 | – | – | 0.331 0.026 | – | – |
| LightGBM, transfer fields only | 0.223 0.005 | 0.069 0.000 | 0.147 0.001 | 0.080 0.007 | 0.014 0.000 | 0.049 0.004 |
| Dataset | Labels | Pretraining | Fine-tuning | LightGBM |
|---|---|---|---|---|
| Synthetic | 17,879 | 10 min | 53 s | 2.5 s |
| MBD-mini | 22,000 | 8 min | 66 s | 2.8 s |
| Client A | 24,000 | 81 s | 31 s | 1.9 s |
| Client B | 15,376 | 5 min | 29 s | 1.5 s |
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Component | Configuration |
|---|---|
| LightGBM | binary objective, learning rate 0.05, 31 leaves, min. 40 rows per leaf, feature fraction 0.8, bagging fraction 0.8 every iteration, L2 1.0, deterministic, up to 3,000 rounds, early stopping after 100 rounds on a stratified 20% split of the training rows |
| Tokens | last 128 events; categorical vocabularies capped at the most frequent training-period levels (8–64 per field), amount in 32 quantile bands, gap 16, recency 16, hour 24; three standardized continuous values |
| Backbone | width 96, 4 layers, 4 heads, feed-forward 256, dropout 0.1, pre-norm, GELU, learned positions; representation = last position mean (192) |
| Field encoder | per event: field embeddings and the three continuous values as tokens, field position embeddings, 2-layer Transformer (width 96, 4 heads, feed-forward 192), concatenation and linear projection to 96 |
| Pretraining | AdamW, learning rate , weight decay 0.01, one-cycle schedule (5% warm-up), batch 128, gradient clipping 1.0; MLM and Nep: 10 epochs (synthetic) or 8; CoLES: 12 or 10 epochs, 4 sub-sequences of 20–120 events per client, temperature 0.1; Mlm-Cl: 10 epochs, contrastive weight 0.5, temperature 0.1, projection to 64 |
| Reconstruction AEs | inputs: last 50 events, amount and hour (plus one-hot event type and currency in the second variant); Transformer width 64, 4 heads, 2 layers, feed-forward 128, 15% of events masked, 5 epochs; MLP autoencoders with layers 256–128 and latent dimension 64, 50 epochs |
| Synthetic | MBD-mini | Client A | Client B | ||||
| Method | Default | Seq. trigger | Churn | Any product | Product 1 | Late payment | Churn (6m) |
| Agg AUC | 0.621 | 0.703 | 0.598 | 0.678 | 0.656 | 0.820 | 0.575 |
| Frozen representation + GBDT | |||||||
| AE, Transformer, masked numeric | 0.5 | +0.3 | +0.3 | +0.3 | 0.1 | +0.1 | +0.2 |
| AE, Transformer, + one-hot | +0.0 | +2.0 ↑∗ | 0.3 | 0.7 | +0.7 | +0.6 | 0.4 |
| AE, MLP, whole sequence | +0.3 | 0.2 | 0.4 | +0.3 | +0.2 | 0.2 | 0.5 |
| Synthetic | MBD-mini | Client A | Client B | ||||
| Method | Default | Seq. trigger | Churn | Any product | Product 1 | Late payment | Churn (6m) |
| Agg AUC | 0.634 | 0.721 | 0.626 | 0.711 | 0.682 | 0.834 | 0.594 |
| Frozen representation + GBDT | |||||||
| AE, Transformer, masked numeric | 0.1 | 0.2 | +0.1 | +0.1 | +1.1 | 0.1 | +0.3 |
| AE, Transformer, + one-hot | +0.1 | +1.0 | 0.7 | +0.2 | +0.4 | +0.1 | +0.3 |
| AE, MLP, whole sequence | +0.5 | 0.2 | 0.3 | +0.3 | +1.0 | +0.1 | +0.2 |
| Synthetic | Client A | Client B | |||
|---|---|---|---|---|---|
| Method | Default | Seq. trigger | Churn | Late payment | Churn (6m) |
| Agg AUC | 0.741 | 0.732 | 0.649 | 0.848 | 0.622 |
| AE, Transformer, masked numeric | 0.1 | +0.3 | 0.1 | +0.0 | +0.0 |
| AE, Transformer, + one-hot | 0.3 | +1.3 ↑∗ | +0.0 | 0.1 | +0.0 |
| AE, MLP, whole sequence | 0.2 | +0.1 | 0.1 | 0.3 | +0.0 |
| AE, MLP, masked events | 0.1 | 0.2 | +0.1 | 0.1 | 0.3 |
| Comparison | Labels | Default | Seq. trigger | Churn | Any product | Product 1 | Late payment | Churn (6m) |
|---|---|---|---|---|---|---|---|---|
| JF vs. JF-rand (pretraining) | 1k | +1.9 ↑∗ | +4.6 ↑∗ | +0.6 | 0.9 | 4.3 ↓∗ | +2.5 ↑∗ | +0.9 ↑ |
| 4k | +1.6 ↑∗ | +3.9 ↑∗ | +1.4 ↑∗ | +0.2 | 0.3 | +0.8 ↑∗ | +2.1 ↑∗ | |
| all | +0.9 ↑ | +0.9 | +0.4 | 0.3 | +0.5 | +0.5 | +0.3 | |
| JF vs. frozen Nep in boosting | 1k | +0.5 | +2.2 ↑∗ | +1.6 ↑ | +1.4 ↑∗ | 1.8 | +1.2 ↑∗ | 1.0 |
| 4k | +0.8 | +2.9 ↑∗ | +0.1 | +1.0 | +1.2 | +0.2 | 0.3 | |
| all | +0.3 | +1.6 ↑∗ | 0.5 | +0.0 | +0.6 | +0.3 | 0.5 |
| Comparison | Labels | Default | Seq. trigger | Churn | Any product | Product 1 | Late payment | Churn (6m) |
|---|---|---|---|---|---|---|---|---|
| Ensemble of Agg and Agg + Trans vs. Agg | 1k | +0.2 | +4.0 ↑∗ | +0.3 | +0.7 ↑∗ | +1.2 ↑∗ | +0.4 ↑∗ | +0.6 ↑∗ |
| 4k | +0.7 ↑∗ | +3.2 ↑∗ | +0.3 | +0.6 ↑∗ | +1.7 ↑∗ | +0.3 ↑∗ | +0.7 ↑∗ | |
| all | +0.3 ↑ | +2.3 ↑∗ | +0.2 | +0.4 ↑∗ | +0.5 | +0.3 ↑∗ | +0.8 ↑∗ | |
| Ensemble with JF-rand vs. Agg + Trans | 1k | +2.1 ↑∗ | 1.6 ↓∗ | +2.0 ↑∗ | +2.2 ↑∗ | +2.9 ↑∗ | +0.3 | 0.6 |
| 4k | +1.1 ↑∗ | 0.2 | +0.5 | +1.8 ↑∗ | +1.7 ↑∗ | +0.5 ↑ | 0.5 | |
| all | +0.7 ↑ | +0.3 | +0.2 | +1.1 ↑∗ | +2.0 ↑∗ | +0.3 | +0.3 |
| Default | Seq. trigger | Churn | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | 1k | 4k | all | 1k | 4k | all | 1k | 4k | all |
| Agg (baseline) | 0.621 (8) | 0.634 (7) | 0.654 (6) | 0.703 (7) | 0.721 (5) | 0.733 (3) | 0.598 (20) | 0.626 (8) | 0.653 (4) |
| AE, Transformer, masked numeric | 0.615 (14) | 0.633 (10) | 0.660 (6) | 0.706 (5) | 0.718 (7) | 0.733 (2) | 0.601 (18) | 0.627 (13) | 0.650 (5) |
| AE, Transformer, + one-hot | 0.620 (13) | 0.636 (8) | 0.657 (7) | 0.723 (8) | 0.730 (7) | 0.741 (4) | 0.595 (22) | 0.619 (10) | 0.649 (5) |
| AE, MLP, whole sequence | 0.624 (10) | 0.639 (9) | 0.652 (7) | 0.701 (12) | 0.719 (7) | 0.731 (4) | 0.594 (18) | 0.623 (12) | 0.648 (10) |
| AE, MLP, masked events | 0.612 (22) | 0.639 (3) | 0.657 (6) | 0.696 (5) | 0.711 (7) | 0.726 (6) | 0.596 (17) | 0.623 (10) | 0.651 (10) |
| Default | Seq. trigger | Churn | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Method | 1k | 4k | all | 1k | 4k | all | 1k | 4k | all |
| Agg (baseline) | 0.702 (11) | 0.722 (5) | 0.741 (3) | 0.699 (7) | 0.718 (5) | 0.732 (4) | 0.602 (14) | 0.626 (11) | 0.649 (4) |
| AE, Transformer, masked numeric | 0.701 (16) | 0.721 (5) | 0.741 (2) | 0.708 (7) | 0.720 (6) | 0.734 (2) | 0.597 (17) | 0.624 (11) | 0.648 (8) |
| AE, Transformer, + one-hot | 0.702 (11) | 0.721 (3) | 0.739 (3) | 0.724 (8) | 0.729 (8) | 0.745 (4) | 0.591 (17) | 0.627 (16) | 0.649 (7) |
| AE, MLP, whole sequence | 0.703 (13) | 0.720 (4) | 0.739 (4) | 0.703 (11) | 0.714 (5) | 0.733 (3) | 0.586 (19) | 0.625 (8) | 0.648 (8) |
| AE, MLP, masked events | 0.701 (12) | 0.722 (7) | 0.741 (1) | 0.704 (7) | 0.714 (7) | 0.730 (5) | 0.596 (17) | 0.626 (14) | 0.650 (4) |
| Any product | Product 1 | |||||
|---|---|---|---|---|---|---|
| Method | 1k | 4k | all | 1k | 4k | all |
| Agg (baseline) | 0.678 (18) | 0.711 (7) | 0.741 (3) | 0.656 (25) | 0.682 (12) | 0.719 (6) |
| AE, Transformer, masked numeric | 0.681 (26) | 0.712 (10) | 0.740 (3) | 0.654 (27) | 0.693 (12) | 0.724 (4) |
| AE, Transformer, + one-hot | 0.671 (55) | 0.713 (6) | 0.740 (2) | 0.662 (34) | 0.686 (9) | 0.724 (11) |
| AE, MLP, whole sequence | 0.682 (22) | 0.714 (15) | 0.739 (5) | 0.658 (22) | 0.691 (24) | 0.733 (7) |
| AE, MLP, masked events | 0.679 (23) | 0.711 (6) | 0.741 (2) | 0.646 (28) | 0.687 (21) | 0.726 (6) |
| Transactions only | Full | |||||
|---|---|---|---|---|---|---|
| Method | 1k | 4k | all | 1k | 4k | all |
| Agg (baseline) | 0.820 (8) | 0.834 (9) | 0.846 (3) | 0.821 (8) | 0.833 (6) | 0.848 (2) |
| AE, Transformer, masked numeric | 0.821 (7) | 0.833 (7) | 0.844 (2) | 0.817 (9) | 0.829 (7) | 0.848 (2) |
| AE, Transformer, + one-hot | 0.825 (3) | 0.835 (7) | 0.846 (3) | 0.824 (1) | 0.834 (6) | 0.848 (1) |
| AE, MLP, whole sequence | 0.817 (16) | 0.835 (9) | 0.844 (2) | 0.816 (16) | 0.832 (11) | 0.846 (4) |
| AE, MLP, masked events | 0.815 (16) | 0.831 (9) | 0.844 (3) | 0.816 (22) | 0.832 (8) | 0.848 (3) |
| Transactions only | Full | |||||
|---|---|---|---|---|---|---|
| Method | 1k | 4k | all | 1k | 4k | all |
| Agg (baseline) | 0.575 (8) | 0.594 (12) | 0.609 (3) | 0.579 (8) | 0.601 (7) | 0.622 (5) |
| AE, Transformer, masked numeric | 0.577 (8) | 0.597 (11) | 0.610 (4) | 0.583 (10) | 0.605 (9) | 0.621 (5) |
| AE, Transformer, + one-hot | 0.571 (11) | 0.597 (13) | 0.614 (4) | 0.576 (10) | 0.599 (13) | 0.622 (4) |
| AE, MLP, whole sequence | 0.570 (11) | 0.595 (10) | 0.611 (3) | 0.570 (11) | 0.601 (7) | 0.622 (5) |
| AE, MLP, masked events | 0.574 (8) | 0.594 (6) | 0.611 (4) | 0.575 (14) | 0.602 (5) | 0.619 (10) |
Financial Time Series
Financial time series studies methods for modeling market and economic data to forecast returns, volatility, risk, and changing regimes.
See all 104 papers