PaMIR: Open Benchmark of Public Credit-Default Datasets

Mikhail Liashkov; Ilyas Varshavskiy; Shuhratjon Khalilbekov; Azizjon Azimi; Bonu Boboeva

cs.LGOct 2, 2026

PaMIR: Open Benchmark of Public Credit-Default Datasets

Authors: Mikhail Liashkov, Ilyas Varshavskiy, Shuhratjon Khalilbekov, Azizjon Azimi, Bonu Boboeva

Abstract

We release PaMIR (Public Arrival-ordered Measurement for Inference in Risk), an open benchmark for credit-default prediction when labels are scarce and arrive late. The field's reference benchmark studies use eight datasets each, only two or four of them public. PaMIR brings together 19 public datasets with binary default labels -- 1.24M loans, firms and card accounts from nine countries -- rebuilt from pinned source snapshots by one leakage-audited recipe and never redistributed; to our knowledge it is the one of its kind as of today. Every model is a single function, scored under a repeated i.i.d. split and a label-delayed stream in which each application is scored on arrival, with AUC reported by label budget; fleet means are withheld unless every dataset is scored. A synthetic-data harness tests generated training rows without letting a generator see held-out rows. This report describes release 0.4.0 of this living benchmark.

Figures & tables

Figure 1: PaMIR pools more public credit-default datasets than any prior multi-dataset study. Each bar counts the datasets with a binary default, bankruptcy or financial-distress target that one study uses together; the filled part is public, the hollow part proprietary or restricted. The counting convention and the sources are given in Section 2.1 and Table 1 .

Figure 2: Under the strictest count the 19 datasets reduce to 14 independent sources, still more than the twelve public datasets of the largest prior study. The three Polish sets are one sample observed at three forecasting horizons [ 22 ] and count once; the three tables labelled as LendingClub-derived count once; the synthetic conorsully is dropped. The remaining twelve datasets are counted as they are.

Figure 3: The 19 datasets span five product families, 1k–266k rows, default rates from 3.2% to 40.9% and 10 to 94 features. One row per dataset, in the order of Table 3 ; the columns give the rows of the harmonized table (log scale), the default rate and the number of features, and the grey label gives the country of origin (a dash where the source names none). Short names are abbreviated from the catalog titles; Table 3 gives the package identifiers.

Figure 4: PaMIR ships metadata and code, rebuilds every table locally from a pinned source snapshot, and reports a fleet mean only for a complete, valid run. Top: what the package contains — the catalogue with each dataset’s source, licence, citation, pinned download specification and harmonization specification; the frozen expected.json contract; and the leakage guards run in continuous integration. Middle: pamir.download(id) fetches the raw file from one of four source kinds, applies the six-step recipe, checks the result against expected.json — a mismatch raises ContractError and nothing is cached — and caches the table as Parquet with its provenance record. Bottom: a model is a single fit_fn (a predict_fn is accepted and wrapped), called by both protocols; every call and failure is counted, and fleet_summary returns auc_mean only when every dataset was scored, every table met its contract and the scores were row-independent.

Figure 5: The day-zero cut asks whether a column could have been known when the application was scored; a univariate threshold misses most post-outcome columns. (a) Kept and removed columns of the Bondora loan book on a loan’s timeline. (b) Univariate ROC AUC of the 31 Bondora columns the cut removes: the earlier rule (drop if AUC

>0.95

) catches one of them, yet the other 30 together predict the outcome perfectly (AUC 1.000, five-fold histogram gradient boosting on 60,000 rows), against 0.768 for the 46 day-zero features. (c) On the SBA loans [ 35 ] the term alone has AUC 0.881, but whether it is a multiple of 12 separates a 3.6% from an 87.7% default rate (indicator AUC 0.932); the recipe drops the term and the two columns derived from it. Measured on the pinned raw snapshots that reproduce expected.json .

Figure 6: The eight leakage guards, their thresholds and their positive controls. Each row gives the check, the leak it was written for, and the test that reconstructs that leak and asserts that the guard fires, together with the case that must not fire. Guards 1–7 read every cached table and run in continuous integration on the seven datasets outside Kaggle; guard 8 runs inside the harmonizer on every download and warns rather than fails.

Figure 7: The audit admitted 19 datasets and rejected seven candidates rather than patching them. Band widths are proportional to the number of datasets. Admitted datasets are shown by product family (as in Table 3 ), rejected ones by the reason for rejection. Candidates screened out earlier by the inclusion criteria of Section 3 are not recorded. Three of the rejected sets — HMEQ, Thomas and Home Credit — are among the twelve public datasets of Baesens et al. [7] .

Figure 8: A dataset enters PaMIR as a recipe, never as a file. A contributor adds a catalogue entry with a pinned snapshot, rebuilds the table locally, runs the test suite on it and freezes the result as the expected.json contract; continuous integration reruns the tests, builds the package and documentation, and the merged change is recorded in the changelog and shipped as a new version. Continuous integration downloads only the seven datasets outside Kaggle, so the data guards on a new Kaggle-hosted dataset run on the contributor’s machine.

Parameter	Default	Meaning
mode	arrival	arrival : each row is scored when it arrives; refresh : the rule of release 0.3.0
lag ( $L$ )	1000	label-maturation delay, in stream positions
max_lag_frac	0.2	cap on the delay: $L_{\text{eff}}=\min(L,\lfloor 0.2\,n\rfloor)$ ; None disables it
k_refit ( $k$ )	10	refit after every $k$ newly resolved defaults
max_n	20000	truncate each stream to its first max_n rows
min_defaults ( $m$ )	3	resolved defaults (and non-defaults) required before the first refit

Table 4: Streaming-protocol parameters. The defaults form the reference setting.

Figure 9: In arrival mode a refit’s model scores only the applications that arrive after it, so the label delay binds for every row. Schematic stream with

N=100

rows, delay

L=20

, warm-up

m=3

and cadence

k=4

(package defaults:

L=1000

capped at

0.2\,N

k=10

m=3

, max_n

=20{,}000

). (a) At step

t

the labels of rows

0,\dots,r-1

, with

r=t+1-L

, are resolved and form the training set; the

L

most recent rows have arrived but not matured. (b) Each refit is fitted on its resolved prefix (line) and scores the rows that arrive after it, up to the next refit (bar); a score, once given, is final. The two strips show which refit owns each row’s evaluated score in arrival mode and, for comparison, in the refresh mode of release 0.3.0, where every refit re-scored all rows from

r

onwards, including rows that had not yet arrived. Row 50 is scored in arrival mode by refit 2, fitted on 26 labels; in refresh mode by refit 3, fitted on 43 labels twelve steps after the row arrived. Rows that arrive before the first refit are never scored and are excluded from the ROC AUC; the last

L

rows are scored, but their labels never resolve inside the stream. (c) Both modes share the trigger: the first refit fires once

m

defaults (and at least

m

non-defaults) have resolved, every later one after

k

newly resolved defaults, and none at the final step.

Figure 10: The two sizings of a synthetic mix answer different questions. One fold with 800 real training rows, synthetic share

s=0.7

and two equally weighted generators, drawn to scale. Under the default sizing all real rows are kept and

n_{\text{syn}}=\mathrm{round}\big(n\,s/(1-s)\big)=1{,}867

synthetic rows are added; under fixed-total sizing the frame is pinned to

N=800

rows, of which 560 are synthetic and 240 are a class-stratified subsample of the real rows. The synthetic part is split by the largest-remainder method, with ties broken by generator name.

Figure 11: Inside each cross-validation fold, generators see only the training rows, and the held-out rows are scored and probed but never augmented. The table is typed statelessly and split by stratified five-fold cross-validation before any generator is fitted. In fold

k

, fresh generators are fitted on the training rows

T_{k}

alone, the planned synthetic rows

S_{k}

are sampled, and the augmented frame trains the augmented arm while the paired baseline trains on

T_{k}

alone; both arms score the held-out rows

E_{k}

. The fidelity battery compares

S_{k}

with

T_{k}

only.

E_{k}

enters the leakage probes, and nothing else, as a comparison set: rows are hashed under

T_{k}

’s schema, and a synthetic row that matches a held-out-only row invalidates the result. After five folds the out-of-fold scores of each arm are pooled, and

\Delta

is reported both pooled and per fold. Numbered badges mark the five invariants of the leakage contract.

Metric	joint	marginal
SDV quality score	0.8484	0.9527
SDV diagnostic score	0.8797	1.0000
KS complement (column mean)	0.9388	0.9742
TV complement (column mean)	0.0553	0.9821
Pearson correlation similarity (pair mean)	0.9877	0.8882
Classifier two-sample AUC	1.0000	0.8527

Table 5: The fidelity battery exposes the defects of two surrogate generators that the aggregate SDV score misranks. Fidelity on south_german , averaged over five folds. joint : row resampling with jitter (dependence kept, schema violated); marginal : independent column resampling (marginals kept, dependence destroyed).

		logistic regression			GBDT
id	$L_{\text{eff}}$	arrival	refresh	i.i.d.	arrival	refresh	i.i.d.
conorsully	200	0.657	0.645	0.674	0.613	0.632	0.643
south_german	200	0.737	0.751	0.791	0.730	0.733	0.783
gastonstat	890	0.771	0.774	0.777	0.787	0.800	0.815
lc_small	1000	0.658	0.660	0.665	0.618	0.620	0.640
taiwan	1000	0.722	0.723	0.735	0.754	0.755	0.786

Table 6: Reference results: under the arrival-mode stream the GBDT baseline leads logistic regression by less than under the i.i.d. split. ROC AUC per dataset at the reference setting (Table 4 ) for the two package baselines, under arrival mode, under the refresh rule of release 0.3.0 and under the i.i.d. protocol on the same 20,000-row truncation (mean of five seeds).

L_{\text{eff}}

is the effective delay. Datasets are grouped by product family as in Table 3 . Commit 2c13c1b , scikit-learn 1.9.1; per-run records, including the label-budget AUCs and refit counts, are written by run_reference.py in benchmarks/reference .

labels behind the score	AUC	datasets	AUC	datasets
	logistic regression		GBDT
0–99	0.606	17	0.600	17
100–299	0.684	17	0.667	17
300–999	0.702	19	0.717	19
1,000–2,999	0.722	17	0.761	17
3,000–9,999	0.721	16	0.774	16

Table 7: Below 300 labels the two baselines are close, with logistic regression ahead in fleet mean; with more labels GBDT pulls ahead. Fleet mean ROC AUC of the rows scored by models fitted on a given number of resolved labels (arrival mode, reference setting), with the number of datasets that reach each group. Groups differ in which datasets they contain, so the curve is descriptive.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

id	source fetched (snapshot)	target (1 = default)
conorsully	Kaggle: conorsully1/credit-score (v1)	Default on personal loan
south_german	UCI: dataset 573 (sha256)	Bad credit risk
gastonstat	GitHub: gastonstat/CreditScoring (commit 250c644)	Loan default
lc_small	Kaggle: itssuru/loan-data (v1)	Not fully paid
taiwan	Kaggle: uciml/default-of-credit-card-clients-dataset (v1)	Default on next month credit card payment
laotse	Kaggle: laotse/credit-risk-dataset (v1)	Loan default

Appendix

Table 8: Every table is fetched from a named source and rebuilt locally. Provenance, pinned snapshot and target definition of each dataset, as fetched by pamir.download (catalog fields download.locator , version and revision ). Every raw file is also pinned by its SHA-256 digest in expected.json ; the UCI archives carry no version and are pinned by the digest alone. Full recipes, including the day-zero drop lists, are returned by pamir.dataset_info(id) .

Explore similar work

May 18, 2026

Data Presentation Over Architecture: Resampling Strategies for Credit Risk Prediction with Tabular Foundation Models

Credit default prediction is a tabular learning problem with severe class imbalance, heterogeneous features, and tight latency budgets. Tabular Foundation Models (TFMs) approach this problem through in-context learning, which makes their predictions sensitive to how the context window is built. We benchmark four classical models and five TFMs on the Home Credit and Lending Club datasets, varying the context-construction strategy (seven options) and the context size (1K to 50K). On both datasets, the choice of context strategy explains more variance in AUC-ROC than the choice of TFM family: balanced and hybrid sampling add 3 to 4 AUC points over uniform sampling, and the gap exceeds the spread between TFMs. With a balanced context of 5K to 10K examples, the strongest TFMs reach the AUC of classical baselines trained on the full data, while also recovering meaningful default-class recall that default-threshold GBDTs do not. We frame this as evidence that context construction, rather than architecture choice, is the primary deployment lever for TFMs in imbalanced credit-risk settings.

Aditya Tanna, Mitul Solanki, Mohamed Bouadi +3

Lexsi Labs

Class-Imbalanced Learning Tabular Foundation Models AI Risk Management Credit Scoring Tabular ICL

Ask PDF

May 11, 2026

V4FinBench: Benchmarking Tabular Foundation Models, LLMs, and Standard Methods on Corporate Bankruptcy Prediction

Corporate bankruptcy prediction is a high-stakes financial task characterized by severe class imbalance and multi-horizon forecasting demands. Public datasets supporting it remain scarce and small: widely used free benchmarks contain between 6,000 and 80,000 company-year observations, while larger resources are behind subscription paywalls. To address this gap, we introduce V4FinBench, a benchmark of over one million company-year records from the Visegràd Group (V4) economies (2006-2021), with 131 financial and non-financial features, six prediction horizons, and a composite distress criterion jointly capturing solvency, profitability, and liquidity deterioration. V4FinBench is designed to support the evaluation of tabular and foundation-model methods under realistic class imbalance, with positive rates between 0.19% and 0.36%. We provide reference evaluations of standard tabular baselines, finetuned TabPFN, and QLoRA-finetuned Llama-3-8B. With imbalance-aware finetuning, TabPFN matches or exceeds gradient boosting at longer time horizons on both

F_1

-score and ROC-AUC. In contrast, Llama-3-8B trails gradient boosting on ROC-AUC at every horizon and is generally weaker on

F_1

-score, with the gap widening sharply beyond the immediate horizon. In an external evaluation on the American Bankruptcy Dataset, the V4FinBench-finetuned TabPFN checkpoint improves over vanilla TabPFN, suggesting that adaptation captures transferable financial-distress structure rather than only V4-specific patterns. V4FinBench is publicly released to support further evaluation and development of prediction methods on realistic financial data.

Marcin Kostrzewa, Sebastian Tomczak, Roman Furman +5

Department of Artificial Intelligence, Wrocław University of Science and Technology, Poland · Opera, Wrocław, Poland · Tooploox, Wrocław, Poland

Class-Imbalanced Learning Tabular Foundation Models LLM Evaluation Benchmark Design Tabular Prediction Finance

Ask PDF

Aug 30, 2026

When Does a Classifier Help an LLM? Classifier-Guided Prompting and Hybrid Classifier-LLM Models for Credit-Default Prediction

Credit-default prediction is an important task in financial decision making. Traditional methods use fitted classifiers such as logistic regression and random forests on tabular features. Large language models (LLMs) have recently been applied to this task through prompting. In this work we study how a fitted classifier and an LLM can be combined for credit-default prediction. We distinguish telling the LLM to imitate a classifier from using the classifier to build the prompt. We hypothesize that a fitted classifier can supply the ranking ability that an LLM prompt lacks. We experiment on the Default of Credit Card Clients dataset, and report recall, F1, and the area under the ROC and precision-recall curves, with bootstrap confidence intervals. We observe that a few-shot LLM has the highest recall (0.47) and F1 (0.50) of any single model but ranks worse than a random forest (AUC-ROC 0.72 against 0.79). Instructing the LLM to imitate a classifier gives no significant change. Pruning the prompt to the classifier's eight most important features raises recall by 0.071 and F1 by 0.032. Adding the classifier's predicted probability to the prompt raises the LLM's AUC-ROC from 0.72 to 0.78, matching the random forest, while keeping 0.118 higher recall than it. The reverse composition, and the use of several classifiers, do not help. We thus recommend a simple classifier-guided prompt for LLM-based credit prediction.

Rishi Datta, Lavanya Prahallad

Amador Valley High School · Pleasanton, California, USA · Research Spark Hub Inc. +1

LLM Prompting Credit Scoring Tabular Prediction

Ask PDF