cs.LGOct 2, 2026
SavePaMIR: Open Benchmark of Public Credit-Default Datasets
Abstract
We release PaMIR (Public Arrival-ordered Measurement for Inference in Risk), an open benchmark for credit-default prediction when labels are scarce and arrive late. The field's reference benchmark studies use eight datasets each, only two or four of them public. PaMIR brings together 19 public datasets with binary default labels -- 1.24M loans, firms and card accounts from nine countries -- rebuilt from pinned source snapshots by one leakage-audited recipe and never redistributed; to our knowledge it is the one of its kind as of today. Every model is a single function, scored under a repeated i.i.d. split and a label-delayed stream in which each application is scored on arrival, with AUC reported by label budget; fleet means are withheld unless every dataset is scored. A synthetic-data harness tests generated training rows without letting a generator see held-out rows. This report describes release 0.4.0 of this living benchmark.
Figures & tables
Figure 1: PaMIR pools more public credit-default datasets than any prior multi-dataset study. Each bar counts the datasets with a binary default, bankruptcy or financial-distress target that one study uses together; the filled part is public, the hollow part proprietary or restricted. The counting convention and the sources are given in Section 2.1 and Table 1 .
Figure 2: Under the strictest count the 19 datasets reduce to 14 independent sources, still more than the twelve public datasets of the largest prior study. The three Polish sets are one sample observed at three forecasting horizons [ 22 ] and count once; the three tables labelled as LendingClub-derived count once; the synthetic conorsully is dropped. The remaining twelve datasets are counted as they are.
Figure 3: The 19 datasets span five product families, 1k–266k rows, default rates from 3.2% to 40.9% and 10 to 94 features. One row per dataset, in the order of Table 3 ; the columns give the rows of the harmonized table (log scale), the default rate and the number of features, and the grey label gives the country of origin (a dash where the source names none). Short names are abbreviated from the catalog titles; Table 3 gives the package identifiers.
Figure 4: PaMIR ships metadata and code, rebuilds every table locally from a pinned source snapshot, and reports a fleet mean only for a complete, valid run. Top: what the package contains — the catalogue with each dataset’s source, licence, citation, pinned download specification and harmonization specification; the frozen expected.json contract; and the leakage guards run in continuous integration. Middle: pamir.download(id) fetches the raw file from one of four source kinds, applies the six-step recipe, checks the result against expected.json — a mismatch raises ContractError and nothing is cached — and caches the table as Parquet with its provenance record. Bottom: a model is a single fit_fn (a predict_fn is accepted and wrapped), called by both protocols; every call and failure is counted, and fleet_summary returns auc_mean only when every dataset was scored, every table met its contract and the scores were row-independent.
Figure 5: The day-zero cut asks whether a column could have been known when the application was scored; a univariate threshold misses most post-outcome columns. (a) Kept and removed columns of the Bondora loan book on a loan’s timeline. (b) Univariate ROC AUC of the 31 Bondora columns the cut removes: the earlier rule (drop if AUC ) catches one of them, yet the other 30 together predict the outcome perfectly (AUC 1.000, five-fold histogram gradient boosting on 60,000 rows), against 0.768 for the 46 day-zero features. (c) On the SBA loans [ 35 ] the term alone has AUC 0.881, but whether it is a multiple of 12 separates a 3.6% from an 87.7% default rate (indicator AUC 0.932); the recipe drops the term and the two columns derived from it. Measured on the pinned raw snapshots that reproduce expected.json .
Figure 6: The eight leakage guards, their thresholds and their positive controls. Each row gives the check, the leak it was written for, and the test that reconstructs that leak and asserts that the guard fires, together with the case that must not fire. Guards 1–7 read every cached table and run in continuous integration on the seven datasets outside Kaggle; guard 8 runs inside the harmonizer on every download and warns rather than fails.
Figure 7: The audit admitted 19 datasets and rejected seven candidates rather than patching them. Band widths are proportional to the number of datasets. Admitted datasets are shown by product family (as in Table 3 ), rejected ones by the reason for rejection. Candidates screened out earlier by the inclusion criteria of Section 3 are not recorded. Three of the rejected sets — HMEQ, Thomas and Home Credit — are among the twelve public datasets of Baesens et al. [7] .
Figure 8: A dataset enters PaMIR as a recipe, never as a file. A contributor adds a catalogue entry with a pinned snapshot, rebuilds the table locally, runs the test suite on it and freezes the result as the expected.json contract; continuous integration reruns the tests, builds the package and documentation, and the merged change is recorded in the changelog and shipped as a new version. Continuous integration downloads only the seven datasets outside Kaggle, so the data guards on a new Kaggle-hosted dataset run on the contributor’s machine.
| Parameter | Default | Meaning |
|---|---|---|
| mode | arrival | arrival : each row is scored when it arrives; refresh : the rule of release 0.3.0 |
| lag ( ) | 1000 | label-maturation delay, in stream positions |
| max_lag_frac | 0.2 | cap on the delay: ; None disables it |
| k_refit ( ) | 10 | refit after every newly resolved defaults |
| max_n | 20000 | truncate each stream to its first max_n rows |
| min_defaults ( ) | 3 | resolved defaults (and non-defaults) required before the first refit |
Table 4: Streaming-protocol parameters. The defaults form the reference setting.
Figure 9: In arrival mode a refit’s model scores only the applications that arrive after it, so the label delay binds for every row. Schematic stream with rows, delay , warm-up and cadence (package defaults: capped at , , , max_n ). (a) At step the labels of rows , with , are resolved and form the training set; the most recent rows have arrived but not matured. (b) Each refit is fitted on its resolved prefix (line) and scores the rows that arrive after it, up to the next refit (bar); a score, once given, is final. The two strips show which refit owns each row’s evaluated score in arrival mode and, for comparison, in the refresh mode of release 0.3.0, where every refit re-scored all rows from onwards, including rows that had not yet arrived. Row 50 is scored in arrival mode by refit 2, fitted on 26 labels; in refresh mode by refit 3, fitted on 43 labels twelve steps after the row arrived. Rows that arrive before the first refit are never scored and are excluded from the ROC AUC; the last rows are scored, but their labels never resolve inside the stream. (c) Both modes share the trigger: the first refit fires once defaults (and at least non-defaults) have resolved, every later one after newly resolved defaults, and none at the final step.
Figure 10: The two sizings of a synthetic mix answer different questions. One fold with 800 real training rows, synthetic share and two equally weighted generators, drawn to scale. Under the default sizing all real rows are kept and synthetic rows are added; under fixed-total sizing the frame is pinned to rows, of which 560 are synthetic and 240 are a class-stratified subsample of the real rows. The synthetic part is split by the largest-remainder method, with ties broken by generator name.
Figure 11: Inside each cross-validation fold, generators see only the training rows, and the held-out rows are scored and probed but never augmented. The table is typed statelessly and split by stratified five-fold cross-validation before any generator is fitted. In fold , fresh generators are fitted on the training rows alone, the planned synthetic rows are sampled, and the augmented frame trains the augmented arm while the paired baseline trains on alone; both arms score the held-out rows . The fidelity battery compares with only. enters the leakage probes, and nothing else, as a comparison set: rows are hashed under ’s schema, and a synthetic row that matches a held-out-only row invalidates the result. After five folds the out-of-fold scores of each arm are pooled, and is reported both pooled and per fold. Numbered badges mark the five invariants of the leakage contract.
| Metric | joint | marginal |
| SDV quality score | 0.8484 | 0.9527 |
| SDV diagnostic score | 0.8797 | 1.0000 |
| KS complement (column mean) | 0.9388 | 0.9742 |
| TV complement (column mean) | 0.0553 | 0.9821 |
| Pearson correlation similarity (pair mean) | 0.9877 | 0.8882 |
| Classifier two-sample AUC | 1.0000 | 0.8527 |
Table 5: The fidelity battery exposes the defects of two surrogate generators that the aggregate SDV score misranks. Fidelity on south_german , averaged over five folds. joint : row resampling with jitter (dependence kept, schema violated); marginal : independent column resampling (marginals kept, dependence destroyed).
| logistic regression | GBDT | ||||||
| id | arrival | refresh | i.i.d. | arrival | refresh | i.i.d. | |
| conorsully | 200 | 0.657 | 0.645 | 0.674 | 0.613 | 0.632 | 0.643 |
| south_german | 200 | 0.737 | 0.751 | 0.791 | 0.730 | 0.733 | 0.783 |
| gastonstat | 890 | 0.771 | 0.774 | 0.777 | 0.787 | 0.800 | 0.815 |
| lc_small | 1000 | 0.658 | 0.660 | 0.665 | 0.618 | 0.620 | 0.640 |
| taiwan | 1000 | 0.722 | 0.723 | 0.735 | 0.754 | 0.755 | 0.786 |
Table 6: Reference results: under the arrival-mode stream the GBDT baseline leads logistic regression by less than under the i.i.d. split. ROC AUC per dataset at the reference setting (Table 4 ) for the two package baselines, under arrival mode, under the refresh rule of release 0.3.0 and under the i.i.d. protocol on the same 20,000-row truncation (mean of five seeds). is the effective delay. Datasets are grouped by product family as in Table 3 . Commit 2c13c1b , scikit-learn 1.9.1; per-run records, including the label-budget AUCs and refit counts, are written by run_reference.py in benchmarks/reference .
| logistic regression | GBDT | |||
|---|---|---|---|---|
| labels behind the score | AUC | datasets | AUC | datasets |
| 0–99 | 0.606 | 17 | 0.600 | 17 |
| 100–299 | 0.684 | 17 | 0.667 | 17 |
| 300–999 | 0.702 | 19 | 0.717 | 19 |
| 1,000–2,999 | 0.722 | 17 | 0.761 | 17 |
| 3,000–9,999 | 0.721 | 16 | 0.774 | 16 |
Table 7: Below 300 labels the two baselines are close, with logistic regression ahead in fleet mean; with more labels GBDT pulls ahead. Fleet mean ROC AUC of the rows scored by models fitted on a given number of resolved labels (arrival mode, reference setting), with the number of datasets that reach each group. Groups differ in which datasets they contain, so the curve is descriptive.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| id | source fetched (snapshot) | target (1 = default) |
|---|---|---|
| conorsully | Kaggle: conorsully1/credit-score (v1) | Default on personal loan |
| south_german | UCI: dataset 573 (sha256) | Bad credit risk |
| gastonstat | GitHub: gastonstat/CreditScoring (commit 250c644) | Loan default |
| lc_small | Kaggle: itssuru/loan-data (v1) | Not fully paid |
| taiwan | Kaggle: uciml/default-of-credit-card-clients-dataset (v1) | Default on next month credit card payment |
| laotse | Kaggle: laotse/credit-risk-dataset (v1) | Loan default |
Table 8: Every table is fetched from a named source and rebuilt locally. Provenance, pinned snapshot and target definition of each dataset, as fetched by pamir.download (catalog fields download.locator , version and revision ). Every raw file is also pinned by its SHA-256 digest in expected.json ; the UCI archives carry no version and are pinned by the digest alone. Full recipes, including the day-zero drop lists, are returned by pamir.dataset_info(id) .