Transfer Learning for Loan Recovery Prediction under Distribution Shifts with Heterogeneous Feature Spaces

Christopher Gerling; Hanqiu Peng; Ying Chen; Stefan Lessmann

q-fin.RMApr 3, 2026

Transfer Learning for Loan Recovery Prediction under Distribution Shifts with Heterogeneous Feature Spaces

Authors: Christopher Gerling, Hanqiu Peng, Ying Chen, Stefan Lessmann

Organizations: Humboldt-Universität zu Berlin, Unter den Linden 6, 10099 Berlin, Germany · National University of Singapore, 10 Lower Kent Ridge Road, Singapore 119076, Singapore

Abstract

Accurate forecasting of recovery rates (RR) is central to credit risk management and regulatory capital determination. In many loan portfolios, however, RR modeling is constrained by data scarcity arising from infrequent default events. Transfer learning (TL) offers a promising avenue to mitigate this challenge by exploiting information from related but richer source domains, yet its effectiveness critically depends on the presence and strength of distributional shifts, and on potential heterogeneity between source and target feature spaces. This paper introduces FT-MDN-Transformer, a mixture-density tabular Transformer architecture specifically designed for TL in RR forecasting across heterogeneous feature sets. The model produces both loan-level point estimates and portfolio-level predictive distributions, thereby supporting a wide range of practical RR forecasting applications. We evaluate the proposed approach in a controlled Monte Carlo simulation that facilitates systematic variation of covariate, conditional, and label shifts, as well as in a real-world transfer setting using the Global Credit Data (GCD) loan dataset as source and a novel bonds dataset as target. Our results show that FT-MDN-Transformer outperforms baseline models when target-domain data are limited, with particularly pronounced gains under covariate and conditional shifts, while label shift remains challenging. We also observe its probabilistic forecasts to closely track empirical recovery distributions, providing richer information than conventional point-prediction metrics alone. Overall, the findings highlight the potential of distribution-aware TL architectures to improve RR forecasting in data-scarce credit portfolios and offer practical insights for risk managers operating under heterogeneous data environments.

Figures & tables

Figure 1: Architecture of the FT–MDN–Transformer with distributional RR predictions.

Panel of Figure 2 — Figure 2: Two-stage schema adaptation comparing shared-only and full-source pretraining followed by expansion to the target schema.

Panel of Figure 3 — Figure 3: Evaluation scenarios used throughout, including zero-shot, target baseline, and transfer learning.

	GCD (Source)	UP5 (Target)
Observation window	1983–2023	1996–2023
Sample size ( $N$ ; raw / final)	331,887 / 14,050	2,170 / 1,725
Instrument type	Secured loans	Unsecured bonds
Observation unit	Loan facility	Bond issuance
Geographic coverage	US and Europe	Predominantly US
Features (pre-dummy; raw / final)	147 / 73	256 / 164

Table 1: Core sample characteristics of the two domains (raw and final after cleaning).

Panel of Figure 4 — Figure 4: Comparison of recovery rate distributions and feature space overlap.

	GCD	UP5	$\Delta$ (Target $-$ Source)
RR mean	0.72	0.48	-0.24
RR standard deviation	0.36	0.33	-0.03
RR skewness	-0.85	0.33	+1.18

Table 2: Key distributional gaps between source and target domains.

Panel of Figure 5 — Figure 5: Correlation heatmaps (31 $\times$ 31) of common macro-financial features in GCD and UP5.

Panel of Figure 6 — Figure 6: Single-loan recovery distributions predicted by FT–MDN–T on UP5. Mixture components capture bimodality that is lost when only the mean is considered.

Figure 7: UP5 portfolio-level recovery distributions and FT–MDN–T based density estimate.

Figure 8: Categorical embeddings versus dummy encoding on UP5. FT–MDN–T benefits from native embeddings, unlike tree-based baselines.

Figure 9: Validation

R^{2}

on UP5 when expanding from 37 to 317 features. Pretraining accelerates fine-tuning and surpasses the scratch baseline.

Panel of Figure 10 — Figure 10: Cross-portfolio transfer from GCD to UP5. Left: $R^{2}$ . Right: MAE. TL shared $\rightarrow$ full shows strong results, followed by TL full $\rightarrow$ full.

Type	Functional Form	Example / Comments
Linear	$\displaystyle X_{j}=\beta_{j}R+\text{intercept}_{j}+\epsilon_{j}$	Captures the proportional effect of recovery. Additive noise $\epsilon_{j}$ and skew can be configured.
Nonlinear	$\displaystyle X_{j}=\beta_{j}f(R)+\epsilon_{j}$	Supports polynomial, sine, exponential, and sigmoid functions. Reflects non-monotonic relationships.
Custom	User-defined function $X_{j}=g(R)+\epsilon_{j}$	Arbitrary Python expressions, e.g., lambda rr: np.square(rr) .
Interaction	$\displaystyle X_{k}=f(X_{a},X_{b})$	Feature constructed from other features via multiplication or addition. Reflects derived ratios and indices.

Table 3: Common feature relationships for synthetic data generation.

Panel of Figure 11 — Figure 11: Monte Carlo simulation workflow from configuration to aggregated results.

Figure 12: Transfer versus target-only performance across heterogeneity regimes. FT–MDN–T outperforms even with missing features, while XGBoost often fails under schema mismatch.

Figure 13:

R^{2}

degradation with increasing shift intensity at 100 target samples. FT–MDN–T is robust under covariate and conditional shifts but deteriorates under label shift, where tree- and MLP-based models fail sharply.

Figure 14: Effect of target sample size across heterogeneity regimes. Transfer accelerates learning, particularly when target data are scarce.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Architecture	Transfer Mechanism
MLP	Two hidden layers (GELU), tuned learning rate and batch size	Pretrain on shared source features; fine-tune on full target schema
XGBoost	Gradient-boosted trees with tuned depth/regularization	Sequential pretraining on shared features; extend to target-only features
FT–Reg (ablation)	FT–Transformer backbone with regression head	Same schema-masked transfer protocol as main model
FT–MDN–Transformer	FT–Transformer backbone + MDN head	Two-stage pretrain $\rightarrow$ fine-tune with schema masking

Appendix

Table 4: Baseline models with architectural specifications and transfer mechanisms.

Cross-Domain Transfer Learning

Cross-domain transfer learning reuses knowledge across datasets, populations, devices, or tasks when their distributions and feature spaces differ.

See all 203 papers

Learning the Pareto Frontier of Predictive Models under Distribution Shift
Aug 1, 2026Yiming Dong, Jiwei Zhao, Yang Young LuDistribution Shift Robustness Vision Foundation Model Adaptation
Department of Biostatistics & Medical Informatics, University of Wisconsin–Madison · Department of Statistics, University of Wisconsin–Madison · Department of Biomedical Engineering, University of Wisconsin–Madison
When is Fine-Tuning Scalable? Certified Recovery from Noisy Data and Linear-Cost Fine-Tuning on Structured Task Shifts
Aug 31, 2025Anastasis Kratsios, Tin Sum Cheng, Bum Jun Kim +1Fine-Tuning OOD Generalization
Department of Mathematics, McMaster University, Canada · Department of Mathematics and Computer Science, University of Basel, Switzerland · Department of Statistical Sciences, University of Toronto, Canada +1
Inferential Evaluation of Surrogate-Derived Models under Covariate Shift
Aug 16, 2026Longtian Shi, Molei Liu, Doudou ZhouDistribution Shift Robustness Covariate Shift
Department of Statistics and Data Science, National University of Singapore · Department of Biostatistics, Peking University Health Science Center, Peking University, Beijing, China · Beijing International Center for Mathematical Research, Peking University, Beijing, China

Aug 1, 2026J/K move · Enter open · S save

Learning the Pareto Frontier of Predictive Models under Distribution Shift

Yiming Dong, Jiwei Zhao, Yang Young Lu

Department of Biostatistics & Medical Informatics, University of Wisconsin–Madison · Department of Statistics, University of Wisconsin–Madison · Department of Biomedical Engineering, University of Wisconsin–Madison

Distribution Shift Robustness Vision Foundation Model Adaptation Cross-Domain Transfer Learning

Open paper AskPDF arXiv

Modern machine learning pipelines increasingly rely on reusing pretrained and foundation models across downstream tasks. These pretrained models can differ not only in performance but also in how they can be used: some only provide black-box predictions, while others may permit white-box access to internal representations that can be probed or fine-tuned. When deployed to the target domain in the presence of distribution shift, no single strategy, including zero-shot application, fine-tuning, or directly training a target-specific model, is uniformly the best. In this work, we propose Frontier Learning, a framework that treats a library of candidate models spanning different training histories and access regimes as complementary sources of information rather than mutually exclusive alternatives. Frontier Learning constructs a unified target-domain feature by concatenating internal representations from white-box candidates as well as prediction outputs from black-box candidates, then fits a lightweight, regularized supervised learner on this concatenated representation using labeled target data. Because the resulting hypothesis class contains predictors obtained by zero-shot reuse, fine-tuning, and direct training as special cases, empirical risk minimization over the frontier learner is guaranteed to be no worse, on the training sample, than any individual baseline. We evaluate the framework in simulations spanning varying degrees of source-target compatibility and in two real-world distribution-shift settings: visual domain adaptation on DomainNet/VisDA and clinical mortality prediction across intensive care unit domains using MIMIC-IV-Notes. Across all settings, Frontier Learning matches or outperforms the strongest individual reuse strategy, with the largest gains arising precisely when no single baseline is reliable across the range of shift considered.

Abstract

Cross-Domain Transfer Learning

Explore similar work

Learning the Pareto Frontier of Predictive Models under Distribution Shift

When is Fine-Tuning Scalable? Certified Recovery from Noisy Data and Linear-Cost Fine-Tuning on Structured Task Shifts

Inferential Evaluation of Surrogate-Derived Models under Covariate Shift