Ranking Prior Alignment for Credit Risk Modeling: When Do External Priors Matter?

Qiye Lu; Jiang Ji; Liang Zhang

cs.LGOct 8, 2026

Ranking Prior Alignment for Credit Risk Modeling: When Do External Priors Matter?

Authors: Qiye Lu, Jiang Ji, Liang Zhang

Organizations: Ant International China

Abstract

Cold-start credit scoring -- deploying models with scarce labeled data, weak features, or minimal capacity -- is a recurring problem in financial machine learning. When a new lending product launches, labeled default data is scarce, feature pipelines are immature, and models must be deployed with minimal capacity to avoid overfitting. Standard defenses operate on the same limited data; what is needed is a source of external regularization grounded in domain knowledge. We propose Ranking Prior Alignment, a model-agnostic framework that distills external ranking priors (from domain experts, teacher models, or LLMs) into any scoring model via a temperature-scaled KL divergence loss. The framework unifies neural (MIL attention) and tree-based (XGBoost custom objective) architectures through a single formulation: L = L_task + gamma(t) * KL(P_agent || P_model), where gamma(t) follows an exponential decay schedule. The method requires no external model at inference, and its tree-based instantiation tolerates annotation noise up to eta = 0.5. On an industrial dataset of over 1.5M merchants, MIL alignment achieves 7/7 positive evaluation cells at 3K bags (1 ID + 3 OOT + 3 degradation metrics; peak Delta AUC = +0.020 on OOT-1), and XGBoost ablation achieves 9/9 positive metrics at 300 bags. Cross-dataset validation on public Amex shows 5/5 positive folds (avg Delta AUC = +0.041). Four model families (MIL, XGBoost, LightGBM, Logistic Regression) and four teacher architectures show that the framework is both model-agnostic and prior-source-independent. We further observe that alignment gains exhibit an inverse-scaling pattern: benefits grow as data abundance N, model capacity C, and feature quality Q decrease, helping practitioners decide when to invest in prior annotation.

Figures & tables

Figure 1. Prior-guided learning is most beneficial under information scarcity. Empirical inverse-scaling relationship:

\Delta\text{AUC}\propto\pi/(N\cdot\mathcal{C}\cdot\mathcal{Q})

(Eq. 3 ). (a) Gains diminish with increasing sample size

N

; (b) diminish with model capacity

\mathcal{C}

; (c) grow with prior quality

\pi

. The largest improvements occur in low-information regimes. Three empirical panels showing alignment gain shrinks with sample size (a) and capacity (b), but scales with prior quality (c).

Method	Align	LLM Inf.	Agnostic	Inv. Scal.	OOT Rob.
TabLLM	Label	Yes	No	No	No
LLM-Boost	Feature	No	Partial	No	No
ForestLLM	Feature	No	Partial	No	No
LAAT	Attrib.	No	Partial	No	No
Ours	Rank	No	Yes	Yes (3D)	Yes

Table 1. Comparison with LLM-Guided Tabular Methods

Figure 2. Ranking Prior Alignment Framework. (A) Construct a ranking prior from any source and convert scores to a probability distribution. (B) Align the model’s ranking distribution with the prior via KL divergence (cross-sample), applicable to MIL attention and tree-based models. (C) Two-phase training with a decaying

\gamma

schedule: strong early alignment followed by gradual release. Detailed architecture diagram showing three phases of the Ranking Prior Alignment framework. Phase A: Prior Construction with LLM Agent, Domain Expert, or Teacher Model producing a validated prior distribution. Phase B: Model-Agnostic Ranking Alignment with MIL attention alignment and XGBoost ranking alignment tracks converging at a cross-sample KL divergence loss. Phase C: Release-the-Prior Training with exponential gamma decay schedule, showing inference requires no prior.

Figure 3. Framework instantiation for two architectures. Both map model outputs to a ranking distribution and minimize the same KL divergence against a shared prior

P_{\mathrm{agent}}

, differing only in how the model-side ranking is computed: MIL aggregates instance-level attention weights to bag-level rankings, while XGBoost ranks batch-level tree predictions directly. Side-by-side comparison of MIL and XGBoost instantiations of ranking alignment. Both map model outputs to a ranking distribution and minimize the same KL divergence against a shared prior distribution, differing only in how the model-side ranking is computed.

Setting	N Bags	Distribution	$\Delta$
93K IID	93,789	IID	+0.0002
3K IID	3,000	IID	+0.0027
10K OOT-1	10,000	OOT (3mo gap)	+0.0018
3K ID (OOF)	3,000	Single-month	+0.0083
3K OOT-1	3,000	OOT (3mo gap)	+0.0202
3K OOT-2	3,000	OOT (9mo gap)	+0.0087

Table 2. MIL Performance under Varying Sample Size (Reference Attention-MIL Baseline). The 3K block is the complete 7-cell evaluation: 1 ID + 3 OOT

\Delta

AUC cells plus 3 degradation-reduction cells (

\delta_{\mathrm{BL}}-\delta_{\mathrm{AL}}

, positive = aligned degrades less) — all 7 in Aligned’s favor.

N Bags	Model	d/n	$\gamma_{\max}$ / $\tau$	BL Test	AL Test	$\Delta$ Test
100	ultra-weak	2/15	1.5/0.3	0.7626	0.8119	+0.049
200	ultra-weak	2/15	1.5/0.3	0.8368	0.8398	+0.003
200	weak	2/20	1.0/0.5	0.8540	0.8678	+0.014
200	std-weak	3/30	0.8/1.0	0.8787	0.8806	+0.002
300	ultra-weak	2/15	1.5/0.3	0.8586	0.8504	$-$ 0.008
500	ultra-weak	2/15	1.5/0.3	0.8599	0.8617	+0.002

Table 3. Data Scarcity and Model Strength (Amex, COLD_START features, teacher prior)

Config	N	Depth	$n$ _est	$\Delta$ OOT	Wins	Pattern
strong_3k	3,000	6	200	+0.0032	2/3	Marginal
weak_3k	3,000	3	30	$-$ 0.0129	1/3	Harmful
weakest_300	300	3	30	+0.0134	3/3	Strong

Table 4. Performance across Model Capacities (Seller_loan)

Family	Strength	$\gamma_{\max}$ / $\tau$	BL	AL	$\Delta$
XGBoost	Ultra-weak (d2/n15)	1.5/0.3	0.832	0.840	+0.008
	Weak (d3/n30)	0.8/0.5	0.886	0.887	+0.001
	Strong (d6/n200)	0.3/1.0	0.883	0.881	$-$ 0.001
LightGBM	Ultra-weak (d2/n15)	1.5/0.3	0.840	0.859	+0.019
	Weak (d3/n30)	0.8/0.5	0.886	0.881	$-$ 0.005
	Strong (d6/n200)	0.3/1.0	0.881	0.869	$-$ 0.012

Table 5. Cross-Model-Family Alignment (Amex, 300 bags, teacher prior)

Feature Set	# Feats	Indiv AUC	BL Test	AL Test	$\Delta$ Test
WEAKER	14	0.52–0.59	0.7349	0.7303	$-$ 0.005
COLD_START	10	0.63–0.70	0.8368	0.8398	+0.003
STRONGER	22	0.75–0.84	0.8627	0.8614	$-$ 0.001

Table 6. Feature Quality Gradient (Amex, 200 bags, teacher prior)

Prior	$\pi$ (Spearman $\rho$ )	XGB $\Delta$ AUC	LGB $\Delta$ AUC
Full teacher	0.665	+0.059	+0.034
Weak teacher	0.663	+0.058	+0.035
Ultra-weak teacher	0.639	+0.047	+0.035
Teacher + 30% noise	0.637	+0.054	+0.036
Teacher + 60% noise	0.557	+0.046	+0.035
Random	0.005	+0.007	+0.025

Table 7. Prior Quality Gradient (100 bags, ultra-weak models, COLD_START, 3 seeds)

Teacher	$\pi$	XGB $\Delta$	LGB $\Delta$	$\Delta/\pi$
XGBoost	0.663	+0.029	+0.028	0.044
Rand. Forest	0.651	+0.022	+0.029	0.034
MLP	0.658	+0.025	+0.029	0.038
Logistic Reg.	0.663	+0.029	+0.028	0.044

Table 8. Prior Source Independence (Amex, 100 bags, 3 seeds)

$N$	BL AUC	AL AUC	$\Delta$ AUC
30	0.553	0.644	+0.091
50	0.582	0.638	+0.057
80	0.601	0.629	+0.027
120	0.617	0.628	+0.011
200	0.628	0.628	+0.000
500	0.642	0.640	$-$ 0.002

Table 9. Logistic Regression:

N

-Scan at the Ultra-Weak Ridge

C=0.001

Q_{5}

Features

Metric	Baseline	Aligned	$\Delta$
ID AUC (5-fold)	0.7412	0.7517	+0.0105
Fold 1	0.8178	0.8304	+0.0126
Fold 2	0.7483	0.7691	+0.0208
Fold 3	0.7060	0.7073	+0.0013
Fold 4	0.7985	0.8296	+0.0311
Fold 5	0.6548	0.6652	+0.0104

Table 10. XGBoost All-Positive Pattern — 300 Bags, Weakened Model (exp_decay

\gamma

: 0.8

\to

0.05)

Schedule	$\gamma$ Range	ID	OOT-1	OOT-2	OOT-3	Wins
Baseline	$\gamma\equiv 0$	0.7412	0.8006	0.7517	0.6986	—
Fixed	$\gamma\equiv 0.3$	0.7531	0.8076	0.7484	0.6840	1/3
Cosine warmup	$0\to 0.3$	0.7522	0.8074	0.7521	0.6805	1/3
Linear decay	$0.5\to 0.1$	0.7470	0.8068	0.7692	0.7080	2/3
Exp decay	$\mathbf{0.8\to 0.05}$	0.7517	0.8075	0.7722	0.7112	3/3
Step decay	$0.5\to 0.1$	0.7514	0.8085	0.7716	0.7091	2/3

Table 11.

\gamma

Schedule Comparison — XGBoost, 300 Bags

$\eta$	OOT-1 $\Delta$	OOT-2 $\Delta$	OOT-3 $\Delta$	OOT Wins
0.0	+0.0069	+0.0205	+0.0126	3/3
0.1	+0.0070	+0.0205	+0.0126	3/3
0.2	+0.0070	+0.0205	+0.0126	3/3
0.3	+0.0070	+0.0205	+0.0125	3/3
0.4	+0.0047	+0.0183	+0.0131	3/3
0.5	+0.0045	+0.0183	+0.0133	3/3

Table 12. XGBoost Noise Tolerance —

\eta

Sweep (exp_decay, 300 Bags)

Figure 4. Robustness checks and ablation studies. Alignment remains beneficial across all prior noise levels

\eta\leq 0.5

for tree-based models, demonstrating tolerance to imperfect priors. Line plot showing

\Delta

AUC across prior noise levels

\eta\in[0, 0.5]

, all three OOT periods maintaining positive gains for XGBoost.

	100 bags		300 bags
$\tau$	XGB $\Delta$	LGB $\Delta$	XGB $\Delta$	LGB $\Delta$
0.1	+0.017	+0.027	+0.003	+0.024
0.3	+0.029	+0.029	+0.013	+0.016
0.5	+0.035	+0.022	+0.009	+0.010
1.0	+0.029	+0.013	+0.007	+0.011
2.0	+0.003	+0.009	+0.008	+0.011

Table 13. Temperature

\tau

Sensitivity (ultra-weak models, teacher prior, COLD_START)

Term	LR-only (n=194)	Pooled (n=216)
Intercept	$-$ 2.443***	$-$ 2.428***
$\log N$	$-$ 0.067***	$-$ 0.072***
$\log\mathcal{C}$	$-$ 0.059***	$-$ 0.060***
$\log\mathcal{Q}$	+0.213 (n.s.)	+0.214 (n.s.)
$\log\pi$	—	+0.064 (n.s.)
Family FE	—	All sig.

Table 14. Log-Linear Scaling Regression (

k=0.06

; 95% CI in brackets; ***

p<0.001

)

Figure 5. Why does prior-guided learning work? A conceptual synthesis. When information is scarce (low

N

\mathcal{C}

\mathcal{Q}

), the prior

P_{\mathrm{agent}}

becomes the dominant signal; a high-quality prior (

\pi

large) tightens

\mathrm{KL}(Q^{*}\|P)

and produces alignment gain. This chain supports the unified empirical pattern:

\Delta\text{AUC}\propto\pi/(N\cdot\mathcal{C}\cdot\mathcal{Q})

(Eq. 3 ). Conceptual diagram showing the causal chain: information scarcity (low N, C, Q) creates the need for priors; when prior quality is high, the prior dominates the likelihood and produces alignment gain. The unified empirical pattern connects all three axes.

Explore similar work

Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk
Aug 8, 2026Gregorius Reynaldi Pratama, Kuo-Kun TsengFeature Attribution Faithfulness of Language Model Explanations
School of Computer Science and Technology Harbin Institute of Technology, Shenzhen Shenzhen, China
Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints for Reliable Decision-Making
Jan 30, 2026Ruoyu Chen, Shangquan Sun, Xiaoqing Guo +8Shortcut Learning
1Michigan State University · 2Nanyang Technological University · 3Hong Kong Baptist University +5
Privacy-Preserving Credit Risk Prediction with Alternative Data
Jun 9, 2026Hongzhe Zhang, Jiarong Xu, Jing He +1AI Risk Management Privacy-Preserving ML
School of Management and Economics, The Chinese University of Hong Kong, Shenzhen (CUHK-Shenzhen), China · School of Management, Fudan University, Shanghai, China · Lerner College of Business and Economics, University of Delaware, Newark, Delaware

Aug 8, 2026, cs.LGJ/K move · Enter open · S save

Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk

Gregorius Reynaldi Pratama, Kuo-Kun Tseng

School of Computer Science and Technology Harbin Institute of Technology, Shenzhen Shenzhen, China

Feature Attribution Faithfulness of Language Model Explanations Ensemble Learning LLM Interpretability Credit Scoring Explainability Evaluation

Open paper AskPDF arXiv

Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters, in tension with supervisory expectations that adverse decisions be explainable. A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale. We build such a system end to end and test whether the second half of the promise holds. The predictive component is a multi-scale stacking ensemble fusing four differently regularised gradient-boosting learners with a residual network through a neural meta-learner trained on out-of-fold predictions. On a public 32,581-application credit dataset it reaches test ROC-AUC 0.9539 (95% CI [0.9462, 0.9616]) and PR-AUC 0.9137, beating the best single model by Delta-AUC = 0.0143 (p = 0.016 under a conservative independence assumption). Our central finding is asymmetric. The ranking gain is real but operationally small: at the F1-optimal threshold the ensemble avoids only six additional missed defaults out of 1,422 against a tuned random forest, cutting cost-weighted loss by under 2%. The narrative layer fails in a way prompt engineering alone does not fix. In an audited case the model named three factors as risk-increasing that the supplied attributions scored as risk-reducing, omitted the dominant driver, and introduced a feature never given to it. We trace this to properties we measure rather than assume: SHAP and LIME agree on which features matter (overlap@10 = 0.80) but not on their order (tau = 0.43, p = 0.18), and the attribution sign for the model's most sensitive input is near a coin flip across applicants (modal-sign share 0.53). Calibration (ECS = 0.117) and perturbation stability (DPD = 0.078) both fall short of our own thresholds. Constrained prompting is necessary but not sufficient: grounding must be verified after generation, not assumed.

Abstract

Explore similar work

Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk

Where Not to Learn: Prior-Aligned Training with Subset-based Attribution Constraints for Reliable Decision-Making

Privacy-Preserving Credit Risk Prediction with Alternative Data