Could Large Language Models work as Post-hoc Explainability Tools in Credit Risk Models?

Wenxi Geng; Dingyuan Liu; Liya Li; Yiqing Wang

q-fin.RMFeb 21, 2026

Could Large Language Models work as Post-hoc Explainability Tools in Credit Risk Models?

Authors: Wenxi Geng, Dingyuan Liu, Liya Li, Yiqing Wang

Organizations: Citigroup, Irving, TX, United States · University of North Carolina at Chapel Hill, Chapel Hill, NC, United States · Georgia Institute of Technology, Atlanta, GA, United States · Southern Methodist University, Dallas, TX, United States

Abstract

Large language models (LLMs) have shown promise in translating model-based explanations into human-readable narratives. This study evaluates whether LLMs can serve as post-hoc explainability interfaces for credit risk models, focusing on their ability to preserve feature-importance rankings and generate autonomous explanations. Using a LendingClub dataset, we compare LLM outputs with SHAP and coefficient-based attributions on three major LLMs, including GPT-4-turbo, Claude-Sonnet-4.5, and Gemini-2.5-Flash. Results indicate that LLMs reliably reproduce reference rankings under controlled prompts but show limited alignment when generating explanations autonomously. These findings suggest that LLMs are best deployed as narrative interfaces rather than substitutes for formal attribution methods in credit risk governance.

Figures & tables

Figure 1: Overall framework of the proposed LLM-based attribution study

Model	PR-AUC	Macro-averaged F1-score	KS
Logistic	0.27	0.46	28.79
XGBoost	0.38	0.63	40.4

Table 1: Key Metrics of Logistic and XGBoost Classification

Base Model	Total Outputs	Invalid Features	Duplicate Features	Malformed JSON
Logistic Regression	1500	0	0	0
XGBoost	1500	1	0	0

Table 2: Schema-level fidelity check for RQ1 outputs

Prompt Setting	LLM Model	Sample ( $N$ )	Mean (95% CI)	Median (Min, Max)
Zero-shot	GPT-4-turbo	500	0.199 (0.186, 0.212)	0.25 (0.00, 0.50)
Zero-shot	Claude-Sonnet-4.5	500	0.064 (0.054, 0.075)	0.00 (0.00, 0.50)
Zero-shot	Gemini-2.5-Flash	498	0.067 (0.057, 0.076)	0.00 (0.00, 0.25)
Few-shot	GPT-4-turbo	500	0.515 (0.494, 0.529)	0.50 (0.00, 1.00)
Few-shot	Claude-Sonnet-4.5	500	0.415 (0.394, 0.436)	0.50 (0.00, 1.00)
Few-shot	Gemini-2.5-Flash	500	0.182 (0.166, 0.198)	0.25 (0.00, 0.75)

Table 3: RQ2 Top-4 Condition: Overlap@4 for Logistic Regression

Prompt Setting	LLM Model	Sample ( $N$ )	Mean (95% CI)	Median (Min, Max)
Zero-shot	GPT-4-turbo	500	0.214 (0.199, 0.248)	0.25 (0.00, 0.75)
Zero-shot	Claude-Sonnet-4.5	500	0.233 (0.218, 0.248)	0.25 (0.00, 0.75)
Zero-shot	Gemini-2.5-Flash	496	0.102 (0.090, 0.114)	0.00 (0.00, 0.25)
Few-shot	GPT-4-turbo	500	0.383 (0.366, 0.400)	0.25 (0.00, 1.00)
Few-shot	Claude-Sonnet-4.5	500	0.321 (0.304, 0.339)	0.25 (0.00, 1.00)
Few-shot	Gemini-2.5-Flash	500	0.204 (0.188, 0.219)	0.25 (0.00, 0.75)

Table 4: RQ2 Top-4 Condition: Overlap@4 for XGBoost

Base Model	Prompt Setting	LLM Model	Valid Outputs	Full list
Logistic Regression	ZeroShot	GPT-4-turbo	500	500
Logistic Regression	ZeroShot	Claude-Sonnet-4.5	500	500
Logistic Regression	ZeroShot	Gemini-2.5-Flash	499	0
Logistic Regression	FewShot	GPT-4-turbo	500	469
Logistic Regression	FewShot	Claude-Sonnet-4.5	500	500
Logistic Regression	FewShot	Gemini-2.5-Flash	500	0

Table 5: Schema-level fidelity check for RQ2 Full-ranking

Prompt	LLM	$N$	O@5	O@10	O@15	O@20
Zero-shot	GPT-4-turbo	500	0.392 (0.381, 0.403)	0.475 (0.465, 0.486)	0.647 (0.638, 0.655)	0.827 (0.823, 0.832)
Zero-shot	Claude-Sonnet-4.5	500	0.210 (0.197, 0.223)	0.537 (0.528, 0.546)	0.773 (0.766, 0.779)	0.892 (0.889, 0.895)
Few-shot	GPT-4-turbo	469	0.609 (0.594, 0.626)	0.670 (0.663, 0.677)	0.769 (0.764, 0.775)	0.947 (0.944, 0.951)
Few-shot	Claude-Sonnet-4.5	500	0.536 (0.519, 0.552)	0.665 (0.658,0.673)	0.762 (0.756,0.767)	0.904 (0.900,0.907)

Table 6: RQ2 Full-Ranking Condition: Overlap@K for Logistic Regression

Prompt	LLM	$N$	O@5	O@10	O@15	O@20
Zero-shot	GPT-4-turbo	500	0.228 (0.215, 0.241)	0.400 (0.390, 0.410)	0.610 (0.603, 0.616)	0.821 (0.818, 0.824)
Zero-shot	Claude-Sonnet-4.5	500	0.242 (0.230, 0.255)	0.487 (0.478, 0.496)	0.667 (0.661, 0.672)	0.859 (0.857, 0.861)
Few-shot	GPT-4-turbo	494	0.423 (0.409, 0.438)	0.536 (0.527, 0.545)	0.686 (0.680, 0.691)	0.892 (0.889, 0.894)
Few-shot	Claude-Sonnet-4.5	500	0.315 (0.298, 0.331)	0.543 (0.534,0.551)	0.697 (0.691,0.702)	0.887 (0.885,0.890)

Table 7: RQ2 Full-Ranking Condition: Overlap@K for XGBoost

Prompt	LLM	$N$	$\tau$ @5	$\tau$ @10	$\tau$ @15	$\tau$ @20
Zero-shot	GPT-4-turbo	500	0.060 (0.018, 0.104)	0.171 (0.140, 0.201)	0.083 (0.071, 0.095)	0.126 (0.115, 0.136)
Zero-shot	Claude-Sonnet-4.5	500	-0.108 (-0.137, -0.078)	-0.238 (-0.260, -0.216)	-0.113 (-0.125, -0.102)	0.159 (0.150, 0.168)
Few-shot	GPT-4-turbo	469	0.096 (0.051, 0.142)	0.287 (0.266, 0.308)	0.365 (0.352, 0.378)	0.399 (0.388, 0.409)
Few-shot	Claude-Sonnet-4.5	500	-0.144 (-0.185, -0.101)	0.123 (0.100, 0.147)	0.260 (0.247, 0.274)	0.358 (0.348, 0.366)

Table 8: RQ2 Full-Ranking Condition: Kendall’s

\tau

for Logistic Regression

Prompt	LLM	$N$	$\tau$ @5	$\tau$ @10	$\tau$ @15	$\tau$ @20
Zero-shot	GPT-4-turbo	500	0.247 (0.215, 0.279)	0.160 (0.137, 0.183)	0.056 (0.039, 0.071)	0.068 (0.054, 0.081)
Zero-shot	Claude-Sonnet-4.5	500	0.217 (0.182, 0.251)	0.256 (0.238, 0.274)	0.166 (0.153, 0.179)	0.115 (0.103, 0.126)
Few-shot	GPT-4-turbo	494	0.204 (0.170, 0.237)	0.247 (0.228, 0.266)	0.210 (0.196, 0.224)	0.154 (0.144, 0.164)
Few-shot	Claude-Sonnet-4.5	500	0.262 (0.227, 0.298)	0.176 (0.153, 0.199)	0.150 (0.134, 0.166)	0.115 (0.101, 0.129)

Table 9: RQ2 Full-Ranking Condition: Kendall’s

\tau

for XGBoost

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Base Model	LLM Model	N	0 hits	1 hits	2 hits	3 hits
Logistic Regression	GPT-4-turbo	500	155	293	52	0
Logistic Regression	Claude-Sonnet-4.5	500	381	110	9	0
Logistic Regression	Gemini-2.5-Flash	498	366	132	0	0
XGBoost	GPT-4-turbo	500	151	275	69	5
XGBoost	Claude-Sonnet-4.5	500	133	273	89	5
XGBoost	Gemini-2.5-Flash	496	303	184	9	0

Appendix

Table 10: Distribution of Feature-Overlap Counts in the RQ2 Top-4 Condition with Zero-Shot Setting

Base Model	LLM Model	N	0 hits	1 hits	2 hits	3 hits	4 hits
Logistic Regression	GPT-4-turbo	500	13	92	267	115	13
Logistic Regression	Claude-Sonnet-4.5	500	79	104	229	85	3
Logistic Regression	Gemini-2.5-Flash	500	211	217	70	2	0
XGBoost	GPT-4-turbo	500	38	220	183	56	3
XGBoost	Claude-Sonnet-4.5	500	79	237	149	33	2
XGBoost	Gemini-2.5-Flash	500	168	262	65	5	0

Appendix

Table 11: Distribution of Feature-Overlap Counts in the RQ2 Top-4 Condition with Few-Shot Setting

Base Model	Experiment	Condition	System Mean	User Mean	Total Mean
Logistic Regression	RQ1	N/A	136	161	297
Logistic Regression	RQ2	Zero Shot	160	278	438
Logistic Regression	RQ2	Few Shot	204	873	1077
XGBoost	RQ1	N/A	137	161	298
XGBoost	RQ2	Zero Shot	161	279	440
XGBoost	RQ2	Few Shot	205	862	1067

Appendix

Table 12: Token Length Estimate

Explore similar work

Communicating Credit Risk with Large Language Models: Evaluation of Explanations from Standard and Alternative Data-Based Models
Aug 18, 2026Sahab Zandi, Noah Kostesku, Christophe Mues +2Explainable Artificial Intelligence Faithfulness of Language Model Explanations
Department of Statistical and Actuarial Sciences, Western University, London, Canada · Department of Computer Science, Western University, London, Canada · University of Southampton Business School, University of Southampton, Southampton, United Kingdom +3
Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk
Aug 8, 2026Gregorius Reynaldi Pratama, Kuo-Kun TsengFeature Attribution Faithfulness of Language Model Explanations
School of Computer Science and Technology Harbin Institute of Technology, Shenzhen Shenzhen, China
Faithfulness Serum: Mitigating the Faithfulness Gap in Textual Explanations of LLM Decisions via Attribution Guidance
Apr 15, 2026Bar Alon, Itamar Zimerman, Lior WolfFaithfulness of Language Model Explanations LLM Interpretability
Blavatnik School of Computer Science, Tel Aviv University

Aug 18, 2026J/K move · Enter open · S save

Communicating Credit Risk with Large Language Models: Evaluation of Explanations from Standard and Alternative Data-Based Models

Sahab Zandi, Noah Kostesku, Christophe Mues, María Óskarsdóttir+1

Department of Statistical and Actuarial Sciences, Western University, London, Canada · Department of Computer Science, Western University, London, Canada · University of Southampton Business School, University of Southampton, Southampton, United Kingdom+3

Explainable Artificial Intelligence Faithfulness of Language Model Explanations LLM Grounding Credit Scoring

Open paper AskPDF arXiv

Credit decisioning is a high-stakes task in which model outputs must be accurate and explainable to support compliant decisions. Although modern credit risk models such as eXtreme Gradient Boosting (XGBoost) and Graph Neural Networks (GNNs) improve predictive performance, their explanations are often too technical for stakeholders creating communication gaps that can shape approvals, denials, and fairness judgments. We examine whether Large Language Models (LLMs) can serve as explanation layers that translate post-hoc explanation artefacts into stakeholder-appropriate risk narratives. Using Freddie Mac single-family loan-level data, we develop three pipelines: standard tabular (XGBoost + SHAP), and two with alternative data, a pure network-based (GNN + GNNExplainer), and a bimodal one (combining tabular and network data). We generate narratives with three LLM configurations: a small fine-tuned LLM (Gemma 3 4B), a large fine-tuned LLM (DeepSeek R1 70B), and a zero-shot commercial LLM (Gemini 2.5). Explanation quality is evaluated through automated checks across all pipelines and a human study of bimodal explanations comparing credit risk professionals and non-professionals on eight decision-relevant dimensions. We have three main findings. First, the pipeline accounts for higher variance in evidence-grounding scores than the language model, meaning that the binding constraint on explanation quality is the evidence representation, not the model used. Second, the explanation narratives reliably name the influential factors but are less reliable when stating the direction of influence, which may be consequential for adverse-action communication. Finally, professionals apply stricter evidentiary standards than non-professionals. We discuss implications for the governance of risk models, including deployment considerations and the value of domain-aligned LLMs in regulated credit settings.

Abstract

Explore similar work

Communicating Credit Risk with Large Language Models: Evaluation of Explanations from Standard and Alternative Data-Based Models

Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk

Faithfulness Serum: Mitigating the Faithfulness Gap in Textual Explanations of LLM Decisions via Attribution Guidance