Conditional Weight of Evidence for Gradient-Boosted Credit Models: Estimation, Additivity and Its Relation to Adverse Action Reasons
Abstract
Traditional weight of evidence assigns a credit characteristic a value based on its association with default across a population. Credit analysis also requires an assessment of that characteristic in the context of an individual applicant's record. We study conditional weight of evidence for this purpose. It measures the change in default log odds from observing a characteristic after the remaining credit information is known. Its sign identifies whether the characteristic adds adverse or favorable information for that applicant, and its magnitude measures the strength of that information. The same characteristic value can therefore receive different assessments for applicants with different payment histories or other credit attributes. We develop estimators using gradient-boosted trees and evaluate their accuracy in a bureau-style simulation. Regressions of the full model's predicted probabilities recover conditional evidence more accurately than separate models fitted to default outcomes, provided the full model is accurate. They also recover a deliberately constructed file-age effect whose direction varies with utilization and is often missed by marginal weight of evidence. Local class-frequency estimates improve on marginal weight of evidence but remain inaccurate for some weak effects. An application to FICO HELOC data illustrates estimation when the population evidence is unknown. Conditional weight of evidence provides an applicant-specific interpretation of the information in a credit characteristic; its population summary, a conditional information value, splits the joint information in the record across blocks when averaged over orderings. Neither is an adverse action reason, which explains the fitted score difference between the applicant and chosen passing records.
Figures & tables
| Block | Fields | Generation given the factors |
|---|---|---|
| Revolving | limit, balance, utilization | ; ; balance util limit |
| Delinquency | accounts 30+, 60+, 90+ days past due; months since delinquency | ; ; ; months since if |
| File age | oldest and average trade age (months) | ; average oldest |
| Credit seeking | inquiries and new accounts in 6 months | inquiries ; new |
| Installment | original amount, balance, fraction outstanding | ; |
| Public record | indicator |
| Sample | Records | Role |
|---|---|---|
| Development | 100,000 | production model, block-excluded models, geometry models, block classes |
| Calibration | 25,000 | Platt recalibration of full and outcome-fitted reduced models; choice of on 4,000 records |
| Evidence | 100,000 | projection regressions; leaf-local counts; |
| Reference | 50,000 | cutoff (riskiest 20 percent declined) and passing band (0.3 in log odds below the cutoff) |
| Test | 25,000 | 3,000 random applicants and 400 declined applicants at which evidence and attribution are evaluated |
| Revolving | Delinq. | File age | Seeking | Install. | Public rec. | |
|---|---|---|---|---|---|---|
| Mean attribution, production model, score band | 0.42 | 0.85 | 0.02 | 0.15 | 0.03 | 0.05 |
| Mean attribution, true log odds, score band | 0.40 | 0.86 | 0.03 | 0.16 | 0.04 | 0.05 |
| Mean attribution, production model, proximity | 0.46 | 0.74 | 0.03 | 0.15 | 0.03 | 0.06 |
| Leading-reason share, production model | 0.26 | 0.57 | 0.05 | 0.07 | 0.00 | 0.06 |
| Leading-reason share, true log odds | 0.23 | 0.56 | 0.06 | 0.08 | 0.00 | 0.07 |
| Leading-reason share, proximity references | 0.33 | 0.51 | 0.01 | 0.09 | 0.01 | 0.05 |
| Applicant | Score-band references | Proximity references | |
| Revolving utilization | 0.53 | 0.53 | 0.27 |
| Revolving limit ($) | 30,809 | 15,936 | 29,493 |
| Revolving balance ($) | 16,229 | 8,176 | 8,089 |
| Accounts 30+ days past due | 1.00 | 0.58 | 1.00 |
| Accounts 60+ days past due | 1.00 | 0.20 | 0.84 |
| Accounts 90+ days past due | 1.00 | 0.10 | 0.40 |
| Revolving | Delinq. | File age | Seeking | Install. | Public rec. | |
| Oracle sd, pointwise | 0.51 | 0.78 | 0.31 | 0.29 | 0.10 | 0.20 |
| Oracle sd, class-level | 0.46 | 0.68 | 0.29 | 0.26 | 0.09 | 0.20 |
| Zero evidence, pointwise target | 0.51 | 0.79 | 0.31 | 0.29 | 0.10 | 0.20 |
| Direct | 0.20 | 0.21 | 0.19 | 0.17 | 0.15 | 0.16 |
| Projection | 0.17 | 0.18 | 0.13 | 0.13 | 0.09 | 0.10 |
| Zero evidence, class-level target | 0.47 | 0.69 | 0.29 | 0.26 | 0.09 | 0.20 |
| Evidence records | Revolving | Delinq. | File age | Seeking | Install. | Public rec. |
|---|---|---|---|---|---|---|
| 2,500 | 0.62 | 0.32 | 0.41 | 0.36 | 0.63 | 0.06 |
| 10,000 | 0.51 | 0.19 | 0.39 | 0.22 | 0.32 | 0.07 |
| 30,000 | 0.36 | 0.17 | 0.29 | 0.16 | 0.20 | 0.11 |
| 100,000 | 0.22 | 0.14 | 0.23 | 0.15 | 0.16 | 0.08 |
| Global WoE, 100,000 | 0.92 | 0.58 | 0.39 | 0.62 | 0.77 | 0.13 |
| Global WoE, 2,500 | 1.01 | 0.66 | 0.45 | 0.71 | 0.98 | 0.14 |
| Revolving | Delinq. | File age | Seeking | Install. | Public rec. | |
| Coalition evidence , oracle mean | 0.43 | 0.62 | 0.05 | 0.19 | 0.12 | 0.05 |
| Leave-one-block-out , oracle mean | 0.19 | 0.47 | 0.07 | 0.05 | 0.00 | 0.03 |
| Coalition excess , oracle mean | 0.24 | 0.15 | 0.02 | 0.14 | 0.12 | 0.01 |
| RMSE of projection estimate of | 0.09 | 0.18 | 0.07 | 0.08 | 0.05 | 0.06 |
| Oracle sd of | 0.68 | 1.14 | 0.28 | 0.57 | 0.35 | 0.35 |
| , all test applicants | 0.23 | 0.35 | 0.00 | 0.08 | 0.00 | 0.05 |
| Simulation | ||||||
|---|---|---|---|---|---|---|
| Revolving | Delinq. | File age | Seeking | Install. | Public rec. | |
| Marginal IV, oracle | 1.43 | 1.51 | 0.07 | 0.75 | 0.68 | 0.12 |
| Conditional IV , oracle | 0.32 | 0.80 | 0.16 | 0.08 | 0.01 | 0.05 |
| Coalition IV , oracle | 0.74 | 1.04 | 0.11 | 0.32 | 0.24 | 0.07 |
| Conditional IV , projection | 0.31 | 0.79 | 0.14 | 0.08 | 0.01 | 0.04 |
| Coalition IV , projection | 0.72 | 1.04 | 0.10 | 0.30 | 0.23 | 0.06 |
| Delinq. | File age | Trades | Inquiries | Revolving | Install. | |
| Corr(direct, projection) | 0.84 | 0.72 | 0.50 | 0.76 | 0.87 | 0.35 |
| Corr(projection, leaf-local default geometry) | 0.75 | 0.58 | 0.14 | 0.48 | 0.76 | 0.04 |
| Corr(leaf-local default geometry, global WoE) | 0.93 | 0.89 | 0.69 | 0.92 | 0.95 | 0.46 |
| Mean coalition evidence | 0.30 | 0.15 | 0.09 | 0.13 | 0.37 | 0.02 |
| Mean leave-one-block-out | 0.30 | 0.14 | 0.04 | 0.13 | 0.33 | 0.04 |
| Mean attribution | 0.37 | 0.22 | 0.11 | 0.18 | 0.36 | 0.01 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Simulation | HELOC | |
|---|---|---|
| Platt slope | 1.00 to 1.03 | 0.81 to 1.18 |
| Calibration slope before Platt | 1.00 to 1.02 | 0.91 to 1.08 |
| Calibration slope | 0.97 to 1.02 | 0.87 to 1.14 |
| Calibration intercept | 0.03 to 0.02 | 0.19 to 0.08 |
| Expected calibration error (deciles) | 0.002 to 0.005 | 0.021 to 0.047 |
| Mean predicted probability | 0.087 to 0.092 | 0.484 to 0.528 |
| Block | Estimator | Evidence MSE | |||
|---|---|---|---|---|---|
| Revolving | Direct | 0.0477 | 0.0405 | 0.0490 | 0.0392 |
| Projection | 0.0477 | 0.0297 | 0.0490 | 0.0284 | |
| Delinquency | Direct | 0.0477 | 0.0375 | 0.0412 | 0.0440 |
| Projection | 0.0477 | 0.0303 | 0.0439 | 0.0341 | |
| File age | Direct | 0.0477 | 0.0476 | 0.0602 | 0.0351 |
| Projection | 0.0477 | 0.0361 | 0.0662 | 0.0177 |
| Revolving | Delinq. | File age | Seeking | Install. | Public rec. | |
|---|---|---|---|---|---|---|
| Direct, combined | 0.198 | 0.210 | 0.187 | 0.165 | 0.153 | 0.159 |
| Direct, sampled declines | 0.236 | 0.278 | 0.232 | 0.195 | 0.190 | 0.195 |
| Projection, combined | 0.168 | 0.185 | 0.133 | 0.128 | 0.093 | 0.104 |
| Projection, sampled declines | 0.190 | 0.263 | 0.169 | 0.158 | 0.108 | 0.143 |
Credit Scoring
Credit scoring uses borrower and transaction data to estimate repayment risk and support fair, calibrated lending decisions.
See all 40 papers