Patching the Blind Spots: Reversible LLM Rule Overlay for Frozen Fraud Detectors
Organizations: †eBay
Abstract
Gradient-boosted trees dominate industrial tabular fraud classification, but even a well-trained model has blind spots, and retraining a deployed model is not always safe. We bring LLM reasoning to patching such a model without modifying it: LLMs read its tree digest and SHAP attributions, reason about the fraud it misses, and propose rules anchored on features it underweights, screened and calibrated into a reversible log-lift overlay at the production cut-off. Single-LLM benefit is high-variance and inverts between datasets, so we pool three diverse backbones by logit averaging, trading peak gain for a bounded worst case. On an experimental detector trained on sampled, selection-biased real-world e-commerce data, tuned continued training regresses, whereas the overlay gains on average +2.26% recall at equal precision (R@P) and +1.60% precision at equal recall (P@R). Replayed offline on a legacy production detector, the overlay, calibrated on one month and frozen, stays more precise at equal recall for seven months (+5.64% P@R). On seven bias-shift variants of a public benchmark it raises P@R on all and keeps R@P near neutral. LLM reasoning over a model's diagnostics can improve a deployed tabular classifier through readable predicates naming the fraud they cover.
Figures & tables
| Dataset | Shift type | #feat | #trees (LGB / RF) | Blend | Split design |
|---|---|---|---|---|---|
| D-Industrial | selection bias; unbiased eval biased train | 500 | 2,219 / 500 | Temporal, six months: train Oct–Nov 2025 (biased, approved-only); validate Dec 2025 and evaluate Jan 1–Mar 7, 2026, both from the unbiased cohort | |
| BAF | public; designed bias-shift variants | 30 | 892 / 200 | Temporal: train months 0–4; evaluate held-out months 5–7 (OOT-1, OOT-2) plus five designed bias-shift variants (Var-I–V) | |
| Production replay | temporal; unbiased eval, seven months | — | — | Temporal: validate Feb 2026; evaluate Mar–Sep 2026 (unbiased cohort) |
| Anchors (production) | |||||||
|---|---|---|---|---|---|---|---|
| Dataset | Split | Precision | Recall | CoT | SlW | MoE | Ensemble (All 3) |
| D-Industrial | OOT | 30.3–56.5% | 23.5–62.9% | / | / | / | / |
| BAF | OOT-1 | 17.3–53.2% | 4.1–41.4% | / | / | / | / |
| OOT-2 | 19.1–52.8% | 3.9–41.2% | / | / | / | / | |
| Var-I | 18.0–56.2% | 3.1–35.3% | / | / | / | / | |
| Var-II | 14.6–51.8% | 6.4–50.1% | / | / | / | / | |
| Month | avg R@P | avg P@R |
|---|---|---|
| Mar | ||
| Apr | ||
| May | ||
| Jun | ||
| Jul | ||
| Aug |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| CT-100 | CT-Tuned | Retrain | Ours | |||||
|---|---|---|---|---|---|---|---|---|
| Dataset (split) | R@P | P@R | R@P | P@R | R@P | P@R | R@P | P@R |
| D-Industrial OOT | ||||||||
| BAF (7 splits, range) | — | |||||||
| Dataset | Split | Full | PerBand | Ens | Ens PerBand | LogLift | Rules-only |
|---|---|---|---|---|---|---|---|
| avg R@P | |||||||
| D-Industrial | OOT | ||||||
| BAF | OOT-1 | ||||||
| BAF | OOT-2 | ||||||
| BAF | Var-I | ||||||
| BAF | Var-II | ||||||
| Dataset | Split | CoT | SlW | MoE | CoT+SlW | CoT+MoE | SlW+MoE | All 3 |
|---|---|---|---|---|---|---|---|---|
| avg R@P | ||||||||
| D-Industrial | OOT | |||||||
| BAF | OOT-1 | |||||||
| BAF | OOT-2 | |||||||
| BAF | Var-I | |||||||
| BAF | Var-II | |||||||
| Production | Overlay | ||||||
| Split | P% | R% | P% | R% | R@P | P@R | |
| D-Ind. OOT | 998 | 56.48 | 23.47 | 57.51 | 24.37 | ||
| 997 | 49.07 | 35.67 | 50.30 | 38.00 | |||
| 996 | 43.29 | 45.80 | 44.06 | 47.67 | |||
| 995 | 37.99 | 52.41 | 38.73 | 54.00 | |||
| 994 | 33.99 | 58.12 | 34.58 | 59.27 | |||
| Month | 998 | 997 | 996 | 995 | 994 | 993 | Avg |
|---|---|---|---|---|---|---|---|
| R@P | |||||||
| Jan 2026 | |||||||
| Feb 2026 | |||||||
| Mar 2026 | |||||||
| P@R | |||||||
| Jan 2026 | |||||||
Financial Fraud Detection
Financial fraud detection identifies suspicious transactions, accounts, claims, and behavior to prevent losses and guide investigation.
See all 147 papers