cs.LGNov 5, 2026

Patching the Blind Spots: Reversible LLM Rule Overlay for Frozen Fraud Detectors

Authors: Bo Qu, Zhurong Wang, Frank Zahradnik, Ming Ouyang

Organizations: †eBay

Abstract

Gradient-boosted trees dominate industrial tabular fraud classification, but even a well-trained model has blind spots, and retraining a deployed model is not always safe. We bring LLM reasoning to patching such a model without modifying it: LLMs read its tree digest and SHAP attributions, reason about the fraud it misses, and propose rules anchored on features it underweights, screened and calibrated into a reversible log-lift overlay at the production cut-off. Single-LLM benefit is high-variance and inverts between datasets, so we pool three diverse backbones by logit averaging, trading peak gain for a bounded worst case. On an experimental detector trained on sampled, selection-biased real-world e-commerce data, tuned continued training regresses, whereas the overlay gains on average +2.26% recall at equal precision (ΔΔR@P) and +1.60% precision at equal recall (ΔΔP@R). Replayed offline on a legacy production detector, the overlay, calibrated on one month and frozen, stays more precise at equal recall for seven months (+5.64% ΔΔP@R). On seven bias-shift variants of a public benchmark it raises ΔΔP@R on all and keeps ΔΔR@P near neutral. LLM reasoning over a model's diagnostics can improve a deployed tabular classifier through readable predicates naming the fraud they cover.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Traceable LLM Reasoning for Fake-Order Fraud Detection

    Jul 25, 2026Siqi You, Bingsong Xu, Zhixian Zheng +4RL for Language Model ReasoningFinancial Fraud Detection