FraudBench: Protocol-Sensitive Benchmarking of Adversarial Robustness for Financial Risk Assessment

Xitong Zeng; Zhaoge Bi; Yitian Yang; Huaming Chen; Quan Z. Sheng

cs.LGAug 25, 2026

FraudBench: Protocol-Sensitive Benchmarking of Adversarial Robustness for Financial Risk Assessment

Authors: Xitong Zeng, Zhaoge Bi, Yitian Yang, Huaming Chen, Quan Z. Sheng

Organizations: School of Electrical and Computer Engineering The University of Sydney Sydney, Australia · School of Computing Macquarie University Sydney, Australia

Abstract

Machine learning models are widely used in financial fraud and credit-risk detection, yet their adversarial robustness remains difficult to evaluate because financial tabular data involve domain-specific constraints, severe class imbalance, and asymmetric attacker capability. We argue that, in this setting, robustness is not only an attribute of the model, but also an attribute of the evaluation protocol. Different ways of enforcing constraints and capability can lead to substantially different robustness conclusions. This paper presents FraudBench, a protocol-sensitive benchmark for adversarial robustness evaluation in financial fraud and credit-risk detection. Rather than treating domain constraints as post-hoc validity checks, FraudBench evaluates the same dataset--model--attack--defence setting under three matched protocols: unconstrained attacks, post-hoc feasibility filtering, and deployment-aware constraint-integrated attacks. FraudBench covers four public financial datasets, and evaluates neural, tree-based, and ensemble models using three attack settings. Our results show that robustness conclusions are highly protocol-sensitive. On Lending Club Loan Data under the white-box setting, post-hoc filtering leaves only 3.7 feasible-flipped examples on average, whereas in-attack projection with attacker mutability masking produces 2,832.3 feasible-flipped examples under the same perturbation budget. The results on IEEE-CIS further show that feasibility and attacker capability are separate axes, while black-box evaluation shows that protocol choice can alter model-family rankings. These findings suggest that fraud robustness evaluation should report predictive degradation and attack feasibility jointly, and should incorporate domain constraints into attack generation rather than treating them as post-processing checks.

Figures & tables

Fig. 1: Overview diagram of the FraudBench benchmark.

Axis	Coverage
Datasets	CCFD, IEEE-CIS, LCLD, Sparkov
Models	MLP, XGBoost, heterogeneous ensemble
Attacks	CAPGD, Square Attack, HopSkipJump partial cross-check
Defences	None, adversarial training, input validation, ensemble
Seeds	42, 123, 456
Primary metric	PR-AUC

TABLE I: FraudBench Experimental Coverage

Dataset	Samples	Raw	Proc.	Pos.	Scenario
CCFD	284,807	30	30	0.17%	Card fraud
IEEE-CIS	590,540	392	537	3.5%	Online fraud
LCLD	1,340,968	63	188	20.0%	Loan default
Sparkov	1,296,675	11	75	0.6%	Simulated card fraud

TABLE II: FraudBench Datasets

Dataset	Main Constraints	C Operator	CAPGD	Square	HSJ
CCFD	None/PCA	None	A/B	A/B	Partial
IEEE-CIS	OHE, non-neg.	OHE+mask	A/B/C1/C2	A/B	Partial
LCLD	$g_{1}$ , inequalities, OHE	$g_{1}$ +mask	A/B/C1/C2	A/B	Partial
Sparkov	OHE, ranges	OHE+mask	A/B/C1/C2	A/B	Partial

TABLE III: Protocol Coverage Matrix

Protocol	Constraints in attack	Post-hoc filtering	Interpretation
A	$\times$	$\times$	Unconstrained adversarial robustness
B	$\times$	$\checkmark$	Feasible subset after filtering unconstrained attacks
C	$\checkmark$	$\checkmark$	Constraint-aware attacker success under deployment conditions

TABLE IV: Comparison of Evaluation Protocols

Dataset	Clean PR-AUC	Robust PR-AUC	Drop
CCFD	$0.683\pm 0.189$	$0.598\pm 0.262$	$-12.5\%$
IEEE-CIS	$0.433\pm 0.033$	$0.075\pm 0.013$	$-82.6\%$
LCLD	$0.308\pm 0.003$	$0.105\pm 0.000$	$-65.9\%$
Sparkov	$0.626\pm 0.033$	$0.005\pm 0.000$	$-99.1\%$

TABLE V: Clean and CAPGD-Robust PR-AUC on the Neural Model Without Defence

Fig. 3: Protocol sensitivity under CAPGD at

\epsilon=0.1

on the no-defence MLP. The top panels show filtered success rate (FSR), with the annotated numbers indicating feasible-flipped counts. The bottom panels show robust PR-AUC. On LCLD, Protocol B retains only a few feasible-flipped attacks from the unconstrained attack, while Protocols C1 and C2 generate feasible attacks directly. On IEEE-CIS, C1 increases feasible attack success through one-hot projection, whereas C2 shows the suppressive effect of mutability masking.

Dataset	Prot.	Robust PR-AUC	Agg. Feas.	Feas.-Flip.	FSR
CCFD	A	0.598 $\pm$ 0.262	1.000	0.0	–
CCFD	B	0.598 $\pm$ 0.262	1.000	0.0	–
IEEE-CIS	A	0.075 $\pm$ 0.013	0.000	0.0	0.000
IEEE-CIS	B	0.433 $\pm$ 0.033	1.000	0.0	–
IEEE-CIS	C1	0.079 $\pm$ 0.005	0.453	93.7	0.474
IEEE-CIS	C2	0.412 $\pm$ 0.033	1.000	4.3	1.000

TABLE VI: CAPGD Protocol Comparison at

\epsilon=0.1

with No Defence

Dataset	Model	A PR-AUC	B PR-AUC	A Feas.	A Flip.	Time
CCFD	MLP	0.651 $\pm$ 0.216	0.651 $\pm$ 0.216	1.000	0.0	8.1
CCFD	XGBoost	0.774 $\pm$ 0.223	0.774 $\pm$ 0.223	1.000	0.7	32.1
CCFD	Ensemble	0.728 $\pm$ 0.237	0.728 $\pm$ 0.237	1.000	0.7	53.2
IEEE-CIS	MLP	0.109 $\pm$ 0.024	0.432 $\pm$ 0.034	0.218	192.0	29.5
IEEE-CIS	XGBoost	0.037 $\pm$ 0.024	0.548 $\pm$ 0.033	0.032	109.3	623.4
IEEE-CIS	Ensemble	0.063 $\pm$ 0.016	0.516 $\pm$ 0.025	0.125	213.7	783.2

TABLE VII: Square Attack results across datasets and model families. A denotes unconstrained attack, and B denotes post-hoc filtering.

Fig. 4: Model-family robustness under Square Attack at

\epsilon=0.1

with no defence. Bars show robust PR-AUC under Protocol A (unconstrained attack) and Protocol B (post-hoc filtering), with error bars indicating one standard deviation across seeds. CCFD shows almost identical Protocol-A and Protocol-B results because no semantic constraint catalogue is defined. In contrast, IEEE-CIS, LCLD, and Sparkov show large protocol-dependent shifts, demonstrating that post-hoc filtering can alter not only metric values but also model-family rankings.

Dataset	HopSkipJump Robust PR-AUC
CCFD	$0.002\pm 0.001$
IEEE-CIS	$0.060\pm 0.042$ (2 seeds)
LCLD	$0.185$ (1 seed)
Sparkov	–

TABLE VIII: Partial HopSkipJump Cross-Check on XGBoost. HopSkipJump is used only as a decision-based black-box cross-check on the tree model. Seed counts are shown in parentheses where fewer than three seeds were run.

Dataset	Best A	Best B	Change
CCFD	XGBoost	XGBoost	No top change
IEEE-CIS	MLP	XGBoost	Top–bottom reversal
LCLD	XGBoost	Ensemble	Top change
Sparkov	XGBoost	XGBoost	Lower-rank change

TABLE IX: Protocol-Induced Model-Family Ranking Changes under Square Attack

Dataset	Prot.	None	Adv. Train	Z Input Val.	Ensemble
CCFD	A	$0.598\pm 0.262$	$0.580\pm 0.225$	$0.572\pm 0.300$	$0.794\pm 0.074$
IEEE-CIS	C2	$0.412\pm 0.033$	$0.453\pm 0.023$	$0.367\pm 0.034$	$0.030\pm 0.008$
LCLD	C2	$0.105\pm 0.000$	$0.316\pm 0.004$	$0.105\pm 0.000$	$0.108\pm 0.001$
Sparkov	C2	$0.501\pm 0.033$	$0.577\pm 0.052$	$0.326\pm 0.013$	$0.093\pm 0.015$

TABLE X: Deployment-Aware Defence Robust PR-AUC with Ensemble Comparison

Explore similar work

Sep 9, 2026

Adversarial Training for Tabular Credit Scoring: A Multi-Attack Robustness Evaluation in P2P Lending

Machine learning-based credit scoring is increasingly central to Peer-to-Peer (P2P) lending, yet its resilience to adversarial manipulation, where applicants strategically alter self-reported inputs to secure favourable decisions, remains poorly understood. Most adversarial-robustness evidence comes from image and text domains and evaluates a single attack against a matching defence, offering little guidance on how defences generalise across attack types in tabular credit data. We address this with a systematic train-test robustness benchmark on a large Lending Club subset, spanning three model families (logistic regression, a feed-forward neural network, and a transformer for tabular data) and four attacks confined to applicant-mutable features: Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), Salt-and-Pepper (S&P) noise, and DeepFool, plus a mixed-attack regime. Across a full grid evaluated with stratified cross-validation, adversarial training sharply improves robustness against the attack it is trained on and transfers well within the gradient-based family, but transfers weakly to non-gradient corruption, so single-attack defences overstate real-world resilience. Mixed training delivers the most balanced robustness across heterogeneous attacks while preserving clean-test performance, supporting multi-attack stress testing in credit-model governance.

Gijs A. F. Niewzwaag, Marijn G. S. Veth, Manuele Massei +1

University of Twente, Faculty of Behavioural, Management and Social Sciences, Department of High-Tech Business and Entrepreneurship, AE Enschede, 7500, Netherlands

Adversarial Training Adversarial Attacks Adversarial Robustness Credit Scoring Finance

Ask PDF

Aug 15, 2026

FinFraudBench: A Heterogeneous Graph Benchmark for Financial Fraud Detection

The increasing complexity of digital financial systems has reshaped financial fraud detection from isolated transaction classification into relational risk reasoning over interconnected financial entities. This shift has motivated graph-based fraud detection, where models identify fraudulent nodes by exploiting dependencies among customers, cards, merchants, categories, and locations. However, despite rapid progress in graph-based methods, existing public benchmarks remain misaligned with real-world financial systems in two important aspects. First, they often simplify financial ecosystems into homogeneous or single-node-type multi-relational graphs, failing to preserve the multi-entity and multi-relational nature of financial data. Second, they rarely provide large-scale heterogeneous financial graph datasets with realistic operating conditions such as extreme class imbalance and limited label availability, making it difficult to assess the practical effectiveness of current methods. To address these gaps, we present FinFraudBench, a heterogeneous graph benchmark for financial fraud detection. FinFraudBench contains two heterogeneous graph datasets (CreditCard-Fraud and BankTrans-Fraud) with up to 8.99M nodes and 89.23M directed typed edges. Each dataset preserves six financial entity types, fourteen directed edge types, and natural fraud rates that mirror deployment constraints. With these datasets, we establish a standardized evaluation protocol covering both ranking and imbalance-sensitive classification metrics, and evaluate representative baselines. Extensive experiments yield empirical insights into current methods' limitations and suggest promising avenues for future research. FinFraudBench is available at https://anonymous.4open.science/r/FinFraudBench-B002.

Yixuan Chen, Hongyu Zhan, Jie Sheng +5

HKUST-GZ · Jilin University · Ant Group +1

Financial Fraud Detection Node Classification Heterogeneous GNNs

Ask PDF

Aug 26, 2026

FinRiskAtlas: Decision-Aligned Evaluation of Large Language Models for Financial Risk Review

Deploying large language models for professional financial review requires more than measuring general financial competence: models must perform the specific review operation required by a workflow and determine whether available evidence is sufficient for a defensible decision. Existing financial benchmarks cover knowledge, reasoning, compliance, and professional tasks, but their evaluation units are often organized around datasets or task formulations rather than the decisions that deployed systems support. We introduce FinRiskAtlas, a Chinese-language benchmark that evaluates financial LLMs along two complementary dimensions: operation execution under fixed evidence states and evidence-state control under evolving review conditions. The static benchmark contains 9,742 instances across 53 task families, including 42 Domain Knowledge families and eleven downstream review operations defined by explicit evaluation contracts. FinRisk-Ask extends this framework through offline replay of 680 pre-action states from 104 de-identified professional trajectories, withholding future evidence during inference and using it only to construct expert-verified evidence targets. Across 33 model configurations, operation-level evaluation yields non-redundant rankings (mean pairwise Spearman correlation 0.42 across downstream operations), and knowledge-based shortlisting can incur up to 18.01 points of regret on individual operations. FinRisk-Ask further shows that entering the Ask branch more frequently does not necessarily improve request targeting or end-to-end evidence acquisition. These results show that broad financial capability scores do not fully capture where models are reliable in professional workflows, motivating evaluation units aligned with the decisions and evidence states that deployed systems must support.

Suyang Zhong, Jingzhe Zhu, Qi Xu +5

Ant International · Xiamen University · Shanghai University +1

LLM Evaluation

Ask PDF