Joint Detection of Fraud and Concept Drift inOnline Conversations with LLM-Assisted Judgment

Ali Şenol; Garima Agrawal; Huan Liu

cs.CLMay 7, 2025

Joint Detection of Fraud and Concept Drift inOnline Conversations with LLM-Assisted Judgment

Authors: Ali Şenol, Garima Agrawal, Huan Liu

Organizations: School of Computing and Augmented Intelligence, Arizona State University, Tempe, 85281, AZ, USA. · Department of Computer Engineering, Tarsus University, Takbaş M. Kartaltepe S., Tarsus, 33400, Mersin, Türkiye. · HumaConn AI Consulting, Arizona, USA.

Abstract

Detecting fake interactions in digital communication platforms remains a challenging and insufficiently addressed problem. These interactions may appear as harmless spam or escalate into sophisticated scam attempts, making it difficult to flag malicious intent early. Traditional detection methods often rely on static anomaly detection techniques that fail to adapt to dynamic conversational shifts. One key limitation is the misinterpretation of benign topic transitions referred to as concept drift as fraudulent behavior, leading to either false alarms or missed threats. We propose a two stage detection framework that first identifies suspicious conversations using a tailored ensemble classification model. To improve the reliability of detection, we incorporate a concept drift analysis step using a One Class Drift Detector (OCDD) to isolate conversational shifts within flagged dialogues. When drift is detected, a large language model (LLM) assesses whether the shift indicates fraudulent manipulation or a legitimate topic change. In cases where no drift is found, the behavior is inferred to be spam like. We validate our framework using a dataset of social engineering chat scenarios and demonstrate its practical advantages in improving both accuracy and interpretability for real time fraud detection. To contextualize the trade offs, we compare our modular approach against a Dual LLM baseline that performs detection and judgment using different language models.

Figures & tables

Figure 1: Ensemble model-based framework for detecting fake conversations, identifying concept drift and determining the drift type (genuine concept drift or fraudulent attempt). (a) Phase 1: preprocessing and the majority-vote ensemble of the three best models with their tuned parameters. (b) Phase 2: OCDD on each flagged dialogue and, on a drift, the LLM judge. In the Dual-LLM framework the ensemble in (a) is replaced by the LLM that also acts as the judge in (b). Conversations are processed one complete dialogue at a time; all evaluations in this paper are retrospective.

Panel of Figure 2 — Figure 2: Illustrative comparison of a spam-like and a fraudulent conversation (excerpts in the style of SEConvo). Top: spam-like verbosity without a topic change. Bottom: fraudulent escalation with a semantic drift into requests for sensitive information.

Figure 3: Workflow used to benchmark the candidate algorithms and identify the most suitable ones for the ensemble model.

Panel of Figure 4 — Figure 4: Phase 2 judge prompt.

Panel of Figure 5 — Figure 5: Phase 1 LLM classification prompt (zero-shot; the few-shot variant prepends labelled training examples). The transcript lists each message as [index] speaker: text .

Set	Dialogues	Malicious / benign	Messages per dialogue	Words per message	Fraudulent drift	Genuine drift	No drift
Training	40	24 / 16	11 (8–16)	48	24	9	7
Test	360	191 / 169	11 (3–20)	47	188	95	77
Held-out	1000	500 / 500	11 (7–23)	47	–	–	–

Table 1: SEConvo conversations used in this study.

Model	Endpoint	Open weights	$ per 1M tokens (in / out)	Role
GPT-6 Sol	openai/gpt6_sol	no	2.00 / 10.00	P1, J
Claude 5 Sonnet	aws/claude5_sonnet	no	3.30 / 16.50	P1, J
Gemini 3.1 Pro	gcp-deepmind/geminipro3_1	no	2.00 / 12.00	P1, J
Llama 4 Maverick	aws/llama4_maverick-17b	yes	0.24 / 0.97	P1, J, J ext
gpt-oss-120b	aws/gpt-oss-120b	yes	0.15 / 0.60	P1, J, J ext
Gemma 4 31B (ASU)	asu-air/gemma4_31b_it	yes	0.14 / 0.40	P1, J, J ext

Table 2: Large language models evaluated.

Model	Acc.	Macro-F1 [95% CI]	Few-shot macro-F1	$p_{\mathrm{Holm}}$	$ per 1k conv.
Claude 5 Sonnet	0.833	0.828 [0.788, 0.864]	0.820	1.000	6.49
Gemma 4 31B (ASU)	0.822	0.820 [0.780, 0.857]	0.852	1.000	0.18
GPT-6 Sol	0.819	0.814 [0.773, 0.853]	0.828	1.000	2.70
GPT-6 Luna	0.819	0.812 [0.771, 0.850]	0.821	1.000	0.16
Gemini 3.1 Pro	0.808	0.802 [0.761, 0.842]	0.842	1.000	6.62
Llama 4 Maverick	0.808	0.802 [0.759, 0.841]	0.803	1.000	0.30

Table 3: Phase 1 on the official test split: LLM classifiers (zero-shot unless stated) versus the Joint ensemble.

System	Macro-F1 (mean $\pm$ SD)	$\Delta$ vs Joint	$p_{\mathrm{Holm}}$
Joint ensemble (trans.)	0.857 $\pm$ 0.038	–	–
SVM (trans.)	0.857 $\pm$ 0.036	+0.000	1.000
Random Forest (trans.)	0.855 $\pm$ 0.044	-0.002	1.000
kNN (trans.)	0.842 $\pm$ 0.035	-0.015	1.000
Joint ensemble (ind.)	0.818 $\pm$ 0.044	-0.039	0.979
Claude 5 Sonnet	0.820 $\pm$ 0.042	-0.036	1.000

Table 4: Phase 1 under 5

\times

5 repeated stratified cross-validation (400 annotated conversations).

Model	Inductive acc.	Inductive macro-F1	Transductive acc.	Transductive macro-F1
kNN	0.614	0.520	0.789	0.787
SVM	0.586	0.466	0.822	0.820
Random Forest	0.722	0.718	0.803	0.802
Naive Bayes	0.756	0.742	0.706	0.670
XGBoost	0.689	0.686	0.739	0.738
k-means	0.456	0.313	0.500	0.487

Table 5: Phase 1 candidate models on the official split (40 train / 360 test) with TF-IDF/SVD fitted on the training texts only (inductive) or on all texts without labels (transductive).

System	Prec.	Rec.	Macro-F1 [95% CI]	$p_{\mathrm{Holm}}$
Joint ensemble (TF-IDF trans.)	0.838	0.964	0.888 [0.868, 0.908]	–
SVM (TF-IDF trans.)	0.834	0.946	0.878 [0.858, 0.899]	0.364
SVM (TF-IDF ind.)	0.744	0.990	0.820 [0.795, 0.844]	$<$ 0.001
SVM (te3-large)	0.816	0.906	0.851 [0.828, 0.873]	0.010
Gemma 4 31B (ASU)	0.695	0.930	0.754 [0.727, 0.780]	$<$ 0.001
Llama 4 Scout (ASU)	0.706	0.840	0.743 [0.717, 0.769]	$<$ 0.001

Table 6: Held-out test, Phase 1: trained on the 400 annotated conversations and tested on 1,000 disjoint SEConvo conversations.

Detector	Official balanced	CV detected	CV localized	CV false drift	CV balanced [95% CI]
OCDD, normal-message ref.	0.486	93%	78%	35%	0.715 [0.656, 0.770]
OCDD, dialogue window	0.450	100%	87%	99%	0.443 [0.420, 0.466]
Fixed rule (message 4)	0.448	100%	87%	99%	0.441 [0.418, 0.464]

Table 7: Drift detection against the annotated drift onset. Official: configuration selected on the 40 training dialogues, scored on the 360 test dialogues. CV: 5-fold cross-validation over the 400 annotated dialogues (316 with and 84 without a drift), configuration selected inside each training fold.

Reference	TF-IDF+SVD	te3-large embeddings
Within-dialogue window	0.443 [0.420, 0.466] (87% / 99%)	0.439 [0.417, 0.462] (87% / 99%)
Normal-message reference	0.581 [0.522, 0.635] (52% / 36%)	0.715 [0.656, 0.770] (78% / 35%)

Table 8: Drift detection under 5-fold cross-validation: reference definition

\times

message representation (configurations selected inside each training fold).

Figure 6: Drift detection under 5-fold cross-validation over the 400 annotated dialogues. (a) Position of the drift detected by OCDD (normal-message reference) relative to the annotated first request for personal information. (b) Localization and false-drift rates and balanced score of the three detectors.

Judge LLM	Fraud. drift	Genuine drift	Joint	No OCDD	Dual- LLM	$\Delta$ vs Dual [95% CI]	$/1k Joint	$/1k Dual
Gemma 4 31B (ASU)	99%	0%	0.872	0.874	0.810	+0.062 [+0.020, +0.104]	0.13	0.31
GPT-6 Sol	97%	0%	0.862	0.865	0.794	+0.068 [+0.029, +0.106]	1.32	4.21
Llama 4 Maverick	92%	20%	0.852	0.860	0.791	+0.062 [+0.025, +0.100]	0.52	0.92
GPT-6 Luna	91%	10%	0.842	0.845	0.780	+0.062 [+0.022, +0.103]	0.069	0.24
Llama 4 Scout (ASU)	93%	5%	0.842	0.842	0.749	+0.093 [+0.052, +0.136]	0.099	0.19
gpt-oss-120b	88%	30%	0.837	0.845	0.742	+0.095 [+0.052, +0.138]	0.14	0.41

Table 9: Phase 2 under 5-fold cross-validation over the 400 annotated dialogues: drift-type accuracy of the judge and macro-F1 of the final fraudulent attempt decision for the Joint framework, the Joint framework without OCDD and the Dual-LLM framework.

Figure 7: Macro-F1 of the final fraudulent attempt decision per judge LLM for the Joint framework, the Joint framework without OCDD and the Dual-LLM framework (95% bootstrap CIs). (a) 5-fold cross-validation over the 400 annotated dialogues. (b) Held-out test with the OCDD configuration selected on the annotated dialogues. The dashed line marks the Joint ensemble’s Phase 1 decision.

LLM	LLM acc.	FN rec.	FP rec.	Judge FP out	Judge TP out	Oracle acc.	Held-out FN rec.
Claude 5 Sonnet	0.828	78%	33%	59%	39%	0.943	–
DeepSeek V4 Pro	0.797	74%	19%	–	–	0.930	–
GPT-6 Luna	0.815	78%	22%	26%	10%	0.935	83%
GPT-6 Sol	0.815	65%	26%	15%	5%	0.930	–
Gemini 3.1 Pro	0.807	74%	30%	26%	12%	0.938	–
Gemma 4 31B (ASU)	0.818	61%	52%	11%	2%	0.945	72%

Table 10: Error overlap between the Joint ensemble and each LLM under cross-validation.

Judge LLM	Whole dialogue	+ OCDD location	+ fixed location	Prefix to OCDD alarm	Prefix to message 4
Gemma 4 31B (ASU)	0.872	0.872	0.877	0.870	0.872
Llama 4 Maverick	0.852	0.850	0.865	0.825	0.822
GPT-6 Luna	0.842	0.825	0.825	0.852	0.862
gpt-oss-120b	0.837	0.845	0.860	0.805	0.802
Llama 4 Scout (ASU)	0.842	0.870	0.869	0.774	0.767

Table 11: Judge-input experiment (5-fold CV, Joint framework): macro-F1 of the final fraudulent attempt decision when the judge receives the whole dialogue, the dialogue plus the drift location (from OCDD or the fixed fourth message), or only the messages up to the alarm.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Abbreviations
Term	Meaning
CI	Confidence interval (95%, bootstrap)
CV	Cross-validation
DBSCAN	Density-based spatial clustering of applications with noise
GMM	Gaussian mixture model
kNN	$k$ -nearest neighbours

Appendix

Table 12: Abbreviations and symbols.

Component	Setting
Text representation	TF-IDF on word 1–2-grams, sublinear tf, English stop words, min. document frequency 2, at most 20,000 terms; truncated SVD to 10 components and $\ell_{2}$ normalization. Fitted inductively (training texts) and transductively (all texts, no labels).
OCDD message spaces	TF-IDF + truncated SVD ( $d\in\{5,10,20\}$ ) and PCA-reduced te3-large embeddings ( $d\in\{10,20,50\}$ ) of individual messages, fitted without labels.
OCDD search grid	design $\in$ {normal-message reference, shift from the opening message}; outlier threshold at the $q$ -quantile of held-out normal-message scores, $q\in\{0.5,0.7,0.8,0.9,0.95\}$ ; $(\ell,\rho)\in\{(1,1.0),(2,0.5),(2,1.0)\}$ ; one-class SVM $\nu\in\{0.1,0.5\}$ , RBF kernel, $\gamma$ = scale, standardized inputs; nested 5-fold cross-fitting within the training dialogues.
OCDD selected	normal-message reference (initiator messages before the first PII request in the training dialogues); embeddings, $d=10$ ; $\nu=0.5$ ; $q=0.9$ ; $\ell=2$ , $\rho=0.5$ (selected on all 400 annotated dialogues; identical design in all CV folds); monitored: initiator messages after the opening one; localization tolerance 2 messages.
LLM decoding	temperature 0.0 where accepted (GPT-6 models reject the parameter); maximum output tokens 600 (Claude, Llama, Gemma), 1,000 (Nova), 3,000 (GPT-6, Gemini, gpt-oss), 4,000 (DeepSeek); outputs cut by the limit are re-requested once with a 4 $\times$ budget (cap 32,000).
LLM classification	JSON output (label, confidence, evidence turns, rationale); unparseable outputs counted as benign; few-shot: 4 class-balanced training examples; 1 run per input.

Appendix

Table 13: Experimental configuration (generated from the run’s configuration file).

Model	Fixed settings	Inner-CV search grid	Selected (official split)
kNN	standardized; Euclidean	n_neighbors $\in$ {3, 5, 7, 9}; weights $\in$ {uniform, distance}	n_neighbors=3, weights=distance
SVM	standardized; class-balanced; probability outputs	kernel $\in$ {rbf, linear}; C $\in$ {0.1, 1.0, 10.0}	C=1.0, kernel=rbf
Random Forest	300 trees; class-balanced	max_depth $\in$ {None, 6}; max_features $\in$ {sqrt, 0.3}	max_depth=None, max_features=sqrt
Naive Bayes	Complement NB on TF-IDF counts	alpha $\in$ {0.1, 0.5, 1.0}	alpha=0.1
XGBoost	300 trees; learning rate 0.05; subsample 0.9	max_depth $\in$ {2, 4}; colsample_bytree $\in$ {0.5, 1.0}	colsample_bytree=0.5, max_depth=2
k-means	10 initializations	n_clusters $\in$ {2}	n_clusters=2

Appendix

Table 14: Phase 1 model configurations. Hyper-parameters are selected by 5-fold stratified inner cross-validation on the training conversations (criterion: macro-F1); the last column gives the values selected on the official split with the transductive TF-IDF/SVD representation.

Judge LLM	Joint [95% CI]	No OCDD	Dual-LLM	$p_{\mathrm{Holm}}$
GPT-6 Sol	0.830 [0.789, 0.868]	0.841	0.814	1.000
Llama 4 Maverick	0.822 [0.781, 0.861]	0.833	0.807	1.000
Gemma 4 31B (ASU)	0.816 [0.773, 0.855]	0.827	0.821	1.000
GPT-6 Luna	0.800 [0.756, 0.844]	0.811	0.795	1.000
gpt-oss-120b	0.800 [0.758, 0.839]	0.811	0.752	0.480
Gemini 3.1 Pro	0.792 [0.749, 0.833]	0.803	0.771	1.000

Appendix

Table 15: Phase 2 on the official test split (360 dialogues), with the OCDD configuration selected on the 40 training dialogues.

Online Anomaly Detection

Online anomaly detection identifies unusual events in data streams as they arrive, adapting to changing conditions while balancing detection speed, accuracy, and false alarms.

See all 60 papers

May 12, 2026

PreScam: A Benchmark for Predicting Scam Progression from Early Conversations

Conversational scams, such as romance and investment scams, are emerging as a major form of online fraud. Unlike one-shot scam lures such as fake lottery or unpaid toll messages, they unfold through multi-turn conversations in which scammers gradually manipulate victims using evolving psychological techniques. However, existing research mainly focuses on static scam detection or synthetic scams, leaving open whether language models can understand how real-world scams progress over time. We introduce PreScam, a benchmark for modeling scam progression from early conversations. Built from user-submitted scam reports, PreScam filters and structures 177,989 raw reports into 11,573 conversational scam instances spanning 20 scam categories. Each instance is hierarchically structured according to the scam lifecycle defined by the proposed scam kill chain, and further annotated at the turn level with scammer psychological actions and victim responses. We benchmark models on two tasks: real-time termination prediction, which estimates whether a conversation is approaching the termination stage, and scammer action prediction, which forecasts the scammer's subsequent actions. Results show a clear gap between surface-level fluency and progression modeling: supervised encoders substantially outperform zero-shot LLMs on real-time termination prediction, while next-action prediction remains only moderately successful even for strong LLMs. Taken together, these results show that current models can capture some scam-related cues, yet still struggle to track how risk escalates and how manipulation unfolds across turns.

Weixiang Sun, Shang Ma, Yiyang Li +5

University of Notre Dame · Arizona State University

LLM Evaluation Cybersecurity

Ask PDF

May 1, 2026

A Low-Latency Fraud Detection Layer for Detecting Adversarial Interaction Patterns in LLM-Powered Agents

Large Language Model (LLM)-powered agents demonstrate strong capabilities in autonomous task execution, tool use, and multi-step reasoning. However, their increasing autonomy also introduces a new attack surface: adversarial interactions can manipulate agent behavior through direct prompt injection, indirect content attacks, and multi-turn escalation strategies. Existing defense strategies focus on prompt-level filtering and rule-based guardrails, which are often insufficient when risk emerges gradually across interaction sequences. In this work, we propose a complementary defense mechanism: a low-latency fraud detection layer for detecting adversarial interaction patterns in LLM-powered agents. Instead of determining whether a single prompt is malicious, our approach models risk over interaction trajectories using structured runtime features derived from prompt characteristics, session dynamics, tool usage, execution context, and fraud-inspired signals. The detection layer can be implemented using lightweight models leading to low-latency real-time deployments. To evaluate the framework, we construct a synthetic corpus of 12,000 multi-turn agent interactions generated from parameterized templates that simulate realistic agentic workflows. Using 42 structured features and an XGBoost classifier, our detector achieves over 9 times faster than LLM-based detectors. Through the experiment and ablation studies, our work suggests that interaction-level behavioral detection should become a core component of deployment-time defense for LLM-powered agents.

Sheldon Yu, Yingcheng Sun, Hanqing Guo +1

University of California, San Diego · UNC at Greensboro · Indiana University Bloomington

AI Agent Security LLM Agent Security AI Agent Monitoring Prompt Injection Attacks on AI Agents

Ask PDF

Jun 6, 2026

SAGE: An LLM-driven Self Reflective Agentic Framework for Fraud Detection

Fraud detection in payment, e-commerce, and telecommunications systems requires accuracy at the individual level, robustness under severe class imbalance, and ease of understanding for risk managers. Existing methods fall at least one of these requirements: automated machine learning systems search a fixed numerical space without semantic awareness of the dataset; graph neural network-based methods require pre-defined relational graphs and remain opaque at the individual-decision level; and the design of general-purpose large language model (LLM) agents does not consider the recall and precision constraints specific to real-world fraud detection. In this paper, we propose SAGE, the first end-to-end LLM-driven multi-agent framework for fraud detection. SAGE coordinates three dedicated agents that make decisions based on a six-layer Data Diagnostic Tree (DDT) and a Markov decision process guided by natural-language gradients, automatically optimizing the model under a fraud-specific reward. On five fraud datasets and five LLM backbones, SAGE wins

96.00\%

of method--dataset comparisons and improves F1 by an average of

40.86\%

over baselines. The code is available at https://github.com/yichenC1c/SAGE.

Yichen Chen, Siying Li, Yuhang Liang +2

National University of Singapore · School of Computing, Singapore 117417 · University of Chinese Academy of Sciences +4

Multi-Agent LLM Systems Financial Fraud Detection LLMs for Cybersecurity LLM Agent Training

Ask PDF

Abstract

Online Anomaly Detection

Explore similar work

PreScam: A Benchmark for Predicting Scam Progression from Early Conversations

A Low-Latency Fraud Detection Layer for Detecting Adversarial Interaction Patterns in LLM-Powered Agents

SAGE: An LLM-driven Self Reflective Agentic Framework for Fraud Detection