Detecting online social engineering (SE) scams often requires examining website workflows, verifying claims against external sources, and connecting evidence across multiple pages. Existing detectors provide limited support for these investigations, leaving analysts to manually gather and validate evidence, which inevitably delays detection and subsequent takedown of these attacks. To address this gap, we present ReDence, an open-source hybrid framework for adaptive, evidence-grounded detection across diverse SE scam categories. ReDence combines two locally hosted models with complementary responsibilities: a fine-tuned multimodal agentic Investigator (Qwen3.5-4B) that performs investigative planning, evidence interpretation, and reporting, and a specialized Decider (Tev1) that handles binary and multiple-choice tasks, including browser-action selection, evidence assessment, and decisions about whether to continue an investigation. By delegating discrete decisions to a specialized model, ReDence reduces reliance on lengthy language-model responses, while preserving the reasoning capabilities required for complex investigations. Our evaluation demonstrates that ReDence achieves 96% accuracy in detecting scams across twelve SE scam categories, substantially outperforming existing detectors while being at least four times faster than comparable agentic models without compromising investigative depth. ReDence also produces structured, evidence-grounded reports that document investigative actions, supporting and contradictory evidence, and collected artifacts, which allow analysts to independently audit detection decisions and utilize the collected evidence to support subsequent mitigation and takedown efforts.
Figures & tables
Fig. 1 : Our framework for developing and evaluating ReDence
SE Scam Category
Ground Truth
Test Set
Ground-truth Categories
Online Purchase
587
637
Counterfeit Products
109
203
Advance-Fee Loans
30
51
Investment
26
59
Debt Collection
37
53
TABLE I: Distribution of websites across the ground-truth and test datasets. Held-out categories are included only in the test set.
Category
Components
Navigation
Header, footer, menus, links, and redirects
Identity
Branding, logos, and claimed organization
Content
Main text, offers, representations, products, and services
User interaction
Forms, requested information, buttons, registration, login, and account access
Organization details
Contact information, policies, legal pages, and company information
Transactions
Payment methods and payment-related claims
TABLE II : Website components explicitly considered in our EXP prompt component.
Fig. 2 : Overview of Mentor’s investigative workflow.
Fig. 3 : PublicTrade’s website (a) The landing page advertises cryptocurrency, forex, and other trading services. (b) The Companies House record for the website’s claimed registration number lists retail sales as its business activity, a mismatch with its claimed purpose.
Fig. 4 : Condensed excerpt from Mentor’s PublicTrade evidence ledger, documenting the website’s registration claim and the corresponding official company record.
Task
Investigator Qwen3.5-4B
Decider Tev1-4B
Open-ended reasoning and generation
Binary and multiple-choice decisions
Planning
Identify the website’s purpose, define objectives, and formulate an investigative plan.
Choose among candidate actions addressing the current objective.
External verification
Identify claims requiring verification and generate search queries.
Select search results or external sources for retrieval.
Browser interaction
Define the interaction’s objective and supply required input values.
Select links, buttons, form options, or other available browser actions.
Evidence assessment
Organize observations, interpret relationships, and formulate evidence questions.
Resolve explicit binary or multiple-choice questions about the available evidence.
Adaptive follow-up
Identify unresolved questions and formulate revisions to the plan.
Choose whether to continue, revise, or conclude the investigation.
TABLE III: Division of responsibilities between ReDence’s Investigator and Decider.
Fig. 5 : Controller routes context and decision requests from Investigator to Decider, and then returns decisions and tool observations back to Investigator.
Configuration
Accuracy
Precision
Recall
F1
Time (Median, s)
GPU Usage (Median, GB)
ReDence (4B)
0.96 (0.95)
0.97 (0.95)
0.98 (0.98)
0.97 (0.96)
43.8 (27.2)
3.4GB
Mentor (27B)
0.82 (0.82)
0.83 (0.82)
0.81 (0.83)
0.82 (0.82)
194.0 (151.2)
33.8GB
URLNet
0.41
0.35
0.19
0.25
1.2 (1.2)
1.5GB
BeyondPhish
0.72
0.87
0.70
0.78
53.7 (1.8)
3.7GB
ScamFerret (GPT 4.1)
0.79
0.84
0.81
0.82
88.5 (41.2)
API-bound
ScamFerret (Llama3.3:70B)
0.71
0.75
0.67
0.71
127.4 (92.7)
45.7GB
TABLE IV: Comparison of detection performance and computational efficiency across the models on the test set. Parenthesized values for detection metrics for ReDence and Mentor indicate performance on the held-out categories , while those for median time indicate inference time excluding website latency.
Fig. 6 : Preview of wego-france.com: (a) Landing page containing the organization (Dick’s) logo, the (fake) timer to induce urgency, and the survey. (b) The fake facebook comment widget without any interactive elements. (c) The fake payment page which is only visible after interacting with the survey and the subsequent page which asks for non-payment information.
Fig. 7 : ReDence report excerpt connecting claimed customer endorsements with observations from testing the comment controls.
Fig. 8 : ScamFerret report excerpt presenting its classification, website category, and narrative explanation.
Conversational scams, such as romance and investment scams, are emerging as a major form of online fraud. Unlike one-shot scam lures such as fake lottery or unpaid toll messages, they unfold through multi-turn conversations in which scammers gradually manipulate victims using evolving psychological techniques. However, existing research mainly focuses on static scam detection or synthetic scams, leaving open whether language models can understand how real-world scams progress over time. We introduce PreScam, a benchmark for modeling scam progression from early conversations. Built from user-submitted scam reports, PreScam filters and structures 177,989 raw reports into 11,573 conversational scam instances spanning 20 scam categories. Each instance is hierarchically structured according to the scam lifecycle defined by the proposed scam kill chain, and further annotated at the turn level with scammer psychological actions and victim responses. We benchmark models on two tasks: real-time termination prediction, which estimates whether a conversation is approaching the termination stage, and scammer action prediction, which forecasts the scammer's subsequent actions. Results show a clear gap between surface-level fluency and progression modeling: supervised encoders substantially outperform zero-shot LLMs on real-time termination prediction, while next-action prediction remains only moderately successful even for strong LLMs. Taken together, these results show that current models can capture some scam-related cues, yet still struggle to track how risk escalates and how manipulation unfolds across turns.
Weixiang Sun, Shang Ma, Yiyang Li +5
University of Notre Dame · Arizona State University
Modern phishing campaigns increasingly evade snapshot-based URL classifiers using interaction gates (e.g., checkbox/slider challenges), delayed content rendering, and logo-less credential harvesters. This shifts URL triage from static classification toward an interactive forensics task: an analyst must actively navigate the page while isolating themselves from potential runtime exploits. We present TraceScope, a decoupled triage pipeline that operationalizes this workflow at scale. To prevent the observer effect and ensure safety, a sandboxed operator agent drives a real GUI browser guided by visual motivation to elicit page behavior, freezing the session into an immutable evidence bundle. Separately, an adjudicator agent circumvents LLM context limitations by querying evidence on demand to verify a MITRE ATT&CK checklist, and generates an audit-ready report with extracted indicators of compromise (IOCs) and a final verdict. Evaluated on 708 reachable URLs from existing dataset (241 verified phishing from PhishTank and 467 benign from Tranco-derived crawling), TraceScope achieves 0.94 precision and 0.78 recall, substantially improving recall over three prior visual/reference-based classifiers while producing reproducible, analyst-grade evidence suitable for review. More importantly, we manually curated a dataset of real-world phishing emails to evaluate our system in a practical setting. Our evaluation reveals that TraceScope demonstrates superior performance in a real-world scenario as well, successfully detecting sophisticated phishing attempts that current state-of-the-art defenses fail to identify.
Following the rapid progress of generative Artificial Intelligence, there is a growing threat posed by conversational scams. These scams often span over multiple weeks or months, gradually build trust and request for money or sensitive information. Existing scam-detection systems mainly focus on isolated messages, which renders them inadequate against this evolving threat. This paper extends single-message phishing detection and presents an explainable agentic system for detecting sophisticated conversational scams. It also introduces ConScamBench-278, an initial public multi-category benchmark for conversational scam detection spanning eight scam types, released to support reproducible evaluation and future expansion. On isolated messages the single-message detector attains 100% phishing recall, while the conversation-level detector identifies all conversational scams in the public LoveFraud02 corpus (83/83) and reaches 97.8% accuracy (95% CI [95.4, 99.0]) on ConScamBench-278. Two user studies (N = 100 and N = 45) further motivate the system: participants report frequently experiencing uncertainty when judging suspicious conversations. In an uncontrolled pre/post comparison, users self-reported trust, self-confidence, and perceived need for AI-based scam detection all increased (p < 0.001, Wilcoxon signed-rank). The system also receives a System Usability Scale score of 74.7 (95% CI [72.5, 76.9]), above the established usability benchmark.
Ahmed Omar Salim Adnan, Yogananda Manjunath, Shivanjali Khare
University of New Haven · M.S. in Data Science · Department of Electrical & Computer Engineering and Computer Science