Spam detection is usually treated as single-message classification, ignoring behavioral patterns that emerge across sequences of messages from the same sender. We study outbound SMS spam detection at the carrier level, where the goal is to predict at send time whether a carrier will flag a message as spam. Using a proprietary production database of 43,741 outbound SMS messages with UTC transmission timestamps and carrier delivery statuses, we report three sets of results. On an 8,749-message held-out test set, a TF-IDF + logistic regression baseline reaches AUC-ROC 0.8721, and a text-only DeBERTa-v3-base classifier reaches AUC-ROC 0.9011 and F1 0.8412. A deployed DistilBERT model augmented with inter-message temporal features reaches accuracy 0.8238, spam-class F1 0.8382, and recall 0.9128 at threshold 0.5 on a separate, class-balanced 20,000-message production evaluation set; these results are not directly comparable with the held-out test results. We also describe TemporalBertSpam, an architecture that encodes each sender's message history with a causal Transformer, and outline the same-split evaluation needed to measure its contribution.
Figures & tables
Model
AUC-ROC
F1
Acc.
TF-IDF + LR
0.8721
–
0.7912
DeBERTa-v3-base
0.9011
0.8412
0.8340
Table 1: Text baselines on the 8,749-message held-out test set (80/20 stratified split). F1 is for the delivered class (platform label convention, Section 5.2 ) and is reported for DeBERTa only.
Metric
Value
Accuracy
0.8238
Precision (spam)
0.7749
Recall (spam)
0.9128
F1 (spam)
0.8382
F1 (delivered)
0.8066
Macro F1
0.8224
Table 2: Deployed DistilBERT + temporal model on the 20,000-message production evaluation set (10,000 spam, 10,000 delivered) at threshold 0.5. Spam is the positive class.
Thr.
Acc.
Prec.
Rec.
F1
FP
FN
0.30
0.814
0.757
0.926
0.833
2972
744
0.35
0.815
0.760
0.920
0.833
2900
796
0.40
0.817
0.764
0.918
0.834
2840
816
0.45
0.821
0.769
0.916
0.836
2752
836
0.50
0.824
0.775
0.913
0.838
2652
872
0.55
0.825
0.779
0.909
0.839
2584
912
Table 3: Threshold sweep on the production evaluation set. Best F1 (0.839) is at threshold 0.55 (bold). FP = false positives; FN = false negatives.
Figure 1: Precision, recall, and F1 across decision thresholds on the production evaluation set (from Table 3 ). The dotted line marks the production threshold of 0.55.
Figure 2: Distribution of predicted pspam on the 20,000-message production evaluation set (0.05-wide bins). The dashed line marks threshold 0.5. Most predictions fall below 0.20 or above 0.70, with few in between.
Bucket
n
Avg. pred.
Observed
Error
[0.0,0.1)
3732
0.025
0.021
0.004
[0.1,0.2)
3380
0.141
0.130
0.011
[0.2,0.3)
660
0.247
0.339
0.093
[0.3,0.4)
204
0.338
0.353
0.015
[0.4,0.5)
244
0.456
0.230
0.227
[0.5,0.6)
248
0.552
0.436
0.116
Table 4: Calibration for predictions below 0.6 (8,468 of 20,000 messages). Observed is the actual spam rate in each bucket; Error is the absolute difference from the average prediction.
Misclassifications in spam and phishing detection are very harmful, as false negatives expose users to attacks while false positives degrade trust. Existing uncertainty-based detectors can flag potential errors, but possibly be deceived and offer limited interpretability. This paper presents X-MAP, an eXplainable Misclassification Analysis and Profilling framework that reveals topic-level semantic patterns behind model failures. X-MAP combines SHAP-based feature attributions with non-negative matrix factorization to build interpretable topic profiles for reliably classified spam/phishing and legitimate messages, and measures each message's deviation from these profiles using Jensen-Shannon divergence. Experiments on SMS and phishing datasets show that misclassified messages exhibit at least two times larger divergence than correctly classified ones. As a detector, X-MAP achieves up to 0.98 AUROC and lowers the false-rejection rate at 95% TRR to 0.089 on positive predictions. When used as a repair layer on base detectors, it recovers up to 97% of falsely rejected correct predictions with moderate leakage. These results demonstrate X-MAP's effectiveness and interpretability for improving spam and phishing detection.
Qi Zhang, Dian Chen, Lance M. Kaplan +4
Virginia Tech, USA · U.S. Army DEVCOM Army Research Laboratory, USA · University of Oslo, Norway +2
Fraudulent messages sent via Short Message Service (SMS) are increasingly obfuscated to evade cost-conscious classifiers in production systems. In Chinese SMS, attackers can exploit a wide range of carefully crafted obfuscation strategies to hide risk-bearing phrases while preserving human readability, making direct classification brittle under real-world latency and throughput constraints. We propose ReCAST, a Restoration-aware Cascaded Stage-wise Training framework for robust obfuscated Chinese SMS classification. ReCAST distills a large teacher model's de-obfuscation ability into a smaller deployable student model by supervising obfuscated span detection, obfuscation type prediction, and text restoration, and then uses the restoration-aware student for downstream risk classification. Experiments on an internally constructed real-world Chinese SMS benchmark show that ReCAST substantially improves classification performance over directly trained baselines under obfuscation. The results suggest that restoration-aware distillation offers a practical path toward robust SMS risk classification with smaller deployable models under production-oriented constraints.
Jieyun Huang, Yi Shen, Kaikai Zhao +7
Unicom Data Intelligence, China Unicom · China Unicom Data Science & Artificial Intelligence Research Institute, China Unicom
Email spam and phishing attacks remain a critical security threat. Adversaries increasingly exploit large language models to craft contextually convincing malicious messages, and existing spam detection systems often struggle to keep pace. Generalization across diverse and evolving attack scenarios is limited, which reduces effectiveness once these systems are deployed in practice. This paper introduces Adaptive Uncertainty-Routed Analysis (AURA), a multimodal email threat detection system that analyzes both the content of an email and its embedded URLs. AURA is built around two layers: the first quantifies prediction uncertainty from a URL classifier, and only ambiguous messages are escalated to a fine-tuned transformer encoder for semantic analysis. The system is evaluated on eight heterogeneous training corpora together with two held-out real-world corpora spanning a decade of adversarial campaigns. AURA reaches a macro F1-score of 0.9858 in-distribution, and on NazPhish-Eval and GuenterTrap-Eval it maintains 0.9502 and 0.9436, respectively, which is evidence of robust generalization under genuine distribution shift.
Omran Berjawi, Walid fahs, Rida Khatoun
Institut Polytechnique de Paris, Télécom Paris, Palaiseau, France · Islamic University of Lebanon, Faculty of Engineering, Wardanieh, Lebanon