Spam detection is usually treated as single-message classification, ignoring behavioral patterns that emerge across sequences of messages from the same sender. We study outbound SMS spam detection at the carrier level, where the goal is to predict at send time whether a carrier will flag a message as spam. Using a proprietary production database of 43,741 outbound SMS messages with UTC transmission timestamps and carrier delivery statuses, we report three sets of results. On an 8,749-message held-out test set, a TF-IDF + logistic regression baseline reaches AUC-ROC 0.8721, and a text-only DeBERTa-v3-base classifier reaches AUC-ROC 0.9011 and F1 0.8412. A deployed DistilBERT model augmented with inter-message temporal features reaches accuracy 0.8238, spam-class F1 0.8382, and recall 0.9128 at threshold 0.5 on a separate, class-balanced 20,000-message production evaluation set; these results are not directly comparable with the held-out test results. We also describe TemporalBertSpam, an architecture that encodes each sender's message history with a causal Transformer, and outline the same-split evaluation needed to measure its contribution.
Figures & tables
Model
AUC-ROC
F1
Acc.
TF-IDF + LR
0.8721
–
0.7912
DeBERTa-v3-base
0.9011
0.8412
0.8340
Table 1: Text baselines on the 8,749-message held-out test set (80/20 stratified split). F1 is for the delivered class (platform label convention, Section 5.2 ) and is reported for DeBERTa only.
Metric
Value
Accuracy
0.8238
Precision (spam)
0.7749
Recall (spam)
0.9128
F1 (spam)
0.8382
F1 (delivered)
0.8066
Macro F1
0.8224
Table 2: Deployed DistilBERT + temporal model on the 20,000-message production evaluation set (10,000 spam, 10,000 delivered) at threshold 0.5. Spam is the positive class.
Thr.
Acc.
Prec.
Rec.
F1
FP
FN
0.30
0.814
0.757
0.926
0.833
2972
744
0.35
0.815
0.760
0.920
0.833
2900
796
0.40
0.817
0.764
0.918
0.834
2840
816
0.45
0.821
0.769
0.916
0.836
2752
836
0.50
0.824
0.775
0.913
0.838
2652
872
0.55
0.825
0.779
0.909
0.839
2584
912
Table 3: Threshold sweep on the production evaluation set. Best F1 (0.839) is at threshold 0.55 (bold). FP = false positives; FN = false negatives.
Figure 1: Precision, recall, and F1 across decision thresholds on the production evaluation set (from Table 3 ). The dotted line marks the production threshold of 0.55.
Figure 2: Distribution of predicted pspam on the 20,000-message production evaluation set (0.05-wide bins). The dashed line marks threshold 0.5. Most predictions fall below 0.20 or above 0.70, with few in between.
Bucket
n
Avg. pred.
Observed
Error
[0.0,0.1)
3732
0.025
0.021
0.004
[0.1,0.2)
3380
0.141
0.130
0.011
[0.2,0.3)
660
0.247
0.339
0.093
[0.3,0.4)
204
0.338
0.353
0.015
[0.4,0.5)
244
0.456
0.230
0.227
[0.5,0.6)
248
0.552
0.436
0.116
Table 4: Calibration for predictions below 0.6 (8,468 of 20,000 messages). Observed is the actual spam rate in each bucket; Error is the absolute difference from the average prediction.