cs.CLAug 21, 2026
SaveEvidence-Consistent Generative Detection under Scenario-Level Distribution Shift
Organizations: Sungkyunkwan University Suwon, Republic of Korea
Abstract
Conventional in-distribution evaluation can overestimate robustness when training and test data share recurring task-specific patterns or surface cues. This risk is especially relevant in social-engineering fraud detection, where attackers can preserve malicious intent while changing the scenario, impersonated entity, or wording. We study this problem as scenario-level out-of-distribution (SL-OOD) detection for SMS and voice phishing, where entire attack scenarios are held out from training while the label space remains fixed. This setting tests whether models can generalize to unseen attack scenarios using decision-relevant evidence rather than familiar scenario-specific cues. Using this SL-OOD evaluation, we find that high in-distribution performance does not reliably predict held-out robustness across feature-, encoder-, and decoder-based baselines. We interpret this gap as scenario memorization: reliance on recurring scenario-specific lexical or entity cues rather than decision-relevant evidence. We propose ECoG, an evidence-consistent generative framework that combines evidence-span supervision with a rationale-label consistency objective during training. On the 0.5B decoder, relative to the same backbone trained without consistency regularization, ECoG raises Macro-F1 on OOD challenging instances by 3.22 points, reduces the share of predictions whose generated rationale supports the opposite label by 4.22 points, and increases token-level overlap with reference evidence spans by 8.38 points; the reduction in prediction-rationale inconsistency is consistent across four decoder backbones. These results suggest that compact generative detectors can benefit from evidence supervision and rationale-label consistency under social-engineering shift.
Figures & tables
| Split | Modality | Scenario | Phish. | Benign | Total |
| Train | SMS | Credit | 8,153 | 7,462 | 15,615 |
| Finance | 3,657 | 8,302 | 11,959 | ||
| Parcel | 7,147 | 5,309 | 12,456 | ||
| Voice | Government | 888 | 818 | 1,706 | |
| Finance | 2,233 | 1,720 | 3,953 | ||
| Validation | SMS | Credit | 1,559 | 1,006 | 2,565 |
Table 1. Dataset composition by split, modality, and scenario. Finance appears in both SMS and Voice.
Figure 1. Overview of ECoG. (1) The training sequence consists of the prompted input and target sequence . (2) A shared decoder produces hidden states . (3) Position masks select prompt and rationale states for auxiliary training-only heads: classification, evidence-span supervision, and rationale–label consistency. (4) The native LM head supervises the generated target sequence through , and all objectives are combined in the weighted joint loss. At inference, all auxiliary heads are discarded, and the decoder generates the label, evidence spans, and rationale from using only the native LM head. Two-panel schematic of the ECoG framework. The left panel, labeled Training: Single-Stage Joint Supervision, shows four numbered steps. Step 1 is a prompt and target sequence, where the prompt is an instruction plus input text and the target is a label, evidence spans, and a rationale. Step 2 is a shared decoder backbone that produces hidden states H. Step 3 is a position-mask state selection block that routes prompt states and rationale states to three auxiliary training-only heads, enclosed in a dashed box: a rationale-label consistency branch, a classification head, and an evidence span head, producing the consistency, classification, and evidence losses. A separate native LM generation head, used at both training and inference, produces the LM loss from the generated target sequence. Step 4 combines all terms into a weighted joint loss that drives a joint parameter update of the shared backbone. The right panel, labeled Inference: Generation-only, shows the auxiliary heads discarded: the prompted input passes through the shared decoder backbone and the LM generation head to produce a generated sequence, which is parsed into a predicted label, evidence spans, and rationale.
| Type | Model / Variant | SMS | Voice | ||||||
| Test | Challenging | Test | Challenging | ||||||
| ID | OOD | ID | OOD | ID | OOD | ID | OOD | ||
| Feature-based | TF-IDF + SVM ( Lee and Park, 2023 ) | 97.29 | 74.92 | 66.92 | 40.13 | 99.13 | 80.33 | 88.15 | 65.95 |
| Neural | BiLSTM ( Boussougou et al., 2025 ) | 98.36 | 86.07 | 80.17 | 58.19 | 98.15 | 85.68 | 88.97 | 74.81 |
| CNN-BiLSTM + Attn. ( Moussavou Boussougou and Park, 2023 ) | 98.60 | 91.81 | 81.48 | 70.32 | 99.13 | 86.40 | 90.11 | 77.72 | |
| Encoder PLM | KoBERT ( Cho and Seo, 2025 ) | 95.70 | 89.15 | 74.96 | 72.42 | 77.77 | 45.57 | 60.64 | 44.46 |
Table 2. Macro-F1, scenario-balanced within each modality; bold and underline mark the best and second-best in each column. Voice Test ID saturation (e.g., HCX-0.5B at ) reflects the limited size of the random-split Voice test set; OOD and Challenging columns are more discriminative.
| Model | SMS | Voice | ||||||
| Test | Challenging | Test | Challenging | |||||
| ID | OOD | ID | OOD | ID | OOD | ID | OOD | |
| HCX-0.5B | 99.65 | 94.97 | 96.80 | 92.71 | 100.00 | 94.56 | 97.36 | 88.98 |
| HCX-1.5B | 99.62 | 95.04 | 96.18 | 93.01 | 99.66 | 95.15 | 97.47 | 87.44 |
| Qwen3-0.6B | 99.40 | 94.93 | 97.88 | 89.75 | 99.66 | 92.05 | 95.66 | 84.78 |
| Qwen3-1.7B | 99.46 | 95.80 | 96.34 | 91.70 | 99.80 | 93.51 | 97.91 | 87.57 |
Table 3. ECoG performance across all decoder backbone sizes, reported as scenario-balanced Macro-F1. HCX-0.5B and Qwen3-0.6B rows duplicate the corresponding ECoG entries in Table 2 for direct scale comparison.
| Method | Cls | Hard | Inc. | Span | BERTScore |
| Basic Baselines | |||||
| ST-SFT | 91.60 | 83.30 | – | – | – |
| Pipeline Baseline | |||||
| 2-Stage SFT | 89.37 | 81.96 | – | – | – |
| Objective Ablation | |||||
| Label + Evi. | 94.66 | 86.57 | – | 14.67 | – |
Table 4. Mechanism ablation on HCX-0.5B. Mechanism ablation on HCX-0.5B; metrics are defined in Section 5.1 (lower is better for Inc., higher for the rest). Evi., Rat., and C denote evidence, rationale, and consistency objectives; + Last uses last-token pooling for the consistency head; Label-Last places the label field last in the target sequence. Dashes mark metrics the method does not produce.
Figure 2. Effect of the consistency weight relative to Joint w/o C ( ). Bars show changes in Hard F1, Inc. reduction, Span F1, and BERTScore; diamonds show Cls F1. Boxed values indicate the selected ECoG configuration ( ). Four bar charts arranged in a two-by-two grid, one per decoder backbone: HCX-0.5B, HCX-1.5B, Qwen3-0.6B, and Qwen3-1.7B. Each chart plots the change in percentage points relative to training without the consistency loss, on the vertical axis, against consistency weights of 0.05, 0.1, and 0.2 on the horizontal axis. Grouped bars show change in Hard F1, inconsistency reduction, span F1, and BERTScore, and diamond markers show change in classification F1. The selected setting of 0.1 is boxed on the axis and marked with an asterisk. Inconsistency reduction is positive at every weight in all four panels, while the other metrics vary in sign and magnitude across backbones; the vertical scale differs by panel, spanning roughly 10 points for HCX-0.5B and 4 points for HCX-1.5B.
Figure 3. ID–OOD robustness gap under scenario shift for trainable configurations, grouped by model family; stars denote ECoG. Lower values on the y-axis (ID minus OOD Macro-F1, in points) indicate stronger robustness. Dotted lines mark the per-panel average gap; panel (c) uses a wider y-range for the KoBERT Voice outlier. Four scatter plots arranged in a two-by-two grid, covering SMS Test, SMS Challenge, Voice Test, and Voice Challenge. Each plots in-distribution Macro-F1 on the horizontal axis against the in-distribution minus out-of-distribution Macro-F1 gap in percentage points on the vertical axis, so lower points indicate stronger robustness. Markers are colored by model family: feature-based and neural, encoder pretrained language models, decoder large language models, and decoder small language models, with ECoG variants shown as red stars. A dotted horizontal line marks the average gap in each panel. ECoG stars sit in the lower-right region of every panel, combining high in-distribution accuracy with small gaps, while several baselines show gaps above 20 percentage points; the vertical range of the Voice Test panel is wider to accommodate an encoder outlier near 32 points.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4. Per-scenario reduction in prediction–rationale inconsistency by ECoG relative to Joint w/o C across backbones. Positive values indicate lower inconsistency under ECoG; negative values indicate increased inconsistency. Background colors denote SMS and voice scenarios. Four bar charts arranged in a two-by-two grid, one per decoder backbone: HCX-0.5B, HCX-1.5B, Qwen3-0.6B, and Qwen3-1.7B. Each shows the reduction in prediction-rationale inconsistency in percentage points achieved by ECoG relative to training without the consistency loss, across five held-out scenarios: Credit, Finance, and Parcel for SMS, and Finance and Government for Voice. Background shading separates the SMS and Voice scenario groups. Most bars are positive, with the largest reduction on SMS Finance for HCX-0.5B at 25.5 points, while a few settings are negative, including Voice Finance for HCX-0.5B and Qwen3-0.6B and SMS Credit for HCX-1.5B. The vertical scale differs by panel, ranging from about 30 points for HCX-0.5B down to about 3 points for Qwen3-1.7B.
| Difficulty Tier | N | HCX-0.5B | Qwen3-0.6B | ||||
| w/o C | ECoG | Cls | w/o C | ECoG | Cls | ||
| disagree | 82 | 95.11 | 96.31 | 93.89 | 93.89 | ||
| disagree | 216 | 92.11 | 94.88 | 93.96 | 93.93 | ||
| disagree | 257 | 86.77 | 91.42 | 86.38 | 86.75 | ||
| All challenging | 555 | 90.08 | 93.49 | 90.44 | 90.60 | ||
Table 5. ECoG vs. Joint w/o C by human-difficulty tier on the retained SMS Challenging set. Scores are instance-level Macro-F1, not scenario-balanced averages; Table 2 reports scenario-balanced results. Difficulty tiers indicate how many of five blind annotators disagreed with the verified label during challenging-set construction.