cs.SEOct 9, 2026
SaveDiagnosing Generalization Boundaries in AI-Based Vulnerability Detection via Calibrated Embedding Distances
Organizations: University of Koblenz, Germany · Fraunhofer ISST, Germany
Abstract
High benchmark scores in learning-based vulnerability detection mask severe degradation under distribution shifts. This paper demonstrates that calibrated embedding geometry exposes these hidden generalization boundaries on syntactically similar vulnerable and patched code pairs. An evaluation of CodeBERT and Qwen2.5-Coder across pattern, project, and temporal shifts on a strictly paired dataset reveals three main findings. First, whitened intermediate-layer embeddings with local nearest-neighbour scoring separate unfamiliar samples with a mean out-of-distribution AUROC of 0.739 (RQ1). Second, distance scores reveal a shift-specific reliability gradient where accuracy drops by over 35 percentage points and cross-project shifts cause the sharpest degradation, even though temporal shifts drift geometrically farthest (RQ2). Third, false positives on patched code systematically correlate with geometric proximity to vulnerable training neighbourhoods, increasing from 9.4% in the lowest-risk decile to 58.8% in the highest (RQ3). These calibrated distances establish a diagnostic signal that maps representation geometry directly to model degradation without requiring auxiliary models.
Figures & tables
Figure 1. Conceptual 2D representation space motivating this study. In-distribution code forms a dense core of paired functions ( ) exhibiting high syntactic overlap despite opposite labels. Under distribution shift, samples drift outward across boundary and distant zones, where calibrated embedding distance exposes model generalization limits. Conceptual 2D scatter plot illustrating how in-distribution pairs form a dense core, while distribution shifts drift outward into boundary and distant zones.
Figure 2. Local neighbourhood proximity mechanism. Although training data is paired, latent distances are asymmetric: (a) A patch whose top- neighbours are dominated by safe samples yields a True Negative. (b) Syntactic overlap traps a patch closer to vulnerable counterparts, inducing a False Positive. Diagram comparing two local k-NN neighbourhoods: panel (a) shows a patch query surrounded by safe samples resulting in a true negative, while panel (b) shows syntactic overlap trapping the patch within vulnerable training samples, inducing a false positive.
Figure 3. Dataset construction pipeline: from multi-source aggregation (Phase 1), through context-preserving cleaning and deduplication (Phase 2), to leakage-free hierarchical partitioning into in-distribution and out-of-distribution groups (Phase 3). Flowchart depicting the three phases of dataset construction: Phase 1 multi-source aggregation from PrimeVul, TitanVul, and VulGate; Phase 2 cleaning and context-preserving deduplication yielding 8,065 unique pairs; and Phase 3 stratified partitioning into 5 distinct evaluation groups.
| Model | Test Acc | Test F1 | Patch-FPR |
|---|---|---|---|
| CodeBERT CE-only | 0.605 | 0.605 | 0.395 |
| CodeBERT CE+SCL | 0.614 | 0.627 | 0.419 |
| Qwen2.5-Coder-7B | 0.627 | 0.635 | 0.394 |
Table 1. Test classification performance. Patch-FPR denotes the fraction of patched samples incorrectly classified as vulnerable.
| Transformation | Method | Best Mean | Avg. Mean |
|---|---|---|---|
| Whitened | cosine NN | 0.739 | 0.613 |
| Raw | cosine NN | 0.710 | 0.576 |
| Raw | Mahalanobis | 0.547 | 0.513 |
Table 2. Effect of embedding transformation and scoring method on OOD separability (aggregated over all layers and model variants).
| Layer | Best Mean | Pattern | Project | Time |
|---|---|---|---|---|
| 0 | 0.500 | 0.500 | 0.500 | 0.500 |
| 1 | 0.711 | 0.668 | 0.750 | 0.758 |
| 2 | 0.720 | 0.685 | 0.746 | 0.779 |
| 3 | 0.739 | 0.697 | 0.776 | 0.793 |
| 4 | 0.734 | 0.699 | 0.774 | 0.794 |
| 5 | 0.735 | 0.706 | 0.774 | 0.790 |
Table 3. Layer-wise OOD separability. Intermediate layers retain the strongest distributional signal.
| Best Mean | Pattern | Project | Time | |
|---|---|---|---|---|
| 1 | 0.739 | 0.720 | 0.745 | 0.794 |
| 3 | 0.721 | 0.693 | 0.776 | 0.740 |
| 5 | 0.703 | 0.673 | 0.774 | 0.735 |
| 10 | 0.676 | 0.659 | 0.750 | 0.709 |
| 20 | 0.650 | 0.623 | 0.721 | 0.676 |
| 30 | 0.631 | 0.609 | 0.713 | 0.651 |
Table 4. Effect of neighbourhood size ( ) on cosine- NN OOD separability. Group columns report marginal maxima.
Figure 4. CodeBERT CE+SCL encoder performance over its own embedding distance (ten equal-frequency bins). Accuracy decreases and patch FPR generally rises with increasing distance from the training distribution. Line graph showing CodeBERT accuracy dropping from 83.4% to 47.9% and patch false positive rate rising from 16.7% to 63.1% across ten equal-frequency embedding distance bins.
Figure 5. Qwen2.5-Coder performance over raw CodeBERT embedding distance. Accuracy decreases with distance, while mean confidence remains comparatively stable. Line graph showing Qwen2.5-Coder LLM accuracy decreasing from 68.0% to 53.9% across distance bins, while LLM confidence remains comparatively stable around 80%.
Figure 6. Shift-specific boundary behaviour and decoupling. Group centroids (circles) and medians (vertical bars) relate raw CodeBERT OOD distance to CE+SCL accuracy (Y-axis), while horizontal ridge curves display the OOD score density for each group. Despite temporal shift drifting furthest, cross-project shift causes the sharpest accuracy degradation. 2D ridge and scatter plot relating embedding distance on the horizontal axis to accuracy on the vertical axis across five evaluation groups, showing cross-project shift causing the lowest accuracy despite temporal shift having higher distance.
Figure 7. Hard-negative pair behaviour (CE+SCL encoder). Pair-exact correctness decreases sharply under OOD shifts, exposing shift-specific failure modes on either the patch (FPR) or the vulnerability (FNR). Line chart showing pair-exact correctness dropping below the 25% random threshold under OOD shifts, while patch false positive rate peaks on project shift and vulnerable false negative rate peaks on temporal shift.
| CWE | Test | OOD | Acc | FPR | FNR | Error |
|---|---|---|---|---|---|---|
| CWE-787 | 234 | 0.546 | 0.722 | 0.316 | 0.239 | 0.278 |
| CWE-120 | 82 | 0.595 | 0.671 | 0.390 | 0.268 | 0.329 |
| CWE-125 | 290 | 0.557 | 0.669 | 0.379 | 0.283 | 0.331 |
| CWE-119 | 500 | 0.566 | 0.596 | 0.428 | 0.380 | 0.404 |
| CWE-476 | 290 | 0.659 | 0.600 | 0.483 | 0.317 | 0.400 |
| CWE-415 | 106 | 0.655 | 0.557 | 0.358 | 0.528 | 0.443 |
Table 5. CWE-level failure modes for the CE+SCL encoder. Performance varies heavily by vulnerability semantics, independent of embedding distance. Snippets may carry multiple CWE labels, thus counts do not sum to total test size.
| Metric | FP | TN | [95% CI] | AUROC |
|---|---|---|---|---|
| Centroid margin | 0.009 | -0.030 | 0.039 [0.027, 0.050] | 0.609 |
| NN margin | -0.012 | -0.040 | 0.029 [0.018, 0.040] | 0.591 |
| Vuln. share ( =10) | 0.511 | 0.460 | 0.051 [0.035, 0.068] | 0.597 |
| Class margin ( =10) | -0.001 | -0.017 | 0.016 [0.012, 0.021] | 0.635 |
| Dist. ratio ( =10) | 1.002 | 1.032 | -0.030 [-0.041, -0.021] | 0.636 |
Table 6. RQ3 proximity analysis for patched FPs vs. TNs ( = FP minus TN; all , two-sided Welch’s -test). For ratios, lower values indicate stronger vulnerable proximity.
Figure 8. Patched FP rate across proximity-risk deciles (class-specific vulnerability margin). Higher deciles indicate greater proximity to vulnerable training neighbourhoods. Line chart of patched false positive rate across ten proximity deciles, surging from 9.4% in the lowest decile to 58.8% in the highest decile.
More in this topic
Software Vulnerability Detection
Software vulnerability detection identifies insecure code, pinpoints weaknesses, and helps developers validate or prevent exploitable bugs.
See all 143 papers