When Models Know but Can't Linearly Say: Relational Encoding Limits Linear Vulnerability Classification in Code LLMs

Rui Melo; Andre Catarino; Claudia Mamede; Corina Pasareanu; Rui Abreu

cs.CROct 19, 2026

When Models Know but Can't Linearly Say: Relational Encoding Limits Linear Vulnerability Classification in Code LLMs

Authors: Rui Melo, Andre Catarino, Claudia Mamede, Corina Pasareanu, Rui Abreu

Organizations: Carnegie Mellon University, Pittsburgh, PA, USA · Faculty of Engineering, University of Porto, Portugal

Abstract

Large language models can detect vulnerable code, yet the form of vulnerability encoding in their representations remains unclear. We analyze 8 code models spanning 6.7B-15B parameters on C programs using three datasets (DELTASECOMMITS, SVEN, and PRECISE-BUGS), comparing two diagnostic tasks on the same representations: identifying what patterns the code instantiates, and determining whether it is vulnerable. Models encode the first with high fidelity. Under the studied paired C-code datasets and linear-probing setup, vulnerability information is more accessible as a paired relational signal-whether code A is riskier than code B-than as a globally separable categorical label. This structural difference has implications for classification: the model reliably ranks which code is riskier but standard classifiers on isolated samples achieve limited accuracy because vulnerable and secure samples form overlapping distributions in representation space.

Figures & tables

Figure 1: Pairwise CWE-type probe AUROC. Heatmap for Qwen2.5-7B-Instruct on DeltaSecommits (vulnerable-side mean-token pooling, Layer 19). The 12 CWE types include memory violations (CWE-119, CWE-120, CWE-125, CWE-787), injection attacks (CWE-78, CWE-89), path traversal (CWE-22), and resource leaks (CWE-401). Most pairs are highly separable (90–100% AUROC, dark green), with weaker separation between similar memory-related types (58–72% AUROC, yellow/orange). Color scale: green indicates strong separation, yellow indicates chance-level performance. This pattern reflects strong functional encoding of vulnerability categories across diverse CWE types.

Figure 2: Vulnerability direction alignment across layers and bug types. Global alignment (black, bold) reaches 87–88% across L3–L23, dropping to 70% at L27. Per-family breakdown (colored lines) shows consistent alignment across bug types: Memory Safety (blue), Injection (orange), Resource Management (green), Information Disclosure (magenta). Family-level alignment closely tracks global alignment across all bug types.

Figure 3: Projection distributions (Qwen2.5-7B-Instruct, mean-token pooling). Left : Distribution of pair-wise projection differences $(h_{\text{vuln}}-h_{\text{secure}})\cdot\hat{d}$ across layers. Most pairs exhibit small positive differences concentrated near zero, with $\sim$ 13% negative. Right : Vulnerable versus secure projections. The tight clustering around the diagonal and strong overlap between distributions indicate modest per-pair separation and no global linear decision boundary.

Figure 4: Cross-family vulnerability direction transfer heatmap (Qwen2.5-7B-Instruct, DeltaSecommits C subset, aggregated across layers L0–L27). Each cell shows per-pair alignment when a direction trained on the row family (on training data) is tested on the column family (on held-out test data), computed per-fold to maintain train/test separation. Values represent mean transfer across all layers. Warm colors (82–87%) indicate strong transfer; cooler colors (70–75%) indicate weaker transfer for injection attacks. The diagonal is undefined (comparing direction to itself). Mean cross-family transfer: 83.9%, well above chance (50%).

Model	L3	L7	L11	L15	L19	L23	L27
CodeLlama-7B-Instruct	$0.610$ $[0.592,0.628]$	$0.626$ $[0.609,0.644]$	$\mathbf{0.644}$ $[0.626,0.659]$	$0.640$ $[0.623,0.656]$	$0.626$ $[0.608,0.643]$	$0.598$ $[0.580,0.615]$	$0.578$ $[0.558,0.594]$
CodeLlama-13B-Instruct	$0.611$ $[0.592,0.628]$	$0.640$ $[0.623,0.656]$	$\mathbf{0.646}$ $[0.629,0.663]$	$0.644$ $[0.627,0.661]$	$0.634$ $[0.616,0.650]$	$0.618$ $[0.600,0.635]$	$0.590$ $[0.572,0.608]$
DeepSeek-Coder-6.7B-Instruct	$0.610$ $[0.591,0.628]$	$0.631$ $[0.614,0.648]$	$\mathbf{0.643}$ $[0.625,0.659]$	$0.635$ $[0.618,0.653]$	$0.630$ $[0.612,0.647]$	$0.614$ $[0.595,0.631]$	$0.607$ $[0.590,0.624]$
Qwen2.5-7B-Instruct	$0.613$ $[0.595,0.632]$	$0.629$ $[0.611,0.646]$	$0.628$ $[0.610,0.645]$	$0.645$ $[0.626,0.661]$	$\mathbf{0.654}$ $[0.637,0.670]$	$0.623$ $[0.605,0.640]$	$0.604$ $[0.585,0.622]$
Qwen2.5-Coder-7B-Instruct	$0.603$ $[0.584,0.622]$	$0.627$ $[0.609,0.644]$	$0.620$ $[0.602,0.637]$	$0.648$ $[0.629,0.665]$	$\mathbf{0.657}$ $[0.640,0.674]$	$0.614$ $[0.595,0.631]$	$0.585$ $[0.566,0.603]$
Qwen2.5-14B-Instruct	$0.610$ $[0.593,0.629]$	$0.625$ $[0.607,0.643]$	$0.608$ $[0.589,0.626]$	$0.620$ $[0.602,0.637]$	$0.631$ $[0.614,0.648]$	$0.643$ $[0.625,0.659]$	$\mathbf{0.646}$ $[0.628,0.662]$

Table 1: Length-controlled binary. Vulnerable vs. secure probe AUROC with 95% bootstrap CIs across layers, for all 8 models. Code length is regressed out before mean-token probing. Bold marks each model’s peak layer.

LLMs for Cybersecurity

Large language models for cybersecurity support threat analysis, vulnerability research, security operations, and cyber defense.

See all 258 papers

Dissecting the Black Box: Circuit-Level Analysis of LLM Vulnerability Detection
May 28, 2026Syafiq Al Atiiq, Chun Zhou, Christian GehrmannLLM Interpretability Software Vulnerability Detection
Lund University Lund, Sweden
Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection
Jun 29, 2026Asif Shahriar, Hongyu Cai, Hadjer Benkraouda +2Software Vulnerability Detection Adversarial Attacks on LLMs
BRAC University · Purdue University · University of Illinois Urbana-Champaign
Probing the Prefill: Detecting Code Vulnerabilities via Latent Activations
Aug 17, 2026Alizishaan KhatriSoftware Vulnerability Detection LLMs for Cybersecurity
Wrynx Inc.

May 28, 2026J/K move · Enter open · S save

Dissecting the Black Box: Circuit-Level Analysis of LLM Vulnerability Detection

Syafiq Al Atiiq, Chun Zhou, Christian Gehrmann

Lund University Lund, Sweden

LLM Interpretability Software Vulnerability Detection LLMs for Cybersecurity

Open paper AskPDF arXiv

Large language models (LLMs) can detect software vulnerabilities, but how do they actually identify vulnerable code? We address this question using mechanistic interpretability; analyzing the internal computations of a neural network to understand its reasoning process.Using Circuit Tracer on Gemma-2-2b, we trace the computational pathways activated when the model classifies 472 C/C++ code samples as vulnerable or safe. Our analysis reveals a surprising finding: the model primarily relies on safety detectors, attention heads that recognize safe coding patterns, rather than directly detecting vulnerability signatures. When these safety detectors fail to activate, the model classifies code as vulnerable. We identify the critical neural components: specific attention heads in early layers (L5, L7) that focus on safety patterns, and Multilayer Perceptron (MLP) neurons in Layer 7 that encode vulnerability-related features. Ablation experiments confirm their causal role; removing Layer 11 drops vulnerability detection accuracy from 100% to 6%, while ablating just 20 neurons in Layer 7 reduces it by 50%.Our findings show that LLM vulnerability detection uses sparse, interpretable circuits (only 16% of model capacity), enabling circuit-level explanations for security predictions and targeted improvements to detection systems.

Abstract

LLMs for Cybersecurity

Explore similar work

Dissecting the Black Box: Circuit-Level Analysis of LLM Vulnerability Detection

Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection

Probing the Prefill: Detecting Code Vulnerabilities via Latent Activations