A Causal Perspective on the Role of Training Dynamics for Interpreting Privacy Risks in Code Models

Hua Yang; Alejandro Velasco; Sen Fang; Bowen Xu; Denys Poshyvanyk

cs.SEDec 8, 2025

A Causal Perspective on the Role of Training Dynamics for Interpreting Privacy Risks in Code Models

Authors: Hua Yang, Alejandro Velasco, Sen Fang, Bowen Xu, Denys Poshyvanyk

Organizations: North Carolina State University · William & Mary

Abstract

Large language models for code (LLM4Code) have greatly improved developer productivity but also raise privacy concerns due to their reliance on open-source repositories containing abundant personally identifiable information (PII). Prior work shows that commercial models can reproduce sensitive PII, yet existing studies largely treat PII as a single category and overlook the heterogeneous risks among different types. We investigate whether distinct PII types vary in their likelihood of being learned and leaked by LLM4Code, and whether this relationship is causal. Our methodology includes building a dataset with diverse PII types, fine-tuning representative models of different scales, computing training dynamics on real PII data, and formulating a structural causal model to estimate the causal effect of learnability on leakage. Results show that leakage risks differ substantially across PII types and correlate with their training dynamics: easy-to-learn instances such as IP addresses exhibit higher leakage, while harder types such as keys and passwords leak less frequently. Ambiguous types show mixed behaviors. This work provides the first causal evidence that leakage risks are type-dependent and offers guidance for developing type-aware and learnability-aware defenses for LLM4Code.

Figures & tables

	Email	Key	IP Address	Name	Username	Password
Regex-based PII Detection	53265	601	10554	0	0	0
StarPII	6137	31855	9608	157751	287461	50564

Table 1. Statistics of detection results for regex match and StarPII

	Email	Key	IP Address	Name	Username	Password
Filtered by Pre-check	471	4438	13388	208	2104	27880
Evaluated by LLM	2953	2356	6029	2356	3425	21544
Exceeding Threshold	2000	2000	1772	2000	2000	1958
Exceeding Threshold Rate	67.73%	84.89%	29.39%	84.89%	58.39%	9.09%

Table 2. Statistics of LLM refinement results

type	GT	regex R	regex P	StarPII R	StarPII P	combined R	combined P
email	223	0.933	0.796	0.973	0.799	0.973	0.795
password	203	0.000	-	0.897	0.451	0.897	0.451
name	228	0.000	-	0.982	0.370	0.982	0.370
username	308	0.000	-	0.773	0.432	0.773	0.432
ip_address	458	0.526	0.801	0.985	0.746	0.985	0.746
key	269	0.227	0.938	0.829	0.209	0.833	0.215

Table 3. Detection Performance by Type: Regex vs StarPII vs Combined. R and P denote recall and precision.

	Pre-Check Rules
	Exactly one "@" symbol must exist; both local and domain parts must be non-empty.
	No leading, trailing, or consecutive (“..”) dots.
	No whitespace, slashes, or parentheses.
Email	Domain Validation: The domain must not start or end with a dot. The top-level domain is verified against the official IANA list.
	IP Address must be syntactically valid and informative, i.e., publicly routable.
IP Address	Private, loopback, link-local, unspecified, multicast, reserved, and broadcast addresses are rejected.

Table 4. Complete list of pre-check rules for each PII type

Figure 6. Dataset maps of the constructed training set for each fine-tuned LLM4Code . Axes represent variability (x) and confidence (y), with samples down-sampled to preserve distribution. Unlike other models, CodeLlama-7b and CodeLlama-13b show many high-confidence yet high-variability samples, suggesting that fine-tuning amplifies their memorization.

Figure 7. Distribution of prediction confidence across PII types for each fine-tuned LLM4Code .

Figure 8. Distribution of prediction variability across PII types for each fine-tuned LLM4Code .

	Email	Key	IP Address	Name	Username	Password
stable-code-3b	42	74	170	56	1 55	90
Qwen2.5-Coder-3B	6 0	16	31	58	63	28
CodeLlama-7b	16	8	27	31	3 0	8
Qwen2.5-Coder-7B	9 3	26	68	85	117	48
CodeLlama-13b	2 6	5	13	2 6	45	9
Qwen2.5-Coder-14B	123	40	77	131	1 27	52

Table 5. Successful PII attack instances by LLM4Code and PII type

PII Type	Pearson	ATE	Pearson	ATE	Pearson	ATE	Pearson	ATE	Pearson	ATE	Pearson	ATE
	stable-code-3b		Qwen2.5-Coder-3B		CodeLlama-7b		Qwen2.5-Coder-7B		CodeLlama-13b		Qwen2.5-Coder-14B
name	-0.06 $\pm$ 0.05	-0.03 $\pm$ 0.01	-0.10 $\pm$ 0.03	-0.03 $\pm$ 0.02	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.03 $\pm$ 0.07	-0.03 $\pm$ 0.03	-0.01 $\pm$ 0.21	0.00 $\pm$ 0.04	-0.02 $\pm$ 0.04	-0.01 $\pm$ 0.02
key	-0.02 $\pm$ 0.05	-0.09 $\pm$ 0.20	-0.11 $\pm$ 0.12	-0.03 $\pm$ 0.01	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.12 $\pm$ 0.14	-0.05 $\pm$ 0.05	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.20 $\pm$ 0.17	-0.10 $\pm$ 0.07
username	-0.63 $\pm$ 0.03	-0.46 $\pm$ 0.08	-0.32 $\pm$ 0.05	-0.09 $\pm$ 0.01	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.34 $\pm$ 0.02	-0.14 $\pm$ 0.01	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.35 $\pm$ 0.01	-0.15 $\pm$ 0.04
password	-0.15 $\pm$ 0.14	-0.03 $\pm$ 0.04	-0.10 $\pm$ 0.02	-0.03 $\pm$ 0.01	0.01 $\pm$ 0.01	0.00 $\pm$ 0.00	-0.08 $\pm$ 0.05	-0.03 $\pm$ 0.01	0.01 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.10 $\pm$ 0.07	-0.03 $\pm$ 0.04
email	-0.39 $\pm$ 0.09	-0.20 $\pm$ 0.10	-0.12 $\pm$ 0.01	-0.02 $\pm$ 0.04	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.17 $\pm$ 0.05	-0.03 $\pm$ 0.03	0.00 $\pm$ 0.00	0.00 $\pm$ 0.00	-0.16 $\pm$ 0.15	-0.03 $\pm$ 0.05

Table 6. Causal Analysis Results (Easy-to-Learn vs Hard-to-Learn)

	stable-code-3b		Qwen2.5-Coder-3B		CodeLlama-7b		Qwen2.5-Coder-7B		CodeLlama-13b		Qwen2.5-Coder-14B
PII Type	Pearson	ATE	Pearson	ATE	Pearson	ATE	Pearson	ATE	Pearson	ATE	Pearson	ATE
name	0.01 $\pm$ 0.02	0.00 $\pm$ 0.01	0.04 $\pm$ 0.02	0.03 $\pm$ 0.00	0.03 $\pm$ 0.02	0.01 $\pm$ 0.00	0.05 $\pm$ 0.03	0.04 $\pm$ 0.06	0.02 $\pm$ 0.06	0.01 $\pm$ 0.04	0.08 $\pm$ 0.01	0.07 $\pm$ 0.02
key	0.01 $\pm$ 0.03	0.02 $\pm$ 0.04	-0.03 $\pm$ 0.06	-0.02 $\pm$ 0.04	0.07 $\pm$ 0.03	0.03 $\pm$ 0.02	0.05 $\pm$ 0.04	0.03 $\pm$ 0.03	0.06 $\pm$ 0.01	0.01 $\pm$ 0.01	0.00 $\pm$ 0.09	-0.01 $\pm$ 0.08
username	-0.32 $\pm$ 0.00	-0.23 $\pm$ 0.10	-0.16 $\pm$ 0.16	-0.10 $\pm$ 0.07	0.02 $\pm$ 0.04	0.02 $\pm$ 0.04	-0.07 $\pm$ 0.04	-0.05 $\pm$ 0.15	0.02 $\pm$ 0.03	0.02 $\pm$ 0.00	0.06 $\pm$ 0.05	0.05 $\pm$ 0.06
password	0.11 $\pm$ 0.07	0.08 $\pm$ 0.03	0.04 $\pm$ 0.00	0.03 $\pm$ 0.05	0.06 $\pm$ 0.02	0.05 $\pm$ 0.01	0.06 $\pm$ 0.02	0.07 $\pm$ 0.02	0.04 $\pm$ 0.01	0.05 $\pm$ 0.01	0.10 $\pm$ 0.01	0.11 $\pm$ 0.06
email	0.04 $\pm$ 0.06	0.06 $\pm$ 0.07	0.07 $\pm$ 0.01	0.04 $\pm$ 0.00	0.03 $\pm$ 0.01	0.03 $\pm$ 0.01	0.12 $\pm$ 0.01	0.08 $\pm$ 0.02	0.03 $\pm$ 0.01	0.03 $\pm$ 0.00	0.13 $\pm$ 0.05	0.08 $\pm$ 0.07

Table 7. Causal Analysis Results (Easy vs Ambiguous to Learn)

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 284 papers

Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs
Mar 5, 2026Patrick Ahrend, Tobias Eder, Xiyang Yang +2Privacy Leakage in Language Models Privacy-Preserving Language Models
Technical University of Munich, Germany
Discovering Universal Activation Directions for PII Leakage in Language Models
Feb 19, 2026Leo Marchyok, Zachary Coalson, Sungho Keum +2Privacy Leakage in Language Models LLM Interpretability
Oregon State University, Corvallis OR, USA · Korea Advanced Institute of Science & Technology, Daejeon, South Korea.
Code-Poisoning Property Inference Attacks
Jul 17, 2026Xukun Luan, Yuhui Gong, Gang Zhang +4Adversarial Attacks
School of Computer Science and Technology, Beijing Institute of Technology · School of Computer Science and Engineering, Northeastern University

Mar 5, 2026J/K move · Enter open · S save

Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs

Patrick Ahrend, Tobias Eder, Xiyang Yang, Zhiyi Pan+1

Technical University of Munich, Germany

Privacy Leakage in Language Models Privacy-Preserving Language Models PII Detection

Open paper AskPDF arXiv

Chain-of-Thought (CoT) prompting improves LLM reasoning but can increase privacy risk by resurfacing personally identifiable information (PII) from the prompt into reasoning traces and outputs, even under policies that instruct the model not to restate PII. We study such direct, inference-time PII leakage using a model-agnostic framework that (i) defines leakage as risk-weighted, token-level events across 11 PII types, (ii) traces leakage curves as a function of the allowed CoT budget, and (iii) compares open- and closed-source model families on a structured PII dataset with a hierarchical risk taxonomy. We find that CoT consistently elevates leakage, especially for high-risk categories, and that leakage is strongly family- and budget-dependent. Increasing the reasoning budget can either amplify or attenuate leakage depending on the base model. We then benchmark lightweight inference-time gatekeepers: a rule-based detector, a TF-IDF + logistic regression classifier, a GLiNER-based NER model, and an LLM-as-judge, using risk-weighted F1, Macro-F1, and recall. No single method dominates across models or budgets, motivating hybrid, style-adaptive gatekeeping policies that balance utility and risk under a common, reproducible protocol.

Abstract

Privacy Leakage in Language Models

Explore similar work

Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs

Discovering Universal Activation Directions for PII Leakage in Language Models

Code-Poisoning Property Inference Attacks