Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs

Patrick Ahrend; Tobias Eder; Xiyang Yang; Zhiyi Pan; Georg Groh

cs.CLMar 5, 2026

Safer Reasoning Traces: Measuring and Mitigating Chain-of-Thought Leakage in LLMs

Authors: Patrick Ahrend, Tobias Eder, Xiyang Yang, Zhiyi Pan, Georg Groh

Organizations: Technical University of Munich, Germany

Abstract

Chain-of-Thought (CoT) prompting improves LLM reasoning but can increase privacy risk by resurfacing personally identifiable information (PII) from the prompt into reasoning traces and outputs, even under policies that instruct the model not to restate PII. We study such direct, inference-time PII leakage using a model-agnostic framework that (i) defines leakage as risk-weighted, token-level events across 11 PII types, (ii) traces leakage curves as a function of the allowed CoT budget, and (iii) compares open- and closed-source model families on a structured PII dataset with a hierarchical risk taxonomy. We find that CoT consistently elevates leakage, especially for high-risk categories, and that leakage is strongly family- and budget-dependent. Increasing the reasoning budget can either amplify or attenuate leakage depending on the base model. We then benchmark lightweight inference-time gatekeepers: a rule-based detector, a TF-IDF + logistic regression classifier, a GLiNER-based NER model, and an LLM-as-judge, using risk-weighted F1, Macro-F1, and recall. No single method dominates across models or budgets, motivating hybrid, style-adaptive gatekeeping policies that balance utility and risk under a common, reproducible protocol.

Figures & tables

Figure 1: Overview of the three-phase evaluation pipeline. In the Injection phase, PII across three risk tiers (A–C) is embedded in the prompt context. During Retrieval, the LLM under test responds in either plain or CoT mode, and token-level leakage is measured. Finally, four Gatekeeper approaches are evaluated on their ability to detect leaked PII , scored via risk-weighted F1.

Rank	Model	Tokens Avg.
1	Mixtral	101
2	Llama 3.3	167
3	Opus	191
4	o3	337
5	Qwen3	449
6	DeepSeek–R1	565

Table 1: Baseline token usage for leakage experiments.

Figure 2: Plain vs. CoT leakage. Fraction of runs (out of 100) with PII leakage across 11 types for three representative models (o3, Llama 3.3, Qwen3).

Figure 3: Leakage vs. reasoning budget. PII leakage across five models as a function of CoT token limit. Lines show median leakage over 90 runs; shaded bands show the interquartile range. Token limit 0 disables CoT .

Panel of Figure 4 — Figure 4: Gatekeeper impact on leakage. Total leaks (blue) and residual leaks after gating (orange) for three gatekeepers (rules, GLiNER2, LLM -as-judge with Opus) across three models (o3, Mixtral, DeepSeek-R1).

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Name	Sex	Job	DoB	IP	MAC	Phone	Company	Credit card	SSN	Email	Avg.
Llama - PLAIN	100	100	100	15	84	100	17	100	0	4	1	56.45
Llama - COT	100	100	100	100	100	100	99	100	91	100	100	99.09
Opus - PLAIN	99	98	100	30	35	37	3	100	0	0	2	45.82
Opus - COT	100	100	100	100	96	100	92	100	0	51	96	85.00
Mixtral - PLAIN	100	100	100	57	100	100	84	100	78	98	100	92.45
Mixtral - COT	100	100	100	97	100	100	100	100	100	97	100	99.45

Appendix

Table 6: Leakage Evaluation Across Models and PII Categories. N=100 for each experiment type (Plain/ CoT ). Values indicate leakage frequency

Entity Type	Accuracy
Credit card number	41.8%
SSN	68.3%
Phone number	82.2%
DOB	88.0%
Email	90.7%
IP address	92.2%

Appendix

Table 7: Performance results broken down by PII type. The table shows the accuracy percentages achieved for each category, ranging from 41.8% for Credit card numbers to 99.8% for Sex.

Panel of Figure 5 — Figure 5: Leakage bar plots across model families. Plain vs. CoT leakage across 11 PII types for six models.

Panel of Figure 6 — Figure 6: Model comparison matrices showing net wins (wins minus losses) across 11 PII types. Green cells indicate the row model outperforms the column model (lower leakage rate); red indicates the opposite. Win/loss uses a 5% tie threshold.

Figure 7: Per-model, per-category PII leakage rates (%) under CoT prompting, illustrating the hierarchical protection of PII types, from near-complete leakage of Group A fields (name, email) to comparatively lower leakage of Group C fields (credit card number, SSN).

Parameter	Value
Total Experiments	2,250
Models Tested	5 (DeepSeek-R1:70b, Llama3.3:70b, Mixtral:8x22b, O3, Qwen3:32b)
Token Limits	5 (0, 138, 345, 690, 1035 tokens)
PII Types	6 (jobtype, phonenumber, ssn, creditcardnumber, name, dob)
Prompts per PII Type	5
Random Seeds	3 (42, 123, 999)

Appendix

Table 9: Token Budget Experiment Hyperparameter Settings

Approach	Model	Recall $\uparrow$	Macro F1 $\uparrow$	Risk-A. F1 $\uparrow$	SPriV $\downarrow$
Rule-based	DeepSeek-R1	0.403	0.457	0.637	0.011
	Llama	0.429	0.489	0.671	0.023
	Mixtral	0.439	0.498	0.694	0.046
	O3	0.432	0.491	0.674	0.034
	Opus	0.340	0.399	0.421	0.022
	Qwen3	0.439	0.499	0.696	0.013

Appendix

Table 10: Complete gatekeeper performance across all approaches and models (CoT experiments). Bold indicates best performance within each approach; underscores indicate second-best performance. Average rows show mean across all 6 models (best and second-best across approach averages are also highlighted). SPriV quantifies the proportion of unmasked PII tokens in the output; lower values indicate better privacy protection.

Panel of Figure 8 — Figure 8: Rule-Based Gatekeeper performance across all six models. Blue bars show total leaks, orange bars show remaining leaks after gatekeeper filtering. Lower orange bars indicate better gatekeeper performance.

Model	PII Type	Support	Blocked	Missed	Recall	F1
DeepSeek-R1	Name	100	0	100	0.000	0.000
DeepSeek-R1	Sex	100	0	100	0.000	0.000
DeepSeek-R1	Job	100	0	100	0.000	0.000
DeepSeek-R1	DoB	65	10	55	0.154	0.267
DeepSeek-R1	IP	100	52	48	0.520	0.684
DeepSeek-R1	MAC	92	92	0	1.000	1.000

Appendix

Table 11: Rule-Based Gatekeeper: Detailed results by model and PII type. Support = actual leaks, Blocked = true positives, Missed = false negatives. Recall = Blocked/Support.

Panel of Figure 9 — Figure 9: ML Classifier Gatekeeper performance across all six models. Blue bars show total leaks, orange bars show remaining leaks after gatekeeper filtering. Lower orange bars indicate better gatekeeper performance.

Model	PII Type	Support	Blocked	Missed	Recall	F1
DeepSeek-R1	Name	100	0	100	0.000	0.000
DeepSeek-R1	Sex	100	100	0	1.000	1.000
DeepSeek-R1	Job	100	10	90	0.100	0.182
DeepSeek-R1	DoB	65	22	43	0.338	0.449
DeepSeek-R1	IP	100	1	99	0.010	0.020
DeepSeek-R1	MAC	92	1	91	0.011	0.022

Appendix

Table 12: ML Classifier Gatekeeper (TF-IDF + Logistic Regression): Detailed results by model and PII type.

Panel of Figure 10 — Figure 10: GLiNER2 Gatekeeper performance across all six models. Blue bars show total leaks, orange bars show remaining leaks after gatekeeper filtering. Lower orange bars indicate better gatekeeper performance.

Model	PII Type	Support	Blocked	Missed	Recall	F1
DeepSeek-R1	Name	100	100	0	1.000	1.000
DeepSeek-R1	Sex	100	100	0	1.000	1.000
DeepSeek-R1	Job	100	100	0	1.000	1.000
DeepSeek-R1	DoB	65	65	0	1.000	0.788
DeepSeek-R1	IP	100	100	0	1.000	1.000
DeepSeek-R1	MAC	92	73	19	0.793	0.844

Appendix

Table 13: GLiNER2 Gatekeeper (NER-based): Detailed results by model and PII type.

Panel of Figure 11 — Figure 11: LLM -as-a-Judge Gatekeeper (O4-mini) performance across all six models. Blue bars show total leaks, orange bars show remaining leaks after gatekeeper filtering. Lower orange bars indicate better gatekeeper performance.

Model	PII Type	Support	Blocked	Missed	Recall	F1
DeepSeek-R1	Name	100	34	66	0.340	0.507
DeepSeek-R1	Sex	53	18	35	0.340	0.424
DeepSeek-R1	Job	100	1	99	0.010	0.020
DeepSeek-R1	DoB	65	20	45	0.308	0.460
DeepSeek-R1	IP	100	17	83	0.170	0.291
DeepSeek-R1	MAC	92	13	79	0.141	0.248

Appendix

Table 14: LLM-as-a-Judge Gatekeeper (O4-mini): Detailed results by model and PII type.

Panel of Figure 12 — Figure 12: LLM -as-a-Judge Gatekeeper (Opus) performance across all six models. Blue bars show total leaks, orange bars show remaining leaks after gatekeeper filtering. Lower orange bars indicate better gatekeeper performance.

Model	PII Type	Support	Blocked	Missed	Recall	F1
DeepSeek-R1	Name	100	100	0	1.000	1.000
DeepSeek-R1	Sex	100	100	0	1.000	1.000
DeepSeek-R1	Job	100	100	0	1.000	1.000
DeepSeek-R1	DoB	65	64	1	0.985	0.780
DeepSeek-R1	IP	100	0	100	0.000	0.000
DeepSeek-R1	MAC	92	0	92	0.000	0.000

Appendix

Table 15: LLM-as-a-Judge Gatekeeper (Opus): Detailed results by model and PII type.

Gatekeeper	Model	SPriV $\downarrow$	N Samples	Missed Detections
Rule-Based	DeepSeek R1	0.0109	1100	565
Rule-Based	Llama 3.3	0.0229	1100	628
Rule-Based	Mixtral	0.0458	1100	614
Rule-Based	o3	0.0341	1100	468
Rule-Based	Opus	0.0222	1100	584
Rule-Based	Qwen3	0.0131	1100	609

Appendix

Table 16: SPriV scores across all gatekeepers and models. Lower SPriV indicates better privacy protection. N Samples = total test samples (1100 per model = 100 per PII type

\times

11 types). Missed Detections = leaked samples that passed through the gatekeeper undetected.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 278 papers

From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves
Feb 27, 2026Haritz Puerto, Haonan Li, Xudong Han +2Large Reasoning Models Privacy Leakage in Language Models
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany · Mohamed bin Zayed University of Artificial Intelligence, UAE · LibrAI
InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants
Oct 25, 2026Pengrun Huang, Yiwei Fu, Luoxi Meng +1Privacy Leakage in Language Models Privacy Protection for LLM Agents
UC San Diego · University of Illinois Urbana-Champaign
Stealing Reasoning Traces from Proprietary LLM APIs
Aug 10, 2026Alexander Panfilov, David Schmotz, Ilia Shumailov +5Model Extraction Attacks Adversarial Attacks
MATS Research · ELLIS Institute Tübingen · Max Planck Institute for Intelligent Systems +4

Feb 27, 2026J/K move · Enter open · S save

From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves

Haritz Puerto, Haonan Li, Xudong Han, Timothy Baldwin+1

Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science, Technical University of Darmstadt and National Research Center for Applied Cybersecurity ATHENE, Germany · Mohamed bin Zayed University of Artificial Intelligence, UAE · LibrAI

Large Reasoning Models Privacy Leakage in Language Models Privacy-Preserving Language Models Instruction Following

Open paper AskPDF arXiv

Large reasoning models (LRMs) produce reasoning traces (RTs) that often contain sensitive information. These leaky thoughts are difficult to control and frequently violate explicit privacy directives. Because RTs can be exposed through prompt injection attacks, this becomes a direct privacy risk to the user. We approach this as a controllability problem: since privacy directives are themselves instructions, improving instruction-following (IF) within the RT provides a direct path to reducing privacy leaks. To this end, we introduce an SFT dataset that teaches models to follow general instructions throughout their reasoning process, and propose Staged Decoding, a simple decoding strategy that decouples RT and answer generation using separate LoRA adapters to maximize IF of each component. We evaluate our approach on six models from two families (1.7B-14B parameters), across two IF benchmarks and two privacy benchmarks. Our method yields substantial improvements, with gains of up to 20.9 points in IF and 51.9 percentage points on privacy benchmarks, though these can come at the cost of task utility due to the trade-off between reasoning performance and IF. Our results show that improving IF in LRMs can significantly enhance privacy, suggesting a promising direction for future privacy-aware LRMs. Our code is available at https://github.com/UKPLab/arxiv2026-controllable-reasoning-models.

Abstract

Privacy Leakage in Language Models

Explore similar work

From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves

InferPriv: Evaluating Inferential Privacy Leakage in LLM Assistants

Stealing Reasoning Traces from Proprietary LLM APIs