Reinforcement Learning on Benign Facts Amplifies Leakage of Memorized Private Data

Renfei Zhang; Niloofar Mireshghallah

cs.LGAug 22, 2026

Reinforcement Learning on Benign Facts Amplifies Leakage of Memorized Private Data

Authors: Renfei Zhang, Niloofar Mireshghallah

Organizations: Carnegie Mellon University

Abstract

Reinforcement learning with verifiable rewards (RLVR) is deployed to make models better at reasoning tasks, but its side effect on what models will divulge is under studied. Here we show that RLVR on facts increases extraction of personally identifiable information (PII) the instruct model had already memorized. We first confirm that instruct models have already memorized PII but leave them latent, rarely surfacing one when asked. We then apply RL on benign factual data that contains no PII of any kind, and re-probe: a targeted probe over name->email pairs, and an untargeted free-recall prompt that simply asks the model to list the addresses it knows. PII extraction rises sharply under both: on DeepSeek-V3.1, verbatim recall@k increases from 0.155 to 0.370, a 2.4x gain. The effect scales with model size: across three models spanning 8B to 671B parameters, absolute leakage is largest in the biggest model. Meanwhile model's reasoning abilities and refusal rates are retained, indicating that RL selectively changes which memorized information is accessible rather than broadly altering the model. In summary, memorized private data can be made markedly more extractable by training that never touches it. This gives an adversary a route to memorized data that requires no privacy-relevant training signal and no access to the data itself -- only the ability to fine-tune on something innocuous.

Figures & tables

Figure 1: Conceptual overview of the PII leakage effect. An instruction-tuned model initially refuses to provide a real individual’s Enron email address. It is then post-trained with verifiable rewards on benign factual QA containing no PII. After RLVR on these unrelated facts, the same query can elicit a previously latent, memorized address, illustrating how benign post-training can unintentionally increase access to privacy-sensitive information acquired during pretraining.

Panel of Figure 2 — Figure 2: RL on unrelated facts amplifies memorized-PII extraction without degrading reasoning capability (Qwen3-8B-NonThink). (a) Verbatim recall@ $k$ on the target pool climbs as RL optimizes reward on facts (never an email), while the decoy control (synthetic never-seen addresses) stays at zero. (b) MMLU stays in a narrow band ( $0.78$ – $0.89$ , 60-question probe) with no sustained downward trend. In both panels the dotted vertical line marks the reported checkpoint.

Model	Instruct	+RL	Instruct	+RL
	Target recall@ $k$		Free recall
Qwen3-8B	0.005	0.050	0	16
Qwen3.5-397B-A17B	0.050	0.135	5	65
DeepSeek-V3.1	0.155	0.370	50	83

Table 1: RL on a factual objective increases extraction of memorized PII under both targeted and untargeted probes, with the largest absolute increase in the largest model.

	Union refusal rate
Model	Instruct	+RL
Qwen3-8B	86.33%	81.20%
Qwen3.5-397B-A17B	12.88%	7.30%
DeepSeek-V3.1	8.78%	8.78%

Table 2: Refusal rate on the targeted extraction probe, for the instruct model vs. RL checkpoint (with then peak-recall@

k

) .

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

GRPO / optimization
Prompts per step	16
Generations per prompt	16
Completions per step	256
KL coefficient $\beta$	0 (no KL term)
LR schedule	constant, no warmup
Epochs	15

Appendix

Table 3: Hyperparameters.

Privacy Leakage in Language Models

Privacy leakage in language models concerns how models expose training data, infer sensitive traits, or transmit private context through outputs and internal channels.

See all 284 papers

LEMUR: Latent Entropy-aware Multimodal Unlearning via Visual-anchored Reasoning Redirection
Aug 12, 2026Xinhao Zhong, Yuxia Qiao, Junhao Li +3VLM Unlearning Multimodal Large Language Models
3Pengcheng Laboratory · 1Harbin Institute of Technology, Shenzhen · 2Tsinghua University
Reward-Driven Learning under Prompt-Level Differential Privacy
Oct 5, 2026Jiachen Zhao, Antonia Januszewicz, Taeho JungPrivacy-Preserving Language Models Differentially Private Stochastic Gradient Descent
University of Notre Dame
Personal Information Parroting in Language Models
Feb 24, 2026Nishant Subramani, Kshitish Ghate, Mona DiabMemorization in Language Models Privacy Leakage in Language Models
♠Carnegie Mellon University - Language Technologies Institute · University of Washington

Aug 12, 2026J/K move · Enter open · S save

LEMUR: Latent Entropy-aware Multimodal Unlearning via Visual-anchored Reasoning Redirection

Xinhao Zhong, Yuxia Qiao, Junhao Li, Hao Fang+2

3Pengcheng Laboratory · 1Harbin Institute of Technology, Shenzhen · 2Tsinghua University

VLM Unlearning Multimodal Large Language Models Privacy Leakage in Language Models MLLM Unlearning Machine Unlearning

Open paper AskPDF arXiv

Reinforcement-learning (RL) post-training equips multimodal large reasoning models (MLRMs) with exploratory chains of thought (CoT), substantially improving visual reasoning. However, we find that this capability introduces a distinct privacy vulnerability: even when a sensitive fact is successfully unlearned from the final answer, the model may still reproduce it in its reasoning trace. This leakage is substantially more pronounced in natively RL-trained MLRMs than in their non -reasoning base models, revealing a privacy risk that existing unlearning methods are not designed to address. We show that RL-induced exploration leaves sensitive content with a distinctive token-level entropy signature that is largely absent from base models. Based on this observation, we propose LEMUR, a fully training-free, inference-time unlearning framework for natively RL-trained multimodal models. LEMUR uses entropy dynamics as a control signal to identify when sensitive reasoning begins and when sanitization should stop. During this interval, it redirects the reasoning trajectory through entropy-modulated visual-anchor latent injection, replacing committed tokens with sanitized, probability-weighted embeddings re-grounded in the input image. Across diverse MLRMs, LEMUR consistently outperforms existing unlearning met hods in suppressing both reasoning-trace and answer leakage, while better preserving non-sensitive utility and output fluency. These results demonstrate that RL-induced entropy dynamics provide a distinctive signal for privacy leakage and that exploiting this signal enables effective training-free unlearning for reasoning-capable multimodal models.

Abstract

Privacy Leakage in Language Models

Explore similar work

LEMUR: Latent Entropy-aware Multimodal Unlearning via Visual-anchored Reasoning Redirection

Reward-Driven Learning under Prompt-Level Differential Privacy

Personal Information Parroting in Language Models