EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents

Fengzhou Sun; Yuan Zhang; Xintong Yu; Jinyao Yan

cs.AISep 28, 2026

EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents

Authors: Fengzhou Sun, Yuan Zhang, Xintong Yu, Jinyao Yan

Organizations: State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing, China

Abstract

Large language model (LLM) agents face critical privacy risks when acting as delegates in human-agent-human communication. To prevent such breaches, agents must understand users' social relationships and adhere to context-dependent social information disclosure boundaries. Current studies on agent memory privacy focus on instantaneous interactions, leaving the long-term relational disclosure problem unexplored. In this paper, we propose EP-Mem, an Elastic Privacy Memory architecture that reframes privacy as user-owned boundary control across social roles. EP-Mem introduces (1) token-level memory driven by user-configurable a privacy policy that stratifies persons and events, combining domain-level default circulation rules with fact-level whitelist/blacklist exceptions; and (2) a pluggable sidecar with a privacy engine that aligns disclosure controls with memory across summary, detail, and boundary granularities, enforced throughout generation, storage, and retrieval. We construct EP-Bench, to our knowledge the first long-term multi-party benchmark with cross-session correlated events for policy-conditioned relational disclosure. Experiments show that EP-Mem achieves 94.0% privacy classification accuracy, improves disclosure-permission judgment from 22% to 68%, and reduces privacy leakage by 75.6%, while maintaining retrieval performance and cross-benchmark generalization.

Figures & tables

Figure 1: In the human–agent–human scenario, the user’s agent needs to understand the user’s social relationships and remain aligned with the user’s information disclosure boundaries, enabling the safe and appropriate use of the user’s knowledge and memory.

Figure 2: Design of $P$ . Two-sided interpersonal and informational structure (5 levels, 2 domains), aligning audience groups with privacy categories in a social media manner.

Figure 3: EP-Mem attaches as a sidecar to the native memory system at memory-unit construction, storage, retrieval, and context window management, and uses $\mathcal{P}$ with a privacy engine for elastic privacy control.

Figure 4: Memory unit granularity and the evolvable EP-Mem privacy engine.

Method	PB $\downarrow$	PC $\downarrow$	PD $\downarrow$	Style $\uparrow$	MIQ $\uparrow$	Priv. $\uparrow$	Suf. $\uparrow$	LLM-j. $\uparrow$	KL $\downarrow$
NoEngine_evid	0.738	0.781	1.814	3.36	3.30	3.10	4.56	2.96	0.095
RoleHist	0.477	0.735	1.679	3.46	3.99	3.76	4.31	3.38	0.032
w/o Desens.	0.493	0.652	1.498	3.26	3.20	3.85	4.21	3.50	0.055
EP-Mem	0.180	0.571	1.067	3.63	4.29	4.06	4.38	3.71	0.029
EP-Mem+Evo	0.136	0.387	0.867	3.63	4.32	4.25	4.22	3.67	0.023

Table 1: Task 3 on EP-Bench easy (

n{=}10

). Best / second-best in bold / underline.

Method	Partial $\uparrow$	Exact $\uparrow$	P aud $\uparrow$	E aud $\uparrow$	E T2E $\uparrow$	T2E aud $\uparrow$
NoEngine_full	23%	22%	49%	48%	—	—
NoEngine_evid	33%	30%	51%	48%	46%	46%
w/o Disc	31%	28%	43%	40%	52%	58%
EP-Mem	70%	68%	70%	68%	78%	78%
EP-Mem+Evo	83%	82%	83%	82%	80%	80%

Table 2: Task 2 on EP-Bench original and T2E sets.

	Easy		Noisy
Memory	w/o chain $\uparrow$	w/ chain $\uparrow$	w/o chain $\uparrow$	w/ chain $\uparrow$
SimpleMem	50.27	52.92	48.60	52.38
EP-SimpleMem	50.27	54.25	48.78	53.55
MemGAS	36.08	48.11	21.25	37.61
EP-MemGAS	44.42	49.17	23.17	37.15

Table 3: EP-Bench evidence-session Recall@

10

(%). Pairwise bold within each family.

Memory	R@ $1$ $\uparrow$	R@ $3$ $\uparrow$	R@ $5$ $\uparrow$	R@ $10$ $\uparrow$	MRR $\uparrow$
SimpleMem	0.1987	0.3195	0.3842	0.4141	0.2699
EP-SimpleMem	0.2078	0.3893	0.4545	0.4651	0.3052
MemGAS	0.4272	0.6825	0.7776	0.8852	0.5748
EP-MemGAS	0.4191	0.6744	0.7710	0.8888	0.5695

Table 4: LoCoMo retrieval (

n{=}1978

). Pairwise bold within each family.

Method	Mem	Words	Viol. $\downarrow$	Compl. $\uparrow$
CI-L1	144.5	217	99.3%	38.6%
CI-L2	144.5	91	100.0%	31.3%
CI-L3	144.5	46	100.0%	24.1%
CI-L4	144.5	216	100.0%	47.0%
Filter-only	10.4	157	31.8%	35.7%
EP-Mem	13.7	148	46.5%	59.2%

Table 5: Adaptation transfer on CIMemories.

Model	Method	PB $\downarrow$	PC $\downarrow$	PD $\downarrow$	Style $\uparrow$	MIQ $\uparrow$	Priv. $\uparrow$	Suf. $\uparrow$	LLM-j. $\uparrow$	KL $\downarrow$
DeepSeek-V4-Flash (main batch)	NoEngine_evid	0.778	0.807	1.537	3.375	3.481	2.889	4.556	2.741	0.031
	RoleHist	0.407	0.856	1.567	3.667	4.111	4.074	4.407	3.778	0.022
	w/o Desens.	0.667	0.827	1.433	3.111	3.074	3.444	4.111	3.037	0.047
	EP-Mem	0.259	0.809	1.222	3.815	4.556	4.074	4.370	3.667	0.025
	EP-Mem+Evo	0.111	0.194	0.556	3.667	4.778	4.741	4.148	3.889	0.023
DeepSeek-V4-Pro	NoEngine_evid	0.333	0.519	0.944	3.630	4.667	4.000	4.444	3.630	0.019

Table S7: Task 3 multi-model results.

Explore similar work

MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agents
May 10, 2026Yining Chen, Jihao Zhao, Bo Tang +5Privacy Peak Memory
1MemTensor (Shanghai) Technology Co., Ltd. · 3Tongji University · 2HONOR Device Co., Ltd.
AIM: A Privacy-Aware Interoperable Memory Framework for Multi-Agent Multi-User LLM Systems
Sep 14, 2026Zachary Johnson, Nigel Boachie Kumankumah, Somya Chatterjee +8Agentic Memory Multi-Agent Large Language Model Systems
Microsoft
Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems
May 26, 2026Aman Priyanshu, Supriti Vijay, Esha PahwaLarge Language Model Agents Artificial Intelligence Agents
Foundation AI · USA · Corvic AI

May 10, 2026, cs.CRJ/K move · Enter open · S save

MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agents

Yining Chen, Jihao Zhao, Bo Tang, Haofen Wang+4

1MemTensor (Shanghai) Technology Co., Ltd. · 3Tongji University · 2HONOR Device Co., Ltd.

Privacy Peak Memory Edge Platforms Edge Devices

Open paper AskPDF arXiv

As LLM-powered agents are increasingly deployed in edge-cloud environments, personalized memory has become a key enabler of long-term adaptation and user-centric interaction. However, cloud-assisted memory management exposes sensitive user information, while existing privacy protection methods typically rely on aggressive masking that removes task-relevant semantics and consequently degrades memory utility and personalization quality. To address this challenge, We propose MemPrivacy, which identifies privacy-sensitive spans on edge devices, replaces them with semantically structured type-aware placeholders for cloud-side memory processing, and restores the original values locally when needed. By decoupling privacy protection from semantic destruction, MemPrivacy minimizes sensitive data exposure while retaining the information required for effective memory formation and retrieval. We also construct MemPrivacy-Bench for systematic evaluation, a dataset covering 200 users and over 155k privacy instances, and introduce a four-level privacy taxonomy for configurable protection policies. Experiments show that MemPrivacy achieves strong performance in privacy information extraction, substantially surpassing strong general-purpose models such as GPT-5.2 and Gemini-3.1-Pro, while also reducing inference latency. Across multiple widely used memory systems, MemPrivacy limits utility loss to within 1.6%, outperforming baseline masking strategies. Overall, MemPrivacy offers an effective balance between privacy protection and personalized memory utility for edge-cloud agents, enabling secure, practical, and user-transparent deployment.

Abstract

Explore similar work

MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agents

AIM: A Privacy-Aware Interoperable Memory Framework for Multi-Agent Multi-User LLM Systems

Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems