CURE:Circuit-Aware Unlearning for LLM-based Recommendation
Authors: Ziheng Chen, Jiali Cheng, Zezhong Fan, Hadi Amiri, Yunzhi Yao, Xiangguo Sun, Yang Zhang
Organizations: Walmart Global Tech, Sunnyvale, CA, USA · University of Massachusetts Lowell, Lowell, MA, USA · Zhejiang University, Hangzhou, China · The Chinese University of Hong Kong, Hong Kong, China · National University of Singapore, Signapore
Recent advances in large language models (LLMs) have opened new opportunities for recommender systems by enabling rich semantic understanding and reasoning about user interests and item attributes. However, as privacy regulations tighten, incorporating user data into LLM-based recommendation (LLMRec) introduces significant privacy risks, making unlearning algorithms increasingly crucial for practical deployment. Despite growing interest in LLMRec unlearning, most existing approaches formulate unlearning as a weighted combination of forgetting and retaining objectives while updating model parameters in a uniform manner. Such formulations inevitably induce gradient conflicts between the two objectives, leading to unstable optimization and resulting in either ineffective unlearning or severe degradation of model utility. Moreover, the unlearning procedure remains largely black-box, undermining its transparency and trustworthiness. To tackle these challenges, we propose CURE, a circuit-aware unlearning framework that disentangles model components into functionally distinct subsets and selectively updates them. Here, a circuit refers to a computational subgraph that is causally responsible for task-specific behaviors. Specifically, we extract the core circuits underlying item recommendation and analyze how individual modules within these circuits contribute to the forget and retain objectives. Based on this analysis, these modules are categorized into forget-specific, retain-specific, and task-shared groups, each subject to function-specific update rules to mitigate gradient conflicts during unlearning. Experiments on real-world datasets show that our approach achieves more effective unlearning than existing baselines.
Figures & tables
Figure 1. Normalized Alignment Values of forget and retain losses on MovieLens-1M using Llama-2 (7B)
Figure 2. Schematics of CURE: (i) Locating Circuits in LLMREC; (ii)Selective Circuits Updating
Figure 3. Unlearning effectivenss and model performance on Movie and Goodreads.
GoodReads
MovieLens-1M
Method
GPT-2
LLaMA-2
GPT-2
LLaMA-2
Retrain
15,800
128,000
29,300
132,300
E2URec
1,900
13,600
3,200
21,800
RecEraser
4,200
32,500
7,100
49,700
SISA
3,700
29,800
6,800
45,300
ROME
600
3,600
1,700
12,100
Table 1. Unlearning time comparison (s) ↓ .
Figure 4. Transparent comparison of circuit-aware unlearning with ROME and WISE
Figure 5. The distribution of gradient conflicts ( cosψ ) on GoodReads. The left and right columns use GPT-2 Large and Lamma-2 7B as backone, respectively
ωr
Llama-2
GPT-2
AUC (↑)
ACC (↑)
JSD (↓)
AUC (↑)
ACC (↑)
JSD (↓)
0.2
71.3
69.1
0.91
69.1
67.5
0.94
0.4
73.5
71.2
0.90
71.2
69.5
0.98
0.6
75.3
72.1
0.91
72.9
70.6
0.98
0.8
74.9
71.8
0.95
72.9
70.5
1.05
Table 2. Ablation study of ωr on GoodRead across different backbone. Best performance is bold and second best is underlined .
Machine unlearning removes the influence of selected data or concepts from trained models while preserving useful behavior and avoiding full retraining.
Generative Recommendation (GenRec) typically leverages Large Language Models (LLMs) to redefine personalization as an instruction-driven sequence generation task. However, fine-tuning on user logs inadvertently encodes sensitive attributes into model parameters, raising critical privacy concerns. Existing Machine Unlearning (MU) techniques struggle to navigate this tension due to the Polysemy Dilemma, where neurons superimpose sensitive data with general reasoning patterns, leading to catastrophic utility loss under traditional gradient or pruning methods. To address this, we propose Utility-aware Contrastive AttenuatioN (U-CAN), a precision unlearning framework that operates on low-rank adapters. U-CAN quantifies risk by contrasting activations and focuses on neurons with asymmetric responses that are highly sensitive to the forgetting set but suppressed on the retention set. To safeguard performance, we introduce a utility-aware calibration mechanism that combines weight magnitudes with retention-set activation norms, assigning higher utility scores to dimensions that contribute strongly to retention performance. Unlike binary pruning, which often fragments network structure, U-CAN develop adaptive soft attenuation with a differentiable decay function to selectively down-scale high-risk parameters on LoRA adapters, suppressing sensitive retrieval pathways and preserving the topological connectivity of reasoning circuits. Experiments on two public datasets across seven metrics demonstrate that U-CAN achieves strong privacy forgetting, utility retention, and computational efficiency.
Zezheng Wu, Rui Wang, Xinghe Cheng +4
Guilin University of Electronic Technology, Guilin, China · University of Southampton, Southampton, United Kingdom · Jinan University, Guangzhou, China +1
Large Language Models (LLMs) often memorize sensitive or harmful information, necessitating effective machine unlearning techniques. While existing parameter-efficient unlearning methods have shown promise, they still struggle with the forget-retain trade-off. This can be attributed to their reliance on parameter importance metrics to identify parameters that are important exclusively for the forget set, which is fundamentally limited by the superposition phenomenon. Due to the polysemantic nature of LLM parameters, such an importance metric may struggle to disentangle parameters associated with the forget and retain sets. In this work, we propose Representation-Guided Low-rank Unlearning (REGLU), a novel approach that leverages the geometric properties of representation spaces to achieve robust and precise unlearning. First, we develop a representation-guided initialization for LoRA that identifies the optimal subspace for selective forgetting. Second, we introduce a regularization loss that constrains the outputs of the LoRA update to lie in the orthogonal complement of the retain set's representation subspace, thereby minimizing interference with the model's performance on the retain set. We evaluate REGLU on the TOFU and WMDP benchmarks across multiple models. Our results demonstrate that REGLU consistently outperforms state-of-the-art baselines, achieving superior unlearning quality while maintaining higher model utility.
Zeguan Xiao, Lang Mo, Yun Chen +4
Shanghai University of Finance and Economics · Southern University of Science and Technology · Deepexi Technology Co. Ltd.
Generative recommendation formulates next-item prediction as autoregressive generation over semantic ID (SID) sequences derived from users' historical interactions, making modern recommender systems structurally similar to large language models (LLMs). As privacy and safety concerns grow, these systems increasingly require concept unlearning to remove sensitive or harmful concepts associated with items. However, existing LLM unlearning methods cannot be directly applied to generative recommendation. Unlike word tokens with explicit semantics, SIDs are abstract identifiers that are often shared by both forget and retain items, leading to severe conflicts between concept removal and recommendation utility preservation. To address this challenge, we propose TRACER, an end-to-end concept unlearning framework based on token reassignment. Rather than directly suppressing shared SIDs, TRACER reassigns concept-related items to alternative tokens that better facilitate forgetting while minimizing side effects on retained items. We further introduce a coherence regularizer to preserve semantic consistency among retain items during unlearning. Experiments on real-world recommendation datasets demonstrate that TRACER effectively removes target concepts while substantially better preserving recommendation utility than existing unlearning baselines.
Ziheng Chen, Jiali Cheng, Zezhong Fan +4
Stony Brook University · University of Massachusetts Lowell · Columbia University +3