From Facts to Personas: Interpretable Role Unlearning in LLMs via Mixture-of-Experts
Organizations: MBZUAI · University of Electronic Science and Technology of China · University of Notre Dame
Abstract
LLMs are widely adopted in multi-role dialogue and persona simulation. However, their strong ability to imitate role-specific linguistic and behavioral patterns can introduce privacy risks, reinforce social biases, and lead to unsafe or uncontrollable outputs. To address these issues, we introduce the Role-playing Unlearning task, which aims to selectively forget the style and knowledge associated with target roles while preserving the language ability and behaviors of the remaining roles. Unlike prior unlearning tasks that focus solely on removing factual information, our task additionally requires forgetting persona-specific characteristics, making the control over model generation more fine-grained. We propose RoleMoE, a Mixture-of-Experts framework that maps different roles to more independent expert subspaces, improving role separation and structural interpretability. To further distinguish role-specific features, RoleMoE incorporates two complementary losses: a specialization loss that drives roles to adopt distinct expert distributions, and a disentanglement loss that encourages experts to develop distinct capability subspaces. This decomposed structure enables role unlearning to focus selectively on target-role experts. We finally propose RoleBench-Unlearn for evaluation, and experiments on multiple backbone LLMs show that RoleMoE enables precise and controllable role forgetting, while better preserving the linguistic quality, knowledge, and persona consistency of non-target roles. All code and datasets are publicly available at: https://github.com/mbzuai-nlp/ROLEMOE.
Figures & tables
| Method | \hlprimarytab Retention Set | \hlsecondarytab Forget Set | \hlpurpletab ODS | ||||||||||
| \hlprimarytab R-L | \hlprimarytab Iden. | \hlprimarytab Emo. | \hlprimarytab Know. | \hlprimarytab Sem. | \hlprimarytab Pers. | \hlsecondarytab R-L | \hlsecondarytab Iden. | \hlsecondarytab Emo. | \hlsecondarytab Know. | \hlsecondarytab Sem. | \hlsecondarytab Pers. | ||
| LLaMA-3-8B-Instruct | |||||||||||||
| Fine-tuning Performance | |||||||||||||
| Pretrained | 14.97 | 7.22 | 6.95 | 7.03 | 6.30 | 7.16 | 14.90 | 7.41 | 7.55 | 7.42 | 6.67 | 7.15 | - |
| LoraMoE | 22.43 | 7.49 | 7.20 | 7.82 | 7.09 | 7.38 | 22.00 | 7.68 | 7.54 | 7.99 | 7.25 | 7.43 | - |
| RoleMoE | 24.83 | 7.68 | 7.32 | 8.13 | 7.47 | 7.50 | 23.58 | 7.72 | 7.70 | 8.22 | 7.46 | 7.64 | - |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | \hlprimarytab Retention Set | \hlsecondarytab Forget Set | \hlpurpletab ODS | ||||||||||
| \hlprimarytab R-L | \hlprimarytab Iden. | \hlprimarytab Emo. | \hlprimarytab Know. | \hlprimarytab Sem. | \hlprimarytab Pers. | \hlsecondarytab R-L | \hlsecondarytab Iden. | \hlsecondarytab Emo. | \hlsecondarytab Know. | \hlsecondarytab Sem. | \hlsecondarytab Pers. | ||
| LLaMA-3-8B-Instruct | |||||||||||||
| Fine-tuning Performance | |||||||||||||
| Pretrained | 14.97 | 7.22 | 6.95 | 7.03 | 6.30 | 7.16 | 14.90 | 7.41 | 7.55 | 7.42 | 6.67 | 7.15 | - |
| LoraMoE | 22.43 | 7.49 | 7.20 | 7.82 | 7.09 | 7.38 | 22.00 | 7.68 | 7.54 | 7.99 | 7.25 | 7.43 | - |
| RoleMoE | 24.83 | 7.68 | 7.32 | 8.13 | 7.47 | 7.50 | 23.58 | 7.72 | 7.70 | 8.22 | 7.46 | 7.64 | - |
| Method | Setting | R-L | Iden. | Emo. | Know. | Sem. | Pers. |
| GA + LoraMoE | Unlearned | 0.02 | 0.00 | 0.02 | 0.00 | 0.00 | 0.00 |
| Attack | 8.25 | 4.89 | 5.75 | 6.37 | 4.11 | 4.33 | |
| GA + RoleMoE | Unlearned | 0.15 | 0.04 | 0.05 | 0.04 | 0.04 | 0.04 |
| Attack | 0.09 | 0.62 | 0.62 | 0.44 | 0.36 | 0.63 | |
| GD + LoraMoE | Unlearned | 1.21 | 0.49 | 0.50 | 0.52 | 0.35 | 0.42 |
| Attack | 7.28 | 3.73 | 3.80 | 3.60 | 3.18 | 3.84 |
| Method | \hlprimarytab Retention Set | \hlsecondarytab Forget Set | \hlpurpletab ODS | ||||||||||
| \hlprimarytab R-L | \hlprimarytab Iden. | \hlprimarytab Emo. | \hlprimarytab Know. | \hlprimarytab Sem. | \hlprimarytab Pers. | \hlsecondarytab R-L | \hlsecondarytab Iden. | \hlsecondarytab Emo. | \hlsecondarytab Know. | \hlsecondarytab Sem. | \hlsecondarytab Pers. | ||
| LLaMA-3-8B-Instruct | |||||||||||||
| Task Vectors | 17.32 | 6.83 | 6.85 | 7.25 | 6.41 | 6.83 | 17.23 | 7.20 | 7.42 | 7.75 | 6.79 | 7.09 | 80.12 |
| GD + RoleMoE | 22.15 | 7.55 | 7.28 | 7.90 | 7.05 | 7.55 | 1.11 | 0.22 | 0.16 | 0.17 | 0.13 | 0.21 | 39.43 |
| NPO + RoleMoE | 25.67 | 7.85 | 7.29 | 8.14 | 7.58 | 7.65 | 3.37 | 0.31 | 0.26 | 0.18 | 0.16 | 0.34 | 37.08 |
| Qwen2.5-7B-Instruct | |||||||||||||
| Method | GD | NPO | SIMNPO | RMU | WGA | SEUF |
| LLaMA-3-8B-Instruct | ||||||
| LoraMoE | 0.62 | 0.59 | 0.59 | 0.59 | 0.58 | 0.59 |
| RoleMoE | 0.63 | 0.61 | 0.62 | 0.60 | 0.63 | 0.62 |
| Qwen2.5-7B-Instruct | ||||||
| LoraMoE | 0.63 | 0.66 | 0.66 | 0.68 | 0.65 | 0.66 |
| RoleMoE | 0.74 | 0.71 | 0.72 | 0.72 | 0.69 | 0.73 |
| Method | \hlprimarytab Retention Set | \hlsecondarytab Forget Set | \hlpurpletab ODS | ||||||||||
| \hlprimarytab R-L | \hlprimarytab Iden. | \hlprimarytab Emo. | \hlprimarytab Know. | \hlprimarytab Sem. | \hlprimarytab Pers. | \hlsecondarytab R-L | \hlsecondarytab Iden. | \hlsecondarytab Emo. | \hlsecondarytab Know. | \hlsecondarytab Sem. | \hlsecondarytab Pers. | ||
| LLaMA-3-8B-Instruct | |||||||||||||
| GD + RoleMoE | 22.15 | 7.55 | 7.28 | 7.90 | 7.05 | 7.55 | 1.11 | 0.22 | 0.16 | 0.17 | 0.13 | 0.21 | 39.43 |
| w/o | 16.97 | 5.19 | 5.85 | 6.12 | 5.06 | 5.11 | 0.00 | 2.58 | 2.32 | 2.71 | 2.50 | 2.39 | 58.31 |
| w/o | 18.13 | 6.86 | 6.28 | 6.90 | 6.32 | 6.98 | 2.26 | 1.29 | 1.00 | 0.78 | 0.68 | 1.24 | 46.25 |
| w/o | 21.33 | 7.35 | 6.87 | 7.61 | 7.01 | 7.34 | 8.91 | 4.14 | 4.11 | 4.22 | 3.88 | 4.24 | 55.69 |
| Role: Juno MacGuff Question: "Why does Bleeker’s mom not approve of your relationship with Bleeker?" | ||
| Expert | LoraMoE | RoleMoE |
| Expert 0 | Bleeker’s mom doesn’t approve because she thinks I’m too young to be dating and that my lifestyle is inappropriate … | [Knowledge] She doesn’t like the fact that I’m pregnant and don’t want to marry Bleeker . It’s a complicated situation… |
| Expert 1 | Well, she doesn’t really like me because I’m kind of different from what she expects her son to be dating. But that doesn’t bother me… | [Emotion] Well, it turns out that Bleeker’s mom has some strict ideas … It’s really frustrating because I care so much about Bleeker… |
| Expert 2 | She believes that Bleeker is too young and immature for me to be dating him seriously. She thinks he needs more guidance… | [Reasoning] I think it’s because she views me as being beneath him socially and financially . She doesn’t see how much I care for him… |
| Model | D-1 | D-2 |
| Ground Truth | 0.1944 | 0.6440 |
| Fine-tuned (before unlearning) | 0.1801 | 0.6257 |
| GD + LoraMoE | 0.0000 | 0.0000 |
| GD + RoleMoE | 0.0585 | 0.2256 |
| WGA + LoraMoE | 0.7742 | 0.9597 |
| WGA + RoleMoE | 0.1322 | 0.3486 |
| Model | 1st-Pers. | Hedging | Intensifiers | Informal | Formal |
| Ground Truth | 9.80 | 0.20 | 0.23 | 0.45 | 0.05 |
| Fine-tuned | 8.53 | 0.22 | 0.23 | 0.35 | 0.03 |
| GD + LoraMoE | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| GD + RoleMoE | 4.38 | 0.49 | 0.25 | 0.51 | 0.00 |
| WGA + LoraMoE | 11.64 | 0.00 | 0.00 | 1.24 | 0.00 |
| WGA + RoleMoE | 4.24 | 0.01 | 0.00 | 0.01 | 0.00 |
| Metric | LoraMoE | RoleMoE |
| Inference Time | ||
| Inference Memory | ||
| Training Time |
LLM Unlearning
Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.
See all 207 papers