cs.LGJul 23, 2026

TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning

Authors: Chaofan Pan, Lingfei Ren, Xiangyu Jiang, Yanhua Li, Xuemei Cao, Xiangkun Wang, Hao Yu, Wei Wei, +1 more

Abstract

Offline Reinforcement Learning (RL) agents are trained on fixed behavioral trajectories, which makes trajectory-level deletion important when selected data must be removed after training. Evaluating such deletion is difficult because a lower membership score can reflect trajectory removal, residual memorization visible to another attack, or policy collapse that destroys useful behavior. We introduce Trajectory-level memOrization and Unlearning in offline RL (TOUR), a benchmark that combines trajectory-level partitioning, matched non-member controls, retraining references, retained-performance anchors, and multi-attack privacy auditing. Across D4RL locomotion experiments and an exploratory AntMaze extension, TOUR shows that common deletion baselines have environment-dependent privacy-utility behavior. Retraining and fine-tuning often provide stronger retained-utility references than uniform GA+Refit, while TrajDeleter remains a useful comparator but is not uniformly stronger under the same audit. Reference-model, threshold, deviation, equivalence, action-error, representation-based, and query-limited attacks further show that a single likelihood-based membership score can overstate deletion quality. In the evaluated settings, conclusions about offline RL unlearning are therefore not stable under single-score auditing. They depend on matched non-member construction, retraining-relative calibration, attack family, retained utility, and explicit scope for diagnostic architecture or component-level evidence.

Explore similar work

CardsList
Jun 2, 2026cs.LG

Exact Unlearning in Reinforcement Learning

We formulate the problem of \emph{exact unlearning} in reinforcement learning, where the goal is to design an efficient framework that enables the removal of any user's data upon deletion request, i.e., the online learner's output after unlearning is \emph{indistinguishable} from what would have been produced had the deleted user never interacted with the learner. For any ρ>0ρ>0, we show that there exists a reinforcement learning (RL) algorithm that is ρρ-TV-stable and supports an exact unlearning procedure whose expected computational cost is only a ρln⁡Tρ\sqrt{\ln T} fraction of the computational cost of retraining from scratch. We construct such a ρρ-TV-stable RL algorithm for tabular Markov decision processes (MDPs), which achieves a regret bound of O(H2SAT+H3S2A+H2.5S2A/ρ)\mathcal{O}(H^2 \sqrt{SAT} + H^3 S^2 A + {H^{2.5} S^2 A}/ρ), where S,A,HS, A, H, and TT denote the number of states, the number of actions, the episode horizon, and the number of episodes, respectively. We also establish a lower bound of Ω(H ⁣SAT ⁣+ ⁣SAH/ρ)Ω(H\sqrt{\!SAT}\! +\! {SAH}/ρ) for ρρ-TV-stable RL algorithms, showing that our algorithm is nearly minimax optimal.
Thanh Nguyen-Tang, Raman Arora
Machine UnlearningComputational Cost
AskPDF
Jun 13, 2026cs.CL

Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning

LLM unlearning has emerged as a cost-effective alternative to full retraining for removing hazardous knowledge from pretrained models while preserving general utility. Recent RL-based methods such as RULE reformulate unlearning as learning a refusal behavior, but their on-policy optimization repeatedly samples from the same forget and retain/boundary prompts throughout training. We identify a critical inefficiency in this process: easy cases quickly converge and provide little useful gradient signal, while hard cases near the forget/retain boundary continue to produce low-reward rollouts that are discarded after a single use. To address this issue, we propose ReRULE, an off-policy replay enhancement for reinforcement unlearning. ReRULE stores low-reward hard-case rollout groups in a replay buffer during early GRPO training and reuses them in later stages through importance-sampled off-policy updates, redirecting computation toward boundary cases that still require learning. Theoretically, we show that ReRULE yields a tighter hard-case convergence bound than pure on-policy RULE. Empirically, ReRULE improves MUSE-Books Retain Quality from 46.3 to 56.2 while adding only 5--11% training time across benchmarks. Its limited improvement on the simpler TOFU setting further supports the intended conditional behavior: replay is most beneficial when the hard/easy disparity is pronounced.
Zirui Pang, Chenlong Zhang, Haosheng Tan +3
Large Language Model UnlearningOff-Policy Reinforcement LearningSafe Reinforcement LearningOptimization Convergence Analysis
AskPDF
Jul 7, 2026cs.LG

Auditing of Unlearning Algorithms

Evaluating whether unlearning algorithms truly remove training data influence remains an open challenge. We propose a practical auditor that computes data-dependent lower bounds on the unlearning parameter ε\varepsilon using membership inference attacks. Evaluating multiple unlearning algorithms, we find a sharp separation: algorithms with rigorous guarantees, such as model clipping and rewind-to-delete, achieve very small ε\varepsilon bounds that do not falsify their unlearning guarantees, whereas empirical methods such as Hessian-based unlearning, interleaved ascent-descent, ascent on the forget set, and fine-tuning on the retain set exhibit large bounds, indicating poor unlearning. Our auditor provides a practical tool for empirically falsifying unlearning claims through a hypothesis-testing framework, and we validate it on CIFAR-100 and Shakespeare text.
Sahasrajit Sarmasarkar, Anastasia Koloskova, Sanmi Koyejo
Machine UnlearningUnlearning AimsMembership Inference AttacksAlgorithm AuditingEmpirical StudyLower Bounds
AskPDF