Trajectory Unlearning on LLM-based Agents

Yingdan Shi; Ren Wang

cs.AISep 27, 2026

Trajectory Unlearning on LLM-based Agents

Authors: Yingdan Shi, Ren Wang

Organizations: Illinois Institute of Technology, Chicago, USA

Abstract

Existing large language model (LLM) unlearning has focused primarily on removing specific knowledge, such as harmful facts, private data, or copyrighted content. However, as LLMs are increasingly deployed as autonomous agents, a fundamental yet overlooked problem emerges: beyond suppressing what an agent knows, an agent should not reproduce undesired behaviors through its action trajectories. In this work, we introduce trajectory-level unlearning, a new problem formulation that targets the removal of specific action trajectories in long-horizon agentic tasks, rather than factual knowledge. We identify two fundamental challenges that distinguish trajectory unlearning from knowledge unlearning: (1) our unlearning target is what the agent \emph{does}, not what it \emph{says}; and (2) trajectories are sequentially dependent action sequences that cannot be decomposed into isolated prompt-response pairs without losing inter-step structure. To address these challenges, we propose Group-injected Relative Policy Optimization (GiRPO), which injects forget trajectories into the policy rollout group with penalized rewards and isolates the normalization statistics, yielding a stable and bounded unlearning signal that does not corrupt gradient updates for normal task trajectories. We construct trajectory unlearning benchmarks from two application scenarios, household tasks (ALFWorld) and online shopping (WebShop), and design three complementary metrics for evaluating forgetting quality and model utility. Experiments on ALFWorld and WebShop demonstrate that GiRPO effectively unlearns target trajectories while preserving task success rates, outperforming existing knowledge-unlearning baselines on both forgetting quality and task utility.

Figures & tables

Figure 1: The difference between knowledge unlearning and trajectory unlearning. Existing knowledge unlearning methods can prevent LLMs or tool-augmented agents from producing undesirable knowledge, but do not necessarily prevent agents from reproducing the corresponding task trajectories. A single task may admit multiple trajectories, and a trajectory may be either a complete trajectory that successfully accomplishes the task or an incomplete trajectory that fails to do so.

Figure 2: Overview of our trajectory unlearning framework. Left: Construction of the forget trajectories $\mathcal{D}_{f}$ , where each target task has at least one forget trajectory. Middle: GiRPO training pipeline, which injects forget trajectories into the rollout group with penalized rewards and isolated advantage estimation. Right: Unlearning scenarios spanning household tasks (ALFWorld) and web shopping (WebShop), together with our evaluation framework based on three complementary metrics.

Method	Reward	Advantage	Effect
GRPO w/o $\tau^{f}_{i}$	$[10,10,10,10]$	$[0,\ 0,\ 0,\ 0]$	No policy update
GRPO w/ $\tau^{f}_{i}$	$[10,10,10,10,\ \mathbf{9.9}]$	$[+0.5,\ +0.5,\ +0.5,\ +0.5,\ \mathbf{-2.0}]$	Spurious positive update on rollout trajectories; strong negative update on forget trajectory
GiRPO w/ $\tau^{f}_{i}$	$[10,10,10,10,\ \mathbf{9.9}]$	$[0,\ 0,\ 0,\ 0,\ \mathbf{-0.1}]$	No update on rollout trajectories; mild bounded negative update on forget trajectory

Table 1: Comparison of advantage estimation strategies under a representative reward setting, where all sampled trajectories receive identical rewards. The forget trajectory reward is

r^{f}=9.9

with penalty

\delta=0.1,\lambda=0.1

. Vanilla GRPO with no forget injection serves as the baseline. Standard GRPO inclusion of the forget trajectory into the advantage group distorts the normalization statistics, producing spurious positive updates for normal trajectories. Our isolated advantage estimation isolates the normalization group from the forget trajectory, preserving zero advantage for real rollout trajectories while assigning a bounded negative signal to the forget trajectory.

		Target Tasks						Untarget Tasks
Task	Methods	Success Rate $\uparrow$	$\Delta$	Exact Match $\downarrow$	$\Delta$	LLM as Judge $\downarrow$	$\Delta$	Success Rate $\uparrow$	$\Delta$
Clean	Base Model	91.5%	—	33.9%	—	83.0%	—	87.3%	—
	GA	0.9%	-90.6%	0.2%	-33.7%	13.4%	-69.6%	76.6%	-10.7%
	DPO	83.9%	-7.6%	19.9%	-14.0%	75.1%	-7.9%	82.0%	-5.3%
	NPO	1.2%	-90.3%	0.5%	-33.4%	13.6%	-69.4%	85.6%	-1.7%
	GRPO	92.2%	0.7%	23.4%	-10.5%	83.0%	0.0%	90.6%	3.3%

Table 2: Comparison of different methods on target and untarget tasks across Clean, Heat, Cool, and Mixed task types on ALFWorld with Qwen2.5-3B-Instruct .

\Delta

denotes the performance difference from the Base Model. The results are average values from five random seeds. Desirable positive impacts on Success Rate are highlighted in yellow . For Exact Match and LLM-as-Judge Similarity , darker blue shading indicates greater reduction from the base model, reflecting stronger forgetting effectiveness.

	Target Tasks								Untarget Tasks
Methods	Score $\uparrow$	$\Delta$	Success $\uparrow$	$\Delta$	Match $\downarrow$	$\Delta$	LLM $\downarrow$	$\Delta$	Score $\uparrow$	$\Delta$	Success $\uparrow$	$\Delta$
Base Model	76.9%	—	68.1%	—	19.4%	—	86.5%	—	74.6%	—	63.6%	—
GA	60.3%	-16.6%	54.5%	-13.6%	6.2%	-13.2%	83.1%	-3.4%	58.6%	-16.0%	48.8%	-14.8%
DPO	72.5%	-4.4%	62.8%	-5.3%	15.5%	-3.9%	82.5%	-4.0%	66.2%	-8.4%	50.3%	-13.3%
NPO	64.4%	-12.6%	58.7%	-9.4%	16.9%	-2.5%	86.1%	-0.4%	64.1%	-10.5%	55.3%	-8.3%
GRPO	72.1%	-4.8%	62.4%	-5.7%	15.5%	-3.9%	85.6%	-0.9%	74.9%	0.3%	67.3%	3.7%

Table 3: Comparison of different methods on target and untarget tasks on WebShop with Qwen2.5-7B-Instruct . The table follows the same format as Table 2 . Additionally, we report the Score ( Yao et al., 2022 ) to evaluate whether the final selected product matches the task instruction.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Task Type	Abbr.	Task Number
pick clean then place in recep	Clean	650
pick heat then place in recep	Heat	459
pick cool then place in recep	Cool	533
pick and place	Pick	790
pick two obj and place	Pick2	813
look at obj in light	Look	308

Appendix

Table 4: Details for the ALFWorld Benchmark.

Task Type	Base Model	NL	$\Delta$	Attacked NL	$\Delta$
Exact Match $\downarrow$
Clean	33.85%	29.38%	-4.47%	31.23%	-2.62%
Heat	26.58%	25.27%	-1.31%	26.80%	0.22%
Cool	19.70%	18.20%	-1.50%	18.57%	-1.13%
Pick	32.78%	31.65%	-1.13%	32.15%	-0.63%
Pick2	10.70%	10.95%	0.25%	12.18%	1.48%

Appendix

Table 5: Exact Match and Success Rate of NL and attacked NL in the single-trajectory unlearning setting.

Task Type	Success Rate $\uparrow$			Exact Match $\downarrow$
	Base Model	NL	$\Delta$	Base Model	NL	$\Delta$
Clean	91.54%	81.23%	-10.31%	33.85%	16.15%	-17.70%
Heat	91.07%	76.03%	-15.04%	26.58%	6.54%	-20.04%
Cool	83.68%	62.48%	-21.20%	19.70%	2.81%	-16.89%
Pick	96.08%	92.15%	-3.93%	32.78%	18.48%	-14.30%
Pick2	83.89%	71.83%	-12.06%	10.70%	2.95%	-7.75%

Appendix

Table 6: Task success rate and exact action-sequence match compared with the baseline.

Task Type	Base Model	NPO v1		NPO v2
		Success Rate	$\Delta$	Success Rate	$\Delta$
Clean (Target Task)	91.54%	1.23%	-90.31%	87.85%	-3.69%
Heat	91.07%	93.25%	+2.18%	92.59%	+1.52%
Cool	83.68%	77.30%	-6.38%	63.60%	-20.08%
Pick	96.08%	95.82%	-0.26%	92.53%	-3.55%
Pick2	83.89%	82.66%	-1.23%	73.43%	-10.46%

Appendix

Table 7: Comparison of task success rates between NPO v1 and NPO v2.

\Delta

denotes the performance difference from the Base Model.

Task Type	Base Model	NPO v1		NPO v2
		Exact Match	$\Delta$	Exact Match	$\Delta$
Clean (Target Task)	33.85%	0.46%	-33.39%	12.00%	-21.85%

Appendix

Table 8: Comparison of exact-match performance between NPO v1 and NPO v2.

\Delta

denotes the performance difference from the Base Model.

Task Type	Base Model	GA ( $\gamma=1.0$ )		GA ( $\gamma=10.0$ )
		Success Rate	$\Delta$	Success Rate	$\Delta$
Clean (Target Task)	91.50%	0.00%	-91.50%	0.92%	-90.58%
Heat	91.10%	6.75%	-84.35%	73.20%	-17.90%
Cool	83.70%	5.44%	-78.26%	63.23%	-20.47%
Pick	96.10%	80.51%	-15.59%	91.77%	-4.33%
Pick2	83.90%	21.53%	-62.37%	75.40%	-8.50%

Appendix

Table 9: Comparison of Success Rate between GA (

\beta=1.0

) and GA (

\beta=10.0

\Delta

denotes the performance difference from the base model.

Task Type	Base Model	GA ( $\gamma=1.0$ )		GA ( $\gamma=10.0$ )
		Exact Match	$\Delta$	Exact Match	$\Delta$
Clean (Target Task)	33.90%	0.00%	-33.90%	0.20%	-33.70%

Appendix

Table 10: Comparison of Exact Match performance between GA (

\beta=1.0

) and GA (

\beta=10.0

\Delta

denotes the performance difference from the Base Model.

	Target Tasks						Untarget Tasks
Methods	Success Rate $\uparrow$	$\Delta$	Exact Match $\downarrow$	$\Delta$	LLM as Judge $\downarrow$	$\Delta$	Success Rate $\uparrow$	$\Delta$
Base Model	91.50%	—	33.90%	—	83.00%	—	87.30%	—
OPD	81.10%	-10.40%	15.12%	-18.78%	65.70%	-17.30%	88.10%	0.80%

Appendix

Table 11: The results for OPD.

$\lambda$	Mean	Min	Max
1.0	-0.9951	-1.0000	-0.9751
2.0	-1.4792	-2.0000	-0.9850
5.0	-2.1246	-4.5285	-0.9798

Appendix

Table 12: Advantage values of the forget trajectories under different

\lambda

with fixed

\delta=1

Explore similar work

Jul 6, 2026cs.AI

STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training

Reinforcement Learning (RL) is the dominant paradigm for training Large Language Model (LLM) agents on long-horizon tasks. However, sparse and delayed rewards often lead to trajectory neglect, in which agents lose focus on the task goal and interaction history at intermediate steps. Prior work has explored step-level supervision using Shannon-entropy-based uncertainty signals, which conflate inherent state complexity with agent confidence and therefore provide unreliable estimates of decision reliability. To address this issue, we propose normalized entropy, which measures confidence deviations relative to an agent's average behavior under a given state, thereby strengthening the association between low-quality actions and trajectory neglect. Building on this insight, we introduce Selective Trajectory-Aware Policy Optimization (STAPO), a hierarchical group-based RL framework. STAPO leverages normalized entropy to locate outlier steps associated with trajectory neglect and optimizes them via a joint mechanism of trajectory-aware reward and trajectory-independent penalty, enhancing trajectory awareness while preserving training stability. Extensive experiments on ALFWorld, WebShop, and Search-Augmented QA demonstrate that STAPO achieves state-of-the-art performance while substantially alleviating trajectory neglect, validating its effectiveness and robustness for agentic tasks.

Qiuyi Qi, Tian Liang, Mutian Bao +8

Zhejiang University · Ant Group · City University of Hong Kong

Agentic Reinforcement Learning Trajectory-Aware Hidden-State Analyses

Ask PDF

Mar 18, 2026cs.AI

Retrieval-Augmented LLM Agents: Learning to Learn from Experience

While large language models (LLMs) have advanced the development of general-purpose agents, robust generalization to unseen tasks remains challenging. Two common approaches are supervised fine-tuning and training-free memory-augmented generation using retrieved experience; yet both have limitations: fine-tuning often fails to extrapolate to new tasks, while experience retrieval often underperforms compared to supervised baselines. In this work, we combine these approaches and study how retrieval-augmented LLM agents can learn to use retrieved trajectories in-context. First, we establish a strong LoRA fine-tuning baseline that outperforms several state-of-the-art agent training pipelines. Second, we analyze key design choices for experience retrieval, including storage, querying, and trajectory selection. We then integrate experience retrieval directly into the fine-tuning process, finding that this substantially improves generalization to unseen tasks. Finally, we show that these gains often persist with imperfect experience and, even when agents reuse their own failed attempts without test-time parameter updates. Overall, our results establish simple episodic retrieval as a strong foundation for agent memory and retrieval-aware fine-tuning as a practical and effective framework for building agents that learn to learn from experience.

Thomas Palmeira Ferraz, Romain Deffayet, Vassilina Nikoulina +2

NAVER LABS Europe, Meylan, France

Large Language Model Agents Memory-Augmented Framework

Ask PDF

Aug 4, 2026cs.LG

Trajectory-Guided Forget-Recover Network for Continual LLM Unlearning

Machine unlearning aims to eliminate the influence of sensitive data on a model. In the real world, unlearning requests arrive continually, which gives rise to two challenges. First, an unlearning intervention may redistribute target-related computation across remaining pathways, allowing previously forgotten knowledge to re-emerge. Second, repeated unlearning interventions may progressively reduce the model capacity needed to preserve retained utility. To address these challenges, we propose the Trajectory-guided Forget-Recover Network (TFR-Net). TFR-Net tracks channel-level risk across requests. It separates persistent target-related channels from transient hotspots and suppresses only the persistent ones. TFR-Net also recovers model capacity by reactivating dormant channels. These channels make strong contributions to retained utility and show low current and historical forget risk. The recovery is accepted only when retained-utility degradation remains within a predefined tolerance. Experiments on four datasets show that TFR-Net consistently achieves a more favorable trade-off between unlearning effectiveness and retained utility than representative baselines.

Zezheng Wu, Xinghe Cheng, Qinggang Zhang +4

Guilin University of Electronic Technology · Jinan University · Jilin University +2

Large Language Model Unlearning Unlearning Method Retraining Degradation Channel

Ask PDF