TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Training

Yuqian Fu; Kaiwen Jiang; Zijun Song; Jiawei Wang; Yuanheng Zhu; Dongbin Zhao

cs.LGOct 13, 2026

TSRL: Trigger-based Supervised Reinforcement Learning for Continual Agent Training

Authors: Yuqian Fu, Kaiwen Jiang, Zijun Song, Jiawei Wang, Yuanheng Zhu, Dongbin Zhao

Organizations: SKL-MAIS, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · University of Science and Technology of China

Abstract

Large language model (LLM) agents excel at complex, multi-step tasks but suffer from catastrophic forgetting during continual learning--acquiring new skills often degrades previously mastered capabilities. Existing post-training paradigms face a fundamental trade-off: Supervised Fine-Tuning (SFT) is sample-efficient but prone to severe forgetting, while Reinforcement Learning (RL) preserves prior capabilities but converges slowly without expert guidance. To bridge this gap, we introduce Trigger-based Supervised Reinforcement Learning (TSRL), a framework that integrates the strengths of both paradigms through an adaptive, state-dependent expert supervision mechanism. TSRL employs RL as the optimization backbone, activating SFT selectively on "hard states" where the policy fails. Theoretically, we show that restricting supervision to failure states derives an upper bound on the cumulative parameter updates, thereby reducing forgetting compared to full SFT. Empirically, TSRL achieves consistent gains across three agent benchmarks (ALFWorld, WebShop, and Deep Search) at both 7B and 32B scales. In continual learning scenarios, our method retains up to 9.6 points more capability on prior tasks than SFT->RL baselines, while matching or exceeding target-task performance.

Figures & tables

Figure 1: Overview of different post-training methods for continual agent learning.

Figure 2: Illustrative case using bimodal Gaussian mixtures. TSRL achieves a favorable trade-off between forgetting mitigation and effective utilization of expert data by leveraging trigger-based supervision.

Method	Sample Efficiency	Forgetting Resistance
Pure SFT	✔	✘
Pure RL	✘	✔
SFT $\to$ RL	✔	✘
TSRL (ours)	✔	✔

Table 1: Comparison of post-training paradigms. ✔ indicates the property is satisfied, and ✘ indicates it is not. TSRL simultaneously achieves both high sample efficiency and forgetting resistance.

Base: Closed-Source Model
Method	ALFWorld	WebShop (Score)	WebShop (Succ.)
Prompting GPT-4o	48.0	31.8	23.7
Prompting Gemini-2.5-Pro	60.3	42.5	35.9
Base: Qwen2.5-7B-Instruct
Prompting Qwen2.5	14.8	26.4	7.8
Prompting ReAct	31.2	46.2	19.5

Table 2: Performance comparison on ALFWorld and WebShop benchmarks (averaged over 3 random seeds). For ALFWorld, we report the average success rate (%); for WebShop, we report both the average score and success rate (%). TSRL consistently achieves performance improvements with both GRPO and DAPO backbones. The plus/minus signs (+/-) indicate the performance change compared to the corresponding SFT→RL baseline.

Method	Continual Learning in WebShop			Continual Learning in ALFWorld
Method	ALFWorld	WebShop (Score)	WebShop (Succ.)	ALFWorld	WebShop (Score)	WebShop (Succ.)
Prior-task Model	74.4	22.0	2.8	24.2	77.2	64.3
SFT	69.3	49.1	18.3	64.2	58.9	29.4
GRPO	74.2	86.7	70.0	78.0	79.3	65.0
SFT $\to$ GRPO	70.3	85.4	60.2	84.3	64.1	58.7
DAPO	72.1	86.9	69.5	88.2	79.1	62.5

Table 3: Continual learning experiments between WebShop and ALFWorld. This table reports performance on both prior and new tasks when a model trained on one task is fine-tuned on another. Prior-task Model denotes the LLM agent after GRPO training on the prior task.

Base: Closed-Source Model
Method	Deep Search		Math Reasoning
Method	BrowseComp	BrowseComp-ZH	AIME 24	AIME 25
Prompting GPT-4o	0.7	6.9	9.3	13.1
Prompting Claude-3.5-Sonnet	0.8	5.9	16.0	7.4
Base: Qwen2.5-32B-Instruct
Prompting Qwen2.5	0.6	6.5	18.4	11.5

Table 4: Performance with Qwen2.5-32B-Instruct on Deep Search (target training task) and Math Reasoning (measuring retention of base model capabilities).

Figure 3: Dynamics of the success/failure ratio within each training batch across optimization steps.

Figure 4: Performance comparison across different trigger threshold

\delta

on ALFWorld and WebShop.

Constant Weight
Method	ALFWorld	WebShop (Score)	WebShop (Succ.)
fixed 1.0	63.5	72.9	52.8
fixed 0.1	69.8	75.1	57.2
Annealing With Different Annealing Steps
25 steps	76.4	78.8	61.2
50 steps	78.2	79.3	60.6

Table 5: Ablation studies on the annealing mechanism.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Primary Goal	How Supervision Enters	Forgetting Control
UFT ( Liu et al., 2025 )	Single-task post-training	Dense unified SFT+RL objective	No explicit continual-learning control
LUFFY ( Yan et al., 2025 )	Reasoning beyond prior capability	Off-policy traces mixed with on-policy RL	Partial, but not continual-specific
Iterative-SFT ( Chen et al., 2025 )	Improve with on-policy data	Alternating collection and updates	Milder drift, no explicit trigger
TSRL	Continual agent training	Sparse state-dependent trigger + annealing	Explicitly designed to reduce forgetting

Appendix

Table A1: Positioning TSRL among representative hybrid SFT-RL methods.

Method	ALFWorld (Prior)	WebShop (Prior)	ScienceWorld (Continual)
Prior-task Model	74.2	70.0	8.5
SFT	69.0	65.4	25.2
GRPO	71.7	68.9	29.3
SFT $\to$ GRPO	68.6	65.7	31.8
DAPO	71.3	68.9	29.5
SFT $\to$ DAPO	68.5	65.4	31.4

Appendix

Table A2: Three-stage continual learning experiment: ALFWorld

\rightarrow

WebShop

\rightarrow

ScienceWorld. Prior-task Model denotes the model after the first two stages, before adaptation to ScienceWorld.

Method	ALFWorld $\rightarrow$ WebShop		WebShop $\rightarrow$ ALFWorld
Method	ALFWorld (Prior)	WebShop (Continual)	ALFWorld (Continual)	WebShop (Prior)
Prior-task Model	74.4	2.8	24.2	64.3
SFT	69.3	18.3	64.2	29.4
GRPO	74.2	70.0	78.0	65.0
SFT $\to$ GRPO	70.3	60.2	84.3	58.7
Iterative-SFT Chen et al. (2025)	74.8	69.8	82.5	64.9

Appendix

Table A3: Comparisons with additional hybrid baselines and fixed-weight joint training. We report success rates for both continual-learning directions.

Method	ALFWorld $\rightarrow$ WebShop		WebShop $\rightarrow$ ALFWorld
Method	ALFWorld (Prior)	WebShop (Continual)	ALFWorld (Continual)	WebShop (Prior)
Prior-task Model	84.5	15.3	23.7	54.9
SFT	77.3	62.5	66.4	25.8
GRPO	83.8	74.2	77.6	49.7
SFT $\to$ GRPO	77.5	79.0	82.5	44.3
TSRL ${}_{\text{GRPO}}$	84.2 (+6.7)	78.8 (-0.2)	83.0 (+0.5)	49.9 (+5.6)

Appendix

Table A4: Continual learning results with Qwen3-4B. We report success rates for both continual-learning directions using the GRPO backbone.

Optimization
Parameter	Value
Actor Learning Rate	1e-6
Batch Size	64
Training Steps	150
Regularization
KL Loss Coefficient	0.01

Appendix

Table A5: WebShop and ALFWorld’s Hyperparameters.

Optimization
Parameter	Value
Actor Learning Rate	1e-6
Batch Size	256
Training Steps	100
Regularization
KL Loss Coefficient	0.001

Appendix

Table A6: Deep Search’s Hyperparameters.

Panel of Figure 16 — Figure A1: Prompt template for WebShop.

Panel of Figure 17 — Figure A2: Prompt template for ALFWorld.

Panel of Figure 18 — Figure A3: Prompt template for Deep Search.

Explore similar work

Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates
Jan 26, 2026Yibo Li, Zijie Lin, Ailin Deng +5Continual Learning for LLM Agents Reinforcement Learning
National University of Singapore, Singapore.
Attribution-Guided Continual Learning for Large Language Models
May 6, 2026Yazheng Liu, Yuxuan Wan, Rui Xu +3Continual Learning for LLMs LLM Fine-Tuning
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Beijing University of Posts and Telecommunications, Beijing, China
Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning
Jun 5, 2026Fatema Siddika, Md Anwar Hossen, Tanwi Mallick +1Continual Learning for LLMs Mixture-of-Experts Language Models
Iowa State University, Ames, USA · Argonne National Laboratory, USA

Jan 26, 2026J/K move · Enter open · S save

Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates

Yibo Li, Zijie Lin, Ailin Deng, Xuan Zhang+4

National University of Singapore, Singapore.

Continual Learning for LLM Agents Reinforcement Learning Test-Time RL Policy Learning RL for LLM Agents Continual RL+1

Open paper AskPDF arXiv

While Large Language Model (LLM) agents excel at general tasks, they inherently struggle with continual adaptation due to the frozen weights after deployment. Conventional reinforcement learning (RL) offers a solution but incurs prohibitive computational costs and the risk of catastrophic forgetting. We introduce Just-In-Time Reinforcement Learning (JitRL), a training-free framework that enables test-time policy optimization without any gradient updates. JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly. These estimates are then used to directly modulate the LLM's output logits. We theoretically prove that this additive update rule is the exact closed-form solution to the KL-constrained policy optimization objective. Extensive experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods. Crucially, JitRL outperforms the performance of computationally expensive fine-tuning methods (e.g., WebRL) while reducing monetary costs by over 30 times, offering a scalable path for continual learning agents. The code is available at https://github.com/liushiliushi/JitRL.

Abstract

Explore similar work

Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates

Attribution-Guided Continual Learning for Large Language Models

Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning