Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates

Yibo Li; Zijie Lin; Ailin Deng; Xuan Zhang; Yufei He; Shuo Ji; Tri Cao; Bryan Hooi

cs.LGJan 26, 2026

Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates

Authors: Yibo Li, Zijie Lin, Ailin Deng, Xuan Zhang, Yufei He, Shuo Ji, Tri Cao, Bryan Hooi

Organizations: National University of Singapore, Singapore.

Abstract

While Large Language Model (LLM) agents excel at general tasks, they inherently struggle with continual adaptation due to the frozen weights after deployment. Conventional reinforcement learning (RL) offers a solution but incurs prohibitive computational costs and the risk of catastrophic forgetting. We introduce Just-In-Time Reinforcement Learning (JitRL), a training-free framework that enables test-time policy optimization without any gradient updates. JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly. These estimates are then used to directly modulate the LLM's output logits. We theoretically prove that this additive update rule is the exact closed-form solution to the KL-constrained policy optimization objective. Extensive experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods. Crucially, JitRL outperforms the performance of computationally expensive fine-tuning methods (e.g., WebRL) while reducing monetary costs by over 30 times, offering a scalable path for continual learning agents. The code is available at https://github.com/liushiliushi/JitRL.

Figures & tables

Figure 1 : While standard RL performs policy gradient updates during training using previous trajectories, JitRL operates at test time. Specifically, it retrieves trajectories relevant to the current state to estimate advantages $A$ , subsequently refining the output logits through a KL-regularized policy optimization objective.

Figure 2 : Overview of the Just-In-Time Reinforcement Learning (JitRL) framework. The system operates in a continuous loop: (1) In the Inference (top), the agent retrieves relevant past experiences $\mathcal{N}(s)$ from the non-parametric memory $\mathcal{M}$ . The base LLM’s logits $z$ are then adjusted in closed-form ( $z^{\prime}=z+\beta\widehat{A}$ ) using the estimated advantage $A$ derived from historical returns, enabling test-time policy improvement without gradient updates. (2) In the Memory Update (bottom), completed trajectories are analyzed by an evaluator to compute discounted returns $G_{t}$ . These new experiences are stored back into $\mathcal{M}$ , allowing the agent to evolve its policy across episodes.

Method	Library		Zork1		Zork3
Method	Avg	Final	Avg	Final	Avg	Final
Static	10.0	10	8.5	10	0.2	0
Memory	13.2	14	22.9	25	1.0	1
Reflexion	15.4	18	26.1	35	1.4	1
AWM	12.7	10	38.8	44	1.9	2
EvoTest	21.5	26	46.8	54	2.6	4

Table 3 : Results on Jericho games. We report: Avg : average score across 50 episodes; Final : final episode score.

Method	Admin		Reddit
Method	Avg	Final	Avg	Final
Gemini-2.5-flash
Static	39.46	39.67	39.60	42.86
Memory	47.91	47.80	53.02	55.04
Reflexion	48.46	50.55	54.88	56.59
AWM	49.46	51.09	55.66	58.14

Table 4 : Comparison of average ( Avg ) and Final success rate between JitRL and baseline methods across different backbone LLMs.

Task	Candidate Action	Base	JitRL	Mechanism Explanation
Find customer reviews	click(CATALOG)	0.90	0.40	While “Catalog” appears semantically intuitive, memory corrects this prior: reviews are located under “Marketing”.
(Site Functionality)	click(MARKETING)	0.70	1.40
Find posts in subreddit	fill(Search, “..”)	0.95	0.45	Global search often yields noisy results. Memory steers the agent toward “Forums” for a deterministic and accurate path.
(Navigation Precision)	click(Forums)	0.80	1.80
Access product inventory	click(Products)	0.95	-0.28	Clicking loads a generic page. Memory identifies “hover” as an efficient shortcut to instantly reveal subcategories.
(UI Mechanics)	hover(Products)	0.40	0.90

Table 7 : Qualitative Analysis of Policy Improvement. We select three representative cases showing how JitRL modifies decision-making. Base and JitRL denote the logits before and after the memory-based update. Compared to Base , JitRL decreases the logits for incorrect options and increases them for correct options. In this way, JitRL corrects semantic priors and optimizes efficiency.

Method	Admin	Reddit
JitRL (Prompt Update)	48.35	54.57
JitRL (Logit Update)	52.31	57.64

Table 8 : Ablation: Logit update vs. prompting on WebArena.

Metric	Static	Memory	Reflexion	AWM	EvoTest	WebRL	JitRL
Cost	$200	$230	$220	$250	$220	$\sim$ $9900	$290

Table 9 : Comparison of training cost.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Parameter	Value
Environment Settings
	Step limit per episode	60
	Number of episodes	50
LLM Configuration
	Temperature	0.8
	Candidate actions ( $\|\mathcal{C}\|$ )	3

Appendix

Table 10 : Hyperparameters for Jericho experiments.

Category	Parameter	Value
Environment Settings
	Step limit per episode	10
	Number of episodes	50
	Random seed	0
LLM Configuration
	Temperature	0.8

Appendix

Table 11 : Hyperparameters for WebArena experiments.

Method	Admin	GitLab	Map	Reddit	Shopping	Avg
WebRL	38.89	23.53	9.68	50.00	17.39	27.27
JitRL	28.57	30.00	20.00	52.63	39.13	32.97

Appendix

Table 12 : Final success rate comparison (%) between WebRL and JitRL on WebArena-Lite using the same base model and training data.

Method	Admin	GitLab	Map	Reddit	Shopping	Average	Cost
Llama-3.1-70B	10.50	16.70	17.10	20.00	4.40	12.70	–
SFT	20.00	20.00	26.70	52.60	13.30	23.00	$640
JitRL	47.22	38.24	25.81	58.33	34.78	40.88	$200
WebRL	58.33	47.06	32.26	62.50	30.43	46.06	$9,900

Appendix

Table 13 : Controlled comparison on WebArena-Lite (Final success rate %) using the same Llama-3.1-70B-Instruct backbone, with separate training and evaluation phases.

Method	Library	Zork1	Zork3
GRPO	13.6	16.2	1.1
JitRL	20.8	42.1	2.3

Appendix

Table 14 : Same-backbone comparison on Jericho (Qwen3-32B, both methods adapt on the test set, Avg score over 50 episodes).

Method	Library	Zork1	Zork3
EvoTest (best baseline)	21.5	46.8	2.6
JitRL (unified)	24.7	51.8	3.3
JitRL	25.9	53.0	3.1

Appendix

Table 15 : Unified pipeline comparison on Jericho (Avg score over 50 episodes).

Method	Admin	GitLab	Map	Reddit	Shopping	Average
EvoTest (best baseline)	44.51	37.94	33.59	51.78	26.67	39.24
JitRL (unified)	52.17	41.18	35.23	56.90	43.33	46.01
JitRL	52.31	40.78	37.66	57.64	41.67	46.98

Appendix

Table 16 : Unified pipeline comparison on WebArena (Avg success rate %).

State / Context	Candidate Action	Base	JitRL	Mechanism Explanation
Library: Lobby	examine desk	0.85	0.35	The base model defaults to exploratory actions.
(Attendant present)	give ID to attendant	0.45	1.65	Memory encodes that this action immediately yields +5 points and unlocks the rare books room.
Zork1: Loud Room	take platinum bar	0.92	0.42	Greedy treasure-grabbing seems intuitive, but fails
(Deafening noise)	echo	0.30	1.50	in the noisy room. JitRL learns that “echo” quiets the room, enabling treasure collection.
Zork3: Cliff	climb down	0.90	0.30	Direct descent without preparation leads to death.
(Holding rope)	tie rope to railing	0.40	1.60	Memory encodes that securing the rope first enables safe descent and progression.

Appendix

Table 17 : Qualitative Analysis of Policy Improvement on Jericho. We select one representative case from each text-based game showing how JitRL modifies decision-making. Base and JitRL denote the logits before and after the memory-based update. The results highlight JitRL’s ability to override exploratory defaults, learn game-specific mechanics, and prioritize high-reward actions.

Website	GitLab	Map	Reddit	Shopping	Admin	Overall
Avg Steps	5.83	9.39	4.57	5.55	6.20	6.28

Appendix

Table 18 : Average Steps per Website.

Memory Size (# entries)	Retrieval (ms)	Avg Score
0–500	15–22	18.1
500–1000	26–27	25.1
1000–1500	29	27.2
1500–2000	38	29.2
2000–2500	47	30.0

Appendix

Table 19 : Inference overhead and performance as memory grows (Library, 50 episodes).

Rollout	Batch Size	PPO Epochs	LR	Val Mean	Val Max	Val Min
8	1	1	1e-6	16.2	35	0
16	1	1	1e-6	13.6	40	0
32	1	1	1e-6	12.0	40	-5
8	1	2	1e-6	10.4	40	0
8	1	4	1e-6	17.1	40	5
8	1	4	1e-5	25.5	45	-5

Appendix

Table 20 : GRPO hyperparameter sweep on Zork1 (Qwen3-32B, 50 training steps).

Explore similar work

No Time Like the Present: Agentic Test-Time Training for LLM Agents
Jul 3, 2026Yanbo Wang, Jinhua Hao, Yuze Shi +2Test-Time Training Large Language Model Training
Kuaishou Technology
Attribution-Guided Continual Learning for Large Language Models
May 6, 2026Yazheng Liu, Yuxuan Wan, Rui Xu +3Large Language Model Fine-Tuning Catastrophic Forgetting
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China · The Beijing University of Posts and Telecommunications, Beijing, China
AgentJet: A Flexible Swarm Training Framework for Agentic Reinforcement Learning
Jun 3, 2026Qingxu Fu, Boyin Liu, Shuchang Tao +2Agentic Reinforcement Learning Offline Reinforcement Learning
Tongyi Lab, Alibaba Group

Jul 3, 2026, cs.LGJ/K move · Enter open · S save

No Time Like the Present: Agentic Test-Time Training for LLM Agents

Yanbo Wang, Jinhua Hao, Yuze Shi, Kun Yuan+1

Kuaishou Technology

Test-Time Training Large Language Model Training Test-Time Scaling

Open paper AskPDF arXiv

LLM agents often degrade over long episodes: as trajectories grow, they revisit explored states, repeat failed actions, and lose strategies that previously worked. Test-time training (TTT) offers a way to adapt model weights to the evolving task state, but existing LLM TTT methods largely adapt once to a fixed input. We study continuous TTT in multi-turn agent episodes, where each update changes the policy that generates later training text. This creates a self-training loop that helps when new trajectory information appears, but can amplify drift when the agent gets stuck and repeatedly trains on similar text. We find that update-text repetition distinguishes these regimes and introduce Agentic Test-Time Training (aTTT), a token-level reweighting method that downweights the loss on tokens appearing in repeated

n

-grams from prior updates while leaving novel tokens fully weighted. To run such updates inside live episodes, we build a concurrent serving system using vLLM's runtime LoRA API, limiting overhead to 1.9

\times

the no-TTT cost. aTTT improves success by up to 5.0 points on ALFWorld and 4.9 points on SWE-bench Lite. The gains concentrate where models already have task competence but drift over long trajectories, suggesting that aTTT mainly preserves existing competence rather than teaching new abilities.

Abstract

Explore similar work

No Time Like the Present: Agentic Test-Time Training for LLM Agents

Attribution-Guided Continual Learning for Large Language Models

AgentJet: A Flexible Swarm Training Framework for Agentic Reinforcement Learning