Learn Now, Use Next, Trust Later: Prequential Test-Time Learning for LLM Agents

Tong Zhao; Reed Li; Yuyang Hu; Yutao Zhu; Haijin Liang; Haibo Shi; Yu Lu; Zhicheng Dou

cs.LGSep 28, 2026

Learn Now, Use Next, Trust Later: Prequential Test-Time Learning for LLM Agents

Authors: Tong Zhao, Reed Li, Yuyang Hu, Yutao Zhu, Haijin Liang, Haibo Shi, Yu Lu, Zhicheng Dou

Organizations: Renmin University of China · Tencent

Abstract

Adapting large language model agents during deployment requires not only retaining past experience, but also turning new observations into timely guidance. Many test-time learning methods, however, acquire knowledge from completed episodes. Feedback from an ongoing interaction may therefore not be distilled into knowledge soon enough to help the next decision. Acquiring knowledge at the granularity of individual transitions could reduce this delay, but raises a separate challenge: a rule that is useful within one episode may not be reliable enough to guide future episodes. Waiting for validation can forfeit immediate benefits, whereas unrestricted reuse can propagate accidental or misattributed guidance. We introduce StepLearn, a nonparametric framework that separates immediate use from persistent trust. It turns informative transitions into hypotheses that can guide the next step, while requiring prospective validation before reuse across episodes. Their predicted effects are checked against subsequent observations outside the source episodes, and only sufficiently supported hypotheses become available for persistent guidance. This process updates external knowledge while keeping all model parameters fixed. Over five rounds on WebArena-Lite and ALFWorld, StepLearn achieves average success rates of 59.9% and 84.0% with GPT-5-mini, and 57.8% and 88.1% with Qwen3.5-35B-A3B, respectively. It outperforms EvoTest, the strongest baseline, by 2.2-12.7 percentage points across the four settings. Learning dynamics further shows that these gains are not restricted to the final repetition, with advantages already present on first task attempts in most settings.

Figures & tables

Figure 1 : Learning during interaction. Unlike episode-level learning, StepLearn acquires knowledge for the next decision and validates it in later episodes before persistent reuse.

Figure 2 : StepLearn overview. (a) Knowledge acquisition turns informative transitions into policy–effect hypotheses. (b) Immediate reuse retrieves runtime and verified knowledge for action selection. (c) Prequential verification fixes predictions before execution and accumulates evidence from non-source episodes for persistent reuse. The example is schematic: verdicts are categories, not counts, and both verified-memory icons denote one store.

Method	WebArena-Lite						ALFWorld
Method	Admin	GitLab	Map	Reddit	Shop.	Avg.	Look	Pick	Clean	Cool	Heat	Two	Avg.
GPT-5-mini
Static	51.4	50.0	23.1	78.9	40.0	46.5	25.0	20.8	19.4	28.6	17.4	20.6	21.6
Memory	57.4	43.3	19.2	75.8	42.7	45.8	75.6	20.8	38.7	68.6	27.0	21.2	40.8
Reflexion	59.3	43.3	23.1	80.0	44.8	47.7	52.2	44.2	52.9	46.7	57.4	58.8	51.8
JitRL	41.7	57.3	24.6	65.3	38.7	43.9	66.7	78.3	45.8	36.2	43.5	51.8	53.3

Table 1 : Average success rates (%). Bold and underline mark the best and second best results per column within each model block.

Figure 3 : Success rates over five task repetitions (rounds), with panel-specific vertical scales. Qwen3.5-35B denotes Qwen3.5-35B-A3B in thinking mode.

Figure 4 : First-attempt (R1) success rates. R1 may include knowledge from earlier tasks. Bars follow legend order.

Configuration	GPT-5-mini		Qwen3.5-35B-A3B
Configuration	WebArena-Lite	ALFWorld	WebArena-Lite	ALFWorld
StepLearn (full reference)	59.9	84.0	57.8	88.1
w/o stepwise acquisition	56.6 ( $-3.3$ )	81.6 ( $-2.4$ )	49.8 ( $-8.0$ )	86.9 ( $-1.2$ )
w/o runtime reuse	56.8 ( $-3.1$ )	81.8 ( $-2.2$ )	51.2 ( $-6.6$ )	87.2 ( $-0.9$ )
w/o retention safeguards	57.0 ( $-2.9$ )	76.4 ( $-7.6$ )	52.0 ( $-5.8$ )	84.2 ( $-3.9$ )

Table 2 : Mechanism ablations: average success rates (%). Parentheses show percentage-point changes from the full reference.

Actor: GPT-5-mini
Auxiliary learner	WebArena-Lite	ALFWorld
GPT-5.4-nano (reference)	59.9	84.0
GPT-5-mini	55.9	86.4
Actor: Qwen3.5-35B-A3B
Qwen3.5-4B (reference)	57.8	88.1
Qwen3.5-9B	56.8	89.0

Table 3 : Auxiliary learners: success rates (%).

Method	WebArena-Lite	ALFWorld
Static	23.8	43.6
Memory	32.8	188.8
Reflexion	25.5	78.4
JitRL	71.4	196.4
AWM	25.4	58.8
EvoTest	85.2	61.2

Table 4 : API and training costs (USD).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	API configuration	Local configuration
Actor	GPT-5-mini	Qwen3.5-35B-A3B
Auxiliary learner	GPT-5.4-nano	Qwen3.5-4B
Temperature	0.0	0.0
Action limit: WebArena / ALFWorld	25 / 50	25 / 50
Knowledge retrieval limit	6	6
Stagnation threshold	2 steps	2 steps

Appendix

Table 5 : Inference and knowledge settings for StepLearn .

Method	Benchmark	H20 GPU-hours	Cost (USD)
SFT	WebArena-Lite	29.8	59.6
SFT	ALFWorld	12.2	24.4
TTRL	WebArena-Lite	38.0	76.0
TTRL	ALFWorld	54.0	108.0

Appendix

Table 6 : Training resources and costs for the retained pipelines.

Explore similar work

Jul 3, 2026cs.LG

No Time Like the Present: Agentic Test-Time Training for LLM Agents

LLM agents often degrade over long episodes: as trajectories grow, they revisit explored states, repeat failed actions, and lose strategies that previously worked. Test-time training (TTT) offers a way to adapt model weights to the evolving task state, but existing LLM TTT methods largely adapt once to a fixed input. We study continuous TTT in multi-turn agent episodes, where each update changes the policy that generates later training text. This creates a self-training loop that helps when new trajectory information appears, but can amplify drift when the agent gets stuck and repeatedly trains on similar text. We find that update-text repetition distinguishes these regimes and introduce Agentic Test-Time Training (aTTT), a token-level reweighting method that downweights the loss on tokens appearing in repeated

n

-grams from prior updates while leaving novel tokens fully weighted. To run such updates inside live episodes, we build a concurrent serving system using vLLM's runtime LoRA API, limiting overhead to 1.9

\times

the no-TTT cost. aTTT improves success by up to 5.0 points on ALFWorld and 4.9 points on SWE-bench Lite. The gains concentrate where models already have task competence but drift over long trajectories, suggesting that aTTT mainly preserves existing competence rather than teaching new abilities.

Yanbo Wang, Jinhua Hao, Yuze Shi +2

Kuaishou Technology

Test-Time Training Large Language Model Training Test-Time Scaling

Ask PDF

May 5, 2026cs.AI

CASCADE: Case-Based Continual Adaptation for Large Language Models During Deployment

Large language models (LLMs) have become a central foundation of modern artificial intelligence, yet their lifecycle remains constrained by a rigid separation between training and deployment, after which learning effectively ceases. This limitation contrasts with natural intelligence, which continually adapts through interaction with its environment. In this paper, we formalise deployment-time learning (DTL) as the third stage in the LLM lifecycle that enables LLM agents to improve from experience during deployment without modifying model parameters. We present CASCADE (CASe-based Continual Adaptation during DEployment), a general and principled framework that equips LLM agents with an explicit, evolving episodic memory. CASCADE formulates experience reuse as a contextual bandit problem, enabling principled exploration-exploitation trade-offs and establishing no-regret guarantees over long-term interactions. This design allows agents to accumulate, select, and refine task-relevant cases, transforming past experience into actionable knowledge. Across 16 diverse tasks spanning medical diagnosis, legal analysis, code generation, web search, tool use, and embodied interaction, CASCADE improves macro-averaged success rate by 20.9% over zero-shot prompting while consistently outperforming gradient-based and memory-based baselines. By reframing deployment as an adaptive learning process, this work establishes a foundation for continually improving AI systems.

Siyuan Guo, Yali Du, Hechang Chen +2

aSchool of Artificial Intelligence, Jilin University. · bEngineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University. · cInternational Center of Future Science, Jilin University. +3

Post-Deployment Adaptation Large Language Model Agents Parameter-Efficient Adaptation

Ask PDF

Sep 28, 2026cs.CL

Harness Learning Enables Generalizable Test-Time Adaptation

A language-model agent is jointly defined by its model and its harness, the executable program that organizes model calls, tool use, and information flow. Because different tasks call for different ways of organizing these operations, the harness needs to be adapted using feedback from the task at hand. We introduce harness learning, which trains a proposer model to revise a solver's harness using execution feedback. We formulate this process as meta-learning over executable programs, with harness revisions playing the role of weight updates in gradient-based adaptation. We train the proposer with reinforcement learning, using the task performance of revised harnesses as the reward. At test time, the proposer uses feedback from successive executions on a new task to refine the harness, without performing any parameter-space update. Experiments on reasoning and multi-hop question answering show that harness learning improves revision quality and that the ability to adapt at test time transfers to unseen tasks. Policies trained on individual revisions can continue improving harnesses over multiple rounds, while the benefits of training on revision sequences vary across settings. These findings suggest a path towards continually learning agents that turn accumulated experience into generalizable improvements.

Alvin Zhang, Xuecheng Liu, Zixuan Wang +6

Agent Harness Efficient Task Adaptation Language-Model Agents Stable Test-Time Adaptation Test Time

Ask PDF