LLM agents operating repeatedly in the same environment, such as a code repository, a database, or an application suite, encounter tasks that share latent structure. Experience can reveal this structure, allowing agents to reuse environment knowledge and learn from environmental feedback. Agents can adapt at test-time using this experience through two channels: retaining it in context and updating their weights. These channels are usually studied separately, leaving unclear what each contributes and when they complement each other. We compare in-context learning (ICL), reinforcement learning with verifiable rewards using GRPO, and self-distillation using SDPO, alone and in combination, using a shared online protocol and modular framework. Agents encounter matched task streams, are evaluated on each task before adapting to it, and learn only from their own attempts and environment feedback. We evaluate 14 adaptation mechanisms and a stateless baseline with Qwen3.5-9B on 109 tasks from four environments drawn from SWE-bench Verified, AppWorld, ScienceWorld, and BIRD-Critic-SQLite, using five task orders per environment. ICL provides the largest standalone gain: its best policy raises mean success across environments from 28.8% to 49.4%. Adding GRPO to this policy achieves the highest mean success, 51.6%. While adding GRPO improves the overall mean under every ICL policy, SDPO has mixed effects. Trace analyses show fewer lookups and shorter attempts under ICL, consistent with reusing knowledge of the shared structure. However, ICL also frequently skips inspection and verification for the current task. Adding GRPO to ICL helps restore these behaviors while preserving its reduced environment lookups. These observations suggest that reinforcement learning with verifiable rewards can refine how agents use retained experience, motivating the joint study of context management and weight updates.
Figures & tables
Figure 1: On average across benchmarks, in-context learning (ICL) gives the largest standalone gain, while ICL with GRPO gives the highest overall gain. Points show five-seed mean success rates (%), using the best mechanism within each adaptation family for each world. Curves connect results without and with ICL for the same weight-update policy. Standard errors are reported in Table 2 .
Figure 2: One round on a task. (1) The adaptation mechanism receives a task. (2) Prepare returns a context c , with which the agent makes k attempts yi in fresh sessions of the environment. (3) Observe receives the attempts and their outcomes oi and updates the mechanism’s non-parametric state. (4) TeacherContexts adds hindsight h for SDPO’s self-teacher. (5) The mechanism updates its parametric state using the weighted SDPO and GRPO losses.
Symbol
Meaning
Symbol
Meaning
W ; X={x1,…,xN}
world; its N tasks
M
adaptation mechanism
x(j)
task at stream position j
θ
current policy weights
m ; k
rounds per task; attempts per round
c ; h
retained context; teacher hindsight
yi∼πθ(⋅∣c,x(j))
multi-turn attempt i
pj=k1∑i=1ksj,i
first-round success rate on task x(j)
oi=(ri,si,fi)
reward, success indicator, feedback
S=N1∑j=1Npj
stream success rate
E={(c,x(j),yi,oi)}i=1k
a round’s experience
ΔS=S−Sbase
gain over the stateless baseline
Table 1: Notation for the online protocol. Indices j , ℓ , and i denote task position, round, and attempt, respectively.
Weight update
ICL policy
SWE-bench Verified (Xarray)
AppWorld (File System)
ScienceWorld (Measurement)
BIRD-Critic- SQLite (Financial)
Mean
Frozen
No ICL
13.4 ± 1.0
57.7 ± 0.8
15.6 ± 0.7
28.5 ± 0.7
28.8 ± 0.4
SDPO
No ICL
16.8 ± 0.8
44.8 ± 1.2
19.2 ± 0.6
27.8 ± 0.6
27.1 ± 0.3
GRPO
No ICL
21.8 ± 1.3
61.3 ± 1.3
31.8 ± 1.6
33.1 ± 1.1
37.0 ± 0.7
Frozen
FIFO · All
39.4 ± 1.8
66.9 ± 3.3
50.3 ± 2.4
28.5 ± 0.7
46.3 ± 1.1
FIFO · Successful
35.1 ± 0.9
77.9 ± 3.0
46.1 ± 5.9
32.6 ± 0.9
47.9 ± 2.1
Prefix · All
35.6 ± 1.9
62.6 ± 5.0
48.0 ± 3.6
27.9 ± 0.7
43.5 ± 1.9
Table 2: Success rates for the adaptation mechanisms and baseline. Mean success rates (%) across five stream seeds, with standard errors in superscripts. Rows are grouped as no adaptation, weight-only, ICL-only, and hybrid. ICL policies are shown as retention · admission. No ICL with frozen weights is the baseline.
Figure 3: Mean gains and wins from adding GRPO or SDPO. Boxes show mean success rates. Yellow arrows indicate adding ICL with the policies listed below; blue and red arrows add GRPO and SDPO, respectively. Horizontal arrow labels show the mean success-rate gain in percentage points (top) and wins out of 20 world–seed stream pairs (bottom), weighting all streams equally.
Figure 4: How single-channel adaptation changes outcomes and attempt length. Top: mean proportions of successful, wrong, and unfinished attempts. An unfinished attempt ends without success or submission. ScienceWorld ends on reaching the goal and has no separate submission decision. Bottom: mean turns per attempt at each task position. Results weight the five stream seeds equally, with 16 attempts per task and stream.
Figure 5: How single-channel adaptation changes behavior along the stream. SWE-bench: mean code-lookup turns per attempt. AppWorld: mean turns containing API-documentation lookups. ScienceWorld: mean explicit look around actions per attempt (environment lookups). BIRD: percentage of all attempts with at least one executed exploratory SQL query. All panels weight the five seeds equally at each position.
Figure 6: Final-task examples across all four benchmarks in seed 4. Each column summarizes all 16 attempts: graded successes and mean code-lookup turns (SWE-bench), mean API lookups (AppWorld), mean environment lookups (ScienceWorld), or the percentage of all attempts with at least one executed exploratory SQL query (BIRD). Below the divider, excerpts summarize attempt 1; t denotes the turn.
Figure 7: Hybrid outcomes and exploration: GRPO preserves reduced lookups while restoring querying. Top: mean proportions of successful, wrong, and unfinished attempts. An unfinished attempt ends without success or submission. Bottom: mean code-lookup turns per attempt on SWE-bench, mean turns containing API-documentation lookups on AppWorld, mean explicit look around actions on ScienceWorld, and percentage of all attempts with at least one executed exploratory SQL query on BIRD. All panels weight the five seeds equally.
Figure 8: GRPO maintains higher late-stream querying across ICL policies on BIRD. Each panel compares ICL with its GRPO and SDPO hybrids under one ICL policy; Frozen is repeated for reference. Querying counts all attempts with at least one executed exploratory SQL query. Rates average 16 attempts per task in a stream.
Figure 9: How weight updates change ICL’s task-specific inspection and verification. (a) Rechecking after edits on SWE-bench; (b) final-state inspection on AppWorld; (c) inspection during melting-point experiments on ScienceWorld; (d) querying on BIRD. Bars show rates (%) averaged equally over five seeds.
Figure 10: How weight updates change ICL behavior: final-task examples across all four benchmarks in seed 0. Each column summarizes all 16 attempts: graded successes and same-check rates (SWE-bench), final-state inspection (AppWorld), state inspection (ScienceWorld), or querying (BIRD), using the metrics of Figure 9 . Below the divider, excerpts summarize attempt 1; t denotes the turn.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
World
Tasks
Action
Turns
Success, reward, feedback
SWE-bench Verified: Xarray
22 instances from pydata/xarray
one block of shell commands per turn, then a submission
40
issue resolved; binary reward; feedback reports how many of the required tests pass
AppWorld: File System
21 train and dev tasks whose first required app is the file system (7 templates, 3 variants each)
Python code calling app APIs in a persistent interpreter
40
all task requirements pass; binary reward; feedback reports how many requirements passed and the first failure
ScienceWorld: Measurement
22 AgentBoard instances of the thermometer and melting-point tasks
one text command
30
all annotated subgoals reached; reward is the fraction reached; feedback reports it
BIRD-Critic-SQLite: Financial
44 tasks on the financial SQLite database
SQL queries, then a submitted fix of one or more statements
5
all test cases pass; reward is the fraction passed; feedback reports it and any error
Appendix
Table 3: Benchmark settings and feedback. Success is each benchmark’s verdict (for ScienceWorld, AgentBoard’s subgoal annotations ( Ma et al., 2024 ) ); reward is the learning signal of GRPO and the selection rule of the All -admission ICL policies, whereas Successful admission selects by success.
Setting
Value
Model
Qwen3.5-9B, non-thinking generation
Attempts per round k
16, sampled at temperature 1.0
Rounds of attempts per task m
2 (frozen baseline, ICL-only and GRPO families); 3 (SDPO family)
Token allowance
task prompt 4,096; retained context 75,776; each attempt, including its observations, 24,576; teacher prompt 106,496; model sequence limit 131,072
replayed conversation with the agent’s actions, observations, and the outcome’s feedback; no limit on the number of episodes beyond the context allowance
Appendix
Table 4: Online evaluation and context settings.
Setting
GRPO
SDPO
Adapter
LoRA
LoRA
Rank
32
32
LoRA α
32
32
Target layers
Linear except output head
Linear except output head
Optimizer
AdamW
AdamW
Learning rate
10−5
10−5
Appendix
Table 5: Weight-update settings. Settings apply both with and without ICL.
Table 6: Success rates by seed for all adaptation mechanisms and the baseline. Success rate (%), separately for seeds 0–4. Mean weights the four worlds equally within each seed, before rounding. Rows are grouped by seed; within each seed, mechanisms follow the grouping and order of Table 2 . Bold marks the highest value in each column within each seed, including ties.
Figure 11: Mean gains and wins from adding ICL and GRPO. Boxes show mean success rates. Each arrow compares its destination with its source: yellow adds ICL, blue adds GRPO. Arrow labels show the mean success-rate gain in percentage points (top) and wins out of 20 world–seed stream pairs (bottom), weighting all streams equally.
Figure 12: Mean gains and wins from adding ICL and SDPO. Boxes show mean success rates. Each arrow compares its destination with its source: yellow adds ICL, red adds SDPO. Arrow labels show the mean success-rate gain in percentage points (top) and wins out of 20 world–seed stream pairs (bottom), weighting all streams equally.
Figure 13: Successful attempts are also shorter with ICL. Mean turns over all attempts (All), successful attempts (Success), and submitted but unsuccessful attempts (Wrong), averaged equally over five seeds. ScienceWorld has no separate Wrong category (N/A). These are full-stream means, not temporal trends; policies and conditioning are defined above.
Figure 14: Multiple operations can share one action. Top: shell commands per action in SWE-bench and API call sites per action in AppWorld. Bottom: percentage of eligible actions containing at least two such operations. Curves show unsmoothed means over five seeds using first-round attempts. Counts describe written operations; measurement rules are given above.
Figure 15: Cumulative ICL gains and retained context. Top: cumulative success-rate gain over Frozen, in percentage points. Bottom: distinct tasks in context before each task begins. All weights are frozen. Curves show means over five seeds; pairing and context counts are defined above.
Figure 16: Adding GRPO to ICL: cumulative gains and retained context. Top: cumulative success-rate gain over the matching frozen-weight ICL policy, in percentage points. Bottom: distinct tasks in the hybrid’s context before each task begins. Curves show means over five seeds, using the pairing and context counts defined above.
Figure 17: Adding SDPO to ICL: cumulative gains and retained context. Top: cumulative success-rate gain over the matching frozen-weight ICL policy, in percentage points. Bottom: distinct tasks in the hybrid’s context before each task begins. Curves show means over five seeds, using the pairing and context counts defined above.
Benchmark
FIFO evictions (%)
Prefix freeze
Retained ( FIFO / Prefix )
Available
AppWorld (File System)
31.0
7.8
12.8 / 6.8
20
ScienceWorld (Measurement)
11.4
17.4
18.6 / 16.4
21
BIRD-Critic-SQLite (Financial)
46.4
16.4
18.8 / 15.4
43
SWE-bench Verified (Xarray)
47.7
4.2
6.4 / 3.2
21
Appendix
Table 7: Context capacity, weights fixed and All admission. Five-seed means. Evictions: preparations evicting episodes. Freeze: first task position where admission stops. Retained: distinct prior tasks before the final task’s first round; Available: total prior tasks.
Figure 18: Reward variation and context admission with GRPO. Each bar partitions training batches by reward variation and admission of a new context episode for one ICL policy. Percentages pool batches across five seeds; category definitions are given above.
Figure 19: Reward variation and context admission with SDPO. Each bar partitions training batches by reward variation and admission of a new context episode for one ICL policy. Percentages pool batches across five seeds; category definitions are given above.
Figure 20: Success rates in two additional environments. Results use seed 4 and 16 attempts per task, with Prefix – Successful for all ICL mechanisms. Bold labels mark the highest success rate in each environment.
Figure 21: Gains from adding context and weight adaptation. Boxes show success rates; arrows show the change in percentage points from source to destination for the experiments in Figure 20 .
Figure 22: Component ablations on the same encountered tasks. Success rates of final seed-4 hybrid agents on their original worlds. Both retains the adapted weights and memory; the other two variants remove one component. Frozen is the fresh baseline from the corresponding round. Each bar averages 16 attempts per task. These are previously encountered tasks, not held-out evaluations.
Figure 23: Transfer of hybrid agents and their components across worlds. Within each matrix, rows identify the source benchmark and columns the evaluation benchmark. Values are success-rate gains in percentage points over the corresponding round’s frozen baseline. Diagonal cells are masked because they evaluate previously encountered tasks. Both, adapter only, and memory only use components of the same final seed-4 hybrid agent.
LLM agents increasingly face long-horizon tasks such as web search and deep research in real-world applications, where accumulated context can cause long-context degradation and reasoning failures. Prior work mitigates this through context management with agent-side context control or fixed strategies such as summarization, which require training the agent itself for adaptation - making it impractical for closed-source agents and ignoring that different agents may require different strategies. We introduce Adaptive Context Management (AdaCoM), which trains an external LLM to manage the context of a frozen agent through flexible modification actions and end-to-end reinforcement learning. Across diverse agents on web search and deep research benchmarks, AdaCoM substantially improves performance by preserving task constraints and progress while pruning stale content. The learned strategies reveal a Fidelity-Reliability Trade-off: agents with higher vanilla ReAct performance benefit from higher-fidelity context preservation, whereas lower-performing agents require more aggressive compression to stay within a reliable reasoning regime. Transfer experiments show that AdaCoM generalizes most effectively across agents with similar capability (measured by vanilla ReAct performance), suggesting a practical path toward reusable context managers for agent systems.
Lu Yi, Runlin Lei, Liuyi Yao +6
1Renmin University of China · Work done during internship at Tongyi Lab, Alibaba Group · 2Tongyi Lab, Alibaba Group +2
Adapting large language model agents during deployment requires not only retaining past experience, but also turning new observations into timely guidance. Many test-time learning methods, however, acquire knowledge from completed episodes. Feedback from an ongoing interaction may therefore not be distilled into knowledge soon enough to help the next decision. Acquiring knowledge at the granularity of individual transitions could reduce this delay, but raises a separate challenge: a rule that is useful within one episode may not be reliable enough to guide future episodes. Waiting for validation can forfeit immediate benefits, whereas unrestricted reuse can propagate accidental or misattributed guidance. We introduce StepLearn, a nonparametric framework that separates immediate use from persistent trust. It turns informative transitions into hypotheses that can guide the next step, while requiring prospective validation before reuse across episodes. Their predicted effects are checked against subsequent observations outside the source episodes, and only sufficiently supported hypotheses become available for persistent guidance. This process updates external knowledge while keeping all model parameters fixed. Over five rounds on WebArena-Lite and ALFWorld, StepLearn achieves average success rates of 59.9% and 84.0% with GPT-5-mini, and 57.8% and 88.1% with Qwen3.5-35B-A3B, respectively. It outperforms EvoTest, the strongest baseline, by 2.2-12.7 percentage points across the four settings. Learning dynamics further shows that these gains are not restricted to the final repetition, with advantages already present on first task attempts in most settings.
Recent advances in large language models (LLMs) have enabled the development of autonomous agents capable of complex reasoning and multi-step problem solving. However, these agents struggle to adapt to specialized environments and do not leverage past interactions, approaching each new task from scratch regardless of their accumulated experience. We introduce Experiential Reflective Learning (ERL), a simple self-improvement framework that enables rapid environment adaptation through experiential learning. ERL reflects on task trajectories and outcomes to generate heuristics, capturing actionable lessons that transfer across tasks. At test time, relevant heuristics are retrieved based on the current task and injected into the agent's context to guide execution. On the Gaia2 benchmark, ERL improves success rate by 7.8% over a ReAct baseline, with large gains in task completion reliability, and outperforms prior experiential learning methods. Through systematic ablations, we find that selective retrieval is essential and that heuristics provide more transferable abstractions than few-shot trajectory prompting. These results demonstrate that reflecting on single-attempt experiences to extract transferable heuristics enables effective agent self-improvement.
Marc-Antoine Allard, Arnaud Teinturier, Victor Xing +1