LLM agents operating repeatedly in the same environment, such as a code repository, a database, or an application suite, encounter tasks that share latent structure. Experience can reveal this structure, allowing agents to reuse environment knowledge and learn from environmental feedback. Agents can adapt at test-time using this experience through two channels: retaining it in context and updating their weights. These channels are usually studied separately, leaving unclear what each contributes and when they complement each other. We compare in-context learning (ICL), reinforcement learning with verifiable rewards using GRPO, and self-distillation using SDPO, alone and in combination, using a shared online protocol and modular framework. Agents encounter matched task streams, are evaluated on each task before adapting to it, and learn only from their own attempts and environment feedback. We evaluate 14 adaptation mechanisms and a stateless baseline with Qwen3.5-9B on 109 tasks from four environments drawn from SWE-bench Verified, AppWorld, ScienceWorld, and BIRD-Critic-SQLite, using five task orders per environment. ICL provides the largest standalone gain: its best policy raises mean success across environments from 28.8% to 49.4%. Adding GRPO to this policy achieves the highest mean success, 51.6%. While adding GRPO improves the overall mean under every ICL policy, SDPO has mixed effects. Trace analyses show fewer lookups and shorter attempts under ICL, consistent with reusing knowledge of the shared structure. However, ICL also frequently skips inspection and verification for the current task. Adding GRPO to ICL helps restore these behaviors while preserving its reduced environment lookups. These observations suggest that reinforcement learning with verifiable rewards can refine how agents use retained experience, motivating the joint study of context management and weight updates.
Figures & tables
Figure 1: On average across benchmarks, in-context learning (ICL) gives the largest standalone gain, while ICL with GRPO gives the highest overall gain. Points show five-seed mean success rates (%), using the best mechanism within each adaptation family for each world. Curves connect results without and with ICL for the same weight-update policy. Standard errors are reported in Table 2 .
Figure 2: One round on a task. (1) The adaptation mechanism receives a task. (2) Prepare returns a context c , with which the agent makes k attempts yi in fresh sessions of the environment. (3) Observe receives the attempts and their outcomes oi and updates the mechanism’s non-parametric state. (4) TeacherContexts adds hindsight h for SDPO’s self-teacher. (5) The mechanism updates its parametric state using the weighted SDPO and GRPO losses.
Symbol
Meaning
Symbol
Meaning
W ; X={x1,…,xN}
world; its N tasks
M
adaptation mechanism
x(j)
task at stream position j
θ
current policy weights
m ; k
rounds per task; attempts per round
c ; h
retained context; teacher hindsight
yi∼πθ(⋅∣c,x(j))
multi-turn attempt i
pj=k1∑i=1ksj,i
first-round success rate on task x(j)
oi=(ri,si,fi)
reward, success indicator, feedback
S=N1∑j=1Npj
stream success rate
E={(c,x(j),yi,oi)}i=1k
a round’s experience
ΔS=S−Sbase
gain over the stateless baseline
Table 1: Notation for the online protocol. Indices j , ℓ , and i denote task position, round, and attempt, respectively.
Weight update
ICL policy
SWE-bench Verified (Xarray)
AppWorld (File System)
ScienceWorld (Measurement)
BIRD-Critic- SQLite (Financial)
Mean
Frozen
No ICL
13.4 ± 1.0
57.7 ± 0.8
15.6 ± 0.7
28.5 ± 0.7
28.8 ± 0.4
SDPO
No ICL
16.8 ± 0.8
44.8 ± 1.2
19.2 ± 0.6
27.8 ± 0.6
27.1 ± 0.3
GRPO
No ICL
21.8 ± 1.3
61.3 ± 1.3
31.8 ± 1.6
33.1 ± 1.1
37.0 ± 0.7
Frozen
FIFO · All
39.4 ± 1.8
66.9 ± 3.3
50.3 ± 2.4
28.5 ± 0.7
46.3 ± 1.1
FIFO · Successful
35.1 ± 0.9
77.9 ± 3.0
46.1 ± 5.9
32.6 ± 0.9
47.9 ± 2.1
Prefix · All
35.6 ± 1.9
62.6 ± 5.0
48.0 ± 3.6
27.9 ± 0.7
43.5 ± 1.9
Table 2: Success rates for the adaptation mechanisms and baseline. Mean success rates (%) across five stream seeds, with standard errors in superscripts. Rows are grouped as no adaptation, weight-only, ICL-only, and hybrid. ICL policies are shown as retention · admission. No ICL with frozen weights is the baseline.
Figure 3: Mean gains and wins from adding GRPO or SDPO. Boxes show mean success rates. Yellow arrows indicate adding ICL with the policies listed below; blue and red arrows add GRPO and SDPO, respectively. Horizontal arrow labels show the mean success-rate gain in percentage points (top) and wins out of 20 world–seed stream pairs (bottom), weighting all streams equally.
Figure 4: How single-channel adaptation changes outcomes and attempt length. Top: mean proportions of successful, wrong, and unfinished attempts. An unfinished attempt ends without success or submission. ScienceWorld ends on reaching the goal and has no separate submission decision. Bottom: mean turns per attempt at each task position. Results weight the five stream seeds equally, with 16 attempts per task and stream.
Figure 5: How single-channel adaptation changes behavior along the stream. SWE-bench: mean code-lookup turns per attempt. AppWorld: mean turns containing API-documentation lookups. ScienceWorld: mean explicit look around actions per attempt (environment lookups). BIRD: percentage of all attempts with at least one executed exploratory SQL query. All panels weight the five seeds equally at each position.
Figure 6: Final-task examples across all four benchmarks in seed 4. Each column summarizes all 16 attempts: graded successes and mean code-lookup turns (SWE-bench), mean API lookups (AppWorld), mean environment lookups (ScienceWorld), or the percentage of all attempts with at least one executed exploratory SQL query (BIRD). Below the divider, excerpts summarize attempt 1; t denotes the turn.
Figure 7: Hybrid outcomes and exploration: GRPO preserves reduced lookups while restoring querying. Top: mean proportions of successful, wrong, and unfinished attempts. An unfinished attempt ends without success or submission. Bottom: mean code-lookup turns per attempt on SWE-bench, mean turns containing API-documentation lookups on AppWorld, mean explicit look around actions on ScienceWorld, and percentage of all attempts with at least one executed exploratory SQL query on BIRD. All panels weight the five seeds equally.
Figure 8: GRPO maintains higher late-stream querying across ICL policies on BIRD. Each panel compares ICL with its GRPO and SDPO hybrids under one ICL policy; Frozen is repeated for reference. Querying counts all attempts with at least one executed exploratory SQL query. Rates average 16 attempts per task in a stream.
Figure 9: How weight updates change ICL’s task-specific inspection and verification. (a) Rechecking after edits on SWE-bench; (b) final-state inspection on AppWorld; (c) inspection during melting-point experiments on ScienceWorld; (d) querying on BIRD. Bars show rates (%) averaged equally over five seeds.
Figure 10: How weight updates change ICL behavior: final-task examples across all four benchmarks in seed 0. Each column summarizes all 16 attempts: graded successes and same-check rates (SWE-bench), final-state inspection (AppWorld), state inspection (ScienceWorld), or querying (BIRD), using the metrics of Figure 9 . Below the divider, excerpts summarize attempt 1; t denotes the turn.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
World
Tasks
Action
Turns
Success, reward, feedback
SWE-bench Verified: Xarray
22 instances from pydata/xarray
one block of shell commands per turn, then a submission
40
issue resolved; binary reward; feedback reports how many of the required tests pass
AppWorld: File System
21 train and dev tasks whose first required app is the file system (7 templates, 3 variants each)
Python code calling app APIs in a persistent interpreter
40
all task requirements pass; binary reward; feedback reports how many requirements passed and the first failure
ScienceWorld: Measurement
22 AgentBoard instances of the thermometer and melting-point tasks
one text command
30
all annotated subgoals reached; reward is the fraction reached; feedback reports it
BIRD-Critic-SQLite: Financial
44 tasks on the financial SQLite database
SQL queries, then a submitted fix of one or more statements
5
all test cases pass; reward is the fraction passed; feedback reports it and any error
Appendix
Table 3: Benchmark settings and feedback. Success is each benchmark’s verdict (for ScienceWorld, AgentBoard’s subgoal annotations ( Ma et al., 2024 ) ); reward is the learning signal of GRPO and the selection rule of the All -admission ICL policies, whereas Successful admission selects by success.
Setting
Value
Model
Qwen3.5-9B, non-thinking generation
Attempts per round k
16, sampled at temperature 1.0
Rounds of attempts per task m
2 (frozen baseline, ICL-only and GRPO families); 3 (SDPO family)
Token allowance
task prompt 4,096; retained context 75,776; each attempt, including its observations, 24,576; teacher prompt 106,496; model sequence limit 131,072
replayed conversation with the agent’s actions, observations, and the outcome’s feedback; no limit on the number of episodes beyond the context allowance
Appendix
Table 4: Online evaluation and context settings.
Setting
GRPO
SDPO
Adapter
LoRA
LoRA
Rank
32
32
LoRA α
32
32
Target layers
Linear except output head
Linear except output head
Optimizer
AdamW
AdamW
Learning rate
10−5
10−5
Appendix
Table 5: Weight-update settings. Settings apply both with and without ICL.
Table 6: Success rates by seed for all adaptation mechanisms and the baseline. Success rate (%), separately for seeds 0–4. Mean weights the four worlds equally within each seed, before rounding. Rows are grouped by seed; within each seed, mechanisms follow the grouping and order of Table 2 . Bold marks the highest value in each column within each seed, including ties.
Figure 11: Mean gains and wins from adding ICL and GRPO. Boxes show mean success rates. Each arrow compares its destination with its source: yellow adds ICL, blue adds GRPO. Arrow labels show the mean success-rate gain in percentage points (top) and wins out of 20 world–seed stream pairs (bottom), weighting all streams equally.
Figure 12: Mean gains and wins from adding ICL and SDPO. Boxes show mean success rates. Each arrow compares its destination with its source: yellow adds ICL, red adds SDPO. Arrow labels show the mean success-rate gain in percentage points (top) and wins out of 20 world–seed stream pairs (bottom), weighting all streams equally.
Figure 13: Successful attempts are also shorter with ICL. Mean turns over all attempts (All), successful attempts (Success), and submitted but unsuccessful attempts (Wrong), averaged equally over five seeds. ScienceWorld has no separate Wrong category (N/A). These are full-stream means, not temporal trends; policies and conditioning are defined above.
Figure 14: Multiple operations can share one action. Top: shell commands per action in SWE-bench and API call sites per action in AppWorld. Bottom: percentage of eligible actions containing at least two such operations. Curves show unsmoothed means over five seeds using first-round attempts. Counts describe written operations; measurement rules are given above.
Figure 15: Cumulative ICL gains and retained context. Top: cumulative success-rate gain over Frozen, in percentage points. Bottom: distinct tasks in context before each task begins. All weights are frozen. Curves show means over five seeds; pairing and context counts are defined above.
Figure 16: Adding GRPO to ICL: cumulative gains and retained context. Top: cumulative success-rate gain over the matching frozen-weight ICL policy, in percentage points. Bottom: distinct tasks in the hybrid’s context before each task begins. Curves show means over five seeds, using the pairing and context counts defined above.
Figure 17: Adding SDPO to ICL: cumulative gains and retained context. Top: cumulative success-rate gain over the matching frozen-weight ICL policy, in percentage points. Bottom: distinct tasks in the hybrid’s context before each task begins. Curves show means over five seeds, using the pairing and context counts defined above.
Benchmark
FIFO evictions (%)
Prefix freeze
Retained ( FIFO / Prefix )
Available
AppWorld (File System)
31.0
7.8
12.8 / 6.8
20
ScienceWorld (Measurement)
11.4
17.4
18.6 / 16.4
21
BIRD-Critic-SQLite (Financial)
46.4
16.4
18.8 / 15.4
43
SWE-bench Verified (Xarray)
47.7
4.2
6.4 / 3.2
21
Appendix
Table 7: Context capacity, weights fixed and All admission. Five-seed means. Evictions: preparations evicting episodes. Freeze: first task position where admission stops. Retained: distinct prior tasks before the final task’s first round; Available: total prior tasks.
Figure 18: Reward variation and context admission with GRPO. Each bar partitions training batches by reward variation and admission of a new context episode for one ICL policy. Percentages pool batches across five seeds; category definitions are given above.
Figure 19: Reward variation and context admission with SDPO. Each bar partitions training batches by reward variation and admission of a new context episode for one ICL policy. Percentages pool batches across five seeds; category definitions are given above.
Figure 20: Success rates in two additional environments. Results use seed 4 and 16 attempts per task, with Prefix – Successful for all ICL mechanisms. Bold labels mark the highest success rate in each environment.
Figure 21: Gains from adding context and weight adaptation. Boxes show success rates; arrows show the change in percentage points from source to destination for the experiments in Figure 20 .
Figure 22: Component ablations on the same encountered tasks. Success rates of final seed-4 hybrid agents on their original worlds. Both retains the adapted weights and memory; the other two variants remove one component. Frozen is the fresh baseline from the corresponding round. Each bar averages 16 attempts per task. These are previously encountered tasks, not held-out evaluations.
Figure 23: Transfer of hybrid agents and their components across worlds. Within each matrix, rows identify the source benchmark and columns the evaluation benchmark. Values are success-rate gains in percentage points over the corresponding round’s frozen baseline. Diagonal cells are masked because they evaluate previously encountered tasks. Both, adapter only, and memory only use components of the same final seed-4 hybrid agent.