Do LLMs Learn from Rewards in Context? : Rethinking the role of reward in In-Context Reinforcement Learning

Minchan Kwon; Seunghee Koh; Sunghyun Baek; Minsung Bae; Junmo Kim

cs.LGOct 8, 2026

Do LLMs Learn from Rewards in Context? : Rethinking the role of reward in In-Context Reinforcement Learning

Authors: Minchan Kwon, Seunghee Koh, Sunghyun Baek, Minsung Bae, Junmo Kim

Organizations: Korea Advanced Institute of Science and Technology (KAIST)

Abstract

LLM agents increasingly improve at inference time by accumulating experience in context rather than by updating parameters. This process is often described as in-context reinforcement learning (ICRL). Whether in-context learning (ICL) can actually play the role of RL, however, has not been tested. We study this question in its simplest form, direct ICRL, where the model conditions directly on raw trajectory-reward pairs, and ask whether the reward acts as a learning signal. Through controlled experiments on four benchmarks across six models, we find that the reward is read, but its effect is small: flipping, randomizing, or removing the reward leaves the improvement curve almost unchanged, and this holds even under meta-prompts that explicitly instruct the model to explore, exploit, or reason over rewards. Trajectories drive improvement, but not through their semantic content: shuffled or corrupted trajectories work as well as real ones. These patterns closely mirror those known in ICL, suggesting that direct ICRL is better understood as a special case of ICL than as inference-time RL. This reframing has implications for agent memory design: ICL factors such as input distribution and demonstrations may matter more than RL elements such as reward shaping and exploration.

Figures & tables

Figure 1 : Direct ICRL overview (left) and controlled interventions for probing its mechanism: reward (middle) and trajectory perturbations (right).

Panel of Figure 2 — Figure 2 : Episode-level performance under different reward conditions on three benchmarks.

Model	No Imp.	Reward-Agnostic	Reward-Sensitive	Total
Qwen-3-4B-Instruct	25 (83.3%)	4 (13.3%)	1 (3.3%)	30
Llama-3.1-8B-Instruct	27 (90.0%)	2 (6.7%)	1 (3.3%)	30
Qwen-3.5-9B	19 (63.3%)	10 (33.3%)	1 (3.3%)	30
GPT-5-nano	11 (36.7%)	19 (63.3%)	0 (0.0%)	30
GPT-5-mini	7 (23.3%)	22 (73.3%)	1 (3.3%)	30
Gemini-3.1-Flash-Lite	12 (40.0%)	16 (53.3%)	2 (6.7%)	30

Table 1: Comparison of improvement types across models on HumanEval+ and MBPP+. Each entry reports the number of problems and the percentage within each model-benchmark setting.

	$\Delta\mathrm{LOO}$		Target Replay Rate		Memory Replay Rate
Condition	$\tau^{-}$	$\tau^{+}$	$\tau^{-}$	$\tau^{+}$
Problem-only	–	–	0.7	1.1	16.3
Trajectory-only	0.416	0.162	3.1	1.8	34.3

Table 2 : Effect of showing trajectories, without any reward. Replay rates are in %.

\Delta\mathrm{LOO}

is undefined for Problem-only because no memory is shown.

	$\Delta\mathrm{LOO}$		Target Replay Rate
Condition	$\tau^{-}$	$\tau^{+}$	$\tau^{-}$	$\tau^{+}$
Trajectory-only (no reward)	0.416	0.162	3.1	1.8
ICRL (true reward)	0.360	0.173	2.5	3.0
Specific-flip	0.474	0.145	3.0	1.6

Table 3 : Effect of the reward field on the replay of the target trajectory. Trajectories are identical across rows; only the reward field differs. Target Replay Rate is in %.

		Target Replay Rate		Memory Replay Rate
Change	Condition	$\tau^{-}$	$\tau^{+}$	$\tau^{-}$	$\tau^{+}$
Reference	Problem-only	0.7	1.1	16.3	16.3
	ICRL	2.5	3.0	30.5	30.5
Position	Order-first	2.4	5.5	29.3	30.0
	Order-last	3.2	13.2	29.4	41.9
Discouraging replay	All Failure	2.3	1.7	27.6	27.3

Table 4 : Effect of changes other than the target’s reward. All replay rates are in %. The

\tau^{-}

and

\tau^{+}

columns report runs in which the target is the lowest- and the highest-reward trajectory, respectively. Problem-only is the level without any memory.

Memory Setting	MBPP	ScienceWorld
RSM	0.250	-32.479
Cross-Task	0.122	-43.000
Random Valid Action	0.102	-68.208
ICRL History	0.245	-42.238
Trajectory Perturbation
Shuffle	0.243	-38.588

Table 5 : Mean reward across MBPP and ScienceWorld. Standard deviations are reported in Table 9 .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Comparison	Mean $\Delta$	95% CI	$p$
ScienceWorld	True vs. Random	$-0.76$	$[-9.40,\ 7.86]$	$0.86$
ScienceWorld	True vs. No Reward	$+1.96$	$[-6.73,\ 10.64]$	$0.65$
MBPP+	True vs. Random	$+0.028$	$[-0.042,\ 0.098]$	$0.43$
MBPP+	True vs. No Reward	$+0.014$	$[-0.064,\ 0.091]$	$0.72$
HumanEval+	True vs. Random	$+0.005$	$[-0.064,\ 0.075]$	$0.88$
HumanEval+	True vs. No Reward	$+0.037$	$[-0.056,\ 0.130]$	$0.42$

Appendix

Table 6 : Task-level paired

t

-tests on final episode scores (

n=40

tasks per benchmark, paired across 3 seeds). All six confidence intervals include zero, and no comparison reaches

p<0.05

. ScienceWorld final scores span roughly 80 points; coding scores are unit-test accuracy in

[0,1]

Figure 3 : Improvement category for accumulated memory settings.

Figure 4 : ScienceWorld ICRL experiment using Llama-3.1-8B-Instruct as a policy network.

Figure 5 : ScienceWorld ICRL experiment with verbal reward.

Figure 6 : ScienceWorld ICRL experiment with Reflexion.

Panel of Figure 7 — Figure 7 : Top-k and Bottom-k memory selection on MBPP+ and HumanEval+.

		$\Delta{\mathrm{LOO}}$
Method Type		$\tau^{-}$	$\tau^{+}$
ICRL	True Reward	0.3601	0.1734
	All Failure	0.4066	0.1468
	Anti prompt	0.3322	0.1632
	CoT	0.4339	0.1615
	Duplicate	0.4327	0.2504

Appendix

Table 7 : Full results of

\Delta{\mathrm{LOO}}

Type	Detail	Target Replay Rate (%)		Memory Replay Rate (%)
Type	Detail	$\tau^{+}$	$\tau^{-}$	$\tau^{+}$	$\tau^{-}$
Problem Only	Problem Only	1.1	0.7	16.3	16.3
ICRL	True Reward	3.0	2.5	30.5	30.5
	All Failure	1.7	2.3	27.3	27.6
	Specific Failure	1.7	2.2	30.6	30.5
	Anti Prompt	2.3	2.4	30.0	30.2

Appendix

Table 8 : Full replay-rate results across memory conditions.

Memory Setting	MBPP	ScienceWorld
RSM	$0.250\pm 0.021$	$-32.479\pm 4.520$
Cross-Task	$0.122\pm 0.017$	$-43.000\pm 4.100$
Random Valid Action	$0.102\pm 0.012$	$-68.208\pm 3.880$
ICRL History	$0.245\pm 0.032$	$-42.238\pm 2.860$
Trajectory Perturbation
Shuffle	$0.243\pm 0.023$	$-38.588\pm 6.090$

Appendix

Table 9 : Mean reward comparison across MBPP and ScienceWorld with standard deviations.

Figure 8 : Example of the Crossover perturbation on an MBPP+ task. Two parent trajectories (a, b), each a valid solution sampled by the model, are mixed line-by-line into a structurally broken child (c). Even though the child code does not compile, using such perturbed trajectories as in-context memory yields performance comparable to using real ICRL trajectories ( Section 6 , Table 9 ).

Figure 9 : Example of the Shuffle perturbation on a ScienceWorld task. The first and last actions are kept fixed; the middle three steps are randomly reordered. The shuffled version is no longer a sensible plan — focus on potato is repeated immediately at step 2, and the agent moves the potato to the red box at step 5 without re-focusing on it — yet using such trajectories as in-context memory yields performance comparable to using the original trajectories ( Section 6 ).

Explore similar work

In-Context Learning as Implicit Policy Gradient
Jul 25, 2026Masahiro Kaneko, Timothy BaldwinRL for Language Models Policy Gradient
MBZUAI
In-Context Reinforcement Learning under Non-Stationarity: A Survey
Jul 1, 2026A Run, Ziluo DingNon-Stationary RL In-Context RL
Independent Researcher · Beijing Innovation Center of Humanoid Robotics
Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy
Aug 3, 2026Jim Dilkes, Vahid Yazdanpanah, Sebastian SteinRL for Language Model Reasoning RL Post-Training
University of Southampton Southampton United Kingdom

Jul 25, 2026, cs.LGJ/K move · Enter open · S save

In-Context Learning as Implicit Policy Gradient

Masahiro Kaneko, Timothy Baldwin

MBZUAI

RL for Language Models Policy Gradient In-Context Learning In-Context RL

Open paper AskPDF arXiv

Recent work has shown that large language models (LLMs) can iteratively improve their outputs by incorporating generated samples and their corresponding evaluation scores as in-context examples. Despite these empirical findings, the theoretical foundations underlying this phenomenon remain poorly understood. In this paper, we show that score-conditioned In-Context Learning (ICL) admits a structural correspondence to policy gradient optimization. We first provide a constructive proof that self-attention mechanisms can implement reward-weighted aggregation analogous to the REINFORCE algorithm under specific weight matrix configurations, and discuss the relationship between this construction and the behavior of pretrained transformers. The correspondence is directional in hidden-state space and holds exactly only under the stated simplifying conditions; we quantify its strength empirically. Within our simplified hidden-state model, we furthermore derive an exact upper bound on the distribution shift induced by a bounded attention update, yielding a trust-region-like analogy to KL-constrained policy optimization. We validate our theory through extensive experiments across multiple LLMs, demonstrating that LLMs effectively utilize score information to shift output distributions toward high-scoring exemplars, and that attention weights exhibit a strong correlation with example scores.

Abstract

Explore similar work

In-Context Learning as Implicit Policy Gradient

In-Context Reinforcement Learning under Non-Stationarity: A Survey

Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policy