Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study

Weiqin Chen; Xinlong Du; Gesualdo Scutari; Harsha Honnappa

stat.MLOct 20, 2026

Is In-Context Reinforcement Learning A Posterior Sampler? An Empirical Study

Authors: Weiqin Chen, Xinlong Du, Gesualdo Scutari, Harsha Honnappa

Organizations: Edwardson School of Industrial Engineering, Purdue University West Lafayette, IN 47907, USA

Abstract

In-context reinforcement learning (ICRL) exploits the impressive in-context learning ability of large language models to zero-shot generalize to new, unseen tasks at inference time without any parameter updates. Among many approaches, the Decision-Pretrained Transformer (DPT) is particularly appealing for its theoretical grounding: under idealized assumptions such as a sufficiently expressive model and infinite pretraining data, DPT can perform Bayesian posterior sampling. Whether this characterization survives in practice remains open. Therefore, we ask: is ICRL, specifically DPT and our proposed DPT variants, a Bayesian posterior sampler in practice? To approach this question, we establish two necessary conditions that a Bayesian posterior sampler must satisfy: sufficient-statistic invariance and the martingale property. We then construct statistical hypothesis tests to statistically assess if practically-trained ICRL models satisfy these conditions. Applying these tests in Gaussian, Bernoulli, and linear bandit settings, we find that both conditions are consistently violated, suggesting that ICRL may not behave as Bayesian posterior samplers in practice. Beyond this negative result, our tests provide a principled toolkit for testing the Bayesian behavior of ICRL.

Figures & tables

Panel of Figure 1 — Figure 1: DPT versus Thompson Sampling in Gaussian bandits: cumulative regret and total variation (TV) distance between action distributions.

	Linear bandits
	Offline		Online
$n$	Sort	Replace	Sort	Replace
$25$	$1.56\times 10^{-18}$	$7.60\times 10^{-7}$	$4.08\times 10^{-22}$	$2.31\times 10^{-4}$
$50$	$1.76\times 10^{-49}$	$1.65\times 10^{-12}$	$2.95\times 10^{-59}$	$1.31\times 10^{-8}$
$100$	$4.88\times 10^{-76}$	$1.21\times 10^{-21}$	$1.05\times 10^{-87}$	$4.79\times 10^{-26}$
$150$	$1.81\times 10^{-67}$	$1.74\times 10^{-42}$	$8.36\times 10^{-69}$	$7.27\times 10^{-33}$

Table 1:

p

-values for sufficient-statistic invariance tests on DPT in Gaussian, Bernoulli, and linear bandits (

N=5000

\alpha=0.05

). Here,

n

denotes the context length.

		Gaussian		Bernoulli
$n$	$m$	Offline	Online	Offline	Online
$50$	$100$	$<10^{-300}$	$<10^{-300}$	$<10^{-300}$	$<10^{-300}$
	$200$	$<10^{-300}$	$<10^{-300}$	$<10^{-300}$	$<10^{-300}$
	$300$	$<10^{-300}$	$<10^{-300}$	$<10^{-300}$	$<10^{-300}$
	$400$	$<10^{-300}$	$<10^{-300}$	$<10^{-300}$	$<10^{-300}$
	$450$	$<10^{-300}$	$<10^{-300}$	$<10^{-300}$	$<10^{-300}$

Table 2:

p

-values for martingale tests on DPT ′ in Gaussian, Bernoulli, and linear bandits (

N=5000

\alpha=0.05

). Here,

n

and

m

denote the starting-context and continuation lengths, respectively.

		Gaussian		Bernoulli
$n$	$m$	Offline	Online	Offline	Online
$50$	$100$	$6.85\times 10^{-110}$	$2.89\times 10^{-248}$	$2.02\times 10^{-5}$	$0.099$
	$200$	$1.33\times 10^{-192}$	$<10^{-300}$	$0.004$	$0.014$
	$300$	$1.20\times 10^{-61}$	$2.04\times 10^{-157}$	$3.09\times 10^{-4}$	$0.028$
	$400$	$1.90\times 10^{-53}$	$9.77\times 10^{-121}$	$0.002$	$0.042$
	$450$	$5.02\times 10^{-57}$	$8.08\times 10^{-246}$	$0.001$	$0.013$

Table 3:

p

-values for martingale tests on GDPT in Gaussian, Bernoulli, and linear bandits (

N=5000

\alpha=0.05

). Here,

n

and

m

denote the starting-context and continuation lengths, respectively.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Environment and data
Parameter	Gaussian	Bernoulli	Linear
Number of arms $K$	$5$	$5$	$10$
Feature dimension $d$	—	—	$2$
Environment prior	$\mu_{a}\sim U[0,1]$	$p_{a}\sim\mathrm{Beta}(1,1)$	$\theta\sim\mathcal{N}(0,I_{2}/2)$
Reward distribution	$\mathcal{N}(\mu_{a},0.09)$	$\mathrm{Bernoulli}(p_{a})$	$\mathcal{N}(x_{a}^{\top}\theta,0.09)$
Context horizon $H$	$500$	$500$	$200$

Appendix

Table 4: Environment, model, and statistical test hyperparameters for DPT, DPT ′ , and GDPT.

Panel of Figure 2 — Figure 2: Performance comparison of DPT and Thompson Sampling in Gaussian bandits under online and offline deployment.

Panel of Figure 3 — Figure 3: Performance comparison of DPT and Thompson Sampling in Bernoulli bandits under online and offline deployment.

Panel of Figure 4 — Figure 4: Performance comparison of DPT and Thompson Sampling in linear bandits under online and offline deployment.

	Linear
	Offline		Online
$n$	Sort	Replace	Sort	Replace
$25$	$1.96\times 10^{-102}$	$5.71\times 10^{-28}$	$1.15\times 10^{-86}$	$4.19\times 10^{-29}$
$50$	$6.30\times 10^{-96}$	$2.33\times 10^{-27}$	$7.27\times 10^{-93}$	$2.22\times 10^{-28}$
$100$	$9.44\times 10^{-108}$	$9.99\times 10^{-18}$	$3.07\times 10^{-99}$	$1.10\times 10^{-21}$
$150$	$1.78\times 10^{-93}$	$3.22\times 10^{-16}$	$8.76\times 10^{-84}$	$3.62\times 10^{-14}$

Appendix

Table 5:

p

-values for sufficient-statistic invariance tests on DPT ′ in Gaussian, Bernoulli, and linear bandits (

N=5000

\alpha=0.05

). Here,

n

denotes the context length.

Panel of Figure 5 — Figure 5: Performance comparison of DPT ′ and Thompson Sampling in Gaussian bandits under online and offline deployment.

Panel of Figure 6 — Figure 6: Per-arm martingale increments for DPT ′ in Gaussian bandits with offline starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

Panel of Figure 7 — Figure 7: Per-arm martingale increments for DPT ′ in Gaussian bandits with online starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

Panel of Figure 8 — Figure 8: Performance comparison of DPT ′ and Thompson Sampling in Bernoulli bandits under online and offline deployment.

Panel of Figure 9 — Figure 9: Per-arm martingale increments for DPT ′ in Bernoulli bandits with offline starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

Panel of Figure 10 — Figure 10: Per-arm martingale increments for DPT ′ in Bernoulli bandits with online starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

Panel of Figure 11 — Figure 11: Performance comparison of DPT ′ and Thompson Sampling in linear bandits under online and offline deployment.

Panel of Figure 12 — Figure 12: Per-arm martingale increments for DPT ′ in linear bandits with offline starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

Panel of Figure 13 — Figure 13: Per-arm martingale increments for DPT ′ in linear bandits with online starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

	Linear
	Offline		Online
$n$	Sort	Replace	Sort	Replace
$25$	$1.37\times 10^{-107}$	$3.64\times 10^{-36}$	$6.02\times 10^{-91}$	$3.81\times 10^{-36}$
$50$	$5.32\times 10^{-91}$	$1.83\times 10^{-41}$	$1.22\times 10^{-81}$	$1.34\times 10^{-32}$
$100$	$1.99\times 10^{-49}$	$2.37\times 10^{-51}$	$2.73\times 10^{-42}$	$1.91\times 10^{-39}$
$150$	$2.60\times 10^{-31}$	$2.88\times 10^{-49}$	$3.15\times 10^{-29}$	$9.43\times 10^{-45}$

Appendix

Table 6:

p

-values for sufficient-statistic invariance tests on GDPT in Gaussian, Bernoulli, and linear bandits (

N=5000

\alpha=0.05

). Here,

n

denotes the context length.

Panel of Figure 14 — Figure 14: Performance comparison of GDPT and Thompson Sampling in Gaussian bandits under online and offline deployment.

Panel of Figure 15 — Figure 15: Per-arm martingale increments for GDPT in Gaussian bandits with offline starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

Panel of Figure 16 — Figure 16: Per-arm martingale increments for GDPT in Gaussian bandits with online starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

Panel of Figure 17 — Figure 17: Performance comparison of GDPT and Thompson Sampling in Bernoulli bandits under online and offline deployment.

Panel of Figure 18 — Figure 18: Per-arm martingale increments for GDPT in Bernoulli bandits with offline starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

Panel of Figure 19 — Figure 19: Per-arm martingale increments for GDPT in Bernoulli bandits with online starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

Panel of Figure 20 — Figure 20: Performance comparison of GDPT and Thompson Sampling in linear bandits under online and offline deployment.

Panel of Figure 21 — Figure 21: Per-arm martingale increments for GDPT in linear bandits with offline starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

Panel of Figure 22 — Figure 22: Per-arm martingale increments for GDPT in linear bandits with online starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

	Gaussian
	Offline		Online
$n$	Sort	Replace	Sort	Replace
$50$	Equal	$6.65\times 10^{-36}$	Equal	$6.60\times 10^{-90}$
$100$	Equal	$2.79\times 10^{-94}$	Equal	$2.54\times 10^{-62}$
$200$	Equal	$5.38\times 10^{-217}$	Equal	$1.46\times 10^{-46}$
$300$	Equal	$8.01\times 10^{-305}$	Equal	$2.11\times 10^{-44}$

Appendix

Table 7:

p

-values for sufficient-statistic invariance tests on permutation-invariant DPT in Gaussian bandits (

N=5000

\alpha=0.05

). “Equal” denotes identical action probabilities and no

p

-value is reported.

Panel of Figure 23 — Figure 23: Performance comparison of permutation-invariant DPT and Thompson Sampling in Gaussian bandits under online and offline deployment.

	Gaussian
	Offline		Online
$n$	Sort	Replace	Sort	Replace
$50$	Equal	$2.46\times 10^{-233}$	Equal	$<10^{-300}$
$100$	Equal	$1.09\times 10^{-293}$	Equal	$<10^{-300}$
$200$	Equal	$<10^{-300}$	Equal	$<10^{-300}$
$300$	Equal	$<10^{-300}$	Equal	$<10^{-300}$

Appendix

Table 8:

p

-values for sufficient-statistic invariance tests on permutation-invariant DPT ′ in Gaussian bandits (

N=5000

\alpha=0.05

). “Equal” denotes identical action probabilities and no

p

-value is reported.

		Gaussian
$n$	$m$	Offline	Online
$50$	$100$	$<10^{-300}$	$<10^{-300}$
	$200$	$<10^{-300}$	$<10^{-300}$
	$300$	$<10^{-300}$	$<10^{-300}$
	$400$	$<10^{-300}$	$<10^{-300}$
	$450$	$<10^{-300}$	$<10^{-300}$

Appendix

Table 9:

p

-values for martingale tests on permutation-invariant DPT ′ in Gaussian bandits (

N=5000

\alpha=0.05

). Here,

n

and

m

denote the starting-context and continuation lengths, respectively.

Panel of Figure 24 — Figure 24: Performance comparison of permutation-invariant DPT ′ and Thompson Sampling in Gaussian bandits under online and offline deployment.

Panel of Figure 25 — Figure 25: Per-arm martingale increments for permutation-invariant DPT ′ in Gaussian bandits with offline starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

Panel of Figure 26 — Figure 26: Per-arm martingale increments for permutation-invariant DPT ′ in Gaussian bandits with online starting contexts, where $m$ and $t$ denote the lengths of the martingale continuation and the starting contexts, respectively.

In-Context RL

In-context reinforcement learning studies how agents adapt their policies from interaction histories without updating model parameters.

See all 45 papers

Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought
May 8, 2026Zixuan Xie, Xinyu Liu, Rohan Chandra +1Reinforcement Learning Transformer
University of Virginia
TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning
Apr 1, 2026Wenxuan Jiang, Yuxin Zuo, Zijian Zhang +8In-Context RL RL for Language Model Reasoning
The Hong Kong Polytechnic University · Meituan-M17 · Institute of Computing Technology, Chinese Academy of Sciences +3
Downstream-Aware Context Selection for Online In-Context Reinforcement Learning
Sep 27, 2026Ruihan A. Li, Shangtong Zhang, Rohan ChandraIn-Context RL In-Context Learning
University of Illinois Urbana-Champaign · University of Virginia

May 8, 2026J/K move · Enter open · S save

Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought

Zixuan Xie, Xinyu Liu, Rohan Chandra, Shangtong Zhang

University of Virginia

Reinforcement Learning Transformer Transformer-Based RL CoT Reasoning Temporal-Difference Learning In-Context Learning+1

Open paper AskPDF arXiv

In-context reinforcement learning (ICRL) refers to the ability of RL agents to adapt to new tasks at inference time without parameter updates by conditioning on additional context. Recent empirical studies further demonstrate that Chain-of-Thought (CoT) generation can amplify this ICRL capability. This paper is the first to provide a theoretical understanding on how CoT interacts with ICRL. We conduct our analysis in a policy evaluation setup with linear Transformer. We prove that with specific Transformer parameters, the CoT generation process is equivalent to repeatedly executing temporal difference learning updates. Additionally, we provide finite sample convergence analysis showing that the policy evaluation error decreases geometrically with CoT length and eventually saturates at a statistical floor determined by the context length. We also prove that the desired Transformer parameters are a global minimizer of the pretraining loss, providing a theoretical understanding on the empirical emergence of those parameters.

Abstract

In-Context RL

Explore similar work

Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought

TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning

Downstream-Aware Context Selection for Online In-Context Reinforcement Learning