What Information Buys in Active Inference

Melih Kandemir; Mustafa Mert Çelikok

cs.LGOct 13, 2026

What Information Buys in Active Inference

Authors: Melih Kandemir, Mustafa Mert Çelikok

Organizations: Department of Mathematics and Computer Science, University of Southern Denmark

Abstract

Information-theoretic exploration is of primary interest in methodological studies of reinforcement learning (RL) and computational cognitive neuroscience, while the established RL theory builds regret guarantees on confidence sets. This situation leaves open the conditions under which the information-driven exploration found in nature is more advantageous than the exploration of the established RL theory. We put the state-of-the-art methods of these two paradigms, Active Inference (AIF) and Bayes-UCBVI, on an equal footing: both act through the same state abstraction of a tabular MDP with $S$ states, and their greedy forms differ only in the exploration term. We find that AIF's information gain decays too fast to serve as a confidence width, locking greedy agents that explore by it out of the optimal action. Calibrating the information gain by a square root yields calibrated AIF, which attains regret of order $\sqrt T$ for an exact abstraction, but no better guarantee than Bayes-UCBVI for a fixed reward. When an adversary chooses rewards from the agent's data, Bayes-UCBVI's value-specific widths lock out, whereas calibrated AIF pays a factor $\sqrt S$ , which no goal-memoryless agent avoids on some MDP whose start states the adversary also chooses. Calibrated information gain thus acts as a reward-agnostic $\ell_1$ confidence set, preferable to value-specific widths when rewards are chosen from the agent's data, not when they are fixed.

Figures & tables

Fixed reward function, with the accuracy of $\phi$ (rows 1–3) and exploration by information (rows 4–8)
#	Agent	Bound	Result	Status
1	any agent acting through $\phi$	$\Omega(\bar{\varepsilon}K+\sqrt{HXT})$	Cor. 20	new
2	UCBVI-CH on $\phi$	$\mathcal{O}(\epsilon_{\text{rep}}K)+\widetilde{\mathcal{O}}(H\sqrt{XAT})$	Thm. 2 (a)	new
3	Bayes-UCBVI on $\phi$	$\mathcal{O}(\epsilon_{\text{rep}}K)+\widetilde{\mathcal{O}}(H^{3/2}\sqrt{XAT})$	Thm. 2 (b)	new
4	AIF and CaI, softmax	$\Omega(K)$	Prop. 6	adapted ${}^{\text{d}}$
5	greedy, $\lambda\,\mathrm{Novelty}^{q}$ , $q\in(\frac{1}{2},1]$	$\Omega(K)$	Thm. 5 (a)	new ${}^{\text{a}}$

Table 1: Regret over

K

episodes of horizon

H

T=HK

steps, with

S=|\mathcal{S}|

X=|\mathcal{X}|\ll S

A=|\mathcal{A}|

, the representation scale

\epsilon_{\text{rep}}

(Definition 1 ) and its bound

\bar{\varepsilon}

(Corollary 20 ), the branching number

\Gamma_{\phi}\leq X

(Theorem 7 ), and the candidate set

\Phi

of abstractions. Each row is a separate result for its agent. Lower bounds hold for every agent of the named kind on some MDP.

\widetilde{\mathcal{O}}

hides

\mathrm{poly}\log(HSXAT/\delta)

. Status marks a result as known, adapted, an instance of a general theorem, or new, and Appendix B states the conditions of each row. (a) PAC form in Kolter and Ng (2009, Thm. 2) . (b) Frequentist counterpart of Lu and Van Roy (2019) . (c) Of Xie et al. (2023, Thm. 1) . (d) Boltzmann exploration in bandits ( Cesa-Bianchi et al., 2017 ) .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
MDP and regret
$\mathcal{S},\mathcal{A}$ ; $S,A$	ground state space and action space; their cardinalities
$P^{},R^{},H$	true transition kernel, deterministic reward $R^{*}:\mathcal{S}\times\mathcal{A}\to[0,1]$ fixed across episodes, horizon
$K$ , $T=HK$	number of episodes, number of steps
$k$ , $h$ , $\tau$	episode index, step index within an episode, step index within a rollout
$s^{k}_{h}$ , $a^{k}_{h}$ ; $s_{k,h}$ , $a_{k,h}$	state and action at step $h$ of episode $k$ (both notations are used)

Appendix

Table 2: Notation.

Work	Setting	Epistemic term	Representation	Guarantee
Kolter and Ng (2009)	tabular MDP	bonus $\propto N^{-1}$	ground states	near Bayes-optimal
Lu and Van Roy (2019)	tabular MDP	$\sqrt{\text{information gain}}$	ground states	Bayesian regret
Tiapkin et al. (2022)	tabular MDP	posterior quantile	ground states	regret
Maillard et al. (2013)	tabular, average reward	optimism	selected from $\Phi$	regret
Ortner et al. (2014)	tabular, average reward	optimism	selected from $\Phi$	regret
Xie et al. (2023)	coverable MDP	optimism over $\mathcal{F}$	implicit in $\mathcal{F}$	regret

Appendix

Table 3: The closest analyses, with their setting, the epistemic term they analyse, how the agent obtains its representation, and the type of guarantee. The setting names the model class, and the objective where it is average reward or adversarial. GP stands for Gaussian process.

Agent (result)	Reward, belief	Value	Epistemic term	Action rule	Abstraction
UCBVI-CH on $\phi$ (Thm. 2 (a))	$\widehat{R}^{k}_{\phi}$ , empirical kernel pooled over steps	max	Hoeffding bonus Eq. ( 7 ), clamps at $H$ and at the previous values	greedy	given
Bayes-UCBVI on $\phi$ (Thm. 2 (b), Prop. 26 )	$\widehat{R}^{k}_{\phi}$ or the goal $R_{k}$ , with $\rho^{k}_{h}$	max	$\kappa^{k}_{h}$ -quantile of $p^{\top}V$	greedy	given
information-driven (Thm. 5 )	$R^{*}$ , $\rho^{k}_{h}$	max	$\lambda\,\mathrm{Novelty}^{q}$ , optional clamp	greedy	given, exact
AIF (Prop. 6 , Alg. 1 )	$\widehat{R}^{k}_{\phi}$ , $\rho^{k}_{h}$	max	$\beta^{-1}(\mathds{H}[\widehat{P}^{k}_{h}]+\mathrm{Novelty})$ , novelty counted twice, clamp	softmax at $\beta$ per step, or posterior over plans (App. F.5 )	given
calibrated AIF (Thm. 7 , Alg. 1 )	$\widehat{R}^{k}_{\phi}$ , $\rho^{k}_{h}$	max	$\Psi^{+}_{k,h}$ of Eq. ( 5 ), clamp	greedy	given
calibrated AIF, uniform prior (Thm. 8 )	goal $R_{k}$ , $\mathrm{Dir}(\mathds{1}+\text{counts})$ on $\mathcal{X}$	max	$\Psi^{\mathrm{u}}_{k,h}$ , clamp at $H-h+1$	greedy	given, exact

Appendix

Table 4: The analysed agents. All Bayesian agents on

\phi

use the instrumental pseudo-state prior with one Dirichlet row per step, and estimate rewards by

\widehat{R}^{k}_{\phi}

, which is

1

at unvisited pairs (Eq. ( 23 )). The column Value states how

V_{h}(x)

is computed from

Q_{h}(x,\cdot)

in Eq. ( 4 ), by a maximum or a soft maximum, the column Epistemic term gives

\Psi

, and “clamp” is the cap at

r_{0}(H-h+1)

RL Exploration

Reinforcement learning exploration studies how agents choose informative actions and states to learn effective policies under uncertainty.

See all 216 papers

Information-Directed Offline-to-Online Reinforcement Learning
May 28, 2026Keru ChenRegret Minimization in RL Bayesian RL
School of Electrical, Computer and Energy Engineering, Arizona State University
Active Inference as a Convex Markov Decision Process
Jul 22, 2026Nikola Milosevic, Nicolás Hinrichs, Nico ScherfMarkov Decision Processes Reinforcement Learning
Max Planck Institute for Human Cognitive and Brain Sciences, Stephanstraße. 1A, 04103 Leipzig, Germany
Subspace Inference Enables Efficient Active Reward Learning from Preferences
Sep 3, 2026Yutai Zhou, Erdem BıyıkBayesian RL Active Preference Learning
Thomas Lord Department of Computer Science University of Southern California, Los Angeles, USA

May 28, 2026J/K move · Enter open · S save

Information-Directed Offline-to-Online Reinforcement Learning

Keru Chen

School of Electrical, Computer and Energy Engineering, Arizona State University

Regret Minimization in RL Bayesian RL Value of Information RL Exploration Offline-To-Online RL

Open paper AskPDF arXiv

Decision-making from offline datasets typically warm-starts a policy or score model from fixed offline data and then refines it with limited online interaction. Offline data reduces uncertainty, but it does not remove the need for exploration; it changes what remains to be explored. We formalise this residual uncertainty by the conditional mutual information

I(χ;τ_{1:T}\mid\mathcal{D}_N)

between a learning target

χ

and the online trajectories after conditioning on the offline dataset. This view leads naturally to information-directed sampling (IDS), a family parameterised by

η\ge 0

that selects actions by trading off instantaneous regret against information gain. We prove a generic offline-to-online Bayesian regret bound for IDS through a ratio certificate: any information-ratio bound satisfied by a reference Thompson-sampling policy over the same randomised policy class is inherited by IDS. In a known-dynamics Bayesian linear-reward model, the conditional mutual information has a log-determinant form, and vanilla IDS (

η=0

) satisfies

\widetilde O\!\left(Hd\min\left\{\sqrt T,\,T\sqrt{C^\dagger_{β,\mathrm{IDS}_0}(N,T)/N}\right\}\right),

where the coverage coefficient is tied to the visitation distribution induced by vanilla IDS itself. We also identify a warm-start regime with a dominated but informative probe in which vanilla IDS selects the probe while Thompson sampling never does, giving a constant-factor Bayesian regret separation. Controlled bandit experiments and D4RL offline-to-online RL experiments validate this mechanism: IDS is most beneficial when offline data is informative but leaves biased or low-probability residual uncertainty that targeted online actions can resolve, a regime shared by offline RL, offline black-box optimization, and Bayesian optimization.

Abstract

RL Exploration

Explore similar work

Information-Directed Offline-to-Online Reinforcement Learning

Active Inference as a Convex Markov Decision Process

Subspace Inference Enables Efficient Active Reward Learning from Preferences