Leveraging the Value of Information in POMDP Planning

Zakariya Laouar; Qi Heng Ho; Zachary Sunberg

cs.AIApr 1, 2026

Leveraging the Value of Information in POMDP Planning

Authors: Zakariya Laouar, Qi Heng Ho, Zachary Sunberg

Organizations: Department of Aerospace Engineering Sciences, University of Colorado Boulder · Kevin T. Crofton Department of Aerospace and Ocean Engineering, Virginia Tech

Abstract

Partially observable Markov decision processes (POMDPs) offer a principled formalism for planning under state and transition uncertainty. Despite advances made towards solving large POMDPs, obtaining performant policies under limited planning time remains a major challenge due to the curse of dimensionality and the curse of history. For many POMDP problems, the value of information (VOI) - the expected performance gain from reasoning about observations - varies over the belief space. We introduce a dynamic programming framework that exploits this structure by conditionally processing observations based on the value of information at each belief. Building on this framework, we propose Value of Information Monte Carlo planning (VOIMCP), a Monte Carlo Tree Search algorithm that allocates computational effort more efficiently by selectively disregarding observation information when the VOI is low, avoiding unnecessary branching of observations. We provide theoretical guarantees on the near-optimality of our VOI reasoning framework and derive non-asymptotic convergence bounds for VOIMCP. Simulation evaluations demonstrate that VOIMCP outperforms baselines on several POMDP benchmarks.

Figures & tables

Panel of Figure 1 — Figure 1: Comparative benchmark results presenting discounted cumulative reward against the number of tree queries over $1000$ trials. The lighter colored ribbons around the Monte Carlo mean display the $95\%$ confidence interval.

Panel of Figure 2 — Figure 2: Tree growth statistics. (Top Row) Maximum tree depth vs. number of tree queries. (Bottom Row) Effective action-observation branching factor vs. number of tree queries. Statistics are computed over 100 trials.

Domain	Algorithm	$c$	$\kappa$	$D$
Tracking	VOIMCP	100	0.03	20
	PO-UCT	100	-	20
	I-UCB POMCP	100	-	20
	Open-Loop	100	-	20
FVRockSample	VOIMCP	10	0.02	20
	PO-UCT	10	-	20

Table 1: Summary of hyperparameters used in experiments.

Panel of Figure 3 — Figure 3: Results for VOIMCP difference between the annealed and standard algorithms, presenting delta discounted cumulative reward over $1000$ trials. The lighter-colored ribbons around the Monte Carlo mean display the $95\%$ confidence interval.

Explore similar work

Belief Embedding Tree Search
Oct 10, 2026Christopher Solinas, Radovan Haluška, Martin Schmid +3Partially Observable RL Belief-Space Planning
University of Alberta · Charles University, Prague · EquiLibre Technologies, Inc. +1
Finite-Time Analysis of MCTS in Continuous POMDP Planning
May 8, 2026Da Kong, Vadim IndelmanMonte Carlo Tree Search Partially Observable Markov Decision Processes
Technion Autonomous Systems Program (TASP) · Technion – Israel Institute of Technology · Stephen B. Klein Faculty of Aerospace Engineering +1
VOiLA: Vectorized Online Planning with Learned Diffusion Models for POMDP Agents
Jun 18, 2026Marcus Hoerger, Rishikesh Joshi, Rahul Shome +2Belief-Space Planning Diffusion-Based Planning
Australian National University, Canberra, ACT, Australia · The University of Sydney, Sydney, NSW, Australia

Oct 10, 2026J/K move · Enter open · S save

Belief Embedding Tree Search

Christopher Solinas, Radovan Haluška, Martin Schmid, Michael Buro+2

University of Alberta · Charles University, Prague · EquiLibre Technologies, Inc.+1

Partially Observable RL Belief-Space Planning Partially Observable Markov Decision Processes Monte Carlo Tree Search

Open paper AskPDF arXiv

Beliefs in a Partially Observable Markov Decision Process (POMDP) are posterior distributions over the hidden state given the agent's history of actions and observations. They summarize everything the history says about the state, so planning in a POMDP is planning in a Markov decision process (MDP) over beliefs. Belief MDPs have continuous state spaces even for finite POMDPs, and exact planning in them is intractable in general. Planners must therefore approximate beliefs and estimate values. Monte Carlo planners do both inside a search tree, with an approximate belief at every node and a value estimate at every leaf. Approximations usually consist of sets of sampled states, with leaf values coming from rollouts, hand-designed bounds, or learned networks. Policies and values computed from these approximations are limited by their quality, and prior methods with learned networks rely on hand-chosen summaries of the sampled states. A belief representation should be computable from the parent node's representation through the POMDP's dynamics, let the search sample from it, and be learned so that it provides the information that policies and value functions need. We present Belief Embedding Tree Search (BETS), a Monte Carlo planner that meets these requirements by representing each node's belief with a generative distribution embedding (GDE) while updating it through the POMDP dynamics. BETS trains its belief model alongside its policy and value networks through experience, with the realized hidden state of the POMDP as the belief model's training target. We evaluate BETS on six configurations of RockSample, LightDark, and PocMan against BetaZero and three planners without learned components. BETS is competitive or leading on all six, while every baseline falls far behind on at least one. Where beliefs are highest-dimensional, it outperforms them by a large margin, even without search.

Abstract

Explore similar work

Belief Embedding Tree Search

Finite-Time Analysis of MCTS in Continuous POMDP Planning

VOiLA: Vectorized Online Planning with Learned Diffusion Models for POMDP Agents