Active Noise Floor Estimation for Reliability-Optimal POMDPs: A Value-of-Noise-Information Approach

Physical sensing and actuation noise floors should inform how much belief resolution a decision-making system can reliably use. We introduce Finite Reliability Representations (FRR), a framework for covering belief spaces by reliability cells: regions within which the optimal action-value function Q*(b,u) varies by at most a tolerance epsilon, uniformly over actions. The framework is formulated on beliefs rather than states and uses a cover rather than an equivalence quotient, because approximate decision-closeness is not transitive in general. A central technical point is that noisy Bayesian updates should not be treated as globally contractive on arbitrary beliefs. We therefore separate three objects: the fixed-observation filter map, the predictive observation law, and the controlled belief-transition kernel. For nonlinear continuous-state systems, FRR is obtained under a reachable-set Lipschitz modulus for the belief-transition kernel. For finite-state POMDPs, the same construction becomes exact on the belief simplex: prediction is linear, Bayesian correction is a normalized positive linear map, sensor noise enters through observation-distribution distinguishability, and actuation uncertainty enters through an action-execution channel. Under the corresponding action-value Lipschitz condition, an FRR cover supports a cell-constant policy whose suboptimality is bounded by 2 epsilon/(1 - gamma). We also introduce reliability entropy, the logarithm of the minimal number of reliability cells, as a measure of certified decision-relevant belief complexity. The framework distinguishes representation sufficiency from fundamental performance floors imposed by sensing, process, and actuation noise. It applies to finite POMDPs, linear-Gaussian filters, locally linearized nonlinear filters, and particle-filter implementations through analytic or empirical certification of reliability cells.

An agent acting under partial observability must decide when to gather information and which observations are worth their cost. Standard POMDPs value information only through its eventual effect on reward. The $ρ$-POMDP framework instead rewards uncertainty reduction directly, through a belief-dependent utility $ρ$, but in practice both the choice of $ρ$ and the weight placed on it are tuned by hand for every task. We show that active inference removes this tuning entirely. Minimizing Expected Free Energy (EFE) is exactly equivalent to solving a $ρ$-POMDP whose utility is expected information gain, and the exploration weight is fixed at $w=1$ because the variational bound expresses pragmatic and epistemic value in the same units (nats). We prove this equivalence for observe-then-commit POMDPs and extend it to factored observation POMDPs, a broader class that covers interleaved observe-act problems such as non-destructive testing and mobile sensing, where gathering information leaves the hidden state unchanged. Experiments support the theory. Across environments ranging from the classic Tiger problem to RockSample and a new Structural Inspection benchmark with over 65,000 states, the untuned weight matches or outperforms reward-only planning at the same horizon, avoids the over-exploration of bonuses tuned per task, and sits near the reward-maximizing knee of the success-reward Pareto frontier. The practical payoff is an exploration objective that works out of the box. In applications such as fault detection and medical screening, where every test has a price and every missed fault has a cost, EFE supplies a belief-dependent utility that is derived rather than tuned.

Adaptive decision-making in biological and artificial intelligence requires balancing the exploitation of known outcomes with the exploration of uncertain alternatives. Although prior work suggests that uncertainty generally promotes exploration, it has typically treated distinct sources of environmental uncertainty as equivalent. We consider environments with latent reward states that drift over time (volatility) and are observed through noisy outcomes (stochasticity). Both increase posterior uncertainty, yet we show they drive optimal exploration in opposite directions: volatility enhances it, stochasticity suppresses it. We establish this asymmetry formally by extending the Gittins index framework to Gaussian state-space bandits with latent dynamics. We further derive Cause-Aware Uncertainty-Sensitive Exploration (CAUSE), a closed-form exploration bonus obtained via control-as-inference that inherits the same monotonicities. CAUSE outperforms standard exploration strategies in environments with heterogeneous noise structure, and also improves on a Gittins-per-arm policy whose rested-bandit optimality does not transfer to restless settings. Learning and exploration are governed by the same noise-inference asymmetry, and the framework predicts that pathological noise inference produces \emph{reversed} rather than merely impaired exploration, with implications for computational accounts of psychiatric conditions.