In dynamical systems, feedback can improve performance by allowing decisions to depend on past observations, rather than being designed a priori. Under full observation, the current state is sufficient for decision-making; under partial observation, the controller must instead estimate the state from measurements. When actions also affect observations, control can have a dual role: actions influence both the state and the information available for future decisions. This interaction can invalidate certainty-equivalent (CE) control, a widely used strategy that applies state feedback policies as if the estimated state were the true state. We ask when this suboptimality leads to a quantitative performance gap that grows with the horizon. We study this question in a linear-quadratic system with bilinear observations by comparing the optimal policy, certainty-equivalent control, and the optimal open-loop (OL) policy. We identify an instance-dependent information-strength parameter q and establish distinct performance regimes as q varies. When q is sufficiently small, we show that no policy can improve over OL control by more than O(1) cumulative cost. For large values of q, we find that CE achieves an Ω(T) advantage over OL control. Our main finding is that for intermediate values of q, we can construct a policy that occasionally probes for information and achieves an Ω(T) advantage over OL control while CE only achieves an O(1) advantage, establishing an Ω(T) gap between CE and optimal control. Our analysis relies on a regret decomposition that captures the tradeoff between the immediate cost of gathering information and its future benefit.
Figures & tables
Regime
Threshold
RT(π⋆,πOL)
RT(πCE,πOL)
Low info.
q<q1
O(1)
O(1)
Separation
q2<q<q3
Ω(T)
O(1)
High info.
q4<q
Ω(T)
Ω(T)
Table 1: RT(π,π′) refers to the regret of policy π′ with respect to π as the benchmark. πOL , πCE , and π⋆ refer to the optimal OL, CE, and optimal policies, respectively.
Figure 1: (a) Depicts the advantage of CE and probing over open-loop; (b) depicts the advantage of probing over CE. The advantages are normalized by T , hence values bounded away from zero are consistent with linear advantage, while values near zero are consistent with bounded advantage. The observed behavior is consistent with the theoretical regimes: probing begins to outperform open-loop control around q2 , while CE’s advantage begins around q4 . For small q , probing incurs costs that outweigh its informational benefits. For large q , CE and probing perform similarly. Appendix H provides the experimental setup and additional plots illustrating how these advantages scale with the horizon.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Main proof dependencies for Section C . An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.
Figure 3: Main proof dependencies for Section D . An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.
Figure 4: Main proof dependencies for Section E . An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.
Figure 5: Main proof dependencies for Section F . An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.
Figure 6: Main proof dependencies for Section G . An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.
Figure 7: Numerical illustration of CE and probing behavior in two different regimes. We plot the cumulative regret RT(π,πOL) over the open-loop policy as a function of the horizon T . In the intermediate q regime q2<q=0.50<q3 (left), the CE regret remains bounded while the probing policy accumulates regret linearly. In the high q regime q=4.00>q4 (right), CE itself accumulates linear regret. The same probing-policy parameters are used in both panels.
Figure 8: (a) Depicts the normalized advantage of CE and probing over open-loop; (b) depicts the normalized advantage of probing over CE. The policy performance as q varies is similar to the scalar experiment in Figure 1 ; we see a clear separation between the two policies for intermediate values of q .
Symbol
Meaning
Model
t,T
Time index and horizon; t=0,…,T .
xt,ut,yt
State, action, and observation.
A,B,C
State transition, action, and bilinear observation coefficients.
wt,zt
Process noise and measurement noise.
Σw,σz2
Process-noise and measurement-noise variances.
Appendix
Table 2: Main notation for the scalar setting ( n=p=1 ). Constants local to individual proofs are omitted.
When a reinforcement learning agent cannot observe the full state, we usually blame its policies: it cannot see enough to represent a good one. We show that in a solvable case the bigger problem lies elsewhere. Even when a good policy is available and the agent's value function is expressive enough to describe it exactly, learning still ends up somewhere far worse. We study a partially observed linear-quadratic problem in which a standard actor-critic learner can be solved in closed form. At our default setting the best policy the agent can represent is already close to optimal, costing 10.4% more than the ideal controller that observes everything. Learning does not find it. The algorithm instead comes to rest at a policy that is 35% worse than the best one available to it, and we can say exactly where and why. The cause is a bias in what the critic learns rather than a limit on what the actor can express. Because the agent cannot attribute what it sees to the part of the state it cannot observe, the critic misreads that unexplained variation as sharp curvature in its own value estimates, and the actor follows that error away from the optimum. We derive closed-form expressions for the resulting policy, for its cost, and for the one design choice that removes the problem, which is how far the learner looks ahead before trusting its own value estimates. Deep reinforcement learning experiments follow these predictions closely. Notably, giving the agent memory of past observations does not help, while changing how far it looks ahead does.
Idil Gözel
University College London · Department of Physics and Astronomy University College London, London, UK
Fundamental limits on the performance of feedback controllers are essential for benchmarking algorithms, guiding sensor selection, and certifying task feasibility -- yet few general-purpose tools exist for computing them. Existing information-theoretic approaches overestimate the information a sensor must provide by evaluating it against the uncontrolled system, producing bounds that degrade precisely when feedback is most valuable. We derive a lower bound on the minimum expected cost of any causal feedback controller under partial observations by applying the Gibbs variational principle to the joint path measure over states and observations. The bound applies to nonlinear, nonholonomic, and hybrid dynamics with unbounded costs and admits a self-consistent refinement: any good controller concentrates the state, which limits the information the sensor can extract, which tightens the bound. The resulting fixed-point equation has a unique solution computable by bisection, and we provide conditions under which the free energy minimization is provably convex, yielding a certifiably correct numerical bound. On a scalar LQG problem the self-consistent bound captures over 80% of the known optimal cost at moderate sensor noise, and on a nonlinear Dubins car tracking problem it remains informative across all noise levels where a bound using the uncontrolled state distribution is vacuous.
Vincent Pacelli, Evangelos A. Theodorou
Daniel Guggenheim School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, USA, 30308
In this paper, we formalize a joint communication-control strategy optimization (JCCO) problem in multi-agent linear systems with quadratic costs, under the common-information-based (CIB) framework from decentralized stochastic control. For computational tractability, we focus on such JCCO problems with partially nested (PN) information structures (ISs). In particular, with a baseline communication protocol that leads to a PN IS, we establish a series of conditions under which the partial nestedness is preserved under the (additional) communication strategies to be optimized, while violating them may cause nonlinearity of the optimal strategies in general, with open-loop communication strategies. We then develop a dynamic-programming-based approach to compute the optimal control strategies of JCCO with open-loop communication strategies, which yields a set of closed-form Riccati Equations. As a byproduct of independent interest, such an approach also offers a way to solve decentralized linear-quadratic control with PN ISs and output feedback, under the CIB framework. Finally, we extend such an approach to JCCOs with closed-loop communication strategies, yielding a more tractable dynamic program than an infinite-dimensional CIB-belief-based one.
Haoyi You, Kaiqing Zhang
University of Maryland, College Park, MD, USA, 20742.