Performance Gaps Between Open-Loop, Certainty-Equivalent, and Optimal Control Under Action-Dependent Observations

Leo Maynard-Zhang; Sunmook Choi; Yahya Sattar; Yassir Jedra; Sarah Dean; Maryam Fazel

cs.LGOct 24, 2026

Performance Gaps Between Open-Loop, Certainty-Equivalent, and Optimal Control Under Action-Dependent Observations

Authors: Leo Maynard-Zhang, Sunmook Choi, Yahya Sattar, Yassir Jedra, Sarah Dean, Maryam Fazel

Organizations: U Washington · Cornell · Imperial College London · U Washington & Amazon

Abstract

In dynamical systems, feedback can improve performance by allowing decisions to depend on past observations, rather than being designed a priori. Under full observation, the current state is sufficient for decision-making; under partial observation, the controller must instead estimate the state from measurements. When actions also affect observations, control can have a dual role: actions influence both the state and the information available for future decisions. This interaction can invalidate certainty-equivalent (CE) control, a widely used strategy that applies state feedback policies as if the estimated state were the true state. We ask when this suboptimality leads to a quantitative performance gap that grows with the horizon. We study this question in a linear-quadratic system with bilinear observations by comparing the optimal policy, certainty-equivalent control, and the optimal open-loop (OL) policy. We identify an instance-dependent information-strength parameter $q$ and establish distinct performance regimes as $q$ varies. When $q$ is sufficiently small, we show that no policy can improve over OL control by more than $O(1)$ cumulative cost. For large values of $q$ , we find that CE achieves an $Ω(T)$ advantage over OL control. Our main finding is that for intermediate values of $q$ , we can construct a policy that occasionally probes for information and achieves an $Ω(T)$ advantage over OL control while CE only achieves an $O(1)$ advantage, establishing an $Ω(T)$ gap between CE and optimal control. Our analysis relies on a regret decomposition that captures the tradeoff between the immediate cost of gathering information and its future benefit.

Figures & tables

Regime	Threshold	$R_{T}(\pi^{\star},\pi^{OL})$	$R_{T}(\pi^{CE},\pi^{OL})$
Low info.	$q<q_{1}$	$O(1)$	$O(1)$
Separation	$q_{2}<q<q_{3}$	$\Omega(T)$	$O(1)$
High info.	$q_{4}<q$	$\Omega(T)$	$\Omega(T)$

Table 1:

R_{T}(\pi,\pi^{\prime})

refers to the regret of policy

\pi^{\prime}

with respect to

\pi

as the benchmark.

\pi^{OL}

\pi^{CE}

, and

\pi^{\star}

refer to the optimal OL, CE, and optimal policies, respectively.

Figure 1: (a) Depicts the advantage of CE and probing over open-loop; (b) depicts the advantage of probing over CE. The advantages are normalized by

T

, hence values bounded away from zero are consistent with linear advantage, while values near zero are consistent with bounded advantage. The observed behavior is consistent with the theoretical regimes: probing begins to outperform open-loop control around

q_{2}

, while CE’s advantage begins around

q_{4}

. For small

q

, probing incurs costs that outweigh its informational benefits. For large

q

, CE and probing perform similarly. Appendix H provides the experimental setup and additional plots illustrating how these advantages scale with the horizon.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 2 — Figure 2: Main proof dependencies for Section C . An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.

Panel of Figure 3 — Figure 3: Main proof dependencies for Section D . An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.

Panel of Figure 4 — Figure 4: Main proof dependencies for Section E . An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.

Panel of Figure 5 — Figure 5: Main proof dependencies for Section F . An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.

Panel of Figure 6 — Figure 6: Main proof dependencies for Section G . An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.

Figure 7: Numerical illustration of CE and probing behavior in two different regimes. We plot the cumulative regret

R_{T}(\pi,\pi^{OL})

over the open-loop policy as a function of the horizon

T

. In the intermediate q regime

q_{2}<q=0.50<q_{3}

(left), the CE regret remains bounded while the probing policy accumulates regret linearly. In the high q regime

q=4.00>q_{4}

(right), CE itself accumulates linear regret. The same probing-policy parameters are used in both panels.

Figure 8: (a) Depicts the normalized advantage of CE and probing over open-loop; (b) depicts the normalized advantage of probing over CE. The policy performance as

q

varies is similar to the scalar experiment in Figure 1 ; we see a clear separation between the two policies for intermediate values of

q

Symbol	Meaning
Model
$t,\ T$	Time index and horizon; $t=0,\ldots,T$ .
$x_{t},\ u_{t},\ y_{t}$	State, action, and observation.
$A,\ B,\ C$	State transition, action, and bilinear observation coefficients.
$w_{t},\ z_{t}$	Process noise and measurement noise.
$\Sigma_{w},\ \sigma_{z}^{2}$	Process-noise and measurement-noise variances.

Appendix

Table 2: Main notation for the scalar setting (

n=p=1

). Constants local to individual proofs are omitted.

Stochastic Optimal Control

Stochastic optimal control studies how to choose actions over time when system dynamics, observations, or outcomes are uncertain.

See all 100 papers

Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysis
Aug 7, 2026Idil GözelRL Control Reinforcement Learning
University College London · Department of Physics and Astronomy University College London, London, UK
Fundamental Limits for Sensor-Based Control via the Gibbs Variational Principle
Mar 19, 2026Vincent Pacelli, Evangelos A. TheodorouFeedback Control Partially Observable Markov Decision Processes
Daniel Guggenheim School of Aerospace Engineering, Georgia Institute of Technology, Atlanta, GA, USA, 30308
Joint Communication-Control Strategy Optimization with Partially Nested Information Structures: The Linear-Quadratic Case
Aug 13, 2026Haoyi You, Kaiqing ZhangStochastic Optimal Control
University of Maryland, College Park, MD, USA, 20742.

Aug 7, 2026J/K move · Enter open · S save

Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysis

Idil Gözel

University College London · Department of Physics and Astronomy University College London, London, UK

RL Control Reinforcement Learning Partially Observable RL Temporal-Difference Learning Actor-Critic Methods Policy Learning

Open paper AskPDF arXiv

When a reinforcement learning agent cannot observe the full state, we usually blame its policies: it cannot see enough to represent a good one. We show that in a solvable case the bigger problem lies elsewhere. Even when a good policy is available and the agent's value function is expressive enough to describe it exactly, learning still ends up somewhere far worse. We study a partially observed linear-quadratic problem in which a standard actor-critic learner can be solved in closed form. At our default setting the best policy the agent can represent is already close to optimal, costing 10.4% more than the ideal controller that observes everything. Learning does not find it. The algorithm instead comes to rest at a policy that is 35% worse than the best one available to it, and we can say exactly where and why. The cause is a bias in what the critic learns rather than a limit on what the actor can express. Because the agent cannot attribute what it sees to the part of the state it cannot observe, the critic misreads that unexplained variation as sharp curvature in its own value estimates, and the actor follows that error away from the optimum. We derive closed-form expressions for the resulting policy, for its cost, and for the one design choice that removes the problem, which is how far the learner looks ahead before trusting its own value estimates. Deep reinforcement learning experiments follow these predictions closely. Notably, giving the agent memory of past observations does not help, while changing how far it looks ahead does.

Abstract

Stochastic Optimal Control

Explore similar work

Learning Suffers More Than the Policy Class Under Partial Observability: A Closed-Form Analysis

Fundamental Limits for Sensor-Based Control via the Gibbs Variational Principle

Joint Communication-Control Strategy Optimization with Partially Nested Information Structures: The Linear-Quadratic Case