PriPG-RL: Privileged Planner-Guided Reinforcement Learning for Partially Observable Systems with Anytime-Feasible MPC

Mohsen Amiri; Mohsen Amiri; Ali Beikmohammadi; Sindri Magnuśson; Mehdi Hosseinzadeh

cs.LGApr 9, 2026

PriPG-RL: Privileged Planner-Guided Reinforcement Learning for Partially Observable Systems with Anytime-Feasible MPC

Authors: Mohsen Amiri, Mohsen Amiri, Ali Beikmohammadi, Sindri Magnuśson, Mehdi Hosseinzadeh

Organizations: The authors are with the Department of Computer and System Science, Stockholm University, 11419 Stockholm, Sweden · The authors are with the School of Mechanical and Material Engineering, Washington State University, Pullman, WA 99164, USA

Abstract

This paper addresses the problem of training a reinforcement learning (RL) policy under partial observability by exploiting a privileged, anytime-feasible planner agent available exclusively during training. We formalize this as a Partially Observable Markov Decision Process (POMDP) in which a planner agent with access to an approximate dynamical model and privileged state information guides a learning agent that observes only a lossy projection of the true state. To realize this framework, we introduce an anytime-feasible Model Predictive Control (MPC) algorithm that serves as the planner agent. For the learning agent, we propose Planner-to-Policy Soft Actor-Critic (P2P-SAC), a method that distills the planner agent's privileged knowledge to mitigate partial observability and thereby improve both sample efficiency and final policy performance. We support this framework with rigorous theoretical analysis. Finally, we validate our approach in simulation using NVIDIA Isaac Lab and successfully deploy it on a real-world Unitree Go2 quadruped navigating complex, obstacle-rich environments.

Figures & tables

Fig. 1: Illustration of the proposed PriPG-RL architecture during training. The planner agent provides guidance exclusively during training and is not used at runtime. The hardware image is included for visualization purposes only and does not represent a closed-loop deployment of the training architecture. A video demonstration of the hardware deployment, along with the complete source code, is available at GitHub. 1 1 1 https://github.com/mohsen1amiri/PriPG-RL_UnitreeGo2.git

Algorithm	Success (%)	Crash (%)	Path opt.	Runtime (s)	Ave. Velocity (m/s)
Accel. SAC [ 8 ]	35.0 $\pm$ 47.7	65.0 $\pm$ 47.7	1.100 $\pm$ 0.073	9.0 $\pm$ 1.3	0.477 $\pm$ 0.045
P2P-SAC	100%	0.0%	1.060 $\pm$ 0.031	9.7 $\pm$ 1.1	0.352 $\pm$ 0.019
REAP [ 15 ]	100%	0.0%	1.10 $\pm$ 0.04	12.26 $\pm$ 1.29	0.353 $\pm$ 0.028

TABLE I: Best-checkpoint metrics (mean

\pm

std, 5 seeds).

Panel of Figure 3 — Fig. 2: Training curves: mean episodic reward (solid lines, left y-axis) and success rate (dashed lines with markers, right y-axis) over environment steps; shaded regions indicate $\pm 1$ standard deviation across seeds.

Fig. 3: Hardware experiments using the Unitree Go2 quadruped. The left subfigure shows a composite image of the Unitree Go2 navigating an obstacle-rich environment under P2P-SAC. The top-right subfigure illustrates the desired velocities generated by the proposed method and the actual velocities measured by the onboard hardware, demonstrating that the velocity constraints are satisfied at all times. The bottom-right subfigure presents the trajectory in Cartesian coordinates along with the goal X-Y position.

Partially Observable RL

Partially observable reinforcement learning studies how agents act when observations hide the true state and decisions must use memory or beliefs.

See all 114 papers

Finite-Time Guarantees for Constrained Reinforcement Learning under Partial Observability
Oct 15, 2026Valentina Zaccaria, Philip Jordan, Tingting Ni +2Reinforcement Learning Constrained RL
University of Padova · SYCAMORE, EPFL
Forager: a lightweight testbed for continual learning with partial observability in RL
May 1, 2026Steven Tang, Xinze Xiong, Anna Hakhverdyan +7RL Benchmarks Loss of Plasticity
Department of Computing Science, University of Alberta, Canada · Alberta Machine Intelligence Institute (Amii) · CIFAR AI Chair
Reinforcement Learning under State and Outcome Uncertainty: A Foundational Distributional Perspective
Sep 21, 2026Larry Preuett, Qiuyi Zhang, Muhammad Aurangzeb AhmadDistributional RL Markov Decision Processes
University of Washington, Tacoma, USA · Google DeepMind, California, USA · University of Washington, Bothell, USA

Oct 15, 2026J/K move · Enter open · S save

Finite-Time Guarantees for Constrained Reinforcement Learning under Partial Observability

Valentina Zaccaria, Philip Jordan, Tingting Ni, Gian Antonio Susto+1

University of Padova · SYCAMORE, EPFL

Reinforcement Learning Constrained RL Partially Observable Markov Decision Processes

Open paper AskPDF arXiv

We study reinforcement learning in constrained partially observable Markov decision processes (CPOMDPs), where an agent maximizes reward subject to a cost constraint using only incomplete information about the underlying state. The lack of state observability may require optimal policies to depend on arbitrarily long observation-action histories, making planning and learning challenging. We leverage recent advances in finite-memory approximations of POMDPs and represent finite history windows as states of a constrained MDP. We then develop a primal--dual algorithm that provides the first non-asymptotic guarantees that jointly bound average reward suboptimality and constraint violation in the original CPOMDP. A key challenge is that this finite-window approximation affects both reward values and the set of feasible policies. Under a filter-stability assumption, we derive bounds relating optimality and feasibility in the finite-state model to those in the original CPOMDP, with approximation errors that decay geometrically with the window length. Experiments in partially observable gridworlds demonstrate the effectiveness of our approach.

Abstract

Partially Observable RL

Explore similar work

Finite-Time Guarantees for Constrained Reinforcement Learning under Partial Observability

Forager: a lightweight testbed for continual learning with partial observability in RL

Reinforcement Learning under State and Outcome Uncertainty: A Foundational Distributional Perspective