Partially Observable RL

RL: Reinforcement Learning

Ask about this topic

Overview

Partially observable reinforcement learning studies how an agent can make good decisions when it sees only incomplete or noisy clues about the world, as in the POMDP setting. Common approaches estimate a belief over hidden states, plan with that belief or a learned world model, or use recurrent networks and transformers to summarize observation and action histories; some agents also choose actions to gather information. These methods are used in settings such as visual navigation, robotics, medical control, and multi-agent tasks where teammates’ or opponents’ states are hidden. A central tension is how to retain the right information over long histories while keeping learning and planning reliable: memory can help, but inaccurate beliefs, critic errors, and weak evaluation can still limit performance.

Latest papers

CardsList

121 papers20 papers in the last four weeks, up 67% on the four weeks before. 0.1% of all new papers.

Jul 29, 2023

Robust Recurrent Reinforcement Learning under Evolving Hidden Disturbances with Application to Rover Wheel Slip

Reinforcement learning (RL) performs well in continuous-control tasks, but evolving hidden disturbances create partial observability: the agent must infer decision-relevant latent dynamics from interaction history. This study investigates how observation history, action history, history length, and network structure affect recurrent Twin Delayed Deep Deterministic Policy Gradient (TD3) agents. Three recurrent architectures are evaluated under controlled disturbances with different temporal characteristics. Results show that action history is particularly important when observed responses depend on previous actions, and that processing past and current action-observation information within a unified temporal sequence improves performance compared with using separate branches. We also introduce H-TD3, which reuses recurrent states generated by the actor to initialize the critic, reducing duplicated sequence processing. The architectures are further tested in a simulation-based differential-drive rover motion-regulation task under hidden asymmetric wheel slip. Recurrent architectures retain their advantage under the physically motivated multiplicative wheel-slip model, while policies trained with abstract temporally structured disturbances transfer more effectively to previously unseen wheel-slip dynamics than policies trained without disturbances. These findings provide practical guidance for recurrent RL under partial observability and evolving hidden disturbances.

Saki Omi, Hyo-Sang Shin, Namhoon Cho +2

Space Robotics Research Group, Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg · School of Aerospace, Transport and Manufacturing, Cranfield University, Cranfield, U.K. · Department of Aerospace Engineering, Seoul National University, Seoul, South Korea

RL for RoboticsReinforcement LearningPartially Observable RLRobust RL
AskPDF