Policy Learning

Follow this topic Ask about this topic

Policy learning studies how to choose actions or interventions over time to improve rewards, welfare, or other goals, often when outcomes are uncertain or feedback is limited. Common approaches include policy-gradient and actor–critic reinforcement learning, value-based and offline learning, imitation from demonstrations, and direct learning of treatment or resource-allocation rules. Current work also tackles long-horizon language-model agents and generative robot policies, using methods such as finer-grained credit assignment, exploration, memory, and policy reuse. A central tension is achieving reliable, safe learning with theoretical guarantees while using expressive policies and imperfect data or rewards efficiently.

Latest papers

488 papers74 papers in the last four weeks, up 48% on the four weeks before. 0.4% of all new papers.

Streaming Deep Reinforcement Learning Finally Works
Oct 18, 2024Mohamed Elsayed, Elena Sorina Lupu, Gautham Vasan +1Reinforcement Learning Actor-Critic Methods
Department of Computing Science, University of Alberta, Edmonton, Canada · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair
Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPs
Oct 4, 2024Max Weltevrede, Caroline Horsch, Matthijs T. J. Spaan +1Reinforcement Learning RL Exploration
Delft University of Technology, Delft, The Netherlands
Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions
Jun 12, 2024Defu Cao, Angela ZhouValue Function Estimation State Abstraction
University of Southern California
Provably Efficient Off-Policy Adversarial Imitation Learning with Convergence Guarantees
May 26, 2024Yilei Chen, Vittorio Giammarino, James Queeney +1Adversarial Imitation Learning Imitation Learning
Division of Systems Engineering Boston University Boston, MA 02215, USA · Department of Computer Science Purdue University West Lafayette, IN 47907, USA · Amazon Robotics North Reading, MA 01864, USA +1
Policy Learning with a Language Bottleneck
May 7, 2024Megha Srivastava, Cedric Colas, Dorsa Sadigh +1Policy Learning LLM-Guided RL
Stanford University · Massachusetts Institute of Technology Inria
Dynamic Strategy Selection
Jul 7, 2023Francesco Cordoni, Alessio Sancetta, Yuqian ZhaoPolicy Learning Portfolio Optimization
Department of Economics, Royal Holloway University of London, Egham TW20 0EX, UK
Topology-Guided Modular Actor-Critic Learning for Continuous Systems under Temporal Objectives
Apr 20, 2023Lening Li, Zhentian Qian, Jianan Xia +7RL Control Reinforcement Learning
Ningbo University, Ningbo, China · Robotics Engineering Program, Worcester Polytechnic Institute, Worcester, MA, USA · Ningbo Zhenhai Silver-Ball Bearing Co., Ltd., Ningbo, China
Policy design in experiments with unknown interference
Nov 16, 2020Davide Viviano, Jess RudderCausal Effect Estimation Policy Optimization
Department of Economics, Harvard University, 1805 Cambridge St, Cambridge, MA 02138. · Department of Economics, Oregon State University, 1500 SW Jefferson Way, Corvallis, OR 97331.

Oct 18, 2024J/K move · Enter open · S save

Streaming Deep Reinforcement Learning Finally Works

Mohamed Elsayed, Elena Sorina Lupu, Gautham Vasan, A. Rupam Mahmood

Department of Computing Science, University of Alberta, Edmonton, Canada · Alberta Machine Intelligence Institute (Amii) · Canada CIFAR AI Chair

Reinforcement Learning Actor-Critic Methods Policy Learning Streaming Algorithms

Open paper AskPDF arXiv

Learning from a stream of experience as it arrives, also known as streaming learning, is a core part of natural learning. However, reliable streaming learning has remained a persistent challenge in modern deep reinforcement learning (RL). Instead, most deep RL algorithms learn from old experience by storing past interactions in a buffer. We show that both classical streaming RL, such as Q-learning and actor-critic, when used with deep neural networks, and batch deep RL, such as PPO, SAC, and DQN, when adapted to the streaming setting, often fail to learn. Across 58 Atari games and 50 continuous-control tasks, we find that these methods, in aggregate, perform close to random policies despite extensive task-specific hyperparameter searches. We call this pattern stream barrier. Here, we introduce Stream-X, a shared recipe for streaming deep RL algorithms that combines signal normalization, representation stabilization, and controlled parameter updates. By applying Stream-X to several base streaming RL algorithms, we provide the first family of deep RL algorithms to overcome the stream barrier. Using one prescribed hyperparameter configuration per algorithm across tasks, Stream-X substantially improves aggregate performance, often on par with batch RL algorithms. Beyond these benchmarks, we demonstrate learning with Stream-X algorithms under nonstationarity and resource constraints. Stream-AC, one of the Stream-X algorithms, repeatedly recovers performance across alternating floor-friction regimes in simulation, outperforming the evaluated PPO and SAC baselines. It also learns a heading tracking task on a robot using proprioceptive and visual features from the on-board camera in a naturally changing laboratory environment. Stream-Q learns a Pong game from pixels directly on an ESP32-S3 microcontroller, a device with limited compute and memory.