Batch Reinforcement Learning

Batch reinforcement learning (BRL) focuses on training reinforcement learning agents using a fixed dataset of pre-collected experiences, eliminating the need for costly and potentially risky real-time interaction with the environment. Current research emphasizes improving sample efficiency through techniques like novel loss functions (e.g., log-loss), multi-step modeling of dynamics to mitigate error compounding, and advanced algorithms such as those inspired by game theory (e.g., Stackelberg learning) or leveraging human feedback for reward shaping. BRL's significance lies in its potential to enable safe and efficient deployment of RL agents in high-stakes applications like robotics and healthcare, where extensive real-world interaction is impractical or undesirable.

Papers

November 14, 2024

Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Search
Amna Najib, Stefan Depeweg, Phillip Swazinna
Reinforcement Learning Offline Reinforcement Learning Policy Learning Batch Reinforcement Learning Policy Search Model Based Policy

October 18, 2024

Streaming Deep Reinforcement Learning Finally Works
Mohamed Elsayed, Gautham Vasan, A. Rupam Mahmood
Reinforcement Learning Deep Reinforcement Learning Q Learning Batch Reinforcement Learning Stream Learning

October 12, 2024

Decision-Point Guided Safe Policy Improvement
Abhishek Sharma, Leo Benac, Sonali Parbhoo, Finale Doshi-Velez
State Action Pair Policy Improvement Safe Policy Batch Reinforcement Learning

September 25, 2024

OffRIPP: Offline RL-based Informative Path Planning
Srikar Babu Gadipudi, Srujan Deolasee, Siva Kailas, Wenhao Luo, Katia Sycara, Woojun Kim
Reinforcement Learning Robotics Domain Action Free Offline Informative Path Planning Real World Experiment Batch Reinforcement Learning

March 8, 2024

Switching the Loss Reduces the Cost in Batch Reinforcement Learning
Alex Ayoub, Kaiwen Wang, Vincent Liu, Samuel Robertson, James McInerney, Dawen Liang, Nathan Kallus, Csaba Szepesvári
Hidden CoST Interest Loss Near Optimal Policy Batch Reinforcement Learning Fitted Q Iteration Small Loss

February 5, 2024

A Multi-step Loss Function for Robust Learning of the Dynamics in Model-based Reinforcement Learning
Abdelhakim Benechehab, Albert Thomas, Giuseppe Paolo, Maurizio Filippone, Balázs Kégl
Loss Function Model Based Reinforcement Learning Robust Learning Multi Step Batch Reinforcement Learning

December 15, 2023

ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent
Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, Manzil Zaheer, Felix Yu, Sanjiv Kumar
Large Language Model Natural Language Question AI Feedback Self Improvement Batch Reinforcement Learning Frontend Component Code Question Answering Benchmark Rest RESTAD NAP

October 9, 2023

Multi-timestep models for Model-based Reinforcement Learning
Abdelhakim Benechehab, Giuseppe Paolo, Albert Thomas, Maurizio Filippone, Balázs Kégl
Model Based Reinforcement Learning Batch Reinforcement Learning Step to Step Dynamic Multi Timestep

October 2, 2023

Sample-Efficiency in Multi-Batch Reinforcement Learning: The Need for Dimension-Dependent Adaptivity
Emmeran Johnson, Ciara Pike-Burke, Patrick Rebeschini
Reinforcement Learning Offline Reinforcement Learning Adaptive Importance Community Need Sample Efficiency Sample Efficient Local Adaptivity Batch Reinforcement Learning

September 28, 2023

Stackelberg Batch Policy Learning
Wenzhuo Zhou, Annie Qu
Game Theoretic Worst Case Regret Guarantee Instance Dependent Regret Batch Reinforcement Learning Batch Policy

August 17, 2023

Reinforced Self-Training (ReST) for Language Modeling
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, Nando de Freitas
Large Language Model Reinforcement Learning Translation Benchmark Batch Reinforcement Learning

June 13, 2023

Stepsize Learning for Policy Gradient Methods in Contextual Markov Decision Processes
Luca Sabbioni, Francesco Corda, Marcello Restelli
Policy Gradient Action Space Model Free Meta Reinforcement Learning Batch Reinforcement Learning Contextual Markov Decision Process Policy Based Algorithm

May 5, 2023

Knowledge Transfer from Teachers to Learners in Growing-Batch Reinforcement Learning
Patrick Emedom-Nnamdi, Abram L. Friesen, Bobak Shahriari, Nando de Freitas, Matt W. Hoffman
LeArning Abstract Knowledge Transfer Actor Critic Good Teacher DeepMind Control Suite Batch Reinforcement Learning Trial and Error

April 27, 2023

BCQQ: Batch-Constraint Quantum Q-Learning with Cyclic Data Re-uploading
Maniraman Periyasamy, Marc Hölle, Marco Wiedmann, Daniel D. Scherer, Axel Plinge, Christopher Mutschler
Deep Reinforcement Learning Quantum Computing Deep Q Batch Reinforcement Learning

February 6, 2023

A Strong Baseline for Batch Imitation Learning
Matthew Smith, Lucas Maystre, Zhenwen Dai, Kamil Ciosek
Markov Decision Process Feature Imitation Strong Baseline Offline Reinforcement Learning Algorithm Sequential Decision Making Batch Reinforcement Learning

January 30, 2023

STEEL: Singularity-aware Reinforcement Learning
Xiaohong Chen, Zhengling Qi, Runzhe Wan
Regret Guarantee Infinite Horizon Batch Reinforcement Learning Steel Manufacturing Singularity Avoidance

October 15, 2022

Near-Optimal Regret Bounds for Multi-batch Reinforcement Learning
Zihan Zhang, Yuhang Jiang, Yuan Zhou, Xiangyang Ji
Optimal Regret Finite Horizon Episodic Reinforcement Learning Batch Reinforcement Learning

September 30, 2022

B2RL: An open-source Dataset for Building Batch Reinforcement Learning
Hsin-Yu Liu, Xiaohan Fu, Bharathan Balaji, Rajesh Gupta, Dezhi Hong
Open Source Replay Buffer Batch Reinforcement Learning

March 11, 2022

Reinforcement Learning for Linear Quadratic Control is Vulnerable Under Cost Manipulation
Yunhan Huang, Quanyan Zhu
Reinforcement Learning Strategic Manipulation Adversarial Manipulation Approximate Dynamic Programming Batch Reinforcement Learning Attack Model

November 8, 2021

Batch Reinforcement Learning from Crowds
Guoxi Zhang, Hisashi Kashima
Reinforcement Learning Reward Function Crowded Environment Synthetic Preference Batch Reinforcement Learning