Robust RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 467% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 153

All topics

Corruption Robust Offline Reinforcement Learning with Human Feedback
Feb 9, 2024Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban +2Offline RL Corruption Robustness
Dept. of Computer Science University of Warwick, UK · Max-Planck Institute for Software Systems, Germany · Independent Researcher London, UK
Robust Recurrent Reinforcement Learning under Evolving Hidden Disturbances with Application to Rover Wheel Slip
Jul 29, 2023Saki Omi, Hyo-Sang Shin, Namhoon Cho +2RL for Robotics Reinforcement Learning
Space Robotics Research Group, Interdisciplinary Centre for Security, Reliability and Trust, University of Luxembourg · School of Aerospace, Transport and Manufacturing, Cranfield University, Cranfield, U.K. · Department of Aerospace Engineering, Seoul National University, Seoul, South Korea
RoboDreamer: Anticipatory Humanoid Locomotion with Predictive State-Space Models
Date pendingZhe Li, Yangyang Wei, Xichen Yuan +4Robot Trajectory Tracking Humanoid Robot Locomotion
BAAI · NJU

Feb 9, 2024, cs.LGJ/K move · Enter open · S save

Corruption Robust Offline Reinforcement Learning with Human Feedback

Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban, Adish Singla+1

Dept. of Computer Science University of Warwick, UK · Max-Planck Institute for Software Systems, Germany · Independent Researcher London, UK

Offline RL Corruption Robustness Robust RL Preference-Based RL RL from Human Feedback

Open paper AskPDF arXiv

We study data corruption robustness for reinforcement learning with human feedback (RLHF) in an offline setting. Given an offline dataset of pairs of trajectories along with feedback about human preferences, an

\varepsilon

-fraction of the pairs is corrupted (e.g., feedback flipped or trajectory features manipulated), capturing an adversarial attack or noisy human preferences. We aim to design algorithms that identify a near-optimal policy from the corrupted data, with provable guarantees. Existing theoretical works have separately studied the settings of corruption robust RL (learning from scalar rewards directly under corruption) and offline RLHF (learning from human feedback without corruption); however, they are inapplicable to our problem of dealing with corrupted data in offline RLHF setting. To this end, we design novel corruption robust offline RLHF methods under various assumptions on the coverage of the data-generating distributions. At a high level, our methodology robustifies an offline RLHF framework by first learning a reward model along with confidence sets and then learning a pessimistic optimal policy over the confidence set. Our key insight is that learning optimal policy can be done by leveraging an offline corruption-robust RL oracle in different ways (e.g., zero-order oracle or first-order oracle), depending on the data coverage assumptions. To our knowledge, ours is the first work that provides provable corruption robust offline RLHF methods.

Momentum

Related topics

Latest papers 153

Corruption Robust Offline Reinforcement Learning with Human Feedback

Robust Recurrent Reinforcement Learning under Evolving Hidden Disturbances with Application to Rover Wheel Slip

RoboDreamer: Anticipatory Humanoid Locomotion with Predictive State-Space Models