Offline RL

Offline reinforcement learning studies how to learn policies from previously collected transitions or demonstrations, which is useful when online trial and error is costly, unsafe, or unavailable. Value-based methods such as Conservative Q-Learning and Implicit Q-Learning, behavior-regularized policy optimization, learned world models, and trajectory or sequence models are common approaches; goal-conditioned and generative methods address long-horizon tasks and complex action distributions. Methods are often compared on benchmarks such as D4RL and OGBench, alongside applications in robotics, healthcare, and control. The central tension is improving beyond the logged behavior while avoiding unreliable value estimates and unsupported actions, so data coverage, uncertainty, and trustworthy evaluation remain key challenges.