Offline RL

RL: Reinforcement Learning

Offline reinforcement learning studies how to learn policies from previously collected transitions or demonstrations, which is useful when online trial and error is costly, unsafe, or unavailable. Value-based methods such as Conservative Q-Learning and Implicit Q-Learning, behavior-regularized policy optimization, learned world models, and trajectory or sequence models are common approaches; goal-conditioned and generative methods address long-horizon tasks and complex action distributions. Methods are often compared on benchmarks such as D4RL and OGBench, alongside applications in robotics, healthcare, and control. The central tension is improving beyond the logged behavior while avoiding unreliable value estimates and unsupported actions, so data coverage, uncertainty, and trustworthy evaluation remain key challenges.

Latest papers

304 papers48 papers in the last four weeks, up 66% on the four weeks before. 0.3% of all new papers.

MAGE: Multi-scale Autoregressive Generation for Offline Reinforcement Learning
Feb 27, 2026Chenxing Lin, Xinhui Gao, Haipeng Zhang +7Offline RL Transformer-Based RL
Fujian Key Laboratory of Urban Intelligent Sensing and Computing, School of Informatics, Xiamen University (XMU), China · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, XMU, China · Hong Kong University of Science and Technology +2
Mollified Value Learning
Feb 26, 2026Hrishikesh Viswanath, Juanwu Lu, S. Talha Bukhari +4Offline RL Robotic RL
Department of Computer Science, Purdue University, USA · College of Engineering, Purdue University, USA · DEVCOM Army Research Laboratory, USA
Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning
Feb 24, 2026Zhangjie Xia, Yu Yang, Pan XuOffline RL Domain Adaptation
New York University · Duke University
Diffusion Modulation via Environment Mechanism Modeling for Planning
Feb 23, 2026Hanping Zhang, Yuhong GuoDiffusion-Based Planning Offline RL
School of Computer Science, Carleton University, Ottawa, Canada
Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learning
Feb 23, 2026Thanh Nguyen, Tung Luu, Tri Ton +2Uncertainty Estimation in RL Offline RL
School of Electrical Engineering, Korea Advanced Institute of Science and Technology, Daejeon 34141, Republic of Korea · Department of Artificial Intelligence, Korea University, Seoul 02841, Republic of Korea
Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixing
Feb 23, 2026Wall Kim, Chaeyoung Song, Hanul KimReinforcement Learning Offline RL
Samsung Electronics, Hwaseong, South Korea · Seoul National University of Science and Technology, Seoul, South Korea
In-Context Planning with Latent Temporal Abstractions
Feb 21, 2026Baiting Luo, Yunuo Zhang, Nathaniel S. Keplinger +3Model-Based Planning Offline RL
Vanderbilt University · William & Mary
Flow Actor-Critic for Offline Reinforcement Learning
Feb 20, 2026Jongseong Chae, Jongeui Park, Yongjae Shin +3Flow-Based Policy Learning Conservative RL
KAIST · UNIST
Learning Optimal and Sample-Efficient Decision Policies with Guarantees
Feb 20, 2026Daqian ShaoReinforcement Learning Sample-Efficient RL
Wolfson College University of Oxford
SMAC: Score-Matched Actor-Critics for Robust Offline-to-Online Transfer
Feb 19, 2026Nathan Samuel de Lara, Florian ShkurtiOffline RL Offline-To-Online RL
Department of Computer Science, University of Toronto, Toronto, Canada · Vector Institute
LexiSafe: Offline Safe Reinforcement Learning with Lexicographic Safety-Reward Hierarchy
Feb 19, 2026Hsin-Jung Yang, Zhanhong Jiang, Prajwal Koirala +3Reinforcement Learning Safe RL
Dept. of Mechanical Engineering Iowa State University Ames, USA · Translational AI Center Iowa State University Ames, USA · Sibley School of Mechanical and Aerospace Engineering Cornell University Ithaca, USA
OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Exploration
Feb 19, 2026Yiqin Yang, Hao Hu, Yihuan Mao +10Preference-Based RL Offline RL
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · Moonshot AI · Tsinghua University +3
Edge Learning via Federated Split Decision Transformers for Metaverse Resource Allocation
Feb 18, 2026Fatih Temiz, Shavbo Salehi, Melike Erol-KantarciWireless Resource Allocation Federated RL
School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, Canada
EpiFlow: Bridging Safety and Performance in Offline Reinforcement Learning via Epigraph-Guided Flow Matching
Feb 8, 2026Manan Tayal, Mumuksh Tayal, Ravi PrakashOffline RL
TAU Intelligence
The impact of temporal resampling on the safety of clinical reinforcement learning: A simulated trial
Feb 6, 2026Thomas Frost, Hrisheekesh Vaidya, Steve HarrisReinforcement Learning Offline RL
University College London, London, United Kingdom · University College London Hospitals NHS Foundation Trust, London, United Kingdom
Flow Matching for Offline Reinforcement Learning with Discrete Actions
Feb 5, 2026Fairoz Nower Khan, Nabuat Zaman Nahim, Ruiquan Huang +2Markov Chains Reinforcement Learning
Department of Computer Science, University of Kentucky · Department of Computing and Information Sciences, Rochester Institute of Technology
Beyond Success Rates: Trainability and Extractability for Offline GCRL
Feb 5, 2026Jan Malte Töpperwien, Aditya Mohan, Marius LindauerRL Benchmarks Offline RL
Institute of Artificial Intelligence (LUH|AI), Leibniz University Hannover · L3S Research Center
COBALT: Contextual Bandit Learning with Offline Trajectories for Multi-Turn Code Generation
Feb 3, 2026Ziru Chen, Dongdong Chen, Ruinan Jin +3Code Generation RL for Code Generation
The Ohio State University. · Microsoft.
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
Feb 3, 2026Jinwoo Choi, Sang-Hyun Lee, Seung-Woo SeoReinforcement Learning Hierarchical RL
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea
Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processes
Feb 2, 2026Mingyuan Xu, Zongqi Xia, Tianxi Cai +2Robust Markov Decision Processes Offline RL
Department of Statistics and Data Science, National University of Singapore · Department of Neurology, University of Pittsburgh · Department of Biostatistics, Harvard T.H. Chan School of Public Health +2
Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment
Jan 30, 2026Mathieu Petitbois, Rémy Portelas, Sylvain LamprierReinforcement Learning Offline RL
Ubisoft La Forge, Bordeaux, France · Université d’Angers, Angers, France
DeFlow: Decoupling Behavior-Prior Modeling and Value Maximization for Offline Policy Extraction
Jan 15, 2026Zhancun Mu, Chi ZhangOffline RL Policy Learning
Peking University, Institute of Artificial Intelligence, China, Beijing.
Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing
Dec 30, 2025Lars van der Laan, Nathan KallusReinforcement Learning KL-Regularized RL
Department of Statistics, University of Washington · Netflix · Cornell University
Infinitesimal Operator World Models: Finite-Sample Guarantees for Offline Reinforcement Learning
Nov 13, 2025Nicolas Hoischen, Petar Bevanda, Max Beier +3Offline RL Model-Based RL
TU Munich · ShanghaiTech University
The Three Regimes of Offline-to-Online Reinforcement Learning
Oct 1, 2025Lu Li, Tianwei Ni, Yihao Sun +1Offline RL RL Fine-Tuning
Mila – Québec AI Institute · Université de Montréal · CIFAR AI Chair
Hybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Making
Aug 22, 2025Yue Pei, Hongming Zhang, Chao Gao +7Reinforcement Learning Offline RL
School of Artificial Intelligence, Beihang University, Beijing, China · Department of Computing Science and Amii, University of Alberta, Edmonton, Canada · Edmonton Research Center, Huawei Canada, Edmonton, Canada +2
RAD: Retrieval High-quality Demonstrations to Enhance Decision-making
Jul 21, 2025Lu Guo, Yixiang Shan, Zhengbang Zhu +5Offline RL Generalization in RL
School of Artificial Intelligence, Jilin University, Changchun, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
Policy Improvement with Style-Specific Demonstrations
Jun 20, 2025Lingfeng Li, Yunlong Lu, Yongyi Wang +1Reinforcement Learning Game-Playing Agents
School of Computer Science, Peking University, Beijing 100871, China
Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
Jun 14, 2025Sara Rajaram, R. James Cotton, Fabian H. SinzOffline RL Robust RL
Institute for Computer Science and Campus Institute for Data Science, University of Göttingen, Göttingen, Germany · University Hospital Tübingen, Tübingen, Germany · International Max Planck Research School for Intelligent Systems, Tübingen, Germany +2
Fully Offline Reinforcement Learning
May 28, 2025Mattie Fellows, Clarisse Wibault, Uljad Berdica +3Bayesian RL Offline RL
Foerster Lab for AI Research (FLAIR), Department of Engineering Science, University of Oxford · Machine Learning Research Group, Department of Engineering Science, University of Oxford

Feb 27, 2026J/K move · Enter open · S save

MAGE: Multi-scale Autoregressive Generation for Offline Reinforcement Learning

Chenxing Lin, Xinhui Gao, Haipeng Zhang, Xinran Li+6

Fujian Key Laboratory of Urban Intelligent Sensing and Computing, School of Informatics, Xiamen University (XMU), China · Key Laboratory of Multimedia Trusted Perception and Efficient Computing, XMU, China · Hong Kong University of Science and Technology+2

Offline RL Transformer-Based RL Sparse-Reward RL

Open paper AskPDF arXiv

Generative models have gained significant traction in offline reinforcement learning (RL) due to their ability to model complex trajectory distributions. However, existing generation-based approaches still struggle with long-horizon tasks characterized by sparse rewards. Some hierarchical generation methods have been developed to mitigate this issue by decomposing the original problem into shorter-horizon subproblems using one policy and generating detailed actions with another. While effective, these methods often overlook the multi-scale temporal structure inherent in trajectories, resulting in suboptimal performance. To overcome these limitations, we propose MAGE, a Multi-scale Autoregressive GEneration-based offline RL method. MAGE incorporates a condition-guided multi-scale autoencoder to learn hierarchical trajectory representations, along with a multi-scale transformer that autoregressively generates trajectory representations from coarse to fine temporal scales. MAGE effectively captures temporal dependencies of trajectories at multiple resolutions. Additionally, a condition-guided decoder is employed to exert precise control over short-term behaviors. Extensive experiments on five offline RL benchmarks against fifteen baseline algorithms show that MAGE successfully integrates multi-scale trajectory modeling with conditional guidance, generating coherent and controllable trajectories in long-horizon sparse-reward settings.