Offline RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

29 papers in the last four weeks, up 222% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 189

All topics

Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing
Dec 30, 2025Lars van der Laan, Nathan KallusReinforcement Learning KL-Regularized RL
Department of Statistics, University of Washington · Netflix · Cornell University
The Three Regimes of Offline-to-Online Reinforcement Learning
Oct 1, 2025Lu Li, Tianwei Ni, Yihao Sun +1Offline RL RL Fine-Tuning
Mila – Québec AI Institute · Université de Montréal · CIFAR AI Chair
Hybrid Sequence Modeling and Reinforced Verification for Controllable Target-Conditioned Decision Making
Aug 22, 2025Yue Pei, Hongming Zhang, Chao Gao +7Reinforcement Learning Offline RL
School of Artificial Intelligence, Beihang University, Beijing, China · Department of Computing Science and Amii, University of Alberta, Edmonton, Canada · Edmonton Research Center, Huawei Canada, Edmonton, Canada +2
RAD: Retrieval High-quality Demonstrations to Enhance Decision-making
Jul 21, 2025Lu Guo, Yixiang Shan, Zhengbang Zhu +5Offline RL Generalization in RL
School of Artificial Intelligence, Jilin University, Changchun, China · School of Computer Science, Shanghai Jiao Tong University, Shanghai, China
Policy Improvement with Style-Specific Demonstrations
Jun 20, 2025Lingfeng Li, Yunlong Lu, Yongyi Wang +1Reinforcement Learning Game-Playing Agents
School of Computer Science, Peking University, Beijing 100871, China
Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
Jun 14, 2025Sara Rajaram, R. James Cotton, Fabian H. SinzOffline RL Robust RL
Institute for Computer Science and Campus Institute for Data Science, University of Göttingen, Göttingen, Germany · University Hospital Tübingen, Tübingen, Germany · International Max Planck Research School for Intelligent Systems, Tübingen, Germany +2
Fully Offline Reinforcement Learning
May 28, 2025Mattie Fellows, Clarisse Wibault, Uljad Berdica +3Bayesian RL Offline RL
Foerster Lab for AI Research (FLAIR), Department of Engineering Science, University of Oxford · Machine Learning Research Group, Department of Engineering Science, University of Oxford
Decision-Centered Abstractions via Orthogonal Estimation of Difference-of-Q Functions
Jun 12, 2024Defu Cao, Angela ZhouValue Function Estimation State Abstraction
University of Southern California
Corruption Robust Offline Reinforcement Learning with Human Feedback
Feb 9, 2024Debmalya Mandal, Andi Nika, Parameswaran Kamalaruban +2Offline RL Corruption Robustness
Dept. of Computer Science University of Warwick, UK · Max-Planck Institute for Software Systems, Germany · Independent Researcher London, UK

Dec 30, 2025, stat.MLJ/K move · Enter open · S save

Soft Fitted Q-Iteration without Bellman Completeness: Occupancy Reweighting and Temperature Annealing

Lars van der Laan, Nathan Kallus

Department of Statistics, University of Washington · Netflix · Cornell University

Reinforcement Learning KL-Regularized RL Q-Learning Offline RL Maximum Entropy RL Policy Learning+1

Open paper AskPDF arXiv

Fitted

Q

-iteration (FQI) is a standard regression-based method for optimal control in offline reinforcement learning, but its stability under function approximation often relies on Bellman completeness, which requires Bellman images of the fitted class to remain in the class. We study Kullback--Leibler (KL)-regularized, or soft, FQI relative to a fixed reference policy without this assumption. Our key insight is that soft control locally inherits the contraction of policy evaluation in a discounted-occupancy norm. At the soft-optimal fixed point, the linearization of the soft Bellman operator is exactly the Bellman operator for the soft-optimal policy, which contracts in its discounted-occupancy norm; projection in the same norm preserves this contraction. Standard soft FQI instead projects under the offline state-action distribution and need not preserve this property. Motivated by this observation, we propose \emph{occupancy-reweighted soft FQI}, which retains standard Bellman targets and least-squares updates while reweighting regressions by discounted-occupancy ratios induced by the current soft policy. Under

Q

-function realizability and local regularity, we establish local contraction and finite-sample convergence with estimated ratios, without Bellman completeness. We then use temperature annealing to convert the local result into global convergence from arbitrary initialization: sufficiently high temperature provides a globally contractive starting regime, while gradual cooling connects successive local contraction regions to any prescribed positive target temperature. Under an action-gap margin condition, switching at a fixed positive temperature to hard FQI with refreshed occupancy weights also yields population and finite-sample convergence to the unregularized optimum.

Momentum

Related topics

Latest papers 189