Offline-To-Online RL

RL: Reinforcement Learning

Follow this topic Ask about this topic

Momentum

5 papers in the last four weeks, up 67% on the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 40

All topics

RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking
May 11, 2026Andrew Choi, Wei XuQ-Learning Robotic RL
Horizon Robotics
SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Data
May 7, 2026Carlo Romeo, Girolamo Macaluso, Alessandro Sestini +1Off-Policy Evaluation Offline-To-Online RL
1Media Integration and Communication Center – University of Florence · 2SEED – Electronic Arts
Adaptive Q-Chunking for Offline-to-Online Reinforcement Learning
May 7, 2026Nandiraju Gireesh, Yuanliang Ju, He WangQ-Learning Offline RL
Peking University · Galbot · University of Toronto
When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning
May 6, 2026Lakshita Dodeja, Ondrej Biza, Shivam Vats +5Q-Learning Robotic RL
1Robotics and AI Institute · 2Brown University · 3Northeastern University
Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning
May 6, 2026Alper Kamil Bozkurt, Xiaoan Xu, Shangtong Zhang +2Sequential Decision Making RL Fine-Tuning
Virginia Commonwealth University, Richmond, VA · Duke University, Durham, NC · University of Virginia, Charlottesville, VA
Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policies
May 1, 2026Yi Wang, Xinchen Li, Pengwei Xie +13Robot Policy Learning Robotic Manipulation
Shanghai Innovation Institute. · AGIBOT Finch. · Columbia University.
OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learning
Apr 20, 2026Xinyu Ma, Mingzhou Xu, Xuebo Liu +4Reward Shaping RL for Language Model Reasoning
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China · Hithink RoyalFlush Information Network, Hangzhou, China · Computer and Information Science, University of Macau, Macau, China
Value Mirror Descent for Reinforcement Learning
Apr 7, 2026Zhichao Jia, Guanghui LanMarkov Decision Processes Reinforcement Learning
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA, 30332.
From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning
Mar 10, 2026Zhanyi Sun, Shuran SongRL for Robotics Robot Policy Learning
1Stanford University.
The Three Regimes of Offline-to-Online Reinforcement Learning
Oct 1, 2025Lu Li, Tianwei Ni, Yihao Sun +1Offline RL RL Fine-Tuning
Mila – Québec AI Institute · Université de Montréal · CIFAR AI Chair

May 11, 2026, cs.AIJ/K move · Enter open · S save

RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking

Andrew Choi, Wei Xu

Horizon Robotics

Q-Learning Robotic RL Sparse-Reward RL Offline-To-Online RL Preference-Based RL

Open paper AskPDF arXiv

Offline-to-online reinforcement learning (RL) improves sample efficiency by leveraging pre-collected datasets prior to online interaction. A key challenge, however, is learning an accurate critic in large state--action spaces with limited dataset coverage. To mitigate harmful updates from value overestimation, prior methods impose pessimism by down-weighting out-of-distribution (OOD) actions relative to dataset actions. While effective, this essentially acts as a behavior cloning anchor and can hinder downstream online policy improvement when dataset actions are suboptimal. We propose RankQ, an offline-to-online Q-learning objective that augments temporal-difference learning with a self-supervised multi-term ranking loss to enforce structured action ordering. By learning relative action preferences rather than uniformly penalizing unseen actions, RankQ shapes the Q-function such that action gradients are directed toward higher-quality behaviors. Across sparse-reward D4RL benchmarks, RankQ achieves competitive overall performance against seven baselines. In vision-based robot learning, RankQ enables effective offline-to-online fine-tuning of a pretrained vision-language-action (VLA) model in a low-data regime, achieving an average simulation success rate 38.2 percentage points higher than the next best method. In a high-data setting, RankQ improves simulation performance by 13.7 percentage points over the next best method and demonstrates strong sim-to-real transfer, increasing real-world cube stacking success from 43.1% to 88.9% relative to the VLA's initial performance.

Momentum

Related topics

Latest papers 40