RL Exploration

RL: Reinforcement Learning

Exploration studies how an agent can gather useful experience when rewards are sparse, the environment is uncertain, or its current policy visits only a narrow part of the state space. Common approaches use uncertainty bonuses, novelty and entropy rewards, model-based planning, goal or skill discovery, memory, and structured or language-guided exploration. These ideas are tested in settings from bandits and sparse-reward navigation to robotics, multi-agent tasks, and language-model reasoning and tool use. A central challenge is to seek information that improves future decisions without wasting experience on irrelevant novelty or destabilizing learning, while maintaining safety and useful behavior.

Latest papers

221 papers29 papers in the last four weeks, up 81% on the four weeks before. 0.2% of all new papers.

SPAARS: Safer RL Policy Alignment through Abstract Exploration and Refined Exploitation of Action Space
Mar 10, 2026Swaminathan S K, Aritra HazraReinforcement Learning Offline-To-Online RL
Department of Computer Science and Engineering, IIT Kharagpur
Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Models
Mar 6, 2026Canyu Chen, Yuguang Yang, Zhewen Tan +10Autonomous Driving VLMs for Autonomous Driving
National Superior College for Engineers, Beihang University · Institute for AI Industry Research, Tsinghua University · School of Electronic Information Engineering, Beihang University +5
Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning
Mar 4, 2026Lei Huang, Xiang Cheng, Chenxiao Zhao +6RL Exploration RL for Language Models
Harbin Institute of Technology · Xiaohongshu Inc.
RAPO: Expanding Exploration for LLM Agents via Retrieval-Augmented Policy Optimization
Mar 3, 2026Siwei Zhang, Yun Xiong, Xi Chen +4Agentic RL RL for Language Model Reasoning
Fudan University · Zhejiang University · UC Davis
Conformal Policy Control
Mar 2, 2026Drew Prinster, Clara Fannjiang, Ji Won Park +4Constrained RL RL Exploration
Prescient Design, Genentech, U.S.A. · Johns Hopkins University, Baltimore, U.S.A. · New York University, NYC, U.S.A. +1
Expanding LLM Agent Boundaries with Strategy-Guided Exploration
Mar 2, 2026Andrew Szot, Michael Kirchhof, Omar Attia +1RL Exploration LLM Agents
Apple
Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewards
Mar 2, 2026Faisal Mohamed, Catherine Ji, Benjamin Eysenbach +1RL Exploration Robotic RL
Mila-Quebec AI Institute · Université de Montréal · Princeton University
Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learning
Mar 2, 2026Naoki Shitanda, Motoki Omura, Tatsuya Harada +1Policy Gradient Methods RL Exploration
The University of Tokyo, Tokyo, Japan · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimization
Feb 26, 2026Zeyuan Liu, Jeonghye Kim, Xufang Luo +2RL for LLM Agents Reinforcement Learning
Microsoft Research · KAIST
$κ$ -Explorer: Minimax Active Model Estimation in MDPs
Feb 23, 2026Xihe Gu, Urbashi Mitra, Tara JavidiFrank-Wolfe Algorithms RL Exploration
Department of Electrical and Computer Engineering, University of California, San Diego · Department of Electrical and Computer Engineering, University of Southern California
Robust Exploration in Directed Controller Synthesis via Reinforcement Learning with Soft Mixture-of-Experts
Feb 22, 2026Toshihide Ubukata, Zhiyao Wang, Enhong Mu +2RL Exploration Generalization in RL
Waseda University, Tokyo, 169-8050, Japan. · Osaka University, Japan. · Southwest University, China. +1
Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning
Feb 22, 2026Zhuoxu Huang, Mengxi Jia, Hao Sun +2RL Exploration RL for VLMs
Aberystwyth University · Institute of Artificial Intelligence (TeleAl), China Telecom · Tsinghua University
Task-Aware Exploration via a Predictive Bisimulation Metric
Feb 21, 2026Dayang Liang, Ruihan Liu, Lipeng Wan +2RL Exploration Visual RL
Department of Automation, Xiamen University, Xiamen, China · Department of Computer Science, Beijing Normal-Hong Kong Baptist University, Zhuhai, China · School of Artificial Intelligence, Xian Jiaotong University, Xian, China +1
Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
Feb 20, 2026Yongjae Shin, Jongseong Chae, Jongeui Park +1Offline-To-Online RL Sample-Efficient RL
KAIST
Avoid What You Know: Divergent Trajectory Balance for GFlowNets
Feb 19, 2026Pedro Dall'Antonia, Tiago da Silva, Daniel Csillag +2Generative Flow Networks Adaptive Sampling
School of Applied Mathmatics, Getulio Vargas Foundation · MBZUAI
Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning
Feb 19, 2026Yonghyeon Jo, Sunwoo Lee, Seungyul HanCooperative MARL Multi-Agent Reinforcement Learning
Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology (UNIST), Ulsan, South Korea 44919
OPRIDE: Offline Preference-based Reinforcement Learning via In-Dataset Exploration
Feb 19, 2026Yiqin Yang, Hao Hu, Yihuan Mao +10Preference-Based RL Offline RL
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences · Moonshot AI · Tsinghua University +3
VAM: Verbalized Action Masking for Controllable Exploration in RL Post-Training -- A Chess Case Study
Feb 18, 2026Zhicheng Zhang, Ziyan Wang, Yali Du +1RL for Language Models RL Exploration
Carnegie Mellon University · King’s College London
Improved Bounds for Reward-Agnostic and Reward-Free Exploration
Feb 18, 2026Oran Ridel, Alon CohenRL Exploration Reward-Free RL
Department of Engineering, Tel Aviv University, Tel Aviv, Israel. · Google Research, Tel Aviv, Israel.
Regret and Sample Complexity of Online Q-Learning via Concentration of Stochastic Approximation with Time-Inhomogeneous Markov Chains
Feb 18, 2026Rahul Singh, Siddharth Chandak, Eric Moulines +2Regret Minimization in RL Q-Learning
MBZUAI, UAE · Stanford University, USA · EPITA, France +1
Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learning
Feb 17, 2026Oswin So, Eric Yang Yu, Songyuan Zhang +3Safe RL Reachability Analysis
Department of Aeronautics and Astronautics, MIT · MIT Lincoln Laboratory
When Remembering and Planning are Worth it: Navigating under Change
Feb 17, 2026Omid Madani, J. Brian Burns, Reza Eghbali +1Robot Navigation Memory-Augmented Agents
The authors were partially affiliated with Brown University.
Partial GFlowNet: Accelerating Convergence in Large State Spaces via Strategic Partitioning
Feb 12, 2026Xuan Yu, Xu Wang, Rui Zhu +2Reinforcement Learning Generative Flow Networks
University of Science and Technology of China (USTC), Hefei, China · Suzhou Institute for Advanced Research, USTC, Suzhou, China · State Key Laboratory of Precision and Intelligent Chemistry, USTC, Hefei, China
Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation
Feb 5, 2026Zhiqi Yu, Zhangquan Chen, Mengting Liu +2RL for Language Model Reasoning Group Relative Policy Optimization
University of Hong Kong · Tsinghua University · Sun Yat-sen University
Thinking Seeds: Restarting On-Policy RL from the Model's Own Past Reasoning States
Jan 29, 2026Lei Yang, Wei Bi, Chenxi Sun +2RL for Language Model Reasoning RL Exploration
TJUNLP Lab, College of Intelligence and Computing, Tianjin University, Tianjin, China · Kuaishou Technology
Tracking Drift: Variation-Aware Entropy Scheduling for Non-Stationary Reinforcement Learning
Jan 27, 2026Tongxi Wang, Zhuoyang Xia, Xinran Chen +1Non-Stationary RL Sequential Decision Making
School of Future Technology, Southeast University, Nanjing, China · School of Automation, Southeast University, Nanjing, China
R $^2$ PO: Decoupling Rollout and Inference Policies for LLM Reasoning
Jan 17, 2026Jingchu Wang, Bingbing Xu, Yige Yuan +4RL for Language Model Reasoning Policy Optimization
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS · University of Chinese Academy of Sciences · National University of Singapore
Experience-Evolving Multi-Turn Tool-Use Agent with Hybrid Episodic-Procedural Memory
Dec 8, 2025Sijia Li, Yuchen Huang, Zifan Liu +6RL Exploration Graph-Based Agent Memory
Hong Kong University of Science and Technology · Microsoft Research Asia · Work done during internship at Microsoft.
Auto-exploration for online reinforcement learning
Dec 6, 2025Caleb Ju, Guanghui LanPolicy Optimization RL Exploration
Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective
Oct 16, 2025Shiqi Chen, Tongyao Zhu, Zian Wang +8RL Exploration Policy Learning
City University of Hong Kong · National University of Singapore · The Hong Kong Polytechnic University +5

Mar 10, 2026J/K move · Enter open · S save

SPAARS: Safer RL Policy Alignment through Abstract Exploration and Refined Exploitation of Action Space

Swaminathan S K, Aritra Hazra

Department of Computer Science and Engineering, IIT Kharagpur

Reinforcement Learning Offline-To-Online RL RL Exploration Safe RL Sample-Efficient RL Curriculum RL

Open paper AskPDF arXiv

Offline-to-online reinforcement learning (RL) offers a promising paradigm for robotics by pre-training policies on safe, offline demonstrations and fine-tuning them via online interaction. However, a fundamental challenge remains: how to safely explore online without deviating from the behavioral support of the offline data? While recent methods leverage conditional variational autoencoders (CVAEs) to bound exploration within a latent space, they inherently suffer from an exploitation gap -- a performance ceiling imposed by the decoder's reconstruction loss. We introduce SPAARS, a curriculum learning framework that initially constrains exploration to the low-dimensional latent manifold for sample-efficient, safe behavioral improvement, then seamlessly transfers control to the raw action space, bypassing the decoder bottleneck. SPAARS has two instantiations: the CVAE-based variant requires only unordered (s,a) pairs and no trajectory segmentation; SPAARS-SUPE pairs SPAARS with OPAL temporal skill pretraining for stronger exploration structure at the cost of requiring trajectory chunks. We prove an upper bound on the exploitation gap using the Performance Difference Lemma, establish that latent-space policy gradients achieve provable variance reduction over raw-space exploration, and show that concurrent behavioral cloning during the latent phase directly controls curriculum transition stability. Empirically, SPAARS-SUPE achieves 0.825 normalized return on kitchen-mixed-v0 versus 0.75 for SUPE, with 5x better sample efficiency; standalone SPAARS achieves 92.7 and 102.9 normalized return on hopper-medium-v2 and walker2d-medium-v2 respectively, surpassing IQL baselines of 66.3 and 78.3 respectively, confirming the utility of the unordered-pair CVAE instantiation.