RL Exploration

RL: Reinforcement Learning

Exploration studies how an agent can gather useful experience when rewards are sparse, the environment is uncertain, or its current policy visits only a narrow part of the state space. Common approaches use uncertainty bonuses, novelty and entropy rewards, model-based planning, goal or skill discovery, memory, and structured or language-guided exploration. These ideas are tested in settings from bandits and sparse-reward navigation to robotics, multi-agent tasks, and language-model reasoning and tool use. A central challenge is to seek information that improves future decisions without wasting experience on irrelevant novelty or destabilizing learning, while maintaining safety and useful behavior.

Latest papers

221 papers29 papers in the last four weeks, up 81% on the four weeks before. 0.2% of all new papers.

Representation-Based Exploration for Language Models: From Test-Time to Post-Training
Oct 13, 2025Jens Tuyls, Dylan J. Foster, Akshay Krishnamurthy +1RL for Language Model Reasoning Test-Time Scaling
Princeton University · Microsoft Research NYC
A Primer on SO(3) Action Representations in Deep Reinforcement Learning
Oct 13, 2025Martin Schuck, Sherif Samy, Angela P. SchoelligRobotic RL RL Exploration
Technical University of Munich · University of Girona
Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
Oct 5, 2025Zishang Jiang, Jinyi Han, Tingyun Li +7RL for Language Model Reasoning LLM-Guided RL
School of Data Science, Fudan University · Shanghai Institute of Artificial Intelligence for Education, East China Normal University · College of Computer Science and Artificial Intelligence, Fudan University +1
Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Exploration
Sep 12, 2025Chirayu Nimonkar, Shlok Shah, Catherine Ji +1Multi-Agent Coordination RL Exploration
Princeton University
Ensemble-MIX: Enhancing Sample-Efficiency in Multi-Agent RL via Selective Ensemble-Based Exploration
Jun 3, 2025Tom Danino, Nahum ShimkinMulti-Agent Reinforcement Learning Sample-Efficient RL
The Andrew and Erna Viterbi Faculty of Electrical & Computer Engineering, Technion- Israel Institute of Technology
A Graph-Based Reinforcement Learning Approach with Frontier Potential Based Reward for Safe Cluttered Environment Exploration
Apr 16, 2025Gabriele Calzolari, Vidya Sumathy, Christoforos Kanellakis +1Reward Shaping Mobile Robotics
Luleå University of Technology · Robotics and AI Group, Department of Computer Science, Electrical and Space Engineering, Luleåa University of Technology, Sweden
An Real-Sim-Real (RSR) Loop Framework for Generalizable Robotic Policy Transfer
Mar 13, 2025Yuxuan Xu, Shiyu Wang, Jinhao Huang +6Robotic Data Collection Sim-to-Real Transfer
Institute for AI Industry Research (AIR), Tsinghua University · Beijing Jiaotong University · The Hong Kong University of Science and Technology (Guangzhou)
ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy
Dec 5, 2024Hongming Li, Zhao Yang, Xiaoxuan Liang +2Reward-Free RL Reinforcement Learning
Zhejiang Lab, Huangzhou, China · Vrije Universiteit Amsterdam, Amsterdam, the Netherlands · Computational NeuroEngineering Laboratory at the University of Florida, Gainesville, USA
Training on Irrelevant States Implies Data Augmentation: Generalization in Contextual MDPs
Oct 4, 2024Max Weltevrede, Caroline Horsch, Matthijs T. J. Spaan +1Reinforcement Learning RL Exploration
Delft University of Technology, Delft, The Netherlands
Adaptive teachers for amortized samplers
Oct 2, 2024Minsu Kim, Sanghyeok Choi, Taeyoung Yun +7Amortized Inference RL Exploration
Mila, KAIST · KAIST · Recursion +1
Active Exploration via Autoregressive Generation of Missing Data
May 29, 2024Tiffany Tianhui Cai, Hongseok Namkoong, Daniel Russo +1RL Exploration Autoregressive Generation
Department of Statistics Columbia University New York, NY 10025, USA · Decision, Risk, and Operations Division Columbia Business School New York, NY 10025, USA · Department of Mathematics Imperial College London London SW7 2AZ, United Kingdom

Oct 13, 2025J/K move · Enter open · S save

Representation-Based Exploration for Language Models: From Test-Time to Post-Training

Jens Tuyls, Dylan J. Foster, Akshay Krishnamurthy, Jordan T. Ash

Princeton University · Microsoft Research NYC

RL for Language Model Reasoning Test-Time Scaling RL Exploration RL Post-Training

Open paper AskPDF arXiv

Reinforcement learning (RL) promises to expand the capabilities of language models, but it is unclear if current RL techniques promote the discovery of novel behaviors, or simply sharpen those already present in the base model. In this paper, we investigate the value of deliberate exploration -- explicitly incentivizing the model to discover novel and diverse behaviors -- and aim to understand how the knowledge in pre-trained models can guide this search. Our main finding is that exploration with a simple, principled, representation-based bonus derived from the pre-trained language model's hidden states significantly improves diversity and pass@k rates -- both for post-training, and in a novel inference-time scaling setting we introduce. For inference-time, exploration with representation-based diversity improves efficiency, consistently improving pass@k rates across a variety of models and reasoning tasks. For example, for Qwen-2.5-14b-Instruct we obtain over 50% improvement in verifier efficiency on almost all tasks. For post-training, we show that integrating this exploration strategy into an RL pipeline improves reasoning performance over that of the initial model and over standard RL post-training. For example, on AIME 2024, our post-trained Qwen-2.5-7b-Instruct's pass@80 matches the pass@256 of GRPO on the same model, demonstrating a 3x improvement in test-time sample efficiency. Overall, our findings suggest that deliberate exploration -- with the right notion of diversity -- is a practical path toward discovery of new behaviors beyond sharpening.