Sample-Efficient RL

RL: Reinforcement Learning

Ask about this topic

Overview

Sample-efficient reinforcement learning aims to reduce the experience needed to learn useful policies, especially when interaction is slow, costly, or risky. Common approaches improve exploration with optimism or uncertainty estimates, reuse data through replay and demonstrations, learn models for planning, or exploit structure such as hierarchy, symmetry, and known dynamics. These methods are studied in settings from tabular and robust decision processes to MuJoCo and Atari control, robotics, and language-model training with verifiable rewards. The field balances empirical gains and PAC or regret guarantees against challenges such as stale replay, model error, limited data coverage, and differences between sample counts and wall-clock cost.

Latest papers

CardsList

253 papers30 papers in the last four weeks, up 150% on the four weeks before. 0.2% of all new papers.

Mar 2, 2026

Efficient RLVR Training via Weighted Mutual Information Data Selection

Reinforcement learning (RL) plays a central role in improving the reasoning and alignment of large language models, yet its efficiency critically depends on how training data are selected. Existing online selection strategies predominantly rely on difficulty-based heuristics, favouring datapoints with intermediate success rates, implicitly equating difficulty with informativeness and neglecting epistemic uncertainty arising from limited evidence. We introduce InSight, an INformation-guided data SamplInG metHod for RL Training, grounded in a weighted mutual information objective. By modeling data outcomes with Bayesian latent success rates, we show that expected uncertainty reduction decomposes into complementary difficulty- and evidence-dependent components, revealing a fundamental limitation of difficulty-only selection. Leveraging this observation, InSight constructs a stable acquisition score based on the mean belief of datapoints' success rather than noisy sampled outcomes, and naturally extends to multi-rollout settings common in reinforcement learning with verifiable rewards (RLVR). Extensive experiments demonstrate that InSight consistently achieves state-of-the-art performance and improves training efficiency, including a +1.41 average gain on Planning & Mathmatics benchmarks, +1.01 improvement on general reasoning, and up to ~2.2x acceleration, with negligible additional computational overhead.

Xinyu Zhou, Boyu Zhu, Haotian Zhang +2

HKUST(GZ) · UCL · Kuaishou Technology +1

Reinforcement LearningRL for Language Model ReasoningSample-Efficient RLReinforcement Learning with Verifiable Rewards
AskPDF
Mar 2, 2026

SEAR: Sample Efficient Action Chunking Reinforcement Learning

Action chunking improves exploration and accelerates value propagation in long-horizon reinforcement learning, but naively applying off-policy methods to the temporally extended action space at reduced decision frequency offsets these gains, leading to poor sample efficiency. Existing online action chunking methods address these issues through computationally expensive critic-only approaches or by relying on offline data. We introduce SEAR, a sample-efficient off-policy algorithm that enables online reinforcement learning with action chunks by addressing both challenges. To handle high-dimensional action sequences, SEAR employs a causal transformer critic trained with multi-horizon targets that provide a training signal for every prefix of an action chunk, effectively increasing the useful gradients per sample. To restore decision frequency, SEAR operates with a receding horizon and random replanning, ensuring uniform state coverage while combining the fast value propagation of large chunks with the reactivity of small ones. SEAR outperforms state-of-the-art online reinforcement learning methods including SimbaV2 on challenging Metaworld manipulation tasks. Beyond online RL, applying SEAR to the offline-to-online method QC improves its performance on the demanding OGBench cube-triple tasks.

C. F. Maximilian Nagy, Onur Celik, Emiliyan Gospodinov +4

Autonomous Learning Robots, Karlsruhe Institute of Technology · FZI Forschungszentrum Informatik, Karlsruhe

Reinforcement LearningSample-Efficient RLOff-Policy RL
AskPDF
Mar 2, 2026

Rethinking Policy Diversity in Ensemble Policy Gradient in Large-Scale Reinforcement Learning

Scaling reinforcement learning to tens of thousands of parallel environments requires overcoming the limited exploration capacity of a single policy. Ensemble-based policy gradient methods, which employ multiple policies to collect diverse samples, have recently been proposed to promote exploration. However, merely broadening the exploration space does not always enhance learning capability, since excessive exploration can reduce exploration quality or compromise training stability. In this work, we theoretically analyze the impact of inter-policy diversity on learning efficiency in policy ensembles, and propose Coupled Policy Optimization which regulates diversity through KL constraints between policies. The proposed method enables effective exploration and outperforms strong baselines such as SAPG, PBT, and PPO across multiple tasks, including challenging dexterous manipulation, in terms of both sample efficiency and final performance. Furthermore, analysis of policy diversity and effective sample size during training reveals that follower policies naturally distribute around the leader, demonstrating the emergence of structured and efficient exploratory behavior. Our results indicate that diverse exploration under appropriate regulation is key to achieving stable and sample-efficient learning in ensemble policy gradient methods. Project page at https://naoki04.github.io/paper-cpo/ .

Naoki Shitanda, Motoki Omura, Tatsuya Harada +1

The University of Tokyo, Tokyo, Japan · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan

Policy Gradient MethodsRL ExplorationSample-Efficient RLKL-Regularized RL
AskPDF
Mar 1, 2026

PAC Guarantees for Reinforcement Learning: Sample Complexity, Coverage, and Structure

When data is scarce or mistakes are costly, average-case metrics fall short. What a practitioner needs is a guarantee: with probability at least 1−δ1-δ, the learned policy is ε\varepsilon-close to optimal after NN episodes. This is the PAC promise, and between 2018 and 2025 the RL theory community made striking progress on when such promises can be kept. We survey that progress. Our organizing tool is the Coverage-Structure-Objective (CSO) framework, proposed here, which decomposes nearly every PAC sample complexity result into three factors: coverage (how data were obtained), structure (intrinsic MDP or function-class complexity), and objective (what the learner must deliver). CSO is not a theorem but an interpretive template that identifies bottlenecks and makes cross-setting comparison immediate. The technical core covers tight tabular baselines and the uniform-PAC bridge to regret; structural complexity measures (Bellman rank, witness rank, Bellman-Eluder dimension) governing learnability with function approximation; results for linear, kernel/NTK, and low-rank models; reward-free exploration as upfront coverage investment; and pessimistic offline RL where inherited coverage is the binding constraint. We provide practitioner tools: rate lookup tables indexed by CSO coordinates, Bellman residual diagnostics, coverage estimation with deployment gates, and per-episode policy certificates. A final section catalogs open problems, separating near-term targets from frontier questions where coverage, structure, and computation tangle in ways current theory cannot resolve.

Joshua Steier

Function Approximation in RLReinforcement LearningSample-Efficient RL
AskPDF
Feb 27, 2026

Actor-Critic Pretraining for Proximal Policy Optimization

Reinforcement learning (RL) actor-critic algorithms enable autonomous learning but often require a large number of environment interactions, which limits their applicability in robotics. Leveraging expert data can reduce the number of required environment interactions. A common approach is actor pretraining, where the actor network is initialized via behavioral cloning on expert demonstrations and subsequently fine-tuned with RL. In contrast, the initialization of the critic network has received little attention, despite its central role in policy optimization. This paper proposes a pretraining approach for actor-critic algorithms like Proximal Policy Optimization (PPO) that uses expert demonstrations to initialize both networks. The actor is pretrained via behavioral cloning, while the critic is pretrained using returns obtained from rollouts of the pretrained policy. The approach is evaluated on 15 simulated robotic manipulation and locomotion tasks. Experimental results show that actor-critic pretraining improves sample efficiency by 86.1% on average compared to no pretraining and by 30.9% to actor-only pretraining.

Andreas Kernbach, Amr Elsheikh, Nicolas Grupp +2

Fraunhofer Institute for Manufacturing Engineering and Automation IPA, Stuttgart, Germany · Institute of Industrial Manufacturing and Management IFF, University of Stuttgart, Germany · Institute for Control Engineering of Machine Tools and Manufacturing Units ISW, University of Stuttgart, Germany +1

Robotic RLSample-Efficient RLActor-Critic MethodsProximal Policy Optimization
AskPDF
Feb 24, 2026

Regret-Guided Search Control for Efficient Learning in AlphaZero

Reinforcement learning (RL) agents achieve remarkable performance but remain far less learning-efficient than humans. While RL agents require extensive self-play games to extract useful signals, humans often need only a few games, improving rapidly by repeatedly revisiting states where mistakes occurred. This idea, known as search control, aims to restart from valuable states rather than always from the initial state. In AlphaZero, prior work Go-Exploit applies this idea by sampling past states from self-play or search trees, but it treats all states equally, regardless of their learning potential. We propose Regret-Guided Search Control (RGSC), which extends AlphaZero with a regret network that learns to identify high-regret states, where the agent's evaluation diverges most from the actual outcome. These states are collected from both self-play trajectories and MCTS nodes, stored in a prioritized regret buffer, and reused as new starting positions. Across 9x9 Go, 10x10 Othello, and 11x11 Hex, RGSC outperforms AlphaZero and Go-Exploit by an average of 77 and 89 Elo, respectively. When training on a well-trained 9x9 Go model, RGSC further improves the win rate against KataGo from 69.3% to 78.2%, while both baselines show no improvement. These results demonstrate that RGSC provides an effective mechanism for search control, improving both efficiency and robustness of AlphaZero training. Our code is available at https://rlg.iis.sinica.edu.tw/papers/rgsc.

Yun-Jui Tsai, Wei-Yu Chen, Yan-Ru Ju +2

Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan · Institute of Information Science, Academia Sinica, Taiwan · College of Computing, Georgia Institute of Technology

Reinforcement LearningGame-Playing AgentsSample-Efficient RLSelf-Play RL
AskPDF
Feb 23, 2026

CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimization

Trajectory Optimization (TO) and Reinforcement Learning (RL) offer complementary strengths for solving optimal control problems. TO efficiently computes locally optimal solutions but can struggle with non-convexity, while RL is more robust to non-convexity at the cost of significantly higher computational demands. CACTO (Continuous Actor-Critic with Trajectory Optimization) was introduced to combine these advantages by learning a warm-start policy that guides the TO solver towards low-cost trajectories. However, scalability remains a key limitation, as increasing system complexity significantly raises the computational cost of TO. This work introduces CACTO-BIC to address these challenges. CACTO-BIC improves data efficiency by biasing initial-state sampling leveraging a property of the value function associated with locally optimal policies; moreover, it reduces computation time by exploiting GPU acceleration. Empirical evaluations show improved sample efficiency and faster computation compared to CACTO. Comparisons with PPO demonstrate that our approach can achieve similar solutions in less time. Finally, experiments on the AlienGO quadruped robot demonstrate that CACTO-BIC can scale to high-dimensional systems and is suitable for real-time applications.

Elisa Alboni, Pietro Noah Crestaz, Elias Fontanari +1

Dept. of Industrial Engineering, University of Trento, Italy · LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France

Actor-Critic MethodsTrajectory OptimizationRobotic RLSample-Efficient RL
AskPDF
Feb 21, 2026

VariBASed: Variational Bayes-Adaptive Sequential Monte-Carlo Planning for Deep Reinforcement Learning

Optimally trading-off exploration and exploitation is the holy grail of reinforcement learning as it promises maximal data-efficiency for solving any task. Bayes-optimal agents achieve this, but obtaining the belief-state and performing planning are both typically intractable. Although deep learning methods can greatly help in scaling this computation, existing methods are still costly to train. To accelerate this, this paper proposes a variational framework for learning and planning in Bayes-adaptive Markov decision processes that coalesces variational belief learning, sequential Monte-Carlo planning, and meta-reinforcement learning. In a single-GPU setup, our new method VariBASeD exhibits favorable scaling to larger planning budgets, improving sample- and runtime-efficiency over prior methods.

Joery A. de Vries, Jinke He, Yaniv Oren +3

Trent AI Ltd, London, United Kingdom · EEMCS, Delft University of Technology, Delft, the Netherlands

Bayesian RLSample-Efficient RLVariational Inference
AskPDF
Feb 20, 2026

Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning

Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.

Yongjae Shin, Jongseong Chae, Jongeui Park +1

KAIST

Offline-To-Online RLSample-Efficient RLRL ExplorationFlow Matching
AskPDF
Feb 20, 2026

Deep Reinforcement Learning for Optimizing Energy Consumption in Smart Grid Systems

The energy management problem in the context of smart grids is inherently complex due to the interdependencies among diverse system components. Although Reinforcement Learning (RL) has been proposed for solving Optimal Power Flow (OPF) problems, the requirement for iterative interaction with an environment often necessitates computationally expensive simulators, leading to significant sample inefficiency. In this study, these challenges are addressed through the use of Physics-Informed Neural Networks (PINNs), which can replace conventional and costly smart grid simulators. The RL policy learning process is enhanced so that convergence can be achieved in a fraction of the time required by the original environment. The PINN-based surrogate is compared with other benchmark data-driven surrogate models. By incorporating knowledge of the underlying physical laws, the results show that the PINN surrogate is the only approach considered in this context that can obtain a strong RL policy even without access to samples from the true simulator. The results demonstrate that using PINN surrogates can accelerate training by 50% compared to RL training without a surrogate. This approach enables the rapid generation of performance scores similar to those produced by the original simulator.

Abeer Alsheikhi, Amirfarhad Farhadi, Azadeh Zamanifar

School of Computer Engineering, Iran university of Science and Technology, Tehran, Iran · Department of Computer Engineering, SR.C, Islamic Azad university, Tehran, Iran

Sample-Efficient RLPower SystemsPhysics-Informed RLOptimal Power FlowNeural Surrogate Modeling
AskPDF
Feb 20, 2026

Learning Optimal and Sample-Efficient Decision Policies with Guarantees

The paradigm of decision-making has been revolutionised by reinforcement learning and deep learning. Although this has led to significant progress in domains such as robotics, healthcare, and finance, the use of RL in practice is challenging, particularly when learning decision policies in high-stakes applications that may require guarantees. Traditional RL algorithms rely on a large number of online interactions with the environment, which is problematic in scenarios where online interactions are costly, dangerous, or infeasible. However, learning from offline datasets is hindered by the presence of hidden confounders. Such confounders can cause spurious correlations in the dataset and can mislead the agent into taking suboptimal or adversarial actions. Firstly, we address the problem of learning from offline datasets in the presence of hidden confounders. We work with instrumental variables (IVs) to identify the causal effect, which is an instance of a conditional moment restrictions (CMR) problem. Inspired by double/debiased machine learning, we derive a sample-efficient algorithm for solving CMR problems with convergence and optimality guarantees, which outperforms state-of-the-art algorithms. Secondly, we relax the conditions on the hidden confounders in the setting of (offline) imitation learning, and adapt our CMR estimator to derive an algorithm that can learn effective imitator policies with convergence rate guarantees. Finally, we consider the problem of learning high-level objectives expressed in linear temporal logic (LTL) and develop a provably optimal learning algorithm that improves sample efficiency over existing methods. Through evaluation on reinforcement learning benchmarks and synthetic and semi-synthetic datasets, we demonstrate the usefulness of the methods developed in this thesis in real-world decision making.

Daqian Shao

Wolfson College University of Oxford

Reinforcement LearningSample-Efficient RLOffline RLCausal RL
AskPDF
Feb 20, 2026

Memory-Based Advantage Shaping for LLM-Guided Reinforcement Learning

In environments with sparse or delayed rewards, reinforcement learning (RL) incurs high sample complexity due to the large number of interactions needed for learning. This limitation has motivated the use of large language models (LLMs) for subgoal discovery and trajectory guidance. While LLMs can support exploration, frequent reliance on LLM calls raises concerns about scalability and reliability. We address these challenges by constructing a memory graph that encodes subgoals and trajectories from both LLM guidance and the agent's own successful rollouts. From this graph, we derive a utility function that evaluates how closely the agent's trajectories align with prior successful strategies. This utility shapes the advantage function, providing the critic with additional guidance without altering the reward. Our method relies primarily on offline input and only occasional online queries, avoiding dependence on continuous LLM supervision. Preliminary experiments in benchmark environments show improved sample efficiency and faster early learning compared to baseline RL methods, with final returns comparable to methods that require frequent LLM interaction.

Narjes Nourzad, Carlee Joe-Wong

University of Southern California · Carnegie Mellon University

Memory-Augmented RLReinforcement LearningLLM-Guided RLSample-Efficient RLSparse-Reward RL
AskPDF
Feb 20, 2026

MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidance

Reinforcement learning (RL) agents often suffer from high sample complexity in sparse or delayed reward settings due to limited prior structure. Large language models (LLMs) can provide subgoal decompositions, plausible trajectories, and abstract priors that facilitate early learning. However, heavy reliance on LLM supervision introduces scalability constraints and dependence on potentially unreliable signals. We propose MIRA (Memory-Integrated Reinforcement Learning Agent), which incorporates a structured, evolving memory graph to guide early training. The graph stores decision-relevant information, including trajectory segments and subgoal structures, and is constructed from both the agent's high-return experiences and LLM outputs. This design amortizes LLM queries into a persistent memory rather than requiring continuous real-time supervision. From this memory graph, we derive a utility signal that softly adjusts advantage estimation to influence policy updates without modifying the underlying reward function. As training progresses, the agent's policy gradually surpasses the initial LLM-derived priors, and the utility term decays, preserving standard convergence guarantees. We provide theoretical analysis showing that utility-based shaping improves early-stage learning in sparse-reward environments. Empirically, MIRA outperforms RL baselines and achieves returns comparable to approaches that rely on frequent LLM supervision, while requiring substantially fewer online LLM queries. Project webpage: https://narjesno.github.io/MIRA/

Narjes Nourzad, Carlee Joe-Wong

University of Southern California, Electrical and Computer Engineering, Los Angeles, CA 90007, USA · Carnegie Mellon University, Electrical and Computer Engineering, Pittsburgh, PA 15213, USA

Memory-Augmented RLReinforcement LearningLLM-Guided RLSample-Efficient RLSparse-Reward RL
AskPDF
Feb 19, 2026

MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning

Existing Reinforcement Learning with Verifiable Rewards (RLVR) algorithms, such as GRPO, rely on rigid, uniform, and symmetric trust region mechanisms that are fundamentally misaligned with the complex optimization dynamics of Large Language Models (LLMs). In this paper, we identify three critical challenges in these methods: (1) inefficient gradient utilization caused by the binary cutoff of hard clipping, (2) insensitive probability mass arising from uniform ratio constraints that ignore the token distribution, and (3) asymmetric signal reliability stemming from the disparate credit assignment ambiguity between positive and negative samples. To bridge these gaps, we propose Mass-Adaptive Soft Policy Optimization (MASPO), a unified framework designed to harmonize these three dimensions. MASPO integrates a differentiable soft Gaussian gating to maximize gradient utility, a mass-adaptive limiter to balance exploration across the probability spectrum, and an asymmetric risk controller to align update magnitudes with signal confidence. Extensive evaluations demonstrate that MASPO serves as a robust, all-in-one RLVR solution, significantly outperforming baselines. Our code is available at: https://github.com/FlyTune/MASPO-RL.

Xiaoliang Fu, Jiaye Lin, Yangyi Fang +7

Meituan · Fudan University · Tsinghua University +2

RL for Language ModelsRL for Language Model ReasoningReinforcement Learning with Verifiable RewardsRobust RLSample-Efficient RL
AskPDF
Feb 19, 2026

Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models

Learning from self-sampled data and sparse environmental feedback remains a fundamental challenge in training self-evolving agents. Temporal credit assignment mitigates this issue by transforming sparse feedback into dense supervision signals. However, previous approaches typically depend on learning task-specific value functions for credit assignment, which suffer from poor sample efficiency and limited generalization. In this work, we propose to leverage pretrained knowledge from large language models (LLMs) to transform sparse rewards into dense training signals (i.e., the advantage function) through retrospective in-context learning (RICL). We further propose an online learning framework, RICOL, which iteratively refines the policy based on the credit assignment results from RICL. We empirically demonstrate that RICL can accurately estimate the advantage function with limited samples and effectively identify critical states in the environment for temporal credit assignment. Extended evaluation on four BabyAI scenarios show that RICOL achieves comparable convergent performance with traditional online RL algorithms with significantly higher sample efficiency. Our findings highlight the potential of leveraging LLMs for temporal credit assignment, paving the way for more sample-efficient and generalizable RL paradigms.

Wen-Tse Chen, Jiayu Chen, Fahim Tajwar +4

Carnegie Mellon University · The University of Hong Kong · Stanford University

Step-Level Credit Assignment in RLSparse-Reward RLSample-Efficient RLLLM-Guided RLIn-Context RL
AskPDF
Feb 18, 2026

Improved Bounds for Reward-Agnostic and Reward-Free Exploration

We study reward-free and reward-agnostic exploration in episodic finite-horizon Markov decision processes (MDPs), where an agent explores an unknown environment without observing external rewards. Reward-free exploration aims to enable εε-optimal policies for any reward revealed after exploration, while reward-agnostic exploration targets εε-optimality for rewards drawn from a small finite class. In the reward-agnostic setting, Li, Yan, Chen, and Fan achieve minimax sample complexity, but only for restrictively small accuracy parameter εε. We propose a new algorithm that significantly relaxes the requirement on εε. Our approach is novel and of technical interest by itself. Our algorithm employs an online learning procedure with carefully designed rewards to construct an exploration policy, which is used to gather data sufficient for accurate dynamics estimation and subsequent computation of an εε-optimal policy once the reward is revealed. Finally, we establish a tight lower bound for reward-free exploration, closing the gap between known upper and lower bounds.

Oran Ridel, Alon Cohen

Department of Engineering, Tel Aviv University, Tel Aviv, Israel. · Google Research, Tel Aviv, Israel.

RL ExplorationReward-Free RLSample-Efficient RL
AskPDF
Feb 18, 2026

Regret and Sample Complexity of Online Q-Learning via Concentration of Stochastic Approximation with Time-Inhomogeneous Markov Chains

We present the first regret bound for classical online Q-learning in infinite-horizon discounted Markov decision processes (MDPs), without relying on optimism or bonus terms. We first analyze Boltzmann Q-learning with decaying temperature and show that its regret depends critically on the suboptimality gap of the MDP: for sufficiently large gaps, the regret is sublinear, while for small gaps it deteriorates and can approach linear growth. To address this limitation, we study a Smoothed εnε_n-Greedy exploration scheme that combines εnε_n-greedy and Boltzmann exploration, for which we prove a gap-robust regret bound of near-O~(N9/10)\tilde{O}(N^{9/10}). We also obtain sample complexity guarantees, with both regret and sample complexity bounds holding with high probability. To analyze these algorithms, we develop a high-probability concentration bound for contractive Markovian stochastic approximation with iterate- and time-dependent transition dynamics. This bound may be of independent interest as the contraction factor in our framework is allowed to converge to one asymptotically.

Rahul Singh, Siddharth Chandak, Eric Moulines +2

MBZUAI, UAE · Stanford University, USA · EPITA, France +1

Regret Minimization in RLQ-LearningSample-Efficient RLRL ExplorationStochastic Approximation
AskPDF
Feb 17, 2026

CDRL: A Reinforcement Learning Framework Inspired by Cerebellar Circuits and Dendritic Computational Strategies

Reinforcement learning (RL) has achieved notable performance in high-dimensional sequential decision-making tasks, yet remains limited by low sample efficiency, sensitivity to noise, and weak generalization under partial observability. Most existing approaches address these issues primarily through optimization strategies, while the role of architectural priors in shaping representation learning and decision dynamics is less explored. Inspired by structural principles of the cerebellum, we propose a biologically grounded RL architecture that incorporate large expansion, sparse connectivity, sparse activation, and dendritic-level modulation. Experiments on noisy, high-dimensional RL benchmarks show that both the cerebellar architecture and dendritic modulation consistently improve sample efficiency, robustness, and generalization compared to conventional designs. Sensitivity analysis of architectural parameters suggests that cerebellum-inspired structures can offer optimized performance for RL with constrained model parameters. Overall, our work underscores the value of cerebellar structural priors as effective inductive biases for RL.

Sibo Zhang, Rui Jing, Liangfu Lv +2

Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin, China.

Reinforcement LearningSample-Efficient RLRobust RLRepresentation Learning for RL
AskPDF
Feb 16, 2026

Near-Optimal Sample Complexity for Online Constrained MDPs

Safety is a fundamental challenge in reinforcement learning (RL), particularly in real-world applications such as autonomous driving, robotics, and healthcare. To address this, Constrained Markov Decision Processes (CMDPs) are commonly used to enforce safety constraints while optimizing performance. However, existing methods often suffer from significant safety violations or require a high sample complexity to generate near-optimal policies. We address two settings: relaxed feasibility, where small violations are allowed, and strict feasibility, where no violation is allowed. We propose a model-based primal-dual algorithm that balances regret and bounded constraint violations, drawing on techniques from online RL and constrained optimization. For relaxed feasibility, we prove that our algorithm returns an ε\varepsilon-optimal policy with ε\varepsilon-bounded violation with arbitrarily high probability, requiring O~(SAH3ε2)\tilde{O}\left(\frac{SAH^3}{\varepsilon^2}\right) learning episodes, matching the lower bound for unconstrained MDPs. For strict feasibility, we prove that our algorithm returns an ε\varepsilon-optimal policy with zero violation with arbitrarily high probability, requiring O~(SAH5ε2ζ2)\tilde{O}\left(\frac{SAH^5}{\varepsilon^2ζ^2}\right) learning episodes, where ζζ is the problem-dependent Slater constant characterizing the size of the feasible region. This result matches the lower bound for learning CMDPs with access to a generative model. Our results demonstrate that learning CMDPs in an online setting is as easy as learning with a generative model and is no more challenging than learning unconstrained MDPs when small violations are allowed.

Chang Liu, Yunfan Li, Lin F. Yang

University of California, Los Angeles

Constrained RLSafe RLSample-Efficient RLModel-Based RL
AskPDF
Feb 2, 2026

VLM-Guided Experience Replay

Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have enabled powerful semantic and multimodal reasoning capabilities, creating new opportunities to enhance sample efficiency, high-level planning, and interpretability in reinforcement learning (RL). While prior work has integrated LLMs and VLMs into various components of RL, the replay buffer, a core component for storing and reusing experiences, remains unexplored. We propose addressing this gap by leveraging VLMs to guide the prioritization of experiences in the replay buffer. Our key idea is to use a frozen, pre-trained VLM as an automated evaluator to identify and prioritize promising sub-trajectories from the agent's experiences. Across scenarios, including game-playing and robotics, spanning both discrete and continuous domains, agents trained with our proposed prioritization method achieve 15-57% higher average success rates and improve sample efficiency by 35-55% compared to previous approaches. Project page: https://esharony.me/projects/vlm-rb/

Elad Sharony, Tom Jurgenson, Orr Krupnik +2

Technion · ForSight Robotics · Nvidia Research

Reinforcement LearningPrioritized Experience ReplaySample-Efficient RL
AskPDF
Jan 30, 2026

Agile Reinforcement Learning through Separable Neural Architecture and Applications

Deep reinforcement learning (RL) is increasingly deployed in resource-constrained environments, yet go-to function approximators - multilayer perceptrons (MLPs) - are often parameter-inefficient due to an imperfect inductive bias for the smooth structure of many value functions. This mismatch can also hinder sample efficiency and slow policy learning in this capacity-limited regime. Although model compression techniques exist, they operate post-hoc and do not improve learning efficiency. Spline-based architectures such as Kolmogorov-Arnold Networks (KANs) have been shown to offer parameter efficiency but are widely reported to exhibit significant computational overhead, especially at scale. In seeking to address these limitations, this work introduces SPAN (SPline-based Adaptive Networks) for RL. SPAN adapts the KHRONOS framework with a learnable preprocessing layer. SPAN is evaluated across discrete (PPO) and high-dimensional continuous (SAC) control tasks, offline settings (Minari/D4RL) and a real-world datacenter HVAC control application. SPAN achieves a 30-50% improvement in sample efficiency and 1.3-9 times higher success rates across benchmarks compared to MLP baselines. Despite incurring a per-step evaluation overhead of 1.2-1.8x, SPAN's superior convergence reliability yields an expected total training cost 1.3-6.3x lower than MLP baselines when accounting for convergence failures. In the HVAC application, SPAN reduces energy consumption in 9 of 12 months relative to MLP while simultaneously achieving a 1.1-3.4x reduction in thermal comfort violations across the evaluation year, demonstrating generalization to real-world engineering control. Furthermore, SPAN demonstrates superior anytime performance and robustness to hyperparameter variations, suggesting it as a viable, high-performance alternative for learning efficient policies in resource-limited settings.

Rajib Mostakim, Reza T. Batley, Sourav Saha

Department of Mechanical Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh · Kevin T. Crofton Department of Aerospace and Ocean Engineering, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA

RL ControlNeural Network Approximation TheorySample-Efficient RLHeating, Ventilation, and Air Conditioning Control
AskPDF
Jan 30, 2026

HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learning

RLVR has become a standard recipe for training LLMs on reasoning tasks with verifiable outcomes, but when rollout generation dominates the cost, efficiency hinges on which prompts are sampled and when. In practice, prompt pools are often static or only weakly coupled to policy progress, so uniform sampling fails to track the moving capability frontier and wastes rollouts on regions that are already solved or still unreachable. Prior methods improve efficiency via filtering, curricula, adaptive rollout allocation, or teacher guidance, but they often assume a fixed pool, which does not support stable on-policy pool growth, or they introduce additional teacher cost and latency. In this work, we propose HeaPA (Heap Sampling and On-Policy Query Augmentation), which maintains a bounded, evolving pool, tracks the frontier with heap-based boundary sampling, grows the pool via on-policy augmentation under lightweight asynchronous validation, and stabilizes correlated queries via topology-aware pool statistics re-estimation and controlled reinsertion. Across two training corpora, two training recipes, and seven benchmarks, HeaPA consistently improves accuracy and reaches target performance with fewer computations at comparable wall-clock time. Analyses attribute the gains to frontier-focused sampling and on-policy pool growth, with more pronounced improvements at mid-to-large model scales. Our training code is publicly available at https://github.com/horizon-llm/HeaPA.

Weiqi Wang, Xin Liu, Binxuan Huang +13

Stores Foundational AI, Amazon Inc., Palo Alto, CA, USA · Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China

RL for Language ModelsRL for Language Model ReasoningAdaptive SamplingReinforcement Learning with Verifiable RewardsRL Post-TrainingSample-Efficient RL
AskPDF
Jan 29, 2026

Can David Beat Goliath? On Multi-Hop Reasoning with Resource-Constrained Agents

Reinforcement learning (RL) trains small language model agents to answer multi-hop questions by retrieving evidence over multiple turns, but reported gains typically rely on thousands of on-policy rollouts per update. We study RL for such agents on commodity GPUs, where each update samples only a few rollouts per question. Most sampled trajectories then retrieve none of the required evidence, so the outcome reward gives the policy little to learn from and small agents settle for answering without retrieval, a failure we call retrieval collapse. DAVID-GRPO addresses this with two mechanisms: (1) expert trajectory seeding places a handful of off-policy expert trajectories into the GRPO groups of the early updates, and (2) evidence-guided continuation rewards evidence coverage and resumes the most promising partial trajectory. The evidence for each training question is constructed from the corpus link graph, without annotated evidence. On six multi-hop QA benchmarks, DAVID-GRPO trained on four RTX 3090 GPUs with 144 rollouts per step brings Qwen2.5-1.5B to 22.6 average EM against 11.9 for the best baseline under the same budget, is comparable to Tree-GRPO trained with 20 times more rollouts, and, unlike the baselines that stop after at most one search, learns to retrieve across turns. The implementation is available at: https://github.com/AsadalJung/David-GRPO

Hojae Han, Heeyun Jung, Jongyoon Kim +1

ETRI · The University of Hong Kong · Seoul National University

RL for Language Model ReasoningMulti-Hop QASample-Efficient RLRL for IRGroup Relative Policy Optimization
AskPDF
Jan 21, 2026

CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning

Agentic Reinforcement Learning (RL) has empowered Large Language Models (LLMs) to utilize tools like Python interpreters for complex problem-solving. However, for parameter-constrained models (e.g., 4B--7B), the exploration phase is often plagued by frequent execution failures, creating noisy trajectories that hinder policy optimization. Under standard outcome-based reward settings, this noise leads to a critical credit assignment issue, where erroneous actions are inadvertently reinforced alongside successful outcomes. Existing mitigations face a dilemma: dense rewards often trigger reward hacking, while supersampling incurs prohibitive computational costs. To address these challenges, we propose CLEANER. Distinct from external filtering methods, CLEANER exploits the model's intrinsic self-correction capabilities to eliminate error-contaminated context directly during data collection. At its core, the Similarity-Aware Adaptive Rollback (SAAR) mechanism autonomously constructs clean, purified trajectories by retrospectively replacing failures with successful self-corrections. Based on semantic similarity, SAAR adaptively regulates replacement granularity from shallow execution repairs to deep reasoning substitutions. By training on these self-purified paths, the model internalizes correct reasoning patterns rather than error-recovery loops. Empirical results on AIME24/25, GPQA, and LiveCodeBench show average accuracy gains of 6%, 3%, and 5% over baselines. Notably, CLEANER matches state-of-the-art performance using only one-third of the training steps, highlighting trajectory purification as a scalable solution for efficient agentic RL. Our models and code are available at GitHub

Tianshi Xu, Yuteng Chen, Meng Li

Peking University · NTU, Singapore

Agentic RLReinforcement LearningRL for Language Model ReasoningSample-Efficient RLRL for LLM Agents
AskPDF
Jan 21, 2026

Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions

Reinforcement learning (RL) has achieved remarkable success in real-world decision-making across diverse domains, including gaming, robotics, online advertising, public health, and natural language processing. Despite these advances, a substantial gap remains between RL research and its deployment in many practical settings. Two recurring challenges often underlie this gap. First, many settings offer limited opportunity for the agent to interact extensively with the target environment due to practical constraints. Second, many target environments often undergo substantial changes, requiring redesign and redeployment of RL systems (e.g., advancements in science and technology that change the landscape of healthcare delivery). Addressing these challenges and bridging the gap between basic research and application requires theory and methodology that directly inform the design, implementation, and continual improvement of RL systems in real-world settings. In this paper, we frame the application of RL in practice as a three-component process: (i) online learning and optimization during deployment, (ii) post- or between-deployment offline analyses, and (iii) repeated cycles of deployment and redeployment to continually improve the RL system. We provide a narrative review of recent advances that address the statistical challenges arising across these three components, including methods for enhancing sample efficiency during online deployment, maximizing data utility for post- or between-deployment inference, and designing sequences of deployments for continual improvement. We also outline future research directions in RL that are use-inspired -- aiming for impactful application of RL in practice.

Asim H. Gazi, Yongyi Guo, Daiqi Gao +3

Department of Computer Science, Harvard University · Department of Statistics, Harvard University · Department of Statistics, University of Wisconsin–Madison +2

Non-Stationary RLReinforcement LearningSample-Efficient RL
AskPDF
Jan 11, 2026

GDEPO: Group Dual-dynamic and Equal-right Advantage Policy Optimization for Sample-Efficient Reinforcement Learning

Automated Theorem Proving (ATP) is a fundamental AI challenge, requiring machine-verifiable proofs in formal languages like Lean to evaluate reasoning. Reinforcement Learning (RL), especially Group Relative Policy Optimization (GRPO), is a mainstream approach. However, GRPO faces two critical issues in ATP: its relative advantage estimation may conflict with binary verifier feedback under composite rewards, and its static sampling strategy discards batches without valid proofs, causing data waste. Repeated training on mastered problems also wastes computation. To address this, we propose Group Dual-dynamic and Equal-right Advantage Policy Optimization (GDEPO), incorporating three core mechanisms: 1) dynamic additional sampling, resampling ineffective batches until a valid proof is found; 2) equal-right advantage, decoupling the advantage function's sign (correctness) from its magnitude (auxiliary rewards) to ensure stable policy updates while increasing effective computation; and 3) dynamic additional iterations, applying extra gradient steps to initially failed but eventually successful samples to accelerate learning. Experiments on MinF2F-test, MathOlympiadBench, and PutnamBench confirm GDEPO's effectiveness, with ablation studies validating its synergistic components. The method enhances data utilization and optimization efficiency, offering a novel training paradigm for ATP.

Zhengqing Yan, Fan Guo, Xinyang Liu +5

State Key Laboratory of Engines, Tianjin University, 300354 Tianjin, China. · Artificial Intelligence Center, Cylingo Group, 100080 Beijing, China.

Group Relative Policy OptimizationReinforcement Learning with Verifiable RewardsSample-Efficient RLAutomated Theorem Proving
AskPDF
Dec 12, 2025

Symmetry-Aware Steering of Equivariant Diffusion Policies: Benefits and Limits

Equivariant diffusion policies (EDPs) combine the expressivity of diffusion models with the generalization and sample efficiency afforded by geometric symmetries. While steering these policies with reinforcement learning (RL) offers a promising fine-tuning mechanism beyond demonstration data, directly applying standard (non-equivariant) RL ignores the symmetries that EDPs are designed to exploit, leading to sample inefficiency and instability. We theoretically establish that the diffusion process of an EDP is equivariant, inducing a group-invariant latent-noise MDP which is well-suited for equivariant diffusion steering. Building on this, we introduce a principled symmetry-aware steering framework and compare standard, equivariant, and approximately equivariant RL strategies across tasks with varying degrees of symmetry. While we identify the practical boundaries of strict equivariance under symmetry breaking, exploiting symmetry during the steering yields substantial benefits---enhancing sample efficiency, preventing value divergence, and achieving strong policy improvements even when EDPs are trained from extremely limited demonstrations.

Minwoo Park, Junwoo Chang, Jongeun Choi +1

Yonsei University, Seoul 03722, Republic of Korea · University of California, Berkeley, CA 94720, USA

Diffusion PolicyRL for Diffusion ModelsEquivariant Policy LearningReinforcement LearningSample-Efficient RL
AskPDF
Dec 6, 2025

Auto-exploration for online reinforcement learning

The exploration-exploitation dilemma in reinforcement learning (RL) is a fundamental challenge to efficient RL algorithms. Existing algorithms for finite state and action discounted RL problems address this by assuming sufficient exploration over both state and action spaces. However, this yields non-implementable algorithms and sub-optimal performance. To resolve these limitations, we introduce a new class of methods with auto-exploration, or methods that automatically explore both state and action spaces. Auto-exploration can be applied in both the tabular and linear function approximation setting. Under algorithm-independent assumptions on the existence of an exploring optimal policy, both settings attain O(ε−2)O(ε^{-2}) sample complexity to solve to εε error. These complexities are novel since they avoid algorithm-dependent parameters seen in prior works, which may be arbitrarily large. The methods are also simple to implement because they are parameter-free. We achieve these results by integrating auto-exploration into policy mirror descent to avoid the (unknown) stationary distribution seen in prior art. In the tabular setting, we introduce a dynamic exploration time with a data-driven stopping time, while for linear function approximation we propose a new sampling distribution based on the discounted visitation distribution that covers a more general class of Markov chains.

Caleb Ju, Guanghui Lan

Policy OptimizationRL ExplorationPolicy LearningSample-Efficient RL
AskPDF
Jun 19, 2025

Energy-Based Transfer for Reinforcement Learning

Reinforcement learning remains sample inefficient, particularly when agents must adapt over time to related but distinct tasks, as in continual learning settings. Efficiency can be improved by transferring knowledge from a pretrained teacher policy to guide exploration in new but related tasks. However, if the new task sufficiently differs from the teacher's training task, the transferred guidance may be sub-optimal and bias exploration toward low-reward behaviors. We propose an energy-based transfer learning method that uses out-of-distribution detection to selectively issue guidance, enabling the teacher to intervene only in states within its training distribution. We theoretically link energy scores to state-visitation patterns under on-policy reinforcement learning. Empirically, our method improves sample efficiency and performance across both single-task and multi-task settings.

Zeyun Deng, Jasorsi Ghosh, Fiona Xie +4

♠Purdue University · ♢AWS AI · ♣Carnegie Mellon University

Transfer Learning in RLTransfer LearningSample-Efficient RLContinual RLMulti-Task RL
AskPDF
Jun 3, 2025

Ensemble-MIX: Enhancing Sample-Efficiency in Multi-Agent RL via Selective Ensemble-Based Exploration

Multi-agent reinforcement learning (MARL) methods have achieved state-of-the-art results on a range of multi-agent tasks. Yet, MARL algorithms typically require significantly more environment interactions than their single-agent counterparts to converge, a problem exacerbated by the difficulty in exploring over a large joint action space and the high variance intrinsic to MARL environments. To tackle these issues, we propose a novel algorithm that combines a decomposed centralized critic with decentralized ensemble learning, incorporating several key contributions. The main component in our scheme is a selective exploration method that leverages ensemble kurtosis. We extend the global decomposed critic with a diversity-regularized ensemble of individual critics and utilize its excess kurtosis to guide exploration toward high-uncertainty states and actions. To improve sample efficiency, we train the centralized critic with a novel truncated variation of the TD(λλ) algorithm, enabling efficient off-policy learning with reduced variance. On the actor side, our suggested algorithm adapts the mixed samples approach to MARL, mixing on-policy and off-policy loss functions for training the actors. This approach balances between stability and efficiency and outperforms purely off-policy learning. The evaluation shows our method outperforms state-of-the-art baselines on standard MARL benchmarks, including a variety of SMAC II maps.

Tom Danino, Nahum Shimkin

The Andrew and Erna Viterbi Faculty of Electrical & Computer Engineering, Technion- Israel Institute of Technology

Multi-Agent Reinforcement LearningSample-Efficient RLRL ExplorationOff-Policy RL
AskPDF