Model-Free RL

RL: Reinforcement Learning

Model-free reinforcement learning studies how agents can choose actions and improve behavior directly from rewards and experience, which is useful when system dynamics are unknown or costly to model. Common approaches include Q-learning and temporal-difference methods, policy gradients and actor-critic algorithms such as SAC and TD3, plus offline learning from fixed datasets. Research spans continuous control, safety and exploration, multi-agent systems, and average-reward problems, with theory increasingly addressing convergence, regret, and sample complexity. A central tension is improving sample efficiency, robustness, and safety while retaining the simplicity of learning without an explicit dynamics model; predictive representations also blur the boundary between model-free and model-based methods.

Latest papers

61 papers13 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees
Oct 20, 2026Chaorui Yao, Yuming Paul Zhang, Lin Yang +1Continuous-Time RL Model-Free RL
Department of Electrical and Computer Engineering, University of California, Los Angeles. · Department of Mathematics and Statistics, Auburn University. · Department of Statistics and Data Science, University of California, Los Angeles.
Continuous-Time Policy Gradient from Discrete Data: Convergence Analysis and Model-Free Algorithms
Oct 20, 2026Martin Hernandez, Xinyan Qing, Mo Zhou +1Continuous-Time RL Model-Free RL
Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics, University of Michigan · School of Mathematical Sciences, Peking University
Reinforcement Learning with Hawkes-Structured Networks for Market Making
Oct 12, 2026Fernando Acero, Philipp Plank, Yufei ZhangHawkes Processes Non-Markovian RL
Department of Mathematics, Imperial College London, United Kingdom
Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning
Oct 11, 2026Gunes Akbas, Naci Saldi, Berkay Anahtarci +1Mean Field Games Model-Free RL
Department of Mathematics, Bilkent University, Ankara 06800, Türkiye · Department of Mathematical Engineering, Özyeğin University, İstanbul 34794, Türkiye
Visualizing the World Model in an Agent's Value Function
Oct 9, 2026Shawheen Naderi, Fatemeh Akherati, Yubei ChenWorld Models Interpretability in RL
University of California, Davis
Randomized Transport Maps for Model-Free Policy-Gradient Mean-Field Control
Oct 8, 2026Adonis Jamal, Samy Mekkaoui, Yadh Hafsi +1Policy Gradient Methods Model-Free RL
École Polytechnique
RLDISCOVER: LLM-driven co-evolution of reinforcement learning algorithms
Oct 6, 2026Haoran Li, Zengle Ge, Xiaomin Yuan +14Automated Algorithm Discovery Evolutionary RL
University of the Chinese Academy of Sciences · Baidu · Tsinghua University +4
Reachability-Aware Diffusion Policy Optimization
Oct 5, 2026Hikmet Simsir, Kutay Demiray, Ozgur S. OguzConstrained RL Diffusion Policy
Department of Computer Engineering, Bilkent University
Taylor Representations for Model-Free RL in Networked MDPs
Oct 4, 2026Salah Chikhi, Abdelhaq Chaoui, Asuman Ozdaglar +1Markov Decision Processes Representation Learning for RL
Massachusetts Institute of Technology
Variance-Aware Fine-Grained Gap-Dependent Bounds for Online Reinforcement Learning
Oct 3, 2026Haochen Zhang, Lingzhou Xue, Zhong ZhengReinforcement Learning Model-Free RL
Department of Statistics, The Pennsylvania State University · Department of Applied and Computational Mathematics and Statistics, University of Notre Dame
Q-Learning for Reachability in MEC-Free MDPs
Oct 1, 2026Lu-Chin Chang, Suguman BansalMarkov Decision Processes Reinforcement Learning
Georgia Institute of Technology
Deep Epistemic Value Functions for Optimistic Exploration
Sep 28, 2026Leander Diaz-Bone, Marco Bagatella, Jonas Hübotter +1Value Function Estimation RL Exploration
ETH Zürich, Switzerland · Max Planck Institute for Intelligent Systems, Germany
GraphHCA: Closed-Form Hindsight Credit Assignment for Long-Horizon LLM Agents
Sep 28, 2026Haodong Zhu, Yangyang Ren, Changbai Li +4Credit Assignment in RL Step-Level Credit Assignment in RL
Beihang University · Zhongguancun Academy · Communication University of China +1
Lifted Bellman Linear Programming for Offline Reinforcement Learning
Sep 21, 2026Hyukjun Yang, Jongchan Park, Narim Jeong +1Reinforcement Learning Offline RL
Korea Advanced Institute of Science and Technology(KAIST) · NAVER LABS
Deep Reinforcement Learning with Buffered Quantile Objectives
Sep 18, 2026Mohammad Alipour-vaezi, Sajad KhodadadianRisk-Sensitive RL Distributional RL
Virginia Tech
Is Monte Carlo Tree Search Just Every-Visit Monte Carlo Control?
Aug 28, 2026Xianyi WuMonte Carlo Tree Search RL Control
ECNU
ProDVI: Programmatic Dynamics Priors for Value Network Initialization
Aug 6, 2026Xinwei Liu, Junyuan Liang, Jianting Zhang +1Reinforcement Learning LLM-Guided RL
1Sun Yat-sen University · 2Purdue University
Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control
Aug 6, 2026Xinwei Liu, Junyuan Liang, Jianting Zhang +1Predictive Representation Learning Continuous Control
Sun Yat-sen University · Purdue University
Exact Model-Free Policy Iteration for Co-safe LTL Planning
Aug 5, 2026Zetong Xuan, Yu WangReinforcement Learning Policy Iteration
Department of Mechanical and Aerospace Engineering, University of Florida, Gainesville, FL 32611 USA
Finite-Time Analysis of Discounted Exponential-Utility Reinforcement Learning
Aug 3, 2026Ankur Naskar, Vivek T A, Aditya Kumar +2Reinforcement Learning Risk-Sensitive RL
Indian Institute of Science, Bengaluru, India · Indian Institute of Technology Madras, Chennai, India
Exact Action Values Are Not Enough: Rollout-Verified Reinforcement Fine-Tuning of a Reasoning Model for Multi-Zone VAV Control
Jul 30, 2026Takumi Shioda, Kohei Terashima, Tatsuo NagaiLanguage Model-Based Control Model-Free RL
The University of Tokyo · Tokyo University of Science
Physics-Aware End-to-End Deep Reinforcement Learning for Quadcopter Control with Actuator Dynamics
Jul 28, 2026Ya-Chia Shen, Woei-Leong ChanRL Control Quadrotor Control
Department of Computer Science and Information Engineering, National Cheng Kung University · Department of Aeronautics and Astronautics, National Cheng Kung University
Planning as Emergent Behavior in Reinforcement Learning with Relational Hidden States
Jul 20, 2026Armin SommerReinforcement Learning Model-Free RL
ETH Zurich, Switzerland
Principled Direction-Free Intrinsic Motivation through Model-Free Epistemic Free-Energy Estimators
Jul 18, 2026Alireza Furutanpey, Schahram DustdarFree Energy Calculation Intrinsic Reward Methods
Coovally, Barcelona, Spain · Distributed Systems Group, TU Vienna, Austria · ICREA Barcelona, Spain
Actor-Critic Learning for Extended Mean Field Control with Deterministic Policies
Jul 13, 2026Ziheng Cheng, Xin Guo, Huyên Pham +1Policy Gradient Continuous-Time RL
University of California, Berkeley · Ecole Polytechnique, CMAP · Imperial College London
Can Reinforcement Learning Efficiently Discover Price Manipulation?
Jul 7, 2026Ioanna-Yvonni Tsaknaki, Andrea Macrì, Fabrizio LilloModel-Free RL Algorithmic Trading
Scuola Normale Superiore, Piazza dei Cavalieri 7, Pisa 56126, Italy.
Mean-Field PhiBE: Continuous-Time Mean-Field Reinforcement Learning from Discrete-Time Data
Jun 25, 2026Erhan Bayraktar, Martin Hernandez, Qinxin Yan +1Actor-Critic Methods Stochastic Optimal Control
Department of Mathematics, University of Michigan, Ann Arbor, MI, USA. · Department of Statistics and Data Science, University of California, Los Angeles, CA, USA. · Program in Applied and Computational Mathematics, Princeton University, Princeton, NJ, USA.
Managing Task Execution for Unknown Workloads in Batteryless IoT: A Hardware-Agnostic Evaluation
Jun 23, 2026Samer Nasser, Henrique Duarte Moura, Ritesh Kumar Singh +2Internet of Things Model-Free RL
University of Antwerp - imec, IDLab Research Group, Antwerp, Belgium
NASDAQ: Normalized Observation Space Dynamics-Augmented Q-Learning
Jun 19, 2026Xinwei Liu, Junyuan Liang, Zicong Hong +2Reinforcement Learning Q-Learning
Sun Yat-sen University, China · EPFL, Switzerland · Purdue University, USA
Inverting the Bellman Equation: From $Q$ -Values to World Models
Jun 19, 2026Alistair Letcher, Mattie Fellows, Alexander D. Goldie +3Reinforcement Learning Model-Free RL
FLAIR, University of Oxford · Google DeepMind · Mila, University of Montreal

Oct 20, 2026J/K move · Enter open · S save

A Model-Free Continuous-Time RL Algorithm from Discrete-Time Off-Policy Data with End-to-End Guarantees

Chaorui Yao, Yuming Paul Zhang, Lin Yang, Yuhua Zhu

Department of Electrical and Computer Engineering, University of California, Los Angeles. · Department of Mathematics and Statistics, Auburn University. · Department of Statistics and Data Science, University of California, Los Angeles.

Continuous-Time RL Model-Free RL Policy Iteration Off-Policy RL

Open paper AskPDF arXiv

In this paper, we study continuous-time reinforcement learning, where the state follows an unknown stochastic differential equation but only a fixed off-policy batch of discrete-time transitions is available. Standard discrete-time Q-learning becomes problematic as the sampling interval shrinks, because differences between action values vanish. Continuous-time alternatives, such as Deep Advantage Updating and martingale q-learning, preserve the action signal but jointly learn a value function and an action-dependent critic. Building on the PhiBE-Q formulation (arXiv:2606.21925), we develop a model-free PhiBE policy iteration (PhiBE-PI) framework that uses a single state-action critic while preserving the continuous-time action-ranking scale. At each iteration, the framework learns the critic directly from sampled increments and uses it to improve the policy. The same batch is reused throughout, without explicitly estimating the drift or diffusion. We further develop an implementation under linear function approximation and establish a local end-to-end suboptimality guarantee for the learned policy. Our analysis quantifies how errors from time discretization, function approximation, and finite-sample policy evaluation propagate through policy iteration. It accounts for the dependence of successive policies on the reused data, so the guarantee does not require a fresh batch at each iteration. Moreover, we identify a bias-variance trade-off: smaller sampling intervals improve the approximation to the continuous-time problem but can increase statistical error at a fixed sample size. Thus, finer time discretization does not necessarily lead to better policy performance.