Model-Free RL

RL: Reinforcement Learning

Ask about this topic

Overview

Model-free reinforcement learning studies how agents can choose actions and improve behavior directly from rewards and experience, which is useful when system dynamics are unknown or costly to model. Common approaches include Q-learning and temporal-difference methods, policy gradients and actor-critic algorithms such as SAC and TD3, plus offline learning from fixed datasets. Research spans continuous control, safety and exploration, multi-agent systems, and average-reward problems, with theory increasingly addressing convergence, regret, and sample complexity. A central tension is improving sample efficiency, robustness, and safety while retaining the simplicity of learning without an explicit dynamics model; predictive representations also blur the boundary between model-free and model-based methods.

Latest papers

CardsList

61 papers13 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Apr 20, 2023

Topology-Guided Modular Actor-Critic Learning for Continuous Systems under Temporal Objectives

We study formal policy synthesis for continuous-state stochastic systems under linear temporal logic specifications. The product of the system with the automaton of the specification has a hybrid state space with sparse rewards. We introduce a generalized optimal backup order, defined in reverse to a topological order over automaton states, that guides value backups and provably preserves optimality. We further present a model-free actor-critic algorithm whose policy evaluation solves a constrained optimization problem by the augmented Lagrangian method, yielding hyperparameter self-tuning, and prove its optimality and convergence in the tabular case. Since integer encodings of automaton states impose a spurious ordinal relationship on functions learned by one network, we dedicate a value and a policy network to each automaton state (modular learning). The algorithm matches or outperforms PPO, DQN, and A2C on CartPole, and on a Dubins car under a temporal specification the topological order and modular learning raise the success rate from 26.0% to 71.5%.

Lening Li, Zhentian Qian, Jianan Xia +7

Ningbo University, Ningbo, China · Robotics Engineering Program, Worcester Polytechnic Institute, Worcester, MA, USA · Ningbo Zhenhai Silver-Ball Bearing Co., Ltd., Ningbo, China

RL ControlReinforcement LearningModel-Free RLActor-Critic MethodsPolicy Learning
AskPDF