Model-Free RL

RL: Reinforcement Learning

Model-free reinforcement learning studies how agents can choose actions and improve behavior directly from rewards and experience, which is useful when system dynamics are unknown or costly to model. Common approaches include Q-learning and temporal-difference methods, policy gradients and actor-critic algorithms such as SAC and TD3, plus offline learning from fixed datasets. Research spans continuous control, safety and exploration, multi-agent systems, and average-reward problems, with theory increasingly addressing convergence, regret, and sample complexity. A central tension is improving sample efficiency, robustness, and safety while retaining the simplicity of learning without an explicit dynamics model; predictive representations also blur the boundary between model-free and model-based methods.

Latest papers

61 papers13 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Topology-Guided Modular Actor-Critic Learning for Continuous Systems under Temporal Objectives
Apr 20, 2023Lening Li, Zhentian Qian, Jianan Xia +7RL Control Reinforcement Learning
Ningbo University, Ningbo, China · Robotics Engineering Program, Worcester Polytechnic Institute, Worcester, MA, USA · Ningbo Zhenhai Silver-Ball Bearing Co., Ltd., Ningbo, China

Apr 20, 2023J/K move · Enter open · S save

Topology-Guided Modular Actor-Critic Learning for Continuous Systems under Temporal Objectives

Lening Li, Zhentian Qian, Jianan Xia, Yawen Wang+6

Ningbo University, Ningbo, China · Robotics Engineering Program, Worcester Polytechnic Institute, Worcester, MA, USA · Ningbo Zhenhai Silver-Ball Bearing Co., Ltd., Ningbo, China

RL Control Reinforcement Learning Model-Free RL Actor-Critic Methods Policy Learning

Open paper AskPDF arXiv

We study formal policy synthesis for continuous-state stochastic systems under linear temporal logic specifications. The product of the system with the automaton of the specification has a hybrid state space with sparse rewards. We introduce a generalized optimal backup order, defined in reverse to a topological order over automaton states, that guides value backups and provably preserves optimality. We further present a model-free actor-critic algorithm whose policy evaluation solves a constrained optimization problem by the augmented Lagrangian method, yielding hyperparameter self-tuning, and prove its optimality and convergence in the tabular case. Since integer encodings of automaton states impose a spurious ordinal relationship on functions learned by one network, we dedicate a value and a policy network to each automaton state (modular learning). The algorithm matches or outperforms PPO, DQN, and A2C on CartPole, and on a Dubins car under a temporal specification the topological order and modular learning raise the success rate from 26.0% to 71.5%.