Model Free Reinforcement Learning

Model-free reinforcement learning (MFRL) aims to learn optimal policies for controlling systems without explicit knowledge of the system's dynamics, focusing on direct interaction and reward maximization. Current research emphasizes improving sample efficiency and robustness through techniques like residual policy learning, adaptive horizon control, and the integration of physics-based models to guide learning, often employing algorithms such as PPO, SAC, TD3, and actor-critic methods with various neural network architectures (e.g., CNNs, RNNs, Transformers). These advancements are significant for applications requiring efficient learning in complex, high-dimensional environments, such as robotics, autonomous navigation, and financial portfolio management, where detailed models are unavailable or computationally intractable.

Papers

November 15, 2024

The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning
Moritz Schneider, Robert Krug, Narunas Vaskevicius, Luigi Palmieri, Joschka Boedecker
Reinforcement Learning Model Free Reinforcement Learning Pre Trained Representation Visual Reinforcement Learning Surprising Effectiveness

October 21, 2024

Learning Quadrotor Control From Visual Features Using Differentiable Simulation
Johannes Heeg, Yunlong Song, Davide Scaramuzza
Reinforcement Learning Policy Gradient Simulation Study Model Free Reinforcement Learning Visual Feature Differentiable Simulation Closed Form Differentiable Expression Quadrotor Control

October 19, 2024

MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning
Suning Huang, Zheyu Zhang, Tianhai Liang, Yihan Xu, Zhehao Kou, Chenhao Lu, Guowei Xu, Zhengrong Xue, Huazhe Xu
Real World Mixture of Expert Model Free Reinforcement Learning Visual Reinforcement Learning DeepMind Control Suite

October 11, 2024

Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient
Wenlong Wang, Ivana Dusparic, Yucheng Shi, Ke Zhang, Vinny Cahill
Recurrent Neural Network Model Based Reinforcement Learning World Model External Sample Model Free Reinforcement Learning Parameter Efficient Play Style

October 9, 2024

Disturbance Observer-based Control Barrier Functions with Residual Model Learning for Safe Reinforcement Learning
Dvij Kalaria, Qin Lin, John M. Dolan
Reinforcement Learning Control Barrier Function Safe Reinforcement Learning Model Free Reinforcement Learning Based Rejection Unknown Disturbance Residual Classifier

October 8, 2024

EVOLvE: Evaluating and Optimizing LLMs For Exploration
Allen Nie, Yi Su, Bo Chang, Jonathan N. Lee, Ed H. Chi, Quoc V. Le, Minmin Chen
Large Language Model Environment Exploration Contextual Bandit Model Free Reinforcement Learning Based Exploration Meta EVOLVE

October 4, 2024

Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation
Jing Yuan Luo, Yunlong Song, Victor Klemm, Fan Shi, Davide Scaramuzza, Marco Hutter
Policy Gradient Model Free Reinforcement Learning Differentiable Simulation Residual Reinforcement Learning Perceptive Locomotion First Order Policy

September 27, 2024

State-free Reinforcement Learning
Mingyu Chen, Aldo Pacchiano, Xuezhou Zhang
State Space Model Free Model Free Reinforcement Learning Reachable Set

September 26, 2024

Model-Free versus Model-Based Reinforcement Learning for Fixed-Wing UAV Attitude Control Under Varying Wind Conditions
David Olivares, Pierre Fournier, Pavan Vasishta, Julien Marzat
Model Based Reinforcement Learning Model Free Model Free Reinforcement Learning Model Predictive Controller Wind Pressure Fixed Wing Free Flight

September 14, 2024

PIP-Loco: A Proprioceptive Infinite Horizon Planning Framework for Quadrupedal Robot Locomotion
Aditya Shirwatkar, Naman Saxena, Kishore Chandra, Shishir Kolathaya
Predictive Control Model Free Reinforcement Learning Long Horizon Quadrupedal Locomotion Safe Locomotion Rolling Locomotion Ability

September 3, 2024

A Deep Reinforcement Learning Framework For Financial Portfolio Management
Jinyang Li
Model Free Reinforcement Learning Portfolio Management Portfolio Selection Portfolio Optimization Problem

August 12, 2024

Sequential sampling without comparison to boundary through model-free reinforcement learning
Jamal Esmaily, Rani Moran, Yasser Roudi, Bahador Bahrami
Consistent Comparison Model Free Reinforcement Learning Decision Boundary Document Boundary Sequential Information Evidence Accumulation

August 5, 2024

Integrating Model-Based Footstep Planning with Model-Free Reinforcement Learning for Dynamic Legged Locomotion
Ho Jae Lee, Seungwoo Hong, Sangbae Kim
Reinforcement Learning Model Free Reinforcement Learning Legged Robot Locomotion Dynamic Locomotion Locomotion Control Footstep Planning

July 31, 2024

ProSpec RL: Plan Ahead, then Execute
Liangliang Liu, Yi Guan, BoRan Wang, Rujia Shen, Yi Lin, Chaoran Kong, Lian Yan, Jingchi Jiang
Reinforcement Learning Decision Making Model Free Reinforcement Learning High Level Plan Optimal Action Future Scenario

July 10, 2024

Real-time system optimal traffic routing under uncertainties -- Can physics models boost reinforcement learning?
Zemian Ke, Qiling Zou, Jiachao Liu, Sean Qian
Reinforcement Learning Real Time Model Free Reinforcement Learning High Uncertainty Value Transportation Network

June 22, 2024

Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models
Yang Zhang, Chenjia Bai, Bin Zhao, Junchi Yan, Xiu Li, Xuelong Li
Multi Agent Reinforcement Learning Multi Agent System Model Free Reinforcement Learning Multi Agent Challenge Transformer Based World Model Decentralized Recommender System

June 19, 2024

Reinforcement Learning to improve delta robot throws for sorting scrap metal
Arthur Louette, Gaspard Lambrechts, Damien Ernst, Eric Pirard, Godefroid Dislaire
Reinforcement Learning Reinforcement Learning Algorithm Model Free Reinforcement Learning Pick and Place Delta Robot Ferrous Scrap

June 6, 2024

Transductive Off-policy Proximal Policy Optimization
Yaozhong Gan, Renye Yan, Xiaoyang Tan, Zhe Wu, Junliang Xing
Proximal Policy Optimization Policy Optimization Model Free Reinforcement Learning Policy Data

May 28, 2024

Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation
Ignat Georgiev, Krishnan Srinivasan, Jie Xu, Eric Heiden, Animesh Garg
Reinforcement Learning Policy Gradient Model Free Reinforcement Learning Differentiable Physic Horizon Graph

May 26, 2024

Multi-State TD Target for Model-Free Reinforcement Learning
Wuhao Wang, Zhiyong Chen, Lepeng Zhang
Reinforcement Learning Model Free Reinforcement Learning Actor Critic Algorithm Deep Deterministic Policy Gradient State Action Pair Temporal Difference

Model Free Reinforcement Learning

Papers

The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning

Learning Quadrotor Control From Visual Features Using Differentiable Simulation

MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning

Drama: Mamba-Enabled Model-Based Reinforcement Learning Is Sample and Parameter Efficient

Disturbance Observer-based Control Barrier Functions with Residual Model Learning for Safe Reinforcement Learning

EVOLvE: Evaluating and Optimizing LLMs For Exploration

Residual Policy Learning for Perceptive Quadruped Control Using Differentiable Simulation

State-free Reinforcement Learning

Model-Free versus Model-Based Reinforcement Learning for Fixed-Wing UAV Attitude Control Under Varying Wind Conditions

PIP-Loco: A Proprioceptive Infinite Horizon Planning Framework for Quadrupedal Robot Locomotion

A Deep Reinforcement Learning Framework For Financial Portfolio Management

Sequential sampling without comparison to boundary through model-free reinforcement learning

Integrating Model-Based Footstep Planning with Model-Free Reinforcement Learning for Dynamic Legged Locomotion

ProSpec RL: Plan Ahead, then Execute

Real-time system optimal traffic routing under uncertainties -- Can physics models boost reinforcement learning?

Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models

Reinforcement Learning to improve delta robot throws for sorting scrap metal

Transductive Off-policy Proximal Policy Optimization

Adaptive Horizon Actor-Critic for Policy Learning in Contact-Rich Differentiable Simulation

Multi-State TD Target for Model-Free Reinforcement Learning