Actor-Critic Pretraining for Proximal Policy Optimization
Authors: Andreas Kernbach, Amr Elsheikh, Nicolas Grupp, René Nagel, Marco F. Huber
Organizations: Fraunhofer Institute for Manufacturing Engineering and Automation IPA, Stuttgart, Germany · Institute of Industrial Manufacturing and Management IFF, University of Stuttgart, Germany · Institute for Control Engineering of Machine Tools and Manufacturing Units ISW, University of Stuttgart, Germany · Materials Testing Institute MPA, Stuttgart, Germany
Reinforcement learning (RL) actor-critic algorithms enable autonomous learning but often require a large number of environment interactions, which limits their applicability in robotics. Leveraging expert data can reduce the number of required environment interactions. A common approach is actor pretraining, where the actor network is initialized via behavioral cloning on expert demonstrations and subsequently fine-tuned with RL. In contrast, the initialization of the critic network has received little attention, despite its central role in policy optimization. This paper proposes a pretraining approach for actor-critic algorithms like Proximal Policy Optimization (PPO) that uses expert demonstrations to initialize both networks. The actor is pretrained via behavioral cloning, while the critic is pretrained using returns obtained from rollouts of the pretrained policy. The approach is evaluated on 15 simulated robotic manipulation and locomotion tasks. Experimental results show that actor-critic pretraining improves sample efficiency by 86.1% on average compared to no pretraining and by 30.9% to actor-only pretraining.
Figures & tables
Fig. 1 : Visualization of the pretraining and fine-tuning approach using expert data with an actor-critic model. First an expert policy is used to pretrain actor and critic, which are subsequently fine-tuned using PPO.
Fig. 2 : Comparison of the learning results in PPO fine-tuning given ACP , AP and NP in three exemplary environments. Returns are averaged over three random seeds for fine-tuning, with shaded regions indicating the standard deviation.
Environment
NP
AP
PIRL
ACP
Sample Reduction
nPPONP
nexp
nPPOfine
nAPtot
nPPOfro
nPPOfine
nPIRLtot
nrol
nPPOfine
nACPtot
NP
AP
PIRL
Walker2D
−
0.6
190.0
190.6
12.2
55.8
68.6
49.5
6.0
56.1
100%
70.6%
18.2%
Hopper
588.0
1.0
86.0
87.0
11.8
76.2
89.0
19.9
10.0
30.9
94.7%
64.5%
65.3%
Ant
544.0
1.0
190.0
191.0
55.3
100.7
157.0
1.7
96.0
98.7
81.9%
48.3%
37.1%
Humanoid
−
0.1
52.0
52.1
45.6
42.4
88.1
1.0
52.0
53.1
100%
-1.9%
39.7%
InvertedD.Pendulum
115.0
10.4
0.5
10.9
1.7
0.3
12.4
1.0
0.5
11.9
89.7%
-9.2%
4.0%
TABLE I : Comparison of required environment steps n for NP , AP , PIRL , and ACP with a fixed amount of expert steps nexp in order to reach Gt=0tar . All steps are reported in thousands and the lowest total steps ntot are bold. The sample reduction gained from ACP is reported in percent and computed as 1−nACPtot/nNP∣AP∣PIRLtot . The symbol − indicates that NP has not reached Gt=0tar in the given training budget and in this case sample reduction ACP vs. NP is defined as 100%.
Fig. 3 : Evaluation of the amount of expert and rollout steps.
The pretraining-finetuning paradigm has facilitated numerous transformative advancements in artificial intelligence research in recent years. However, in the domain of reinforcement learning (RL) for robot locomotion, individual skills are often learned from scratch despite the high likelihood that some generalizable knowledge is shared across all task-specific policies belonging to the same robot embodiment. This work aims to define a paradigm for pretraining neural network models that encapsulate such knowledge and can subsequently serve as a basis for warm-starting the RL process in classic actor-critic algorithms, such as Proximal Policy Optimization (PPO). We begin with a task-agnostic exploration-based data collection algorithm to gather diverse, dynamic transition data, which is then used to train a Proprioceptive Inverse Dynamics Model (PIDM) through supervised learning. The pretrained weights are then loaded into both the actor and critic networks to warm-start the policy optimization of actual tasks. We systematically validated our proposed method with 9 distinct robot locomotion RL environments comprising 3 different robot embodiments, showing significant benefits of this initialization strategy. Our proposed approach on average improves sample efficiency by 36.2% and task performance by 4.3% compared to random initialization. We further present key ablation studies and empirical analyses that shed light on the mechanisms behind the effectiveness of this method.
Jiale Fan, Andrei Cramariuc, Tifanny Portela +1
Section of Microengineering, EPFL, 1015 Lausanne, Switzerland · Robotic Systems Lab, ETH Zurich, 8092 Zürich, Switzerland · ETH AI Center, ETH Zurich, 8092 Zürich, Switzerland
For reinforcement learning in the real world online exploration is expensive A common practice in robotic reinforcement learning is to incorporate additional data to improve sample efficiency Expert demonstration data is often crucial for solving hard exploration tasks with sparse rewards While prior data is used to augment experience and pretrain models we show that the design of existing algorithms fails to achieve the sample efficiency that is possible in this setting due to a failure to use pretrained policies effectively We propose XQCfD which extends the sample-efficient XQC actor-critic to learn from demonstrations using augmented replay buffers pretrained policies and stationary policy architectures designed to avoid rapidly unlearning the strong initial policy like prior works We show our stationary network architecture enables policy improvement out-of-distribution better than standard network architectures due to its higher entropy predictions XQCfD achieves state of the art performance across a range of complex manipulation tasks with sparse rewards from the popular Adroit Robomimic and MimicGen benchmarks -- notably with a low update-to-data ratio and no ensemble networks
Daniel Palenicek, Florian Vogt, Joe Watson +3
Technical University of Darmstadt · KTH Royal Institute of Technology · University of Oxford +2
Offline-to-online (O2O) reinforcement learning aims to leverage policies pretrained on static datasets while improving them through online interaction. However, directly reusing an offline-trained critic can hinder online fine-tuning: as the policy and data distribution change rapidly, value estimates inherited from offline training may become misaligned with the online environment, leading to inaccurate policy improvement and inefficient exploration. To address this problem, we introduce Critic-Free Pretraining: an efficient paradigm that completely abandons the approach of offline critic training, allowing a freshly initialized critic to adapt without inheriting biased estimates. CFP is compatible with various mainstream O2O algorithms and consistently matches or improves upon conventional O2O algorithms across a diverse set of tasks, with particularly pronounced gains on several challenging tasks.