Actor-Critic Pretraining for Proximal Policy Optimization

Andreas Kernbach; Amr Elsheikh; Nicolas Grupp; René Nagel; Marco F. Huber

cs.LGFeb 27, 2026

Actor-Critic Pretraining for Proximal Policy Optimization

Authors: Andreas Kernbach, Amr Elsheikh, Nicolas Grupp, René Nagel, Marco F. Huber

Organizations: Fraunhofer Institute for Manufacturing Engineering and Automation IPA, Stuttgart, Germany · Institute of Industrial Manufacturing and Management IFF, University of Stuttgart, Germany · Institute for Control Engineering of Machine Tools and Manufacturing Units ISW, University of Stuttgart, Germany · Materials Testing Institute MPA, Stuttgart, Germany

Abstract

Reinforcement learning (RL) actor-critic algorithms enable autonomous learning but often require a large number of environment interactions, which limits their applicability in robotics. Leveraging expert data can reduce the number of required environment interactions. A common approach is actor pretraining, where the actor network is initialized via behavioral cloning on expert demonstrations and subsequently fine-tuned with RL. In contrast, the initialization of the critic network has received little attention, despite its central role in policy optimization. This paper proposes a pretraining approach for actor-critic algorithms like Proximal Policy Optimization (PPO) that uses expert demonstrations to initialize both networks. The actor is pretrained via behavioral cloning, while the critic is pretrained using returns obtained from rollouts of the pretrained policy. The approach is evaluated on 15 simulated robotic manipulation and locomotion tasks. Experimental results show that actor-critic pretraining improves sample efficiency by 86.1% on average compared to no pretraining and by 30.9% to actor-only pretraining.

Figures & tables

Fig. 1 : Visualization of the pretraining and fine-tuning approach using expert data with an actor-critic model. First an expert policy is used to pretrain actor and critic, which are subsequently fine-tuned using PPO.

Panel of Figure 2 — Fig. 2 : Comparison of the learning results in PPO fine-tuning given ACP , AP and NP in three exemplary environments. Returns are averaged over three random seeds for fine-tuning, with shaded regions indicating the standard deviation.

Environment	NP		AP		PIRL			ACP			Sample Reduction
	$n^{\mathrm{NP}}_{\mathrm{PPO}}$	$n^{\mathrm{exp}}$	$n^{\mathrm{fine}}_{\mathrm{PPO}}$	$n^{\mathrm{tot}}_{\mathrm{AP}}$	$n^{\mathrm{fro}}_{\mathrm{PPO}}$	$n^{\mathrm{fine}}_{\mathrm{PPO}}$	$n^{\mathrm{tot}}_{\mathrm{PIRL}}$	$n^{\mathrm{rol}}$	$n^{\mathrm{fine}}_{\mathrm{PPO}}$	$n^{\mathrm{tot}}_{\mathrm{ACP}}$	NP	AP	PIRL
Walker2D	$-$	0.6	190.0	190.6	12.2	55.8	68.6	49.5	6.0	56.1	100%	70.6%	18.2%
Hopper	588.0	1.0	86.0	87.0	11.8	76.2	89.0	19.9	10.0	30.9	94.7%	64.5%	65.3%
Ant	544.0	1.0	190.0	191.0	55.3	100.7	157.0	1.7	96.0	98.7	81.9%	48.3%	37.1%
Humanoid	$-$	0.1	52.0	52.1	45.6	42.4	88.1	1.0	52.0	53.1	100%	-1.9%	39.7%
InvertedD.Pendulum	115.0	10.4	0.5	10.9	1.7	0.3	12.4	1.0	0.5	11.9	89.7%	-9.2%	4.0%

TABLE I : Comparison of required environment steps

n

for NP , AP , PIRL , and ACP with a fixed amount of expert steps

n^{\mathrm{exp}}

in order to reach

G_{t=0}^{\mathrm{tar}}

. All steps are reported in thousands and the lowest total steps

n^{\mathrm{tot}}

are bold. The sample reduction gained from ACP is reported in percent and computed as

1-n^{\mathrm{tot}}_{\mathrm{ACP}}/n^{\mathrm{tot}}_{\mathrm{NP|AP|PIRL}}

. The symbol

-

indicates that NP has not reached

G_{t=0}^{\mathrm{tar}}

in the given training budget and in this case sample reduction ACP vs. NP is defined as 100%.

Panel of Figure 4 — Fig. 3 : Evaluation of the amount of expert and rollout steps.

Robotic RL

Robotic reinforcement learning uses trial-and-error learning to help robots control, navigate, and manipulate objects.

See all 297 papers

Pretraining in Actor-Critic Reinforcement Learning for Locomotion
Oct 14, 2025Jiale Fan, Andrei Cramariuc, Tifanny Portela +1RL for Robotics Reinforcement Learning
Section of Microengineering, EPFL, 1015 Lausanne, Switzerland · Robotic Systems Lab, ETH Zurich, 8092 Zürich, Switzerland · ETH AI Center, ETH Zurich, 8092 Zürich, Switzerland
XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policies
May 11, 2026Daniel Palenicek, Florian Vogt, Joe Watson +3Robotic RL Learning from Demonstration
Technical University of Darmstadt · KTH Royal Institute of Technology · University of Oxford +2
Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning
Aug 11, 2026Daoyi Li, Yixian Zhang, Wenbo Ding +2Reinforcement Learning Critic-Free RL
Tsinghua University

Oct 14, 2025J/K move · Enter open · S save

Pretraining in Actor-Critic Reinforcement Learning for Locomotion

Jiale Fan, Andrei Cramariuc, Tifanny Portela, Marco Hutter

Section of Microengineering, EPFL, 1015 Lausanne, Switzerland · Robotic Systems Lab, ETH Zurich, 8092 Zürich, Switzerland · ETH AI Center, ETH Zurich, 8092 Zürich, Switzerland

RL for Robotics Reinforcement Learning Robot Locomotion Actor-Critic Methods

Open paper AskPDF arXiv

The pretraining-finetuning paradigm has facilitated numerous transformative advancements in artificial intelligence research in recent years. However, in the domain of reinforcement learning (RL) for robot locomotion, individual skills are often learned from scratch despite the high likelihood that some generalizable knowledge is shared across all task-specific policies belonging to the same robot embodiment. This work aims to define a paradigm for pretraining neural network models that encapsulate such knowledge and can subsequently serve as a basis for warm-starting the RL process in classic actor-critic algorithms, such as Proximal Policy Optimization (PPO). We begin with a task-agnostic exploration-based data collection algorithm to gather diverse, dynamic transition data, which is then used to train a Proprioceptive Inverse Dynamics Model (PIDM) through supervised learning. The pretrained weights are then loaded into both the actor and critic networks to warm-start the policy optimization of actual tasks. We systematically validated our proposed method with 9 distinct robot locomotion RL environments comprising 3 different robot embodiments, showing significant benefits of this initialization strategy. Our proposed approach on average improves sample efficiency by 36.2% and task performance by 4.3% compared to random initialization. We further present key ablation studies and empirical analyses that shed light on the mechanisms behind the effectiveness of this method.

Abstract

Robotic RL

Explore similar work

Pretraining in Actor-Critic Reinforcement Learning for Locomotion

XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policies

Critic-Free Pretraining for Efficient Online Reinforcement Learning Fine-Tuning