Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control

Siwei Ju; Jan Tauberschmidt; Oleg Arenz; Peter van Vliet; Jan Peters

cs.ROApr 3, 2026

Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control

Authors: Siwei Ju, Jan Tauberschmidt, Oleg Arenz, Peter van Vliet, Jan Peters

Organizations: Intelligent Autonomous Systems Lab, Department of Computer Science, TU Darmstadt, Germany. · Porsche AG, Stuttgart, Germany. · German Research Center for Artificial Intelligence (DFKI), Germany.

Abstract

Learning high-performance control policies that remain consistent with expert behavior is a fundamental challenge in robotics. Reinforcement learning can discover high-performing strategies but often departs from desirable human behavior, whereas imitation learning is limited by demonstration quality and struggles to improve beyond expert data. We propose a behavior-constrained reinforcement learning framework that improves beyond demonstrations while explicitly controlling deviation from expert behavior. Because expert-consistent behavior in dynamic control is inherently trajectory-level, we introduce a receding-horizon predictive mechanism that models short-term future trajectories and provides look-ahead rewards during training. To account for the natural variability of human behavior under disturbances and changing conditions, we further condition the policy on reference trajectories, allowing it to represent a distribution of expert-consistent behaviors rather than a single deterministic target. Empirically, we evaluate the approach in high-fidelity race car simulation using data from professional drivers, a domain characterized by extreme dynamics and narrow performance margins. The learned policies achieve competitive lap times while maintaining close alignment with expert driving behavior, outperforming baseline methods in both performance and imitation quality. Beyond standard benchmarks, we conduct human-grounded evaluation in a driver-in-the-loop simulator and show that the learned policies reproduce setup-dependent driving characteristics consistent with the feedback of top-class professional race drivers. These results demonstrate that our method enables learning high-performance control policies that are both optimal and behavior-consistent, and can serve as reliable surrogates for human decision-making in complex control systems.

Figures & tables

Fig. 1: Application workflow of the proposed method in professional motorsports. A learned policy acts as a behavior-consistent digital driver model for virtual setup evaluation in a driver-in-the-loop simulation environment.

Fig. 2: Overview of the policy structure with the components and their corresponding dimensions.

Fig. 3: A schematic diagram for reward terms.

exp	prediction	$\boldsymbol{r^{s}}$	$\boldsymbol{r^{p}}$	$\boldsymbol{\alpha^{s}}$	lap time (Track A)	MRO (Track A)
A ours	yes	$r^{s,t},r^{s,\psi}$	$r^{p,t},r^{p,\psi}$	$\hat{R}=0.009$	66.229 $\pm$ 0.0259	0.2891 $\pm$ 0.0052
B step	no	$r^{s,t}$	$r^{p,t}$	1	$66.961\pm 0.299$	$0.4547\pm 0.0320$
C adv	no	$r^{s,\text{ad}}$	$r^{p,t}$	0.1	66.775 $\pm$ 0.0143	$\textbf{0.1341}\pm\textbf{0.0035}$
D CRL	no	0	$r^{p,t}$	0	67.209 $\pm$ 0.275	$0.7686\pm 0.0098$

TABLE I: Experiment definition and results.

Panel of Figure 5 — Fig. 4: Learning curves of Lap Time and Mean Offset of Experiment B, C, and D, compared with ours. For lap time, we plot the minimum value over the seeds, as the objective is to optimize performance. For MRO, we plot the average value across the seeds to assess imitation quality. The shaded area in both plots represents the range covered the seeds, providing insight into the variability of training outcomes.

Fig. 5: The Pareto fronts for different methods. Since both lap time and Mean Reference Offset (MRO) need to be minimized, the optimal solutions lie in the lower-left corner of the plot. Each point represents a lap from the training epochs that lie on the Pareto front within its respective experiment group. For improved readability, we have omitted data points that are not on the Pareto front.

exp	lap time (B)	MRO (B)	lap time (C)	MRO (C)
A ours	66.892 $\pm$ 0.0256	0.2417 $\pm$ 0.0166	69.688 $\pm$ 0.0473	0.2989 $\pm$ 0.0109
B step	67.060 $\pm$ 0.0318	0.2743 $\pm$ 0.0208	69.318 $\pm$ 0.0510	0.7296 $\pm$ 0.0976
C adv	67.402 $\pm$ 0.0215	0.0789 $\pm$ 0.0054	69.822 $\pm$ 0.0365	0.1535 $\pm$ 0.0136
D CRL	67.458 $\pm$ 0.2242	0.7053 $\pm$ 0.0096	69.910 $\pm$ 0.1182	0.9703 $\pm$ 0.0101

TABLE II: Experiments for track B and C

Fig. 6: Single lap compared to the reference trajectory.

Fig. 7: Driving lines of agent laps and their corresponding reference trajectories.

exp	scaling	lap time	mean ref offset
S-I: fixed scaling	0.5	$\textbf{66.120}\pm 0.0216$	$0.3799\pm 0.0243$
S-II: fixed scaling	1.6	$66.226\pm 0.0631$	$\textbf{0.2732}\pm 0.0070$
S-III: auto scaling	auto	$\textbf{66.163}\pm 0.0229$	$\textbf{0.2629}\pm 0.0096$

TABLE III: Experiments for automatic scaling

Panel of Figure 11 — Fig. 8: Learning progress of experiment S-I, S-II und S-II, using reward coefficient $\alpha^{s}$ 0.5, 1.6 and automatic coefficient tuning with imitation target $\hat{R}=0.9$

exp	DA	data	lap time	MRO
DA-demos	no	train	$66.156\pm 0.035$	$0.2861\pm 0.0165$
DA-RTD	yes	train	$66.078\pm 0.040$	$0.2934\pm 0.0093$
DA-demos	no	test	$66.866\pm 0.0257$	$0.4279\pm 0.0176$
DA-RTD	yes	test	$\boldsymbol{66.639}\pm 0.0197$	$0.4587\pm 0.0067$

TABLE IV: Ablation experiment for data augmentation

exp	pred	reward	lap time	MRO
P I	no	$r^{p}$	$66.708\pm 0.0560$	$0.8820\pm 0.0586$
P II	yes	$r^{p}$	$\textbf{66.454}\pm\textbf{0.0379}$	$0.8000\pm 0.0608$
P III	no	$r^{p}+r^{s}$	$66.530\pm 0.0560$	$0.4984\pm 0.0690$
P IV	yes	$r^{p}+r^{s}$	$\textbf{66.316}\pm\textbf{0.0535}$	$\textbf{0.3512}\pm 0.0120$

TABLE V: Ablation experiments for prediction

Fig. 9: Lap time distributions across three vehicle setups. Each curve shows ranked lap times with linear trends indicating consistency.

Fig. 10: Time-series comparison of control and state variables for 10 laps from Setup 1 and Setup 2.

n_step	8192
num_env	8
$\gamma$	0.998
$\lambda$	0.98
entcoe	1e-5
learning rate	1e-4
activation	ReLU

TABLE VI: PPO Hyperparameters

$\lambda_{\psi}$	1
Bézier degree	5
n_control_point	10
learning rate	1e-3
predictor layers	[128, 128]

TABLE VII: Trajectory Predictor Parameters

$\alpha_{\psi}$	0.8
$\alpha_{d}$	0.01
scaling_learning	1e-4
scaling_init_value	0.5
$\hat{R}$ (auto scaling only)	0.009

TABLE VIII: Reward and automatic reward scaling

learning rate	1e-3
batchsize	128
$N_{pt}$	2,976,947
num_epoch	1
$\alpha_{\text{reg}}$	1
$\alpha_{\psi,\text{pt}}$	1e-6

TABLE IX: Pretrain Hyperparameters

$\boldsymbol{\alpha^{s}}$	lap time	MRO
1	66.443 $\pm$ 0.029	0.4159 $\pm$ 0.0468
0.8	66.673 $\pm$ 0.020	0.4108 $\pm$ 0.0108
0.6	67.530 $\pm$ 0.038	0.4984 $\pm$ 0.0690
0.4	66.543 $\pm$ 0.038	0.6428 $\pm$ 0.0296
0.2	66.560 $\pm$ 0.028	0.7093 $\pm$ 0.0401

TABLE X: Additional experiment result for Exp B

$\boldsymbol{\alpha^{s}}$	lap time	MRO
1	$68.588\pm 0.027$	$0.1407\pm 0.0134$
0.9	$68.638\pm 0.053$	$0.1519\pm 0.0097$
0.7	$68.496\pm 0.066$	$0.1371\pm 0.0097$
0.5	$67.960\pm 0.037$	$0.1335\pm 0.0166$
0.3	$67.630\pm 0.054$	$\textbf{0.1283}\pm 0.0073$
0.1	$66.796\pm 0.029$	$\textbf{0.1240}\pm 0.0058$

TABLE XI: Additional experiment result for Exp C

Fig. 11: Comparison of four different race car setups by ranking the driven laps by lap time. From Setup 1 to 4, the mechanical balance progressively shifts rearward. The horizontal lines represent the average lap time for each setup.

Constrained RL

Constrained reinforcement learning optimizes task performance while keeping safety risks, resource use, or other costs within explicit limits.

See all 220 papers

Disagreement-Regularized Imitation Learning for Image-Based Continuous Control with Gaussian and Beta Policies
Sep 29, 2026Irving Giovani Bronzatti Petrazzini, Eric Aislan AntoneloImitation Learning Behavior Cloning
Department of Automation and Systems Engineering, Federal University of Santa Catarina, Florian´opolis, Santa Catarina, Brazil.
TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels
Apr 17, 2025Siow Meng Low, Ze Gong, Akshat KumarRL Control Credit Assignment in RL
School of Computing and Information Systems, Singapore Management University, Singapore · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
Learning from Imperfect Demonstrations via Temporal Behavior Tree-Guided Trajectory Repair
Apr 5, 2026Aniruddh G. Puranic, Sebastian Schirmer, John S. Baras +1Robot Imitation Learning RL for Robotics
Institute of Systems Research (ISR), University of Maryland, College Park, United States · DLR German Aerospace Center, Department of Unmanned Aircraft, Braunschweig, Germany

Sep 29, 2026J/K move · Enter open · S save

Disagreement-Regularized Imitation Learning for Image-Based Continuous Control with Gaussian and Beta Policies

Irving Giovani Bronzatti Petrazzini, Eric Aislan Antonelo

Department of Automation and Systems Engineering, Federal University of Santa Catarina, Florian´opolis, Santa Catarina, Brazil.

Imitation Learning Behavior Cloning Visual RL Vision-Based Robot Control

Open paper AskPDF arXiv

Purpose: Behavior cloning can accumulate errors when a learned controller visits states outside the demonstrated distribution. This study evaluates whether Disagreement-Regularized Imitation Learning (DRIL), which converts disagreement among cloned policies into a reinforcement-learning reward, improves image-based continuous control. Methods: A controlled CarRacing study combines Gaussian and Beta learner policies, demonstrations from either a clipped Gaussian expert or an intrinsically bounded Beta expert, one or 20 trajectories, deterministic and stochastic evaluation, and three retained stages: behavior cloning, the highest 10-episode training-score checkpoint, and the final DRIL checkpoint. The disagreement ensemble contains five Gaussian policies in every variant. Each retained policy is evaluated over 100 procedurally generated episodes. Results: Score-selected DRIL produced its largest gains in the few-demonstration setting, improving over the strongest behavior-cloning mean by 61% with clipped-action demonstrations and by 112% with bounded-action demonstrations. With 20 trajectories, the advantage of DRIL narrowed; in the bounded-action regime, Beta behavior cloning remained about 7% above the best DRIL checkpoint. The experiments also show that the informativeness of the disagreement reward changes with the learner representation and training stage. Conclusion: DRIL can substantially improve few-demonstration visual continuous control, while bounded Beta policies provide strong behavior-cloning performance when more demonstrations are available. The results highlight the joint importance of learner support,ensemble response, and checkpoint selection.

Abstract

Constrained RL

Explore similar work

Disagreement-Regularized Imitation Learning for Image-Based Continuous Control with Gaussian and Beta Policies

TraCeS: Learning Per-Timestep Constraint-Violation Credit from Sparse Trajectory-Level Labels

Learning from Imperfect Demonstrations via Temporal Behavior Tree-Guided Trajectory Repair