Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control
Authors: Siwei Ju, Jan Tauberschmidt, Oleg Arenz, Peter van Vliet, Jan Peters
Organizations: Intelligent Autonomous Systems Lab, Department of Computer Science, TU Darmstadt, Germany. · Porsche AG, Stuttgart, Germany. · German Research Center for Artificial Intelligence (DFKI), Germany.
Learning high-performance control policies that remain consistent with expert behavior is a fundamental challenge in robotics. Reinforcement learning can discover high-performing strategies but often departs from desirable human behavior, whereas imitation learning is limited by demonstration quality and struggles to improve beyond expert data. We propose a behavior-constrained reinforcement learning framework that improves beyond demonstrations while explicitly controlling deviation from expert behavior. Because expert-consistent behavior in dynamic control is inherently trajectory-level, we introduce a receding-horizon predictive mechanism that models short-term future trajectories and provides look-ahead rewards during training. To account for the natural variability of human behavior under disturbances and changing conditions, we further condition the policy on reference trajectories, allowing it to represent a distribution of expert-consistent behaviors rather than a single deterministic target. Empirically, we evaluate the approach in high-fidelity race car simulation using data from professional drivers, a domain characterized by extreme dynamics and narrow performance margins. The learned policies achieve competitive lap times while maintaining close alignment with expert driving behavior, outperforming baseline methods in both performance and imitation quality. Beyond standard benchmarks, we conduct human-grounded evaluation in a driver-in-the-loop simulator and show that the learned policies reproduce setup-dependent driving characteristics consistent with the feedback of top-class professional race drivers. These results demonstrate that our method enables learning high-performance control policies that are both optimal and behavior-consistent, and can serve as reliable surrogates for human decision-making in complex control systems.
Figures & tables
Fig. 1: Application workflow of the proposed method in professional motorsports. A learned policy acts as a behavior-consistent digital driver model for virtual setup evaluation in a driver-in-the-loop simulation environment.
Fig. 2: Overview of the policy structure with the components and their corresponding dimensions.
Fig. 3: A schematic diagram for reward terms.
exp
prediction
rs
rp
αs
lap time (Track A)
MRO (Track A)
A ours
yes
rs,t,rs,ψ
rp,t,rp,ψ
R^=0.009
66.229 ± 0.0259
0.2891 ± 0.0052
B step
no
rs,t
rp,t
1
66.961±0.299
0.4547±0.0320
C adv
no
rs,ad
rp,t
0.1
66.775 ± 0.0143
0.1341±0.0035
D CRL
no
0
rp,t
0
67.209 ± 0.275
0.7686±0.0098
TABLE I: Experiment definition and results.
Fig. 4: Learning curves of Lap Time and Mean Offset of Experiment B, C, and D, compared with ours. For lap time, we plot the minimum value over the seeds, as the objective is to optimize performance. For MRO, we plot the average value across the seeds to assess imitation quality. The shaded area in both plots represents the range covered the seeds, providing insight into the variability of training outcomes.
Fig. 5: The Pareto fronts for different methods. Since both lap time and Mean Reference Offset (MRO) need to be minimized, the optimal solutions lie in the lower-left corner of the plot. Each point represents a lap from the training epochs that lie on the Pareto front within its respective experiment group. For improved readability, we have omitted data points that are not on the Pareto front.
exp
lap time (B)
MRO (B)
lap time (C)
MRO (C)
A ours
66.892 ± 0.0256
0.2417 ± 0.0166
69.688 ± 0.0473
0.2989 ± 0.0109
B step
67.060 ± 0.0318
0.2743 ± 0.0208
69.318 ± 0.0510
0.7296 ± 0.0976
C adv
67.402 ± 0.0215
0.0789 ± 0.0054
69.822 ± 0.0365
0.1535 ± 0.0136
D CRL
67.458 ± 0.2242
0.7053 ± 0.0096
69.910 ± 0.1182
0.9703 ± 0.0101
TABLE II: Experiments for track B and C
Fig. 6: Single lap compared to the reference trajectory.
Fig. 7: Driving lines of agent laps and their corresponding reference trajectories.
exp
scaling
lap time
mean ref offset
S-I: fixed scaling
0.5
66.120±0.0216
0.3799±0.0243
S-II: fixed scaling
1.6
66.226±0.0631
0.2732±0.0070
S-III: auto scaling
auto
66.163±0.0229
0.2629±0.0096
TABLE III: Experiments for automatic scaling
Fig. 8: Learning progress of experiment S-I, S-II und S-II, using reward coefficient αs 0.5, 1.6 and automatic coefficient tuning with imitation target R^=0.9
exp
DA
data
lap time
MRO
DA-demos
no
train
66.156±0.035
0.2861±0.0165
DA-RTD
yes
train
66.078±0.040
0.2934±0.0093
DA-demos
no
test
66.866±0.0257
0.4279±0.0176
DA-RTD
yes
test
66.639±0.0197
0.4587±0.0067
TABLE IV: Ablation experiment for data augmentation
exp
pred
reward
lap time
MRO
P I
no
rp
66.708±0.0560
0.8820±0.0586
P II
yes
rp
66.454±0.0379
0.8000±0.0608
P III
no
rp+rs
66.530±0.0560
0.4984±0.0690
P IV
yes
rp+rs
66.316±0.0535
0.3512±0.0120
TABLE V: Ablation experiments for prediction
Fig. 9: Lap time distributions across three vehicle setups. Each curve shows ranked lap times with linear trends indicating consistency.
Fig. 10: Time-series comparison of control and state variables for 10 laps from Setup 1 and Setup 2.
n_step
8192
num_env
8
γ
0.998
λ
0.98
entcoe
1e-5
learning rate
1e-4
activation
ReLU
TABLE VI: PPO Hyperparameters
λψ
1
Bézier degree
5
n_control_point
10
learning rate
1e-3
predictor layers
[128, 128]
TABLE VII: Trajectory Predictor Parameters
αψ
0.8
αd
0.01
scaling_learning
1e-4
scaling_init_value
0.5
R^ (auto scaling only)
0.009
TABLE VIII: Reward and automatic reward scaling
learning rate
1e-3
batchsize
128
Npt
2,976,947
num_epoch
1
αreg
1
αψ,pt
1e-6
TABLE IX: Pretrain Hyperparameters
αs
lap time
MRO
1
66.443 ± 0.029
0.4159 ± 0.0468
0.8
66.673 ± 0.020
0.4108 ± 0.0108
0.6
67.530 ± 0.038
0.4984 ± 0.0690
0.4
66.543 ± 0.038
0.6428 ± 0.0296
0.2
66.560 ± 0.028
0.7093 ± 0.0401
TABLE X: Additional experiment result for Exp B
αs
lap time
MRO
1
68.588±0.027
0.1407±0.0134
0.9
68.638±0.053
0.1519±0.0097
0.7
68.496±0.066
0.1371±0.0097
0.5
67.960±0.037
0.1335±0.0166
0.3
67.630±0.054
0.1283±0.0073
0.1
66.796±0.029
0.1240±0.0058
TABLE XI: Additional experiment result for Exp C
Fig. 11: Comparison of four different race car setups by ranking the driven laps by lap time. From Setup 1 to 4, the mechanical balance progressively shifts rearward. The horizontal lines represent the average lap time for each setup.
Purpose: Behavior cloning can accumulate errors when a learned controller visits states outside the demonstrated distribution. This study evaluates whether Disagreement-Regularized Imitation Learning (DRIL), which converts disagreement among cloned policies into a reinforcement-learning reward, improves image-based continuous control. Methods: A controlled CarRacing study combines Gaussian and Beta learner policies, demonstrations from either a clipped Gaussian expert or an intrinsically bounded Beta expert, one or 20 trajectories, deterministic and stochastic evaluation, and three retained stages: behavior cloning, the highest 10-episode training-score checkpoint, and the final DRIL checkpoint. The disagreement ensemble contains five Gaussian policies in every variant. Each retained policy is evaluated over 100 procedurally generated episodes. Results: Score-selected DRIL produced its largest gains in the few-demonstration setting, improving over the strongest behavior-cloning mean by 61% with clipped-action demonstrations and by 112% with bounded-action demonstrations. With 20 trajectories, the advantage of DRIL narrowed; in the bounded-action regime, Beta behavior cloning remained about 7% above the best DRIL checkpoint. The experiments also show that the informativeness of the disagreement reward changes with the learner representation and training stage. Conclusion: DRIL can substantially improve few-demonstration visual continuous control, while bounded Beta policies provide strong behavior-cloning performance when more demonstrations are available. The results highlight the joint importance of learner support,ensemble response, and checkpoint selection.
Irving Giovani Bronzatti Petrazzini, Eric Aislan Antonelo
Department of Automation and Systems Engineering, Federal University of Santa Catarina, Florian´opolis, Santa Catarina, Brazil.
Ensuring safe behavior in reinforcement learning (RL) is challenging when safety constraints are implicit and cannot be densely measured. In many settings, supervision is limited to coarse approvals or rejections of whole trajectories (e.g., whether a rollout remained within an unknown safety threshold). We propose TraCeS (Trajectory-based Constraint Estimation for Safety), a method for learning per-timestep violation credit from such sparse trajectory-level labels. TraCeS trains a sequential violation estimator whose per-step credits factorize the predicted probability that a trajectory has not yet violated the constraint, and integrates this learned signal into constrained policy optimization. The method requires neither a known cost function nor a known threshold, and remains compatible with standard continuous-control algorithms. We provide a theoretical analysis of the approximation gap introduced by the learning objective, and demonstrate empirically that TraCeS improves constraint satisfaction and feedback efficiency over baselines across multiple continuous-control benchmarks, including long-horizon tasks and settings with noisy or inconsistent labels.
Siow Meng Low, Ze Gong, Akshat Kumar
School of Computing and Information Systems, Singapore Management University, Singapore · Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
Learning robot control policies from demonstrations is a powerful paradigm, yet real-world data is often suboptimal, noisy, or otherwise imperfect, posing significant challenges for imitation and reinforcement learning. In this work, we present a formal framework that leverages Temporal Behavior Trees (TBT), an extension of Signal Temporal Logic (STL) with Behavior Tree semantics, to repair suboptimal trajectories prior to their use in downstream policy learning. Given demonstrations that violate a TBT specification, a model-based repair algorithm corrects trajectory segments to satisfy the formal constraints, yielding a dataset that is both logically consistent and interpretable. The repaired trajectories are then used to extract potential functions that shape the reward signal for reinforcement learning, guiding the agent toward task-consistent regions of the state space without requiring knowledge of the agent's kinematic model. We demonstrate the effectiveness of this framework on discrete grid-world navigation and continuous single and multi-agent reach-avoid tasks, highlighting its potential for data-efficient robot learning in settings where high-quality demonstrations cannot be assumed.
Aniruddh G. Puranic, Sebastian Schirmer, John S. Baras +1
Institute of Systems Research (ISR), University of Maryland, College Park, United States · DLR German Aerospace Center, Department of Unmanned Aircraft, Braunschweig, Germany