Learning from Imperfect Demonstrations via Temporal Behavior Tree-Guided Trajectory Repair
Authors: Aniruddh G. Puranic, Sebastian Schirmer, John S. Baras, Calin Belta
Organizations: Institute of Systems Research (ISR), University of Maryland, College Park, United States · DLR German Aerospace Center, Department of Unmanned Aircraft, Braunschweig, Germany
Learning robot control policies from demonstrations is a powerful paradigm, yet real-world data is often suboptimal, noisy, or otherwise imperfect, posing significant challenges for imitation and reinforcement learning. In this work, we present a formal framework that leverages Temporal Behavior Trees (TBT), an extension of Signal Temporal Logic (STL) with Behavior Tree semantics, to repair suboptimal trajectories prior to their use in downstream policy learning. Given demonstrations that violate a TBT specification, a model-based repair algorithm corrects trajectory segments to satisfy the formal constraints, yielding a dataset that is both logically consistent and interpretable. The repaired trajectories are then used to extract potential functions that shape the reward signal for reinforcement learning, guiding the agent toward task-consistent regions of the state space without requiring knowledge of the agent's kinematic model. We demonstrate the effectiveness of this framework on discrete grid-world navigation and continuous single and multi-agent reach-avoid tasks, highlighting its potential for data-efficient robot learning in settings where high-quality demonstrations cannot be assumed.
Figures & tables
Figure 1: Overview of the proposed framework. The TBT specification T governs both the repair of suboptimal demonstrations and the construction of the TBT monitor and reward potentials used during RL training.
Figure 2: Grid-World environment: (a) : Blue indicates start location at the top-left, red is the goal at the bottom-right, and the black cells are obstacles; (b) Example demonstrations that violate the TBT specification by visiting an unsafe cell. (c) Heatmaps of the extracted potential functions from the cleaned trajectories; (d) Comparison of the mean episodic rewards and success rates.
Figure 3: Reach-avoid navigation task: (a) A close-up of our experiment setup showing a mobile robot (indicated by a car-like object) tasked with navigating to a uniformly randomized 2D goal location (green cylinder); (b) Multi-agent extension of the single-agent environment. The robots are indicated by red and blue car-like objects and the goals are indicated by red and blue cylinders. (c)–(e) Each robot must avoid two types of obstacles: a traversable soft constraint (purple discs) and hard constraints that inhibit motion upon collision (cyan cubes). These hazards incur costs that the agent must minimize while learning to safely navigate to the goal.
Figure 4: Comparisons between the proposed framework and RL baselines in the Safety-Gymnasium reach-avoid tasks.
Hyperparameter
Grid-World
SG-Single
SG-Multi
Horizon
300
1000
1000
Learning rate
10−3
3×10−4
3×10−4
Discount factor γ
0.95
0.99
0.99
Target KL
−
0.2
0.2
Batch size
64
256
256
# Epochs
4
10
10
Table 1: PPO Hyperparameters for grid-world, safety-gymnasium single-agent (SG-Single), and multi-agent (SG-Multi). Unlisted hyperparameters are set to the default values in [ 19 ] ; the neural networks are all fully connected.