Learning from Imperfect Demonstrations via Temporal Behavior Tree-Guided Trajectory Repair
Authors: Aniruddh G. Puranic, Sebastian Schirmer, John S. Baras, Calin Belta
Organizations: Institute of Systems Research (ISR), University of Maryland, College Park, United States · DLR German Aerospace Center, Department of Unmanned Aircraft, Braunschweig, Germany
Learning robot control policies from demonstrations is a powerful paradigm, yet real-world data is often suboptimal, noisy, or otherwise imperfect, posing significant challenges for imitation and reinforcement learning. In this work, we present a formal framework that leverages Temporal Behavior Trees (TBT), an extension of Signal Temporal Logic (STL) with Behavior Tree semantics, to repair suboptimal trajectories prior to their use in downstream policy learning. Given demonstrations that violate a TBT specification, a model-based repair algorithm corrects trajectory segments to satisfy the formal constraints, yielding a dataset that is both logically consistent and interpretable. The repaired trajectories are then used to extract potential functions that shape the reward signal for reinforcement learning, guiding the agent toward task-consistent regions of the state space without requiring knowledge of the agent's kinematic model. We demonstrate the effectiveness of this framework on discrete grid-world navigation and continuous single and multi-agent reach-avoid tasks, highlighting its potential for data-efficient robot learning in settings where high-quality demonstrations cannot be assumed.
Figures & tables
Figure 1: Overview of the proposed framework. The TBT specification T governs both the repair of suboptimal demonstrations and the construction of the TBT monitor and reward potentials used during RL training.
Figure 2: Grid-World environment: (a) : Blue indicates start location at the top-left, red is the goal at the bottom-right, and the black cells are obstacles; (b) Example demonstrations that violate the TBT specification by visiting an unsafe cell. (c) Heatmaps of the extracted potential functions from the cleaned trajectories; (d) Comparison of the mean episodic rewards and success rates.
Figure 3: Reach-avoid navigation task: (a) A close-up of our experiment setup showing a mobile robot (indicated by a car-like object) tasked with navigating to a uniformly randomized 2D goal location (green cylinder); (b) Multi-agent extension of the single-agent environment. The robots are indicated by red and blue car-like objects and the goals are indicated by red and blue cylinders. (c)–(e) Each robot must avoid two types of obstacles: a traversable soft constraint (purple discs) and hard constraints that inhibit motion upon collision (cyan cubes). These hazards incur costs that the agent must minimize while learning to safely navigate to the goal.
Figure 4: Comparisons between the proposed framework and RL baselines in the Safety-Gymnasium reach-avoid tasks.
Hyperparameter
Grid-World
SG-Single
SG-Multi
Horizon
300
1000
1000
Learning rate
10−3
3×10−4
3×10−4
Discount factor γ
0.95
0.99
0.99
Target KL
−
0.2
0.2
Batch size
64
256
256
# Epochs
4
10
10
Table 1: PPO Hyperparameters for grid-world, safety-gymnasium single-agent (SG-Single), and multi-agent (SG-Multi). Unlisted hyperparameters are set to the default values in [ 19 ] ; the neural networks are all fully connected.
This paper introduces an interpretation of Temporal Behavior Trees (TBTs) as Reward-Petri-Nets (RPNs) for reinforcement learning (RL). Designing reward functions for complex, long-horizon robotic tasks is notoriously difficult, especially when tasks have hierarchical structure and temporal constraints. TBTs extend conventional behavior trees (BTs) used in robotic applications by incorporating temporal properties into their leaf nodes. This allows TBTs to represents not only the behavioral task structure defined by BT operators such as Sequence, Fallback, and Parallel, but also the task's temporal constraints. In this work, the constraints are specified in the leaf nodes using Linear Temporal Logic. In order to inform RL rewards using TBTs, we provide a translation from TBT into a Petri Net (PN) and show how rewards can be automatically assigned based on the TBT's structure, resulting in a RPN. In a series of increasingly challenging environments, we demonstrate how TBT-based rewards enable learning where vanilla RL fails, improve sample efficiency, and offer flexible, intuitive control over the learning progress. We showcase the learning impact by using different reward distribution schemes and TBT structures.
Till Schmeil, Günther Waxenegger-Wilfing, Sebastian Schirmer
University of Würzburg, Germany · German Aerospace Center (DLR), Germany
Executing Linear Temporal Logic (LTL) instructions with learned robot policies faces two practical challenges: demonstrations may cover individual behaviors without containing the temporal compositions requested at deployment, and semantic success predicates often provide little useful motor guidance through quantitative robustness. We address these challenges by separating behavior learning from temporal composition. From the same offline demonstrations, we train an unconditioned multimodal diffusion policy and a semantic predictor that estimates which semantic outcomes are likely to follow a proposed action sequence. These predictions provide a learned guidance signal without requiring hand-designed robustness measures for semantic task objectives. At deployment, an LTL automaton tracks instruction progress and scores the policy's proposed actions according to their predicted semantic outcomes. Neither learned component receives the specification during training, allowing demonstrated behaviors to be reused in longer, previously unseen temporal compositions without retraining. For additional runtime safety constraints with informative continuous margins, an optional robustness-based controller locally refines the selected actions. Experiments in a navigation environment, CALVIN manipulation, and on a real robot demonstrate reliable execution of complex temporal instructions and additional safety constraints, with substantial improvements over temporal-logic baselines.
Moritz Zoellner, Anastasios Manganaris, Ahmed H. Qureshi +1
Behavior Trees (BTs) are widely adopted for complex task execution in robotics, providing modular, reactive control but lacking formal guarantees. However, existing correct-by-construction synthesis from Linear Temporal Logic (LTL) cannot express quantitative timing constraints. This letter synthesizes correct-by-construction BTs from Signal Temporal Logic (STL) specifications. The workspace is modeled as a timed transition system and abstracted into a zone graph, and an augmented state space tracking both logical progress and timing constraints is introduced. A hierarchical fixed-point algorithm computes winning sets for an STL fragment encompassing safety, reachability, response, recurrence, and persistence, yielding BT subtrees with a runtime constraint function. Correctness guarantees are proven and complexity bounds are derived. Simulations demonstrate specification satisfaction with strictly positive robustness, and a physical quadrotor experiment with six STL specifications validates practical deployability.
Jiaheng Dong, Jingyi Huang, Liang Han
Sino-French Engineer School,Beihang University, Beijing 100191, China