CABTO: Context-Aware Behavior Tree Grounding for Robot Manipulation

Synthesizing deployable robot behavior trees (BTs) from natural language (NL) requires grounding to ensure every generated BT references only skills a robot can actually execute. Existing LLM-based BT synthesis approaches often place this grounding responsibility on the prompt author. This makes deployment brittle when the author does not know which skills the robot can execute, how those skills are parameterized, or how the robot runtime software constrains valid BT structure. This paper proposes a contract-grounded BT synthesis architecture in which a coding agent queries a robot-side Model Context Protocol (MCP) server to retrieve an explicit contract consisting of a skill library, permitted BT operators, and optional BT composition templates, before synthesizing a BT for validation and execution. In our framework, non-expert operators issue NL commands without knowledge of robot implementation details, while a robot runtime validation gate enforces correctness before execution. We evaluate two LLMs, a closed model (Sonnet 4.6) and a smaller open-source model (Gemma4:31b), across 110 simulated tasks in PyRoboSim and 14 tasks on a physical Husarion Panther robot. Results show that contract grounding enables near-perfect BT validation and high task success, that BT composition templates substantially recover success on reactive control-flow tasks for the smaller model, and that the architecture transfers to physical hardware running a Nav2 stack opaque to both operator and agent.

Large language models have been widely used for robotic task planning, often taking advantage of representations such as Behavior Trees (BTs). Vision-Language Models (VLMs) have extended these works by grounding the generated plans in the observed scene. However, existing methods are either text-only or rely on large proprietary VLMs, while no dataset pairs visual observations and task instructions with executable and ROS2-compatible BTs. We address this gap with a multi-stage teacher pipeline that converts 1,622 Open X-Embodiment episodes into an augmented multimodal BT dataset containing 2,433 examples. We use this dataset to fine-tune compact and open-source VLMs, ranging from 500M to 4B parameters, using parameter-efficient fine-tuning (PEFT). We then evaluate the generated BTs offline in terms of syntactic correctness and by executing them on 15 household tasks in BEHAVIOR-1K. Our best model, Gemma-3 4B, achieves perfect BT validity and an 87% success rate, outperforming Claude Opus 4.8 and approaching GPT-5, while running locally. Finally, our ablation studies show that adding visual observations increases task success from 40% to 87%, while data augmentation increases BT validity from 65% to 100%.

Behavior Trees (BTs) are widely adopted for complex task execution in robotics, providing modular, reactive control but lacking formal guarantees. However, existing correct-by-construction synthesis from Linear Temporal Logic (LTL) cannot express quantitative timing constraints. This letter synthesizes correct-by-construction BTs from Signal Temporal Logic (STL) specifications. The workspace is modeled as a timed transition system and abstracted into a zone graph, and an augmented state space tracking both logical progress and timing constraints is introduced. A hierarchical fixed-point algorithm computes winning sets for an STL fragment encompassing safety, reachability, response, recurrence, and persistence, yielding BT subtrees with a runtime constraint function. Correctness guarantees are proven and complexity bounds are derived. Simulations demonstrate specification satisfaction with strictly positive robustness, and a physical quadrotor experiment with six STL specifications validates practical deployability.