BT-Agent: Deploying Onboard Robot Agents with Behavior Trees and Multi-Fidelity Evaluation

Fabio Hübel; Adam Lilja; Yue Yin; Siming He; Claire Tomlin; Shankar Sastry; Jonas Frey; Marco Pavone

cs.ROOct 9, 2026

BT-Agent: Deploying Onboard Robot Agents with Behavior Trees and Multi-Fidelity Evaluation

Authors: Fabio Hübel, Adam Lilja, Yue Yin, Siming He, Claire Tomlin, Shankar Sastry, Jonas Frey, Marco Pavone

Organizations: Stanford University · University of California, Berkeley · NVIDIA

Abstract

Deploying language-model agents onboard mobile robots requires task reasoning and responsive execution within limited compute and memory. We propose BT-Agent, an onboard architecture combining LLM-driven task decisions with predefined behavior-tree (BT) skills. These skills enforce execution conditions and return structured failure feedback. We demonstrate fully onboard operation with a 9B model on a quadruped across five task categories. Intervention tests show that local checks stop ongoing navigation without waiting for another model decision, while the agent remains active to retry or report constraints. To compare agent designs efficiently, we introduce a 161-task multi-fidelity benchmark, evaluated in text on all tasks and in Habitat on 39. Text supports 10.9 $\times$ higher evaluation throughput, while simulation reveals how perception and motion failures change task outcomes. BT-Agent's largest success gains over the strongest Act/ReAct baseline occur at 4B and 9B: approximately 10 and 5 percentage points. Ablations identify structured tool calls and informative failure feedback as the most useful components.

Figures & tables

	BT planning / execution					LLM robot agents								Onboard
Feature	Wake et al. [ 10 ]	Ahmad et al. [ 12 ]	BTGen- Bot-2 (ER) [ 11 ]	Contract- grounded BT [ 13 ]	PA-BT [ 9 ]	SayCan [ 2 ]	Inner Monologue [ 3 ]	CaP- Agent0 [ 5 ]	ROSClaw [ 16 ]	Thea [ 14 ]	Robo- Harness [ 15 ]	VLAs-as- Tools [ 22 ]	Cortex [ 23 ]	PRISM [ 17 ]	Ours
Skill contracts	$(\checkmark)$	$\checkmark$	$(\checkmark)$	$(\checkmark)$	$\checkmark$	$(\checkmark)$	$(\checkmark)$	$(\checkmark)$	$(\checkmark)$	$\checkmark$	$(\checkmark)$	$(\checkmark)$	$(\checkmark)$	$(\checkmark)$	$\checkmark$
BT	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\times$	$\checkmark$
Guards	$(\checkmark)$	$(\checkmark)$	$(\checkmark)$	$(\checkmark)$	$\checkmark$	$\times$	$\times$	$\times$	$\checkmark$	$\checkmark$	$\times$	$(\checkmark)$	$(\checkmark)$	$(\checkmark)$	$\checkmark$
Agentic	$(\checkmark)$	$\checkmark$	$\checkmark$	$\times$	–	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$(\checkmark)$	$\checkmark$
Recovery	$(\checkmark)$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	$\checkmark$

TABLE I: Features of cited configurations. Skill contracts: declared preconditions and per-call result/status semantics (partial: incomplete contract); Guards: enforced constraints; Agentic: feedback-driven model decisions; Recovery: post-failure correction; Onboard: robot-borne inference.

\checkmark

: reported;

(\checkmark)

: partial;

\times

: not reported/not included; –: not applicable. Partial guards use interface validation or model-mediated checks; PRISM varies by planner.

Fig. 2: Physics-aligned task construction. Simulator geometry and camera records produce scene memory, navigation graphs, and task definitions. Habitat and Isaac tasks receive text counterparts; graded simulator evaluation covers 39 primary Habitat tasks. Instance masks are construction evidence, not VLM input.

Skill	Inputs	Effect
query	description, exclusions	look up objects in scene memory
surround_ survey	target	inspect five cameras; reveal visible objects and additional scene information
navigate	object, viewpoint	move to a stored object viewpoint
capture	object, camera	record photographic evidence
press	object	activate a reachable interaction target
push	object	move an object toward its configured target

TABLE II: Benchmark skill semantics. Blue: context; orange: physical; green: terminal.

Example instruction	Required evidence
Photograph something used to seat a child for a meal.	New image of an appropriate seat.
Find the stool near the round mirror and photograph it.	Survey reveals the target; new image.
Photograph the bench cushion and report the word printed on it.	New image and a report of the word.
Photograph every black leather-and-brass bar chair; report how many.	Three distinct captures and a report of three.
Find the kayak for the lake outing.	Report that the kayak cannot be found.

TABLE III: Example instructions and required evidence (shortened). The first four are from the additional home; the last is from the primary suite.

Fig. 3: Shared safe task across physics and text. Physical views combine an interaction excerpt with scripted survey and capture. The map route is schematic; the capture is displayed upright with an enlarged target.

Fault group	Injected behavior
Localization	Stale or missing pose; unreported position jump.
Navigation progress	No movement, timeout, or arrival short of the target.
Route obstruction	Reported blockage, temporary wedge, disabled edge, or unavailable route.
Viewpoint	Incorrect viewpoint despite reported arrival.
Interaction	Press stops short without activating the target.
Perception	Capture analysis fails to produce evidence.

TABLE IV: Mechanisms in the text fault schedule, grouped by effect. Faults fire only on eligible skill calls.

	4B			9B			27B
	O6	A8	All	O6	A8	All	O6	A8	All
Raw privileged API
CaP-X single	17.5	8.1	12.1	26.7	2.5	12.9	44.2	47.5	46.1
CaP-X multi	7.5	11.9	10.0	16.7	4.4	9.6	50.8	55.6	53.6
Shared skill library
CaP-X single	16.7	4.4	9.6	32.5	36.9	35.0	42.5	45.6	44.3

TABLE VI: Robosuite SR (%), 20 repetitions/task. O6: six CaP-X reference tasks; A8: eight extensions; All: fourteen tasks. CaP-X uses single- or multi-turn programs over its privileged API or shared skills, without an action-count cap. Bold/underline: best/second per column, including ties.

Fig. 4: Text–Habitat comparison: 39 tasks, 117 episodes/method. (a,b) Bold: highest observed; gray $\Delta$ : Habitat minus text (percentage points). (c) Descriptive association of task means over four methods and three seeds in two configurations of one home; marker size: coincident tasks; $\rho$ : Spearman correlation. (d) Pooled paired outcomes: 168/293 text successes also succeed in Habitat.

Fig. 5: Onboard deployment across five tasks. Left: task locations with the long-horizon trajectory and capture points. HOME: start/return pose; d 1 /d 2 : yellow panel/chest. Right: recorded observations, shortened prompts, and operator-adjudicated successes over three trials/task.

Fig. 6: Oracle mean time and agent throughput on 16 Habitat tasks; logarithmic axes. Throughput excludes timeouts and ends at the last non-timeout completion. Habitat uses one RTX 5090, with an asynchronously batched 9B VLM on one RTX PRO 6000. Triangle: serial real-robot execution, excluding setup, charging, and debugging.

Fig. 7: Reactive stopping across nine interventions. Commanded velocity reaches zero a median 19 ms after local acceptance; measured speed reflects subsequent locomotion control and physical settling. Battery trials also include sitting down.

Explore similar work

Multimodal Behavior Tree Generation: A Small Vision-Language Model for Robot Task Planning
Mar 6, 2026Riccardo Andrea Izzo, Cristiano Battistini, Gianluca Bardaro +1Large Language Model-Based Robot Planning Synthetic Data Generation for Robotics
Department of Electronics, Information, and Bioengineering, Politecnico di Milano, Milan, Italy
Contract-Grounded Behavior Tree Synthesis via Coding Agents
Jul 13, 2026Jonathan Salfity, Robert Blake Anderson, Mitch PryorLarge Language Model-Based Robot Planning LLM-Based Program Synthesis
Nuclear and Applied Robotics Group, Department of Mechanical Engineering, The University of Texas at Austin
MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Models
Jun 3, 2026Zhichao Yang, Yuanze Hu, Haojie Hao +7Mobile GUI Automation Latent Visual Reasoning
Beihang University · Northwestern Polytechnical University · Institute of Software, Chinese Academy of Sciences +2

Mar 6, 2026, cs.ROJ/K move · Enter open · S save

Multimodal Behavior Tree Generation: A Small Vision-Language Model for Robot Task Planning

Riccardo Andrea Izzo, Cristiano Battistini, Gianluca Bardaro, Matteo Matteucci

Department of Electronics, Information, and Bioengineering, Politecnico di Milano, Milan, Italy

Large Language Model-Based Robot Planning Synthetic Data Generation for Robotics VLMs for Robotics Behavior Trees

Open paper AskPDF arXiv

Large language models have been widely used for robotic task planning, often taking advantage of representations such as Behavior Trees (BTs). Vision-Language Models (VLMs) have extended these works by grounding the generated plans in the observed scene. However, existing methods are either text-only or rely on large proprietary VLMs, while no dataset pairs visual observations and task instructions with executable and ROS2-compatible BTs. We address this gap with a multi-stage teacher pipeline that converts 1,622 Open X-Embodiment episodes into an augmented multimodal BT dataset containing 2,433 examples. We use this dataset to fine-tune compact and open-source VLMs, ranging from 500M to 4B parameters, using parameter-efficient fine-tuning (PEFT). We then evaluate the generated BTs offline in terms of syntactic correctness and by executing them on 15 household tasks in BEHAVIOR-1K. Our best model, Gemma-3 4B, achieves perfect BT validity and an 87% success rate, outperforming Claude Opus 4.8 and approaching GPT-5, while running locally. Finally, our ablation studies show that adding visual observations increases task success from 40% to 87%, while data augmentation increases BT validity from 65% to 100%.

Abstract

Explore similar work

Multimodal Behavior Tree Generation: A Small Vision-Language Model for Robot Task Planning

Contract-Grounded Behavior Tree Synthesis via Coding Agents

MIRAGE: Mobile Agents with Implicit Reasoning and Generative World Models