Learning Structured Robot Policies from Vision-Language Models via Synthetic Neuro-Symbolic Supervision
Authors: Alessandro Adami, Tommaso Tubaldo, Marco Todescato, Ruggero Carli, Pietro Falco
Organizations: University of Padova, Dept. of Information Engineering, Italy. · Polytechnic of Bari Dept. of Electrical and Information Engineering, Italy. · Fraunhofer Italia Research, 39100, Bozen, Italy.
Vision-Language Models (VLMs) have recently demonstrated strong capabilities in mapping multimodal observations to robot behaviors. However, most current approaches rely on end-to-end visuomotor policies that remain opaque and difficult to analyze, limiting their use in real-world robotic applications. In contrast, classical robotic systems often rely on structured policy representations that provide interpretability, modularity, and reactive execution. This work investigates how foundation models can be specialized to generate structured robot policies grounded in multimodal perception, bridging high-dimensional learning and symbolic control. We propose a neuro-symbolic approach in which a VLM synthesizes executable Behavior Tree policies from visual observations, natural language instructions, and structured system specifications. To enable scalable supervision without manual annotation, we introduce an automated pipeline that generates a synthetic multimodal dataset of domain-randomized scenes paired with instruction-policy examples produced by a foundation model. By decoupling structured task decomposition under constrained symbolic grammars from hardware-specific motor control, we demonstrate that a 12B-parameter model can learn structured spatial-symbolic mappings required for executable BT synthesis, solely through in-silico supervision. Real-world physical experiments on two heterogeneous robotic manipulators confirm that these structurally constrained policies achieve zero-shot transfer to real-world environments. The results emphasize that the data bottleneck in robotic planning can be bypassed by procedurally synthesizing high-fidelity, neuro-symbolic training data.
Figures & tables
Fig. 1: Overview of the proposed framework. First, high-fidelity simulation is used to procedurally generate domain-randomized synthetic scenes and structured metadata. A large foundation model then converts these observations into a synthetic multimodal supervision dataset of task instructions and executable Behavior Trees. This dataset is subsequently used to fine-tune a 12B-parameter vision-language model for constrained symbolic policy synthesis. At inference time, the fine-tuned model receives a real-world observation, a task instruction, and system specifications, and outputs a Behavior Tree representing an interpretable and reactive robot policy for task execution.
Primitive
Type
Functionality Description
MovePose
Action
Moves end-effector to a 6D goal pose.
OpenGripper
Action
Fully opens the parallel gripper fingers.
CloseGripper
Action
Closes the gripper until contact or limit.
MoveDown
Action
Moves vertically down.
is_at_pose
Condition
Verifies if the gripper is in a pose.
is_grasped
Condition
Checks if an object is in the gripper.
TABLE I: Library of Atomic Robotic Primitives.
Fig. 2: Examples of synthetic tabletop scenes used in dataset generation.
Element
Description and Role
Image ( I )
RGB rendering from MuJoCo, providing spatial context and object localization data (compacted as 224 × 224 during training phase).
Instruction ( U )
A human-centric command (e.g., ”Place the pink prism on the green box”) defining the high-level goal.
Role & Objective ( S )
Part of S , defining the agent as an expert interpreter tasked with generating BTs.
Node Library ( S )
A formal list of atomic Actions ( MovePose , OpenGripper ) and Conditions ( is_grasped ) available to the agent.
Metadata ( S )
A list of unique object identifiers (e.g., pink_prism_1 ) and home for the robot.
Safety Rules ( S )
Explicit instructions for mandatory z -offsets and the enforcement of a flat hierarchy.
TABLE II: Elements of the Multimodal Training Dataset.
ID
Module Map
Rank
Context (tokens)
Dropout
Learn. Rate
Dataset Size
λp
LOSS
PPL
PVU (%)
TL (h)
A1
heavy
16
4400
0.05
1⋅10−4
10k
1
0.605
1.831
96.506
15.12
A2
medium
16
4400
0.05
1⋅10−4
10k
1
0.615
1.850
98.580
12.92
A3
lite
16
4400
0.05
1⋅10−4
10k
1
0.6203
1.859
98.411
12.59
A4
heavy
8
4400
0.05
1⋅10−4
10k
1
0.608
1.837
98.529
15.18
A5
heavy
32
4400
0.05
1⋅10−4
10k
1
0.6117
1.844
95.741
15.12
A6
heavy
16
1200
0.05
1⋅10−4
10k
1
0.675
1.964
66.979
8.41
TABLE III: Training Metrics and Resource Utilization on NVIDIA A40. The reported model A1 is the one used for real-world tests and SOTA comparison.
Fig. 4: Example of real-world images, representing scenarios coherent with the synthetic dataset.
ID
TSR (%)
JVR (%)
KER (%)
TD
LC
ND
MEN
SC (%)
IL (s)
GPT-5 zero-shot
0
0
0
0
0
0
0
0
N.A.
GPT-5 one-shot
80
80
0
6.2
40.4
0.58
64.0
80
N.A.
Gemini 3 Flash zero-shot
0
0
0
0
0
0
0
0
N.A.
Gemini 3 Flash one-shot
100
100
0
6.2
33.6
0.56
52.2
100
N.A.
Pixtral-12B
0
0
0
0
0
0
0
0
52.80
A1
100
100
0
6.0
23.1
0.59
30.1
100
61.85
TABLE IV: Ablation Study: Impact of Hyperparameters on BT Synthesis Quality. All the models are on 20 different user requests U that are coherent with the training set (in distribution scenarios) and with real-world I , despite zero real-world training samples . A1-18 are fine-tuned specialists (10k synthetic training examples). GPT-5 and Gemini 3 Flash are prompted generalists with no task-specific training. Results are not directly comparable but illustrate the trade-off between fine-tuning cost and zero-shot structural compliance.
Fig. 5: Real-world experimental platforms used to validate hardware-agnostic BT execution: the Franka Emika Panda and the UR5e.
Fig. 6: Example of BT obtained asking to the trained model to ”Stack the yellow parallepiped on top of the blue parallelepiped”.
Fig. 7: Execution of the BT reported in Fig. 6 .
Fig. 8: Validation on unseen components. The images show the setup for the zero-shot generalization test. The model successfully identifies the specific parts and their relative positions, translating high-level user commands into executable Behavior Trees for tasks.
Fig. 9: Execution in the real world of an in-distribution task with out-of-distribution and unseen objects.
Department of Computer Science, Tufts University, Medford, MA, USA · Complex Dynamical Systems, Austrian Institute of Technology GmbH (AIT), Vienna, Austria