Learning Structured Robot Policies from Vision-Language Models via Synthetic Neuro-Symbolic Supervision
Authors: Alessandro Adami, Tommaso Tubaldo, Marco Todescato, Ruggero Carli, Pietro Falco
Organizations: University of Padova, Dept. of Information Engineering, Italy. · Polytechnic of Bari Dept. of Electrical and Information Engineering, Italy. · Fraunhofer Italia Research, 39100, Bozen, Italy.
Vision-Language Models (VLMs) have recently demonstrated strong capabilities in mapping multimodal observations to robot behaviors. However, most current approaches rely on end-to-end visuomotor policies that remain opaque and difficult to analyze, limiting their use in real-world robotic applications. In contrast, classical robotic systems often rely on structured policy representations that provide interpretability, modularity, and reactive execution. This work investigates how foundation models can be specialized to generate structured robot policies grounded in multimodal perception, bridging high-dimensional learning and symbolic control. We propose a neuro-symbolic approach in which a VLM synthesizes executable Behavior Tree policies from visual observations, natural language instructions, and structured system specifications. To enable scalable supervision without manual annotation, we introduce an automated pipeline that generates a synthetic multimodal dataset of domain-randomized scenes paired with instruction-policy examples produced by a foundation model. By decoupling structured task decomposition under constrained symbolic grammars from hardware-specific motor control, we demonstrate that a 12B-parameter model can learn structured spatial-symbolic mappings required for executable BT synthesis, solely through in-silico supervision. Real-world physical experiments on two heterogeneous robotic manipulators confirm that these structurally constrained policies achieve zero-shot transfer to real-world environments. The results emphasize that the data bottleneck in robotic planning can be bypassed by procedurally synthesizing high-fidelity, neuro-symbolic training data.
Figures & tables
Fig. 1: Overview of the proposed framework. First, high-fidelity simulation is used to procedurally generate domain-randomized synthetic scenes and structured metadata. A large foundation model then converts these observations into a synthetic multimodal supervision dataset of task instructions and executable Behavior Trees. This dataset is subsequently used to fine-tune a 12B-parameter vision-language model for constrained symbolic policy synthesis. At inference time, the fine-tuned model receives a real-world observation, a task instruction, and system specifications, and outputs a Behavior Tree representing an interpretable and reactive robot policy for task execution.
Primitive
Type
Functionality Description
MovePose
Action
Moves end-effector to a 6D goal pose.
OpenGripper
Action
Fully opens the parallel gripper fingers.
CloseGripper
Action
Closes the gripper until contact or limit.
MoveDown
Action
Moves vertically down.
is_at_pose
Condition
Verifies if the gripper is in a pose.
is_grasped
Condition
Checks if an object is in the gripper.
TABLE I: Library of Atomic Robotic Primitives.
Fig. 2: Examples of synthetic tabletop scenes used in dataset generation.
Element
Description and Role
Image ( I )
RGB rendering from MuJoCo, providing spatial context and object localization data (compacted as 224 × 224 during training phase).
Instruction ( U )
A human-centric command (e.g., ”Place the pink prism on the green box”) defining the high-level goal.
Role & Objective ( S )
Part of S , defining the agent as an expert interpreter tasked with generating BTs.
Node Library ( S )
A formal list of atomic Actions ( MovePose , OpenGripper ) and Conditions ( is_grasped ) available to the agent.
Metadata ( S )
A list of unique object identifiers (e.g., pink_prism_1 ) and home for the robot.
Safety Rules ( S )
Explicit instructions for mandatory z -offsets and the enforcement of a flat hierarchy.
TABLE II: Elements of the Multimodal Training Dataset.
ID
Module Map
Rank
Context (tokens)
Dropout
Learn. Rate
Dataset Size
λp
LOSS
PPL
PVU (%)
TL (h)
A1
heavy
16
4400
0.05
1⋅10−4
10k
1
0.605
1.831
96.506
15.12
A2
medium
16
4400
0.05
1⋅10−4
10k
1
0.615
1.850
98.580
12.92
A3
lite
16
4400
0.05
1⋅10−4
10k
1
0.6203
1.859
98.411
12.59
A4
heavy
8
4400
0.05
1⋅10−4
10k
1
0.608
1.837
98.529
15.18
A5
heavy
32
4400
0.05
1⋅10−4
10k
1
0.6117
1.844
95.741
15.12
A6
heavy
16
1200
0.05
1⋅10−4
10k
1
0.675
1.964
66.979
8.41
TABLE III: Training Metrics and Resource Utilization on NVIDIA A40. The reported model A1 is the one used for real-world tests and SOTA comparison.
Fig. 4: Example of real-world images, representing scenarios coherent with the synthetic dataset.
ID
TSR (%)
JVR (%)
KER (%)
TD
LC
ND
MEN
SC (%)
IL (s)
GPT-5 zero-shot
0
0
0
0
0
0
0
0
N.A.
GPT-5 one-shot
80
80
0
6.2
40.4
0.58
64.0
80
N.A.
Gemini 3 Flash zero-shot
0
0
0
0
0
0
0
0
N.A.
Gemini 3 Flash one-shot
100
100
0
6.2
33.6
0.56
52.2
100
N.A.
Pixtral-12B
0
0
0
0
0
0
0
0
52.80
A1
100
100
0
6.0
23.1
0.59
30.1
100
61.85
TABLE IV: Ablation Study: Impact of Hyperparameters on BT Synthesis Quality. All the models are on 20 different user requests U that are coherent with the training set (in distribution scenarios) and with real-world I , despite zero real-world training samples . A1-18 are fine-tuned specialists (10k synthetic training examples). GPT-5 and Gemini 3 Flash are prompted generalists with no task-specific training. Results are not directly comparable but illustrate the trade-off between fine-tuning cost and zero-shot structural compliance.
Fig. 5: Real-world experimental platforms used to validate hardware-agnostic BT execution: the Franka Emika Panda and the UR5e.
Fig. 6: Example of BT obtained asking to the trained model to ”Stack the yellow parallepiped on top of the blue parallelepiped”.
Fig. 7: Execution of the BT reported in Fig. 6 .
Fig. 8: Validation on unseen components. The images show the setup for the zero-shot generalization test. The model successfully identifies the specific parts and their relative positions, translating high-level user commands into executable Behavior Trees for tasks.
Fig. 9: Execution in the real world of an in-distribution task with out-of-distribution and unseen objects.
While vision-language-action models have demonstrated impressive zero-shot manipulation capabilities, they remain fundamentally black box policies that are difficult to interpret, adapt, or correct when they inevitably fail. In this work, we propose ARCHITECT, a framework that treats robot policy acquisition as an interactive program synthesis task. ARCHITECT leverages the reasoning capabilities of LLM coding agents to synthesize modular robot programs that utilize a suite of perception and control tools. Unlike end-to-end models where distribution shift leads to unpredictable, cascading failures, our modular architecture allows users to isolate failures and localize feedback at the level of abstraction required. We introduce an iterative process where a human supervisor provides natural language corrections to steer the policy. These corrections are grounded in the policy code by program execution traces and distilled into a persistent skill library, a form of long-term in-context learning which enables the agent to accumulate a repertoire of reusable, interpretable behaviors. In a benchmark evaluation on a Franka Panda robot, ARCHITECT outperforms state-of-the-art VLA models and program synthesis baselines on complex, long-horizon tasks, including articulated object manipulation and cloth folding. Our results demonstrate that the synthesized skill library enables the system to transfer to novel tasks with decreasing human intervention, providing a steerable and data-efficient alternative to black-box robot learning. Website: https://robo-architect.github.io/
Daphne Chen, Archit Ritesh Jain, Eric Goossen +4
University of Washington · 2Microsoft Research · 3Massachusetts Institute of Technology
Scaling robot learning to long-horizon tasks remains a formidable challenge. While end-to-end policies often lack the structural priors needed for effective long-term reasoning, traditional neuro-symbolic methods rely heavily on hand-crafted symbolic priors. To address the issue, we introduce ENAP (Emergent Neural Automaton Policy), a framework that allows a bi-level neuro-symbolic policy adaptively emerge from visuomotor demonstrations. Specifically, we first employ adaptive clustering and an extension of the L* algorithm to infer a Mealy state machine from visuomotor data, which serves as an interpretable high-level planner capturing latent task modes. Then, this discrete structure guides a low-level reactive residual network to learn precise continuous control via behavior cloning (BC). By explicitly modeling the task structure with discrete transitions and continuous residuals, ENAP achieves high sample efficiency and interpretability without requiring task-specific labels. Extensive experiments on complex manipulation and long-horizon tasks demonstrate that ENAP outperforms state-of-the-art (SoTA) end-to-end VLA policies by up to 27% in low-data regimes, while offering a structured representation of robotic intent (Fig. 1).
Yiyuan Pan, Xusheng Luo, Hanjiang Hu +2
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA
Enabling robots to learn long-horizon manipulation tasks from a handful of demonstrations remains a central challenge in robotics. Existing neuro-symbolic approaches often rely on hand-crafted symbolic abstractions, semantically labeled trajectories or large demonstration datasets, limiting their scalability and real-world applicability. We present a scalable neuro-symbolic framework that autonomously constructs symbolic planning domains and data-efficient control policies from as few as one to thirty unannotated skill demonstrations, without requiring manual domain engineering. Our method segments demonstrations into skills and employs a Vision-Language Model (VLM) to classify skills and identify equivalent high-level states, enabling automatic construction of a state-transition graph. This graph is processed by an Answer Set Programming solver to synthesize a PDDL planning domain, which an oracle function exploits to isolate the minimal, task-relevant and target relative observation and action spaces for each skill policy. Policies are learned at the control reference level rather than at the raw actuator signal level, yielding a smoother and less noisy learning target. Known controllers can be leveraged for real-world data augmentation by projecting a single demonstration onto other objects in the scene, simultaneously enriching the graph construction process and the dataset for imitation learning. We validate our framework primarily on a real industrial forklift across statistically rigorous manipulation trials, and demonstrate cross-platform generality on a Kinova Gen3 robotic arm across two standard benchmarks. Our results show that grounding control learning, VLM-driven abstraction, and automated planning synthesis into a unified pipeline constitutes a practical path toward scalable, data-efficient, expert-free and interpretable neuro-symbolic robotics.
Pierrick Lorang, Johannes Huemer, Timothy Duggan +3
Department of Computer Science, Tufts University, Medford, MA, USA · Complex Dynamical Systems, Austrian Institute of Technology GmbH (AIT), Vienna, Austria