Build on Priors: Vision--Language--Guided Neuro-Symbolic Imitation Learning for Data-Efficient Real-World Robot Manipulation
Authors: Pierrick Lorang, Johannes Huemer, Timothy Duggan, Kai Goebel, Patrik Zips, Matthias Scheutz
Organizations: Department of Computer Science, Tufts University, Medford, MA, USA · Complex Dynamical Systems, Austrian Institute of Technology GmbH (AIT), Vienna, Austria
Enabling robots to learn long-horizon manipulation tasks from a handful of demonstrations remains a central challenge in robotics. Existing neuro-symbolic approaches often rely on hand-crafted symbolic abstractions, semantically labeled trajectories or large demonstration datasets, limiting their scalability and real-world applicability. We present a scalable neuro-symbolic framework that autonomously constructs symbolic planning domains and data-efficient control policies from as few as one to thirty unannotated skill demonstrations, without requiring manual domain engineering. Our method segments demonstrations into skills and employs a Vision-Language Model (VLM) to classify skills and identify equivalent high-level states, enabling automatic construction of a state-transition graph. This graph is processed by an Answer Set Programming solver to synthesize a PDDL planning domain, which an oracle function exploits to isolate the minimal, task-relevant and target relative observation and action spaces for each skill policy. Policies are learned at the control reference level rather than at the raw actuator signal level, yielding a smoother and less noisy learning target. Known controllers can be leveraged for real-world data augmentation by projecting a single demonstration onto other objects in the scene, simultaneously enriching the graph construction process and the dataset for imitation learning. We validate our framework primarily on a real industrial forklift across statistically rigorous manipulation trials, and demonstrate cross-platform generality on a Kinova Gen3 robotic arm across two standard benchmarks. Our results show that grounding control learning, VLM-driven abstraction, and automated planning synthesis into a unified pipeline constitutes a practical path toward scalable, data-efficient, expert-free and interpretable neuro-symbolic robotics.
Figures & tables
Figure 1: Our forklift system loading two pallets from the ground and unloading them onto a truck. Such system builds on priors (perception, VLM for annotation and controls) to imitate long-horizon and complex task planning and acting from only 10 base skills demonstrations and 1 extra adaptation skill trajectory.
Figure 2: Training Pipeline. Two shared input columns feed three processing lanes, each producing one component of the learned system Φ (right). Shared input: Raw demonstrations (vision, proprioception, vocabulary Λ , change-point breakpoints T^ ) are augmented by projecting waypoint skeletons onto all scene objects ( N→N×K trajectories). Augmented data feeds all three lanes. Lane A (object detection): OWLv2 distills a compact YOLOv8 student online. Detections are lifted to 3D and combined with proprioception to form ego-centric trajectories τ={pobj−pee,at} , tracked across time. Delivers the YOLOv8 Detector . Lane B (symbolic abstraction): A VLM classifies each skill segment by comparing before/after scene images against Λ , building graph G that an ASP solver converts into PDDL domain σ=⟨E,F,S,O⟩ . Delivers PDDL Domain σ , and feeds σ back (dashed) to Lane C. Lane C (control learning—sequential after Lane B): The Oracle reads σ to construct minimal, ego-centric observations s~toi per operator. Diffusion sub-policies with Transformer termination predictors βi,j are assembled into per-operator skill automata. Delivers Skill Automata {πi} .
Figure 3: Execution Pipeline. At execution time, the system proceeds through five stages. (1) Scene grounding: YOLOv8 detects task-relevant objects, 3D positions are estimated, and relational predicates are evaluated to construct the initial symbolic state s0 . The user specifies a goal sg as a partial state. (2) Symbolic planning: MetricFF solves the PDDL instance T=⟨E,F,O,s0,sg⟩ in milliseconds, producing a human-readable, inspectable plan P . (3) Hierarchical skill execution: For each operator oi , the Oracle ϕ resolves symbolic entity bindings to physical detections and constructs the ego-centric observation. Diffusion sub-policies execute closed-loop at ∼ 20 Hz; a Transformer termination predictor βi,j gates transitions between action steps. The tracker updates the symbolic state after each operator completes. (4) Generalization: Three levels of generalization are handled by distinct mechanisms—relative observations for intra-task generalization, replanning for inter-task generalization, and on-the-fly OWLv2 distillation for cross-object transfer. (5) Termination: Execution terminates when st⊨sg .
Figure 4: Pallet alignment problem with respect to ADAPT kinematics. Relevant states include: rear body pose qR=[xR,yR,θR]T , fork tip pose qT=[xT,yT,θT,zT]T , and target pallet pose qP=[xP,yP,θP,zP]T , along with the articulation angle γ and control inputs v and γ˙ .
Figure 6: Validation domains: We validate our approach on a different embodiment (Kinova robotic arm) across two domains: Kitchen , where the arm must organize objects on a kitchen table, and Stacking , where it must stack cubes in the order of their size. The architecture mirrors the forklift domain, with the perception stack replaced by OWLv2 for object detection and depth data for 3D position estimation. From a single human demonstration, the agent extracts waypoints, projects them onto new object configurations, and executes the corresponding controls to complete each task end-to-end. We further validate a knowledge distillation pathway, OWLv2 supervising a lightweight YOLOv8 model, and demonstrated controls supervising compact diffusion policy networks, confirming that the architecture can be deployed on resource-constrained hardware without sacrificing task performance.
Figure 7: Success rate of the neurosymbolic approach as a function of the number of demonstrations for real-world forklift pallet management. The task consists of loading a pallet, navigating to a target area, and unloading it under randomized relative position to the pallet. We report two criteria: Strong SR (no significant collisions or pallet displacement) and Soft SR (task completed but minor collisions or pallet pushing allowed). The dashed line indicates the classical control baseline, which achieves 98% under both criteria. Performance improves monotonically with additional demonstrations, with Soft SR approaching the baseline at 30 demonstrations.
Task
Accuracy
Node matching (state equivalence, boolean)
85%
Edge annotation (skill label, one word)
98%
Table 1: VLM graph construction accuracy using Gemini 3 Flash.
Figure 8: VLM-constructed state-transition graph with three added nodes for visual completeness (left) and its bisimulation reduction (right), used as input to the ASP solver for automated action model induction. A bigger version of both graphs is attached in Appendix. A .
Figure 9: Distribution of performance (success/failure) of the forklift pallet loading task over the forklift reset region of the state. Each episode reset pose is represented by an arrow parametrized with origin (xinit,yinit,yawinit) relative to the pallet location. Blue arrows represent successes, while Red arrows represent failures. Reset configurations are sampled from a Gaussian distribution around a nominal pose. The three subfigures correspond to policies trained with 30, 10, and 5 demonstrations, respectively. No strong global correlation is observed between task success and initial pose. In the 5-demonstration setting, performance shows higher sensitivity to x , with a smooth success region of approximately 50cm . Failures are primarily correlated with perception noise (lightning notably impacts the detections) and the number of demonstrations.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Representative observations merged into a single graph node by Gemini 3 Flash, guided by a semantic domain hint defining high-level states in terms of pallet load status, number of pallets present in the scene, and forklift location. Each row shows four raw visual observations that the VLM abstracted into a single node despite significant viewpoint, lighting, and clutter variation. The domain hint provided to the VLM described high-level states as configurations differing in whether a pallet is currently loaded on the forklift, how many pallets remain in the scene, and the forklift’s coarse location (loading bay vs. unloading bay). This abstraction capacity is key to reducing the number of nodes in the graph and ensuring that semantically equivalent observations are not split across multiple nodes.
Figure 11: State-transition graph constructed by Gemini 3 Flash over 5 demonstrations per skill. Nodes represent clusters of visually equivalent high-level states (boolean VLM matching); edges are annotated with single-word skill labels. Three nodes were manually completed to ensure full connectivity.
Figure 12: Gallery of raw visual observations grouped by node. Each cell shows the set of demonstration frames the VLM merged into a single abstract state, illustrating the degree of intra-node visual variability handled by the matching mechanism.
Vision-Language Models (VLMs) have recently demonstrated strong capabilities in mapping multimodal observations to robot behaviors. However, most current approaches rely on end-to-end visuomotor policies that remain opaque and difficult to analyze, limiting their use in real-world robotic applications. In contrast, classical robotic systems often rely on structured policy representations that provide interpretability, modularity, and reactive execution. This work investigates how foundation models can be specialized to generate structured robot policies grounded in multimodal perception, bridging high-dimensional learning and symbolic control. We propose a neuro-symbolic approach in which a VLM synthesizes executable Behavior Tree policies from visual observations, natural language instructions, and structured system specifications. To enable scalable supervision without manual annotation, we introduce an automated pipeline that generates a synthetic multimodal dataset of domain-randomized scenes paired with instruction-policy examples produced by a foundation model. By decoupling structured task decomposition under constrained symbolic grammars from hardware-specific motor control, we demonstrate that a 12B-parameter model can learn structured spatial-symbolic mappings required for executable BT synthesis, solely through in-silico supervision. Real-world physical experiments on two heterogeneous robotic manipulators confirm that these structurally constrained policies achieve zero-shot transfer to real-world environments. The results emphasize that the data bottleneck in robotic planning can be bypassed by procedurally synthesizing high-fidelity, neuro-symbolic training data.
Alessandro Adami, Tommaso Tubaldo, Marco Todescato +2
University of Padova, Dept. of Information Engineering, Italy. · Polytechnic of Bari Dept. of Electrical and Information Engineering, Italy. · Fraunhofer Italia Research, 39100, Bozen, Italy.
Scaling imitation learning to diverse multi-task robot manipulation remains challenging due to suboptimal demonstrations, behavioral multi-modality, and destructive interference across tasks. While skill-based methods offer a promising direction by decomposing behaviors into reusable abstractions, existing approaches often learn skills that are either biased toward linguistic structure or lack semantic alignment across tasks, limiting generalization. In this work, we propose AtomSkill, a novel framework that learns a semantically aligned Atomic Skill Space from demonstrations and enables robust long-horizon execution through keypose imagination. Our method introduces: (1) semantic contrastive skill alignment, which partitions demonstrations into variable-length atomic skills and employs a contrastive objective to jointly enforce semantic consistency and temporal coherence, yielding a compact and reusable skill library; and (2) action decoding with keypose imagining, where the policy predicts both a skill's terminal keypose and immediate actions, thereby supporting progress-aware skill transitions. During inference, an atomic skill diffusion sampler generates plausible skill sequences, while predicted keyposes autonomously trigger smooth skill chaining. Extensive experiments in simulation and real-world settings show that AtomSkill consistently outperforms state-of-the-art imitation learning and skill-based baselines. Project page: https://atom-skill.github.io.
Yihang Zhu, Weiqing Wang, Shijie Wu +2
ShanghaiTech University, Shanghai, China · InstAdapt
Scaling robot learning to long-horizon tasks remains a formidable challenge. While end-to-end policies often lack the structural priors needed for effective long-term reasoning, traditional neuro-symbolic methods rely heavily on hand-crafted symbolic priors. To address the issue, we introduce ENAP (Emergent Neural Automaton Policy), a framework that allows a bi-level neuro-symbolic policy adaptively emerge from visuomotor demonstrations. Specifically, we first employ adaptive clustering and an extension of the L* algorithm to infer a Mealy state machine from visuomotor data, which serves as an interpretable high-level planner capturing latent task modes. Then, this discrete structure guides a low-level reactive residual network to learn precise continuous control via behavior cloning (BC). By explicitly modeling the task structure with discrete transitions and continuous residuals, ENAP achieves high sample efficiency and interpretability without requiring task-specific labels. Extensive experiments on complex manipulation and long-horizon tasks demonstrate that ENAP outperforms state-of-the-art (SoTA) end-to-end VLA policies by up to 27% in low-data regimes, while offering a structured representation of robotic intent (Fig. 1).
Yiyuan Pan, Xusheng Luo, Hanjiang Hu +2
Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA