Build on Priors: Vision--Language--Guided Neuro-Symbolic Imitation Learning for Data-Efficient Real-World Robot Manipulation
Authors: Pierrick Lorang, Johannes Huemer, Timothy Duggan, Kai Goebel, Patrik Zips, Matthias Scheutz
Organizations: Department of Computer Science, Tufts University, Medford, MA, USA · Complex Dynamical Systems, Austrian Institute of Technology GmbH (AIT), Vienna, Austria
Enabling robots to learn long-horizon manipulation tasks from a handful of demonstrations remains a central challenge in robotics. Existing neuro-symbolic approaches often rely on hand-crafted symbolic abstractions, semantically labeled trajectories or large demonstration datasets, limiting their scalability and real-world applicability. We present a scalable neuro-symbolic framework that autonomously constructs symbolic planning domains and data-efficient control policies from as few as one to thirty unannotated skill demonstrations, without requiring manual domain engineering. Our method segments demonstrations into skills and employs a Vision-Language Model (VLM) to classify skills and identify equivalent high-level states, enabling automatic construction of a state-transition graph. This graph is processed by an Answer Set Programming solver to synthesize a PDDL planning domain, which an oracle function exploits to isolate the minimal, task-relevant and target relative observation and action spaces for each skill policy. Policies are learned at the control reference level rather than at the raw actuator signal level, yielding a smoother and less noisy learning target. Known controllers can be leveraged for real-world data augmentation by projecting a single demonstration onto other objects in the scene, simultaneously enriching the graph construction process and the dataset for imitation learning. We validate our framework primarily on a real industrial forklift across statistically rigorous manipulation trials, and demonstrate cross-platform generality on a Kinova Gen3 robotic arm across two standard benchmarks. Our results show that grounding control learning, VLM-driven abstraction, and automated planning synthesis into a unified pipeline constitutes a practical path toward scalable, data-efficient, expert-free and interpretable neuro-symbolic robotics.
Figures & tables
Figure 1: Our forklift system loading two pallets from the ground and unloading them onto a truck. Such system builds on priors (perception, VLM for annotation and controls) to imitate long-horizon and complex task planning and acting from only 10 base skills demonstrations and 1 extra adaptation skill trajectory.
Figure 2: Training Pipeline. Two shared input columns feed three processing lanes, each producing one component of the learned system Φ (right). Shared input: Raw demonstrations (vision, proprioception, vocabulary Λ , change-point breakpoints T^ ) are augmented by projecting waypoint skeletons onto all scene objects ( N→N×K trajectories). Augmented data feeds all three lanes. Lane A (object detection): OWLv2 distills a compact YOLOv8 student online. Detections are lifted to 3D and combined with proprioception to form ego-centric trajectories τ={pobj−pee,at} , tracked across time. Delivers the YOLOv8 Detector . Lane B (symbolic abstraction): A VLM classifies each skill segment by comparing before/after scene images against Λ , building graph G that an ASP solver converts into PDDL domain σ=⟨E,F,S,O⟩ . Delivers PDDL Domain σ , and feeds σ back (dashed) to Lane C. Lane C (control learning—sequential after Lane B): The Oracle reads σ to construct minimal, ego-centric observations s~toi per operator. Diffusion sub-policies with Transformer termination predictors βi,j are assembled into per-operator skill automata. Delivers Skill Automata {πi} .
Figure 3: Execution Pipeline. At execution time, the system proceeds through five stages. (1) Scene grounding: YOLOv8 detects task-relevant objects, 3D positions are estimated, and relational predicates are evaluated to construct the initial symbolic state s0 . The user specifies a goal sg as a partial state. (2) Symbolic planning: MetricFF solves the PDDL instance T=⟨E,F,O,s0,sg⟩ in milliseconds, producing a human-readable, inspectable plan P . (3) Hierarchical skill execution: For each operator oi , the Oracle ϕ resolves symbolic entity bindings to physical detections and constructs the ego-centric observation. Diffusion sub-policies execute closed-loop at ∼ 20 Hz; a Transformer termination predictor βi,j gates transitions between action steps. The tracker updates the symbolic state after each operator completes. (4) Generalization: Three levels of generalization are handled by distinct mechanisms—relative observations for intra-task generalization, replanning for inter-task generalization, and on-the-fly OWLv2 distillation for cross-object transfer. (5) Termination: Execution terminates when st⊨sg .
Figure 4: Pallet alignment problem with respect to ADAPT kinematics. Relevant states include: rear body pose qR=[xR,yR,θR]T , fork tip pose qT=[xT,yT,θT,zT]T , and target pallet pose qP=[xP,yP,θP,zP]T , along with the articulation angle γ and control inputs v and γ˙ .
Figure 6: Validation domains: We validate our approach on a different embodiment (Kinova robotic arm) across two domains: Kitchen , where the arm must organize objects on a kitchen table, and Stacking , where it must stack cubes in the order of their size. The architecture mirrors the forklift domain, with the perception stack replaced by OWLv2 for object detection and depth data for 3D position estimation. From a single human demonstration, the agent extracts waypoints, projects them onto new object configurations, and executes the corresponding controls to complete each task end-to-end. We further validate a knowledge distillation pathway, OWLv2 supervising a lightweight YOLOv8 model, and demonstrated controls supervising compact diffusion policy networks, confirming that the architecture can be deployed on resource-constrained hardware without sacrificing task performance.
Figure 7: Success rate of the neurosymbolic approach as a function of the number of demonstrations for real-world forklift pallet management. The task consists of loading a pallet, navigating to a target area, and unloading it under randomized relative position to the pallet. We report two criteria: Strong SR (no significant collisions or pallet displacement) and Soft SR (task completed but minor collisions or pallet pushing allowed). The dashed line indicates the classical control baseline, which achieves 98% under both criteria. Performance improves monotonically with additional demonstrations, with Soft SR approaching the baseline at 30 demonstrations.
Task
Accuracy
Node matching (state equivalence, boolean)
85%
Edge annotation (skill label, one word)
98%
Table 1: VLM graph construction accuracy using Gemini 3 Flash.
Figure 8: VLM-constructed state-transition graph with three added nodes for visual completeness (left) and its bisimulation reduction (right), used as input to the ASP solver for automated action model induction. A bigger version of both graphs is attached in Appendix. A .
Figure 9: Distribution of performance (success/failure) of the forklift pallet loading task over the forklift reset region of the state. Each episode reset pose is represented by an arrow parametrized with origin (xinit,yinit,yawinit) relative to the pallet location. Blue arrows represent successes, while Red arrows represent failures. Reset configurations are sampled from a Gaussian distribution around a nominal pose. The three subfigures correspond to policies trained with 30, 10, and 5 demonstrations, respectively. No strong global correlation is observed between task success and initial pose. In the 5-demonstration setting, performance shows higher sensitivity to x , with a smooth success region of approximately 50cm . Failures are primarily correlated with perception noise (lightning notably impacts the detections) and the number of demonstrations.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 10: Representative observations merged into a single graph node by Gemini 3 Flash, guided by a semantic domain hint defining high-level states in terms of pallet load status, number of pallets present in the scene, and forklift location. Each row shows four raw visual observations that the VLM abstracted into a single node despite significant viewpoint, lighting, and clutter variation. The domain hint provided to the VLM described high-level states as configurations differing in whether a pallet is currently loaded on the forklift, how many pallets remain in the scene, and the forklift’s coarse location (loading bay vs. unloading bay). This abstraction capacity is key to reducing the number of nodes in the graph and ensuring that semantically equivalent observations are not split across multiple nodes.
Figure 11: State-transition graph constructed by Gemini 3 Flash over 5 demonstrations per skill. Nodes represent clusters of visually equivalent high-level states (boolean VLM matching); edges are annotated with single-word skill labels. Three nodes were manually completed to ensure full connectivity.
Figure 12: Gallery of raw visual observations grouped by node. Each cell shows the set of demonstration frames the VLM merged into a single abstract state, illustrating the degree of intra-node visual variability handled by the matching mechanism.
University of Padova, Dept. of Information Engineering, Italy. · Polytechnic of Bari Dept. of Electrical and Information Engineering, Italy. · Fraunhofer Italia Research, 39100, Bozen, Italy.