Multi-CONDOR: Learning Multistable Motion Primitives from Demonstration
Authors: Gabriele Giuseppe Di Marzo, Mariano Ramírez Montero, Maximilian Stölzle, Rodrigo Pérez-Dattari, Cosimo Della Santina
Organizations: Department of Cognitive Robotics, Delft University of Technology, Delft, The Netherlands. · Department of Computer, Control and Management Engineering (DIAG), Sapienza University of Rome, Italy. · Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA. · Department of Robotics, Perception and Learning, KTH Royal Institute of Technology, Stockholm, Sweden.
Dynamical-system-based motion policies provide a framework for learning reactive robot motions with stability and convergence properties. Many manipulation tasks, however, admit several distinct goals or a continuous set of acceptable outcomes, requiring motion primitives that preserve these alternatives. Accommodating both goal structures within a unified framework with continuous, autonomous dynamics remains an open challenge. We propose a CONDOR-based framework, multi-CONDOR, for learning motion primitives with either multiple discrete attractors or a continuous attracting goal contour. The approach combines imitation learning with losses that encourage agreement between the learned task dynamics and latent reference systems encoding the desired attraction structure. We provide a Lyapunov-based analysis of the convergence properties of the learned motion dynamics. We evaluate the framework on simulated benchmarks with discrete goals, qualitative examples with continuous goal contours, and robotic grasping experiments. In simulation, multi-CONDOR reduces trajectory RMSE by approximately 72% compared with separately learned single-goal motion primitives selected by a state machine. For tasks with two and three goals, respectively, it reduces the fraction of evaluated initial states ending outside the goal tolerance from StableNODE's 0.31% and 7.29% to 0% in both settings. Experiments with a Franka Emika robot arm demonstrate successful grasping of a container by either of its two handles using motions learned from demonstrations. Together, these results show multi-CONDOR combines motion reproduction with the flexibility to converge to alternative valid outcomes.
Figures & tables
Fig. 1: Overview of multi-CONDOR: an encoder ψ maps task states x(t) to latent states y(t) , and a function ϕ maps these to task velocities x˙(t) . Both are jointly trained as parts of a single neural network. Training aligns the encoded task dynamics with prescribed latent reference dynamics featuring multiple attractors (shown above “Latent space”), encouraging the learned vector field to inherit their multistability while reproducing the demonstrations D .
Fig. 2: Matching loss for two sampled initial task states x0,b . Encoded task rollouts yt,bT=ψθ(xt,b) are compared with reference rollouts yt,bL initialized at the same latent state ψθ(x0,b) .
Fig. 3: Reference dynamics fL in two dimensions: two isolated stable attractors (blue dots, left) and a continuous attracting circle (dashed blue line, right).
Method
RMSE
DTWD
FD
Spurious attractors (%)
Distance from goals
2 attractors
StableNODE ( α=ϵ=10−2 )
17.65±0.64
8.37±1.35
32.91±4.10
0.31±0.63
0.38±0.21
Single-attractor CONDORs + state machine
77.85±3.94
81.32±2.96
106.28±9.11
9.22±9.23
3.71±0.70
Multi-CONDOR (only imitation, λstable=λmap=0 )
15.25±6.53
5.69±1.48
20.68±8.03
16.98±20.89
13.33±16.38
Multi-CONDOR (no mapping loss, λmap=0 )
32.00±32.89
23.62±42.24
48.20±69.99
13.33±35.19
21.25±48.09
Multi-CONDOR (Ours)
21.58±11.30
8.82±6.97
24.26±16.15
0.00±0.00
3.10±0.97
TABLE I: Performance metrics for the 2-attractor and 3-attractor datasets. We report the mean and standard deviation over the total number of tasks (3 for 2-attractor and 4 for 3-attractor) and over 5 random seeds. The best results are shown in bold.
Fig. 4: Learned dynamics of multi-CONDOR (left) and StableNODE [ 12 ] (right) on two three-attractor benchmark tasks. Demonstrations are white, generated trajectories red, and attractors blue. StableNODE produces additional spurious attractors away from the intended goals.
Fig. 5: Kinesthetic demonstrations approaching the two grasping goals.
Fig. 6: Robot executions reaching either grasping goal using a single vector field learned with multi-CONDOR.
Fig. 7: Visualization of provided demonstrations (in green) for the multi-goal grasping task and four multi-CONDOR trajectories (in red) which successfully grasped the box by the handles. Note how the trajectories converge to the desired goals despite starting from initial conditions which are not the same as the provided demonstrations.
Fig. 8: Example of learned dynamics using multi-CONDOR for a continuous, closed star shape (left) without any imitation, and an example of a case with a circular shape as a continuous goal with two subgroups of demonstrations to imitate (right).
Learning motion policies from expert demonstrations is an essential paradigm in modern robotics. While end-to-end models aim for broad generalization, they require large datasets and computationally heavy inference. Conversely, learning dynamical systems (DS) provides fast, reactive, and provably stable control from very few demonstrations. However, existing DS learning methods typically model isolated tasks and struggle to reuse demonstrations for novel behaviors. In this work, we formalize the problem of combining isolated demonstrations within a shared workspace to enable generalization to unseen tasks. The Gaussian Graph is introduced, which reinterprets spatial components of learned motion primitives as discrete vertices with connections to one another. This formulation allows us to bridge continuous control with discrete graph search. We propose two frameworks leveraging this graph: Stitching, for constructing time-invariant DSs, and Chaining, giving a sequence-based DS for complex motions while retaining convergence guarantees. Simulations and real-robot experiments show that these methods successfully generalize to new tasks where baseline methods fail.
Kilian Freitag, Alvin Combrink, Nadia Figueroa
Department of Electrical Engineering, Chalmers University of Technology, Gothenburg, Sweden · GRASP Lab, University of Pennsylvania, PA 19104, USA
We present GHOST, a framework for learning visuomotor manipulation policies that generalize beyond the training distribution. GHOST factorizes control into (i) a high-level policy that predicts the next sub-goal as a distribution over 3D end-effector poses from multi-view RGB-D observations, and (ii) a low-level goal-conditioned controller that executes embodiment-specific actions. To condition image-based policies on 3D goals, we introduce a simple spatial interface that projects predicted goals into the image plane and represents them as end-effector heatmaps. Across a suite of manipulation tasks, this hierarchical factorization consistently improves performance and robustness compared to a flat Diffusion Policy. Further, we show that this hierarchical interface also makes it easy to incorporate human demonstrations without relying on (noisy) action retargeting. As sub-goals are largely embodiment-agnostic, we train the high-level policy on human video to specify how learned skills should be applied and composed, while keeping the low-level policy trained purely on robot data. This hierarchy enables adaptation to novel objects and task variations using a small number of human demonstrations.
Sriram Krishna, Ben Eisner, Haotian Zhan +5
Robotics Institute, Carnegie Mellon University · UMass Amherst
Behavior cloning for robotic manipulation is highly sensitive to the coverage of its training data, yet collecting diverse successful demonstrations remains costly. We study reinforcement learning (RL) not as a deployable controller, but as a data generator for downstream visuomotor imitation learning. Our key premise is that multiple successful solution modes cover a broader range of successful trajectories than repeatedly collecting a single narrow solution. We introduce DiscoDemo, which trains a privileged latent-conditioned RL generator with a parallel reverse curriculum and a success-gated diversity objective to discover diverse solutions while preserving task success. The trained generator produces multiple successful closed-loop behaviors, which are safety-filtered and exported as a static dataset for downstream imitation learning. Across four manipulation tasks, DiscoDemo produces broader successful-state coverage and improves downstream visuomotor policies in both simulation and the real world. Averaged across tasks, downstream success increases to 62% from 40% for the strongest baseline in simulation, and to 78% from 39% on the real robot, respectively. These results suggest that successful solution diversity improves imitation learning for a pretrained flow-matching VLA and remains complementary to corrective data collection. Project page: https://davian-robotics.github.io/DiscoDemo