Imitation Learning via Discrete and Rhythmic Primitive Composition

Rafid Saif; Auke Ijspeert; Hsiu-Chin Lin

cs.ROOct 11, 2026

Imitation Learning via Discrete and Rhythmic Primitive Composition

Authors: Rafid Saif, Auke Ijspeert, Hsiu-Chin Lin

Organizations: McGill University, and Mila - Quebec AI Institute, Canada. · École Polytechnique Fédérale de Lausanne (EPFL), Switzerland.

Abstract

Long-horizon robotic manipulation is inherently composed of semantically meaningful sub-tasks. While Movement Primitives (MPs) can model these components, their composition typically relies on manual engineering or sample-inefficient reinforcement learning. Existing supervised methods are limited by the lack of MP labels in standard datasets and often fail to generalize across discrete and rhythmic movements. We propose Neural Dynamic Composition (NDC), a supervised framework to compose Dynamic Movement Primitives (DMPs) using only raw trajectories for supervision, given a single example of each primitive. Our approach exploits parallel combinations of DMPs as a differentiable relaxation of categorical primitive selection, and utilizes a warp-invariant loss formulation to identify primitives despite spatial or temporal variations. A staged training curriculum is employed to learn the selection policy and DMP parameters, unifying discrete and rhythmic movements in one framework. Evaluated on 2D handwriting and 3D robotic tasks, our method outperforms baselines on most tasks and metrics. An ablation study demonstrates how each design choice contributes to the system's performance.

Figures & tables

Fig. 1: Examples of sequential tasks with a mixture of discrete and rhythmic movements.

Fig. 2: Overview of NDC. NDC leverages a pre-built skill library, using DMPs fitted to a single segmented skill demonstration. Conditioned on a task context, it predicts probabilities for each skill in the library across the sequence, and associated primitive parameters. The probabilities are used to combine DMPs, and rolled out into a resulting trajectory. By computing two losses with respect to the demonstration trajectory, the network is trained in a fully supervised manner.

Fig. 3: Evolution of combined trajectories (top) and predicted probabilities (bottom) for an example with fixed parameters.

Fig. 4: DMP rollouts with varying spatial scale and boundary conditions. Raw velocities converge to scaled nominal velocities. Meanwhile, velocities normalized along the movement dimension converge to the nominal normalized velocity, even over the finite horizons used in training.

	DTW ( $\downarrow$ )
	reps $=1$	reps $=2$	reps $=3$
BC	$2.800{\color[rgb]{0.5,0.5,0.5}\pm 6.667}$	$0.664{\color[rgb]{0.5,0.5,0.5}\pm 0.939}$	$1.404{\color[rgb]{0.5,0.5,0.5}\pm 0.989}$
NDMP	$0.048{\color[rgb]{0.5,0.5,0.5}\pm 0.039}$	$0.124{\color[rgb]{0.5,0.5,0.5}\pm 0.041}$	$0.122{\color[rgb]{0.5,0.5,0.5}\pm 0.040}$
NDMP-PD	$0.038{\color[rgb]{0.5,0.5,0.5}\pm 0.025}$	$0.077{\color[rgb]{0.5,0.5,0.5}\pm 0.011}$	$0.076{\color[rgb]{0.5,0.5,0.5}\pm 0.019}$
NDC (ours)	$\mathbf{0.026}{\color[rgb]{0.5,0.5,0.5}\pm 0.016}$	$\mathbf{0.037}{\color[rgb]{0.5,0.5,0.5}\pm 0.014}$	$\mathbf{0.038}{\color[rgb]{0.5,0.5,0.5}\pm 0.008}$

TABLE I: Average DTW distances across varying numbers of movement repetitions, on the test set.

		Success Rate (%) ( $\uparrow$ )
Task	$K$	BC	NDMP	NDMP-PD	NDC P2 (ours w/o term.)	NDC (ours)
Wipe $\rightarrow$ Press $\rightarrow$ Reach	3	$23.0{\color[rgb]{0.5,0.5,0.5}\pm 23.0}$	$8.0{\color[rgb]{0.5,0.5,0.5}\pm 2.0}$	$57.0{\color[rgb]{0.5,0.5,0.5}\pm 21.0}$	$\mathbf{87.0}{\color[rgb]{0.5,0.5,0.5}\pm 21.0}$	$\mathbf{87.0}{\color[rgb]{0.5,0.5,0.5}\pm 21.0}$
Retreat $\rightarrow$ Slide Left $\rightarrow$ Reach	3	$53.0{\color[rgb]{0.5,0.5,0.5}\pm 23.0}$	$47.0{\color[rgb]{0.5,0.5,0.5}\pm 17.0}$	$57.0{\color[rgb]{0.5,0.5,0.5}\pm 22.0}$	$67.0{\color[rgb]{0.5,0.5,0.5}\pm 16.0}$	$\mathbf{80.0}{\color[rgb]{0.5,0.5,0.5}\pm 17.0}$
Circle $\rightarrow$ Press $\rightarrow$ Slide Left	3	$10.0{\color[rgb]{0.5,0.5,0.5}\pm 10.0}$	$60.0{\color[rgb]{0.5,0.5,0.5}\pm 29.0}$	$76.0{\color[rgb]{0.5,0.5,0.5}\pm 16.0}$	$\mathbf{95.0}{\color[rgb]{0.5,0.5,0.5}\pm 10.0}$	$91.0{\color[rgb]{0.5,0.5,0.5}\pm 14.0}$
Slide Left $\rightarrow$ Reach $\rightarrow$ Wipe	3	$47.0{\color[rgb]{0.5,0.5,0.5}\pm 16.0}$	$54.0{\color[rgb]{0.5,0.5,0.5}\pm 16.0}$	$94.0{\color[rgb]{0.5,0.5,0.5}\pm 16.0}$	$\mathbf{98.0}{\color[rgb]{0.5,0.5,0.5}\pm 1.0}$	$\mathbf{98.0}{\color[rgb]{0.5,0.5,0.5}\pm 1.0}$
Reach $\rightarrow$ Wipe $\rightarrow$ Retreat	3	$16.0{\color[rgb]{0.5,0.5,0.5}\pm 17.0}$	$28.0{\color[rgb]{0.5,0.5,0.5}\pm 13.0}$	$91.0{\color[rgb]{0.5,0.5,0.5}\pm 6.0}$	$\mathbf{94.0}{\color[rgb]{0.5,0.5,0.5}\pm 3.0}$	$93.0{\color[rgb]{0.5,0.5,0.5}\pm 4.0}$

TABLE II: Task success rates across various sub-task sequences in simulation.

Fig. 5: Rollouts of the digit-writing experiment.

Fig. 6: Simulation and real-world robot setups shown side-by-side.

	Success Rate (%) ( $\uparrow$ )
Prim.	BC	NDMP	NDMP-PD	NDC P2	NDC
Slide Left	$64.0{\color[rgb]{0.5,0.5,0.5}\pm 41.0}$	$80.0{\color[rgb]{0.5,0.5,0.5}\pm 28.0}$	$96.0{\color[rgb]{0.5,0.5,0.5}\pm 8.0}$	$\mathbf{100.0}{\color[rgb]{0.5,0.5,0.5}\pm 0.0}$	$99.0{\color[rgb]{0.5,0.5,0.5}\pm 4.0}$
Wipe	$18.0{\color[rgb]{0.5,0.5,0.5}\pm 16.0}$	$21.0{\color[rgb]{0.5,0.5,0.5}\pm 10.0}$	$90.0{\color[rgb]{0.5,0.5,0.5}\pm 8.0}$	$\mathbf{93.0}{\color[rgb]{0.5,0.5,0.5}\pm 3.0}$	$\mathbf{93.0}{\color[rgb]{0.5,0.5,0.5}\pm 3.0}$
Press	$12.0{\color[rgb]{0.5,0.5,0.5}\pm 8.0}$	$12.0{\color[rgb]{0.5,0.5,0.5}\pm 27.0}$	$48.0{\color[rgb]{0.5,0.5,0.5}\pm 34.0}$	$86.0{\color[rgb]{0.5,0.5,0.5}\pm 17.0}$	$\mathbf{92.0}{\color[rgb]{0.5,0.5,0.5}\pm 11.0}$
Circle	$20.0{\color[rgb]{0.5,0.5,0.5}\pm 20.0}$	$38.0{\color[rgb]{0.5,0.5,0.5}\pm 13.0}$	$\mathbf{93.0}{\color[rgb]{0.5,0.5,0.5}\pm 4.0}$	$88.0{\color[rgb]{0.5,0.5,0.5}\pm 9.0}$	$88.0{\color[rgb]{0.5,0.5,0.5}\pm 9.0}$
Slide Right	$10.0{\color[rgb]{0.5,0.5,0.5}\pm 17.0}$	$80.0{\color[rgb]{0.5,0.5,0.5}\pm 26.0}$	$97.0{\color[rgb]{0.5,0.5,0.5}\pm 6.0}$	$\mathbf{100.0}{\color[rgb]{0.5,0.5,0.5}\pm 0.0}$	$\mathbf{100.0}{\color[rgb]{0.5,0.5,0.5}\pm 0.0}$

TABLE III: Per-primitive success rates.

	Goal-Prediction Error (mm) ( $\downarrow$ )
Task	NDMP	NDMP-PD	NDC P2	NDC
Slide Left	$77.2{\color[rgb]{0.5,0.5,0.5}\pm 26.3}$	$137.9{\color[rgb]{0.5,0.5,0.5}\pm 42.4}$	$12.3{\color[rgb]{0.5,0.5,0.5}\pm 8.5}$	$\mathbf{8.7}{\color[rgb]{0.5,0.5,0.5}\pm 5.2}$
Press	$66.0{\color[rgb]{0.5,0.5,0.5}\pm 26.0}$	$67.2{\color[rgb]{0.5,0.5,0.5}\pm 28.4}$	$14.6{\color[rgb]{0.5,0.5,0.5}\pm 7.8}$	$\mathbf{10.5}{\color[rgb]{0.5,0.5,0.5}\pm 6.8}$
Slide Right	$114.3{\color[rgb]{0.5,0.5,0.5}\pm 20.4}$	$280.4{\color[rgb]{0.5,0.5,0.5}\pm 21.0}$	$\mathbf{11.6}{\color[rgb]{0.5,0.5,0.5}\pm 6.2}$	$11.9{\color[rgb]{0.5,0.5,0.5}\pm 7.3}$
All	$80.2{\color[rgb]{0.5,0.5,0.5}\pm 30.5}$	$138.1{\color[rgb]{0.5,0.5,0.5}\pm 85.3}$	$13.1{\color[rgb]{0.5,0.5,0.5}\pm 7.8}$	$\mathbf{10.1}{\color[rgb]{0.5,0.5,0.5}\pm 6.3}$

TABLE IV: Goal-prediction error.

Fig. 7: Example of predicted v.s. actual goals.

Fig. 8: Ablations of NDC.

	Success Rate (%) ( $\uparrow$ )
Task	NDMP	NDMP-PD	NDC P2 (ours w/o term.)	NDC (ours)
Retreat $\rightarrow$ Slide Left $\rightarrow$ Press	$0.00{\color[rgb]{0.5,0.5,0.5}\pm 0.00}$	$50.00{\color[rgb]{0.5,0.5,0.5}\pm 52.70}$	$90.00{\color[rgb]{0.5,0.5,0.5}\pm 31.62}$	$\mathbf{100.00}{\color[rgb]{0.5,0.5,0.5}\pm 0.00}$
Circle $\rightarrow$ Slide Left $\rightarrow$ Slide Right	$6.67{\color[rgb]{0.5,0.5,0.5}\pm 25.82}$	$53.33{\color[rgb]{0.5,0.5,0.5}\pm 51.64}$	$66.67{\color[rgb]{0.5,0.5,0.5}\pm 48.80}$	$\mathbf{73.33}{\color[rgb]{0.5,0.5,0.5}\pm 45.77}$
Reach $\rightarrow$ Wipe $\rightarrow$ Press $\rightarrow$ Slide Left	$13.33{\color[rgb]{0.5,0.5,0.5}\pm 35.19}$	$60.00{\color[rgb]{0.5,0.5,0.5}\pm 50.71}$	$86.67{\color[rgb]{0.5,0.5,0.5}\pm 35.19}$	$\mathbf{93.33}{\color[rgb]{0.5,0.5,0.5}\pm 25.82}$
Overall	$6.67{\color[rgb]{0.5,0.5,0.5}\pm 26.67}$	$54.44{\color[rgb]{0.5,0.5,0.5}\pm 50.38}$	$81.11{\color[rgb]{0.5,0.5,0.5}\pm 40.51}$	$\mathbf{88.89}{\color[rgb]{0.5,0.5,0.5}\pm 33.49}$

TABLE V: Real World Experiment Results.

Explore similar work

Compositional Motion Generation from Demonstration with Object-Centric Neural Fields
Jul 8, 2026Ahmet Ercan Tekden, Yasemin BekirogluGeneralization in Robotic Manipulation Robot Motion Generation
Department of Electrical Engineering, Chalmers University of Technology, SE-412 96 Gothenburg, Sweden. · Department of Computer Science, University College London, WC1E 6BT London, U.K.
Learning Expressive and Compositional Motion Representation via Spectral Skills
Sep 29, 2026Feiyang Wu, Chenxiao Gao, Chen Yang +3Robot Skill Learning Humanoid Robot Control
Georgia Institute of Technology
DROM: A Language-Guided Diffusion Framework for Multi-Skill Robotic Manipulation
Sep 29, 2026Vincenzo Pomponi, Rocco Felici, Paolo Franceschi +5Long-Horizon Robotic Manipulation Robot Skill Learning
Institute of Systems and Technologies for Sustainable Production (ISTePS), Department of Innovative Technologies, University of Applied Science and Arts of Southern Switzerland (SUPSI), Via la Santa 1, Lugano, CH-6900, Ticino, Switzerland. · Istituto Dalle Molle di studi sull’intelligenza artificiale (IDSIA), Department of Innovative Technologies, University of Applied Science and Arts of Southern Switzerland (SUPSI), Via la Santa 1, Lugano, CH-6900, Ticino, Switzerland. · Mechanical Department, Politecnico di Milano (PoliMi), Via Giuseppe Candiani, Milan, 20155, Italy. +1

Jul 8, 2026J/K move · Enter open · S save

Compositional Motion Generation from Demonstration with Object-Centric Neural Fields

Ahmet Ercan Tekden, Yasemin Bekiroglu

Department of Electrical Engineering, Chalmers University of Technology, SE-412 96 Gothenburg, Sweden. · Department of Computer Science, University College London, WC1E 6BT London, U.K.

Generalization in Robotic Manipulation Robot Motion Generation Robotic Manipulation Learning from Demonstration

Open paper AskPDF arXiv

Compositionality, by organizing complex behavior as combinations of simpler elements, enables robot learning that is scalable and data efficient. Leveraging this principle, we propose a generative learning-from-demonstration framework that enables compositional modeling of robotic behavior by connecting perception and motion through shared object-level representations. We render scenes from object-centric neural representations that integrate canonical neural fields with latent-conditioned deformations, capturing positional and geometric variations in a smooth, consistent, and interpretable way. For motion generation, a temporal mixture-of-experts (MoE) employs a gating mechanism to combine object-conditioned movement primitives over time, producing complete trajectories. This spatial-temporal compositionality maintains the data efficiency of movement primitives while grounding motion in visual structure, enabling systematic generalization across diverse scene configurations. In simulation, long-horizon manipulation tasks are successfully completed using the proposed model, which requires significantly less training data than other image-based baselines. Real-world experiments further demonstrate the method's robustness to noise, its ability to generalize at the category level through language-based segmentation models, and its capacity to operate directly on 3D scene representations.

Abstract

Explore similar work

Compositional Motion Generation from Demonstration with Object-Centric Neural Fields

Learning Expressive and Compositional Motion Representation via Spectral Skills

DROM: A Language-Guided Diffusion Framework for Multi-Skill Robotic Manipulation