Demonstration-Based Robotic Feeding Using Dynamic Movement Primitives
Authors: Kenzhi Iskandar Wong, Kitti Thamrongaphichartkul, Sri Harsha Turlapati, Yanshuo Lu, Kanichi Doi, Boyu Ma, Bin Zhou, Lin Yang, Ananda Sidarta, Lek Syn Lim, Arun-Kumar Kaliya-Perumal, Jianfei Yang, Domenico Campolo
Organizations: Robotics Research Centre, School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore. · School of Science and Technology, Singapore University of Social Sciences, Singapore. · Department of Mechano-Informatics, University of Tokyo, Japan. · R3 Institute Singapore, Nanyang Technological University, Singapore.
This paper presents a demonstration-based robotic feeding framework that integrates human demonstration, Dynamic Movement Primitive-based motion planning, vision-based scene perception, and robot control. A human operator first demonstrates the scooping and feeding motions while the position and Euler-angle orientation of the spoon are recorded using a motion-capture system. The recorded trajectory is segmented into scooping and feeding motions, which are encoded using separate Dynamic Movement Primitives. Before task execution, an RGB-D camera estimates the participant's mouth position, while two AprilTags are used for robot-frame registration and tray localization. The detected tray pose is used to position the scooping motion, whereas the detected mouth position defines the positional goal of the feeding motion. The approach, scooping, and feeding trajectories are then executed sequentially by a Kinova Gen3 manipulator through a ROS~2-based framework. A real-world experiment demonstrates the feasibility of the integrated pipeline and shows that the learned spoon motions can be reproduced within the detected experimental scene.
Figures & tables
Fig. 1: Overview of the demonstration-based robotic feeding framework. The scooping and feeding motions are recorded using a motion-capture system and encoded as DMPs. Before task execution, the mouth position is detected once using an RGB-D camera, while two AprilTags are used for camera-to-robot frame registration and tray localization. The learned DMP weights, task-relevant configurations, and initial mouth position are used to generate the desired end-effector motion for the robotic manipulator.
Fig. 2: Experimental setup for the robotic feeding task. A Kinova Gen3 manipulator equipped with a spoon performs the scooping and feeding motions, while an RGB-D camera is used to estimate the participant’s mouth position. Two AprilTags are used for camera-to-robot frame registration and tray localization.
Fig. 3: Mouth-perception results under different head poses and mouth configurations.
Fig. 4: Sequential snapshots of the robotic feeding task. From left to right, the manipulator approaches the bowl, executes the learned scooping motion, transports the spoon toward the participant, and reaches the initially detected mouth position.
Fig. 5: Planned end-effector trajectories visualized within the RGB-D point cloud of the experimental scene. The approach trajectory yapproach(t) moves the robot toward the pre-scooping pose, while yscoop(t) and yfeed(t) denote the DMP-generated scooping and feeding trajectories, respectively. The coordinate frames indicate the planned spoon position and orientation along each trajectory.
Coding agents have demonstrated enormous success in solving complex programming problems. To leverage their potential for robot systems, this work introduces Robot Agentic Programming from Demonstrations (RAPID), which automatically generates, verifies, and refines robot programs, given a single visual human demonstration. The iterative agentic loop of code refinement requires several key ingredients: (i) a testable task specification, (ii) action primitives for robot execution, and (iii) an interactive environment for program execution and verification. RAPID infers all three from the demonstration automatically. To make the resulting program reusable beyond the demonstration setting, RAPID uses an object-centric relational program representation that focuses on the underlying structure of the demonstrated strategy rather than the specific motion per se: it expresses the action primitives as trajectory-optimization programs that realize object-level motion effects, while composing them through relational constraints that capture scene-specific geometry at run time. We evaluated RAPID in simulation on eight challenging contact-rich nonprehensile manipulation tasks as well as general prehensile manipulation tasks in the LIBERO-Pro benchmark. We also successfully deployed it on a real Franka arm and evaluated on all eight nonprehensile tasks. In all experiments, RAPID demonstrated strong performance, with generalization over object pose, shape, material, and environment. Website: https://yuyaoliu.me/projects/rapid.
Yuyao Liu, Jiayuan Mao, David Hsu +2
Massachusetts Institute of Technology · National University of Singapore · University of Pennsylvania +1
Robotic manipulation often requires object specific programming, manual data annotation, or calibrated perception pipelines, which limits rapid deployment in practical settings. Learning from demonstration offers a more direct alternative, but collecting demonstrations can still demand human teleoperation or kinesthetic teaching. This paper presents a self supervised visual manipulation method in which a robot automatically generates demonstrations around a target pose and learns relative pose corrections directly from wrist mounted RGB images. The proposed pipeline uses ROS~2 and Isaac Sim to collect labeled image-pose pairs without requiring explicit camera to robot extrinsic calibration. Separate datasets are generated for planar refinement and coarse three dimensional approach, and a convolutional network is trained to regress relative translation and rotation from single frame RGB observations. During execution, a coarse to fine controller first approaches the object using models trained with height variation and then refines the final alignment using planar data. The method is evaluated both in simulation and on a real UR5e collaborative robot equipped with a gripper and a monocular camera. In simulation, the refinement stage reduces the final planar dispersion from 9.69 mm to 5.38 mm. In real world experiments, the system performs end to end grasp attempts on three physical objects and reaches success rates of 66.6% and 63.6% for two objects without object rotation, while still maintaining partial robustness under rotated conditions. These results show that automatically generated demonstrations can support practical visual manipulation with limited setup effort, while also exposing remaining challenges in depth prediction and object dependent generalization.
Andres Rivas, Anselmo R. Cukla, Rodrigo S. Guerra +2
Robotics and AI Lab, Technological University of Uruguay, Rivera, Uruguay · Center of Technology, Federal University of Santa Maria, Santa Maria, Brazil · Computer Science Center, Federal University of Rio Grande, Rio Grande, Brazil
Learning from demonstration (LfD) provides a developmental framework through which robots can develop motor skills by observing and imitating human dynamics, reducing reliance on explicit programming to teach a skill to a robot. The resulting human-like robot motion is recognised as a key factor in building trust and enabling natural collaboration in human-robot interaction. This paper presents a framework for learning human-like robot motion from demonstration, including data collection, probabilistic trajectory learning, and perceptual user evaluation. A dataset of 3,142 handwriting demonstrations was collected from 22 participants across all 52 Latin alphabet character-case combinations via a touchscreen teleoperation interface, capturing planar position, contact force, and timing. Building on the widely used Gaussian Mixture Model and Gaussian Mixture Regression approach for learning from demonstration, the framework is extended in this work by incorporating force and normalised time dimensions to enable richer representation of human dynamics, and adapting it to handle non-continuous, multi-segment trajectories, enabling generalisation across demonstrations. A user study with 21 participants evaluated the perceived human-likeness of the generated trajectories using a continuous scale anchored between robotic and human-like motion, normalised to 0-100 where 50 represents the neutral midpoint. The generated trajectories achieved an overall human-likeness score of 71.50 (SD=22.56), indicating that the majority of trajectories were perceived as more human-like. Participants identified geometric positioning and trajectory sequence as the most influential perceptual factors, and reported positive attitudes toward human-like robot behaviour. The datasets are released as open-source, providing a reproducible benchmark for developing and evaluating human-like robot motion methods.
Alperen Kenan, Paul Bremner, Manuel Giuliani
Bristol Robotics Laboratory, University of the West of England, Bristol, United Kingdom · Faculty of Electrical Engineering, Kempten University of Applied Sciences, Kempten, Germany