POIL: Point-based One-Shot Imitation Learning with Stable Dynamical Systems
Authors: Sang Min Kim, Jinwoo Seo, Hyeongjun Heo, Junho Lee, Yonghyeon Lee, Young Min Kim
Organizations: Department of Electrical and Computer Engineering, Seoul National University, Seoul, Republic of Korea. · Department of Artificial Intelligence, Yonsei University, Seoul, Republic of Korea.
We present POIL, a point-based one-shot imitation learning framework with stable dynamical systems. While one-shot imitation avoids collecting extensive demonstrations, successful one-shot manipulation requires not only transferring a demonstrated trajectory to a novel object but also executing it robustly under changing scene conditions, grasp configurations, and external disturbances. POIL addresses both problems through a shared representation: a set of 3D points on the object's functional part, used jointly for trajectory transfer and closed-loop execution. The one-shot transfer from the demonstrated trajectory is enabled with point correspondences. POIL grounds the shared functional part with a multi-modal large language model, and transfers the trajectory across viewpoint, pose, and object category changes. During execution, multi-view tracking observes the same points online, and Point-set BCSDM drives them in closed loop by projecting per-point velocities onto a single rigid-body twist computed from the tracked points alone. This extends stable dynamical models from an SE(3) pose to a point set without requiring a known 3D model or pose estimator. We show that at the goal the controller becomes a gradient flow on the classical SO(3) potential, so its terminal phase inherits the almost-global convergence of that potential under a rigid-object assumption. Across simulation and real-robot experiments, POIL transfers a single demonstration across object category, grasp pose, and goal geometry, while recovering from external disturbances during execution. Project page: https://sangminkim-99.github.io/poil
Figures & tables
Approach
No action pretrain.
Object gen.
Closed loop
Conv. guarantee
Meta / in-context [ 4 , 6 ]
✗
in-dist.
✓
✗
Correspondence [ 9 , 10 ]
✓
func. part ‡
✗
✗
Alignment [ 7 , 8 ]
✓
intra-cat.
✓ †
✗
Analytical SDS [ 11 , 12 ]
✓
✗
✓
✓
POIL (ours)
✓
func. part ‡
✓
✓
† Closed-loop alignment to an intermediate pose, then open-loop replay.
TABLE I: One-shot imitation learning approaches for manipulation, compared along the four axes discussed in Sec. I.
Fig. 1: Overview of POIL. The transfer stage grounds the functional part, establishes point correspondences, and transfers the demonstrated point trajectory to the target scene. The execution stage tracks the same points online and drives them with Point-set BCSDM.
Fig. 2: Handle detection from multi-view mug images. Multi-view prompting enables Rex-Omni to localize the handle using cross-view context.
Fig. 3: Trajectory adaptation to a new fixture. User-clicked correspondences between the demonstration’s straight rack (left) and the target’s L-shaped rack (right) define a TPS warp Φ . Applying Φ to every waypoint of the reference trajectory yields the adapted trajectory in the target scene.
TABLE II: Control robustness under varying initial poses. Success rate (%) over 100 trials. Near denotes the region used to collect demonstrations for PP-50, and Far samples poses outside it.
Fig. 4: Functional-part point transfer under viewpoint changes. We report part point error (left) and the fraction of transferred points within a distance threshold (right). The ablation without multi-view prompting (MVP) isolates the effect of cross-view reasoning.
Fig. 5: Demonstration sequences for the control tasks. Object motion is rendered from blue to red.
Fig. 6: Success rate on Mug Insertion versus the initial pose’s xy distance to the training region (20 cm bins; shaded: in-distribution). The 50-demo Point Policy degrades with distance, while ours stays at 100%.
Fig. 7: End-to-end execution under object-centric closed-loop control. Object motion is rendered from blue to red. (a) Teapot pouring with perturbation recovery. (b) Mug hanging with a side-lying target mug and a different feasible grasp.
Fig. 8: Trajectory adaptation under goal-geometry changes: a cube placed in a taller box, a mug hung on an L-hook rod, and a plate seated in a tilted rack slot. Top: demonstration. Bottom: execution along the trajectory adapted to the target fixture.
Fig. 9: Representative failures in the side-lying mug evaluation. Left: the mug is detected in all views, but the hidden handle causes Rex-Omni [ 25 ] to ground the rack as the handle (red box). Right: a feasible initial grasp leads to mug-floor collision, outside the current feasibility check.
Fig. 10: Real-world cross-category transfer from simulation (top) and human hand demonstration (bottom). Robot and object motion is rendered from blue (initial) to red (final), with tracked point trajectories overlaid.
Fig. 11: Disturbance recovery during sim-to-real mug hanging. A human pulls the gripper and rotates the mug within the grasp (red; intermediate frames below), yet the handle reaches the demonstrated goal (bottom right).
Learning robot manipulation policies with deep neural networks from a single demonstration remains highly challenging, as even small deviations from the demonstrated trajectory can quickly compound into failure, while collecting substantial online interaction data is costly. We propose ReGIL, a retrieval-guided imitation learning framework that treats a single demonstration as an external memory. ReGIL repeatedly queries this static memory throughout training to simultaneously guide exploration, generate the regularization buffer, and construct rewards. Specifically, it computes rewards through local temporal alignment between the current trajectory and the retrieved segment, providing step-wise and informative feedback for policy improvement. We evaluate ReGIL on robotic manipulation tasks from the LIBERO and Meta-World benchmarks under the single demonstration setting. ReGIL outperforms prior baselines in both success rate and training efficiency. In real-robot experiments, using only one demonstration and less than one hour of online training, ReGIL achieves over 75% success rate across three manipulation tasks with randomness in both initial robot pose and target position. These results demonstrate that leveraging the single demonstration as reusable memory can provide more than static supervision for efficient robot learning. More details can be found on our website: https://regil2026.github.io/
Yuying Zhang, Francesco Verdoja, Wenyan Yang +1
School of Electrical Engineering Aalto University, Finland
Object interaction tasks have been a focus of advances in imitation learning. End-to-end methods, dominated by diffusion and flow-based variants have shown leaps in performance while sacrificing interpretability. Object-centric and pose-informed variants have had a role in learning from demonstration in manipulation tasks. In this paper, we revisit a few modern imitation learning benchmarks for object interactions, with the aim of composing a framework that repurposes principled theories of manipulation, preserving both performance and interpretability. For image observations, lightweight training is proposed for model-free pose estimation of the target object, using self-supervision over the demonstration data available for imitation learning. This information is then used to inform a manipulation mode-aware retrieval of a demonstration, a mode-aware transformation, a replan step that connects to the retrieval point while preserving mode constraints, and finally rolling out the transformed demonstration. These compose four key steps of the proposed RE4 framework, evaluated over state-based and image-based benchmarks in Push-T and Robomimic. An adversarial benchmark that evaluates sparse data regions of image-based Push-T showcases the robustness, further bolstered by indications from low-data regime experiments. The current work shows promise in using simple interpretable building blocks to learn manipulation skills.
Relational object rearrangement (ROR) tasks (e.g., insert flower to vase) require a robot to manipulate objects with precise semantic and geometric reasoning. Existing approaches either rely on pre-collected demonstrations that struggle to capture complex geometric constraints or generate goal-state observations to capture semantic and geometric knowledge, but fail to explicitly couple object transformation with action prediction, resulting in errors due to generative noise. To address these limitations, we propose Imagine2Act, a 3D imitation-learning framework that incorporates semantic and geometric constraints of objects into policy learning to tackle high-precision manipulation tasks. We first generate imagined goal images conditioned on language instructions and reconstruct corresponding 3D point clouds to provide robust semantic and geometric priors. These imagined goal point clouds serve as additional inputs to the policy model, while an object-action consistency strategy with soft pose supervision explicitly aligns predicted end-effector motion with generated object transformation. This design enables Imagine2Act to reason about semantic and geometric relationships between objects and predict accurate actions across diverse tasks. Experiments in both simulation and the real world demonstrate that Imagine2Act outperforms previous state-of-the-art policies. More visualizations can be found at https://sites.google.com/view/imagine2act.
Liang Heng, Jiadong Xu, Yiwen Wang +6
CFCS, School of Computer Science, Peking University · PrimeBot · AGIBOT