POIL: Point-based One-Shot Imitation Learning with Stable Dynamical Systems

Sang Min Kim; Jinwoo Seo; Hyeongjun Heo; Junho Lee; Yonghyeon Lee; Young Min Kim

cs.ROSep 24, 2026

POIL: Point-based One-Shot Imitation Learning with Stable Dynamical Systems

Authors: Sang Min Kim, Jinwoo Seo, Hyeongjun Heo, Junho Lee, Yonghyeon Lee, Young Min Kim

Organizations: Department of Electrical and Computer Engineering, Seoul National University, Seoul, Republic of Korea. · Department of Artificial Intelligence, Yonsei University, Seoul, Republic of Korea.

Abstract

We present POIL, a point-based one-shot imitation learning framework with stable dynamical systems. While one-shot imitation avoids collecting extensive demonstrations, successful one-shot manipulation requires not only transferring a demonstrated trajectory to a novel object but also executing it robustly under changing scene conditions, grasp configurations, and external disturbances. POIL addresses both problems through a shared representation: a set of 3D points on the object's functional part, used jointly for trajectory transfer and closed-loop execution. The one-shot transfer from the demonstrated trajectory is enabled with point correspondences. POIL grounds the shared functional part with a multi-modal large language model, and transfers the trajectory across viewpoint, pose, and object category changes. During execution, multi-view tracking observes the same points online, and Point-set BCSDM drives them in closed loop by projecting per-point velocities onto a single rigid-body twist computed from the tracked points alone. This extends stable dynamical models from an SE(3) pose to a point set without requiring a known 3D model or pose estimator. We show that at the goal the controller becomes a gradient flow on the classical SO(3) potential, so its terminal phase inherits the almost-global convergence of that potential under a rigid-object assumption. Across simulation and real-robot experiments, POIL transfers a single demonstration across object category, grasp pose, and goal geometry, while recovering from external disturbances during execution. Project page: https://sangminkim-99.github.io/poil

Figures & tables

Approach	No action pretrain.	Object gen.	Closed loop	Conv. guarantee
Meta / in-context [ 4 , 6 ]	✗	in-dist.	✓	✗
Correspondence [ 9 , 10 ]	✓	func. part ‡	✗	✗
Alignment [ 7 , 8 ]	✓	intra-cat.	✓ †	✗
Analytical SDS [ 11 , 12 ]	✓	✗	✓	✓
POIL (ours)	✓	func. part ‡	✓	✓
† Closed-loop alignment to an intermediate pose, then open-loop replay.

TABLE I: One-shot imitation learning approaches for manipulation, compared along the four axes discussed in Sec. I.

Fig. 1: Overview of POIL. The transfer stage grounds the functional part, establishes point correspondences, and transfers the demonstrated point trajectory to the target scene. The execution stage tracks the same points online and drives them with Point-set BCSDM.

Fig. 2: Handle detection from multi-view mug images. Multi-view prompting enables Rex-Omni to localize the handle using cross-view context.

Fig. 3: Trajectory adaptation to a new fixture. User-clicked correspondences between the demonstration’s straight rack (left) and the target’s L-shaped rack (right) define a TPS warp $\Phi$ . Applying $\Phi$ to every waypoint of the reference trajectory yields the adapted trajectory in the target scene.

Policy	#Demos	PnP Cube		Mug Insert.		Reshelving		Hanging Bag
Policy	#Demos	Near	Far	Near	Far	Near	Far	Near	Far
PP [ 21 ]	1	100	54	33	25	71	31	51	45
PP [ 21 ]	50	100	100	100	84	100	57	100	100
IP [ 6 ]	1	100	100	0	0	4	3	23	19
Ours	1	100	100	100	100	100	100	100	100
PP: Point Policy [ 21 ] , IP: Instant Policy [ 6 ] .

TABLE II: Control robustness under varying initial poses. Success rate (%) over 100 trials. Near denotes the region used to collect demonstrations for PP-50, and Far samples poses outside it.

Fig. 4: Functional-part point transfer under viewpoint changes. We report part point error (left) and the fraction of transferred points within a distance threshold (right). The ablation without multi-view prompting (MVP) isolates the effect of cross-view reasoning.

Fig. 5: Demonstration sequences for the control tasks. Object motion is rendered from blue to red.

Fig. 6: Success rate on Mug Insertion versus the initial pose’s xy distance to the training region (20 cm bins; shaded: in-distribution). The 50-demo Point Policy degrades with distance, while ours stays at 100%.

Fig. 7: End-to-end execution under object-centric closed-loop control. Object motion is rendered from blue to red. (a) Teapot pouring with perturbation recovery. (b) Mug hanging with a side-lying target mug and a different feasible grasp.

Fig. 8: Trajectory adaptation under goal-geometry changes: a cube placed in a taller box, a mug hung on an L-hook rod, and a plate seated in a tilted rack slot. Top: demonstration. Bottom: execution along the trajectory adapted to the target fixture.

Fig. 9: Representative failures in the side-lying mug evaluation. Left: the mug is detected in all views, but the hidden handle causes Rex-Omni [ 25 ] to ground the rack as the handle (red box). Right: a feasible initial grasp leads to mug-floor collision, outside the current feasibility check.

Fig. 10: Real-world cross-category transfer from simulation (top) and human hand demonstration (bottom). Robot and object motion is rendered from blue (initial) to red (final), with tracked point trajectories overlaid.

Fig. 11: Disturbance recovery during sim-to-real mug hanging. A human pulls the gripper and rotates the mug within the grasp (red; intermediate frames below), yet the handle reaches the demonstrated goal (bottom right).

Explore similar work

ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstration
Jun 8, 2026Yuying Zhang, Francesco Verdoja, Wenyan Yang +1Robot Skill Learning Robotic RL
School of Electrical Engineering Aalto University, Finland
RE4: Transformation-aware Imitation of Object Interactions Using Manipulation Modes
Jun 23, 2026Arsh Chawla, Rahul ShomeImitation Learning Robotic Manipulation
Australian National University
Imagine2Act: Leveraging Object-Action Motion Consistency from Imagined Goals for Robotic Manipulation
Sep 21, 2025Liang Heng, Jiadong Xu, Yiwen Wang +6Robot Skill Learning Language-Conditioned Robot Manipulation
CFCS, School of Computer Science, Peking University · PrimeBot · AGIBOT

Jun 8, 2026J/K move · Enter open · S save

ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstration

Yuying Zhang, Francesco Verdoja, Wenyan Yang, Ville Kyrki

School of Electrical Engineering Aalto University, Finland

Robot Skill Learning Robotic RL Imitation Learning Robotic Manipulation Data-Efficient Robot Learning Interactive Imitation Learning+1

Open paper AskPDF arXiv

Learning robot manipulation policies with deep neural networks from a single demonstration remains highly challenging, as even small deviations from the demonstrated trajectory can quickly compound into failure, while collecting substantial online interaction data is costly. We propose ReGIL, a retrieval-guided imitation learning framework that treats a single demonstration as an external memory. ReGIL repeatedly queries this static memory throughout training to simultaneously guide exploration, generate the regularization buffer, and construct rewards. Specifically, it computes rewards through local temporal alignment between the current trajectory and the retrieved segment, providing step-wise and informative feedback for policy improvement. We evaluate ReGIL on robotic manipulation tasks from the LIBERO and Meta-World benchmarks under the single demonstration setting. ReGIL outperforms prior baselines in both success rate and training efficiency. In real-robot experiments, using only one demonstration and less than one hour of online training, ReGIL achieves over 75% success rate across three manipulation tasks with randomness in both initial robot pose and target position. These results demonstrate that leveraging the single demonstration as reusable memory can provide more than static supervision for efficient robot learning. More details can be found on our website: https://regil2026.github.io/

Abstract

Explore similar work

ReGIL: Retrieval-Guided Imitation Learning from a Single Demonstration

RE4: Transformation-aware Imitation of Object Interactions Using Manipulation Modes

Imagine2Act: Leveraging Object-Action Motion Consistency from Imagined Goals for Robotic Manipulation