Authors: Hieu Bui, Ziyan Gao, Yuya Hosoda, Joo-Ho Lee
Organizations: Graduate School of Information Science and Engineering, Ritsumeikan University, Japan. · Japan Advanced Institute of Science and Technology (JAIST). · College of Information Science and Engineering, Ritsumeikan University, Japan.
As one of the simplest non-prehensile manipulation skills, pushing has been widely studied as an effective means to rearrange objects. Existing approaches, however, typically rely on multi-step push plans composed of pre-defined pushing primitives with limited application scopes, which restrict their efficiency and versatility across different scenarios. In this work, we propose a unified pushing policy that incorporates a lightweight prompting mechanism into a flow matching policy to guide the generation of reactive, multimodal pushing actions. The visual prompt can be specified by a high-level planner, enabling the reuse of the pushing policy across a wide range of planning problems. Experimental results demonstrate that the proposed unified pushing policy not only outperforms existing baselines but also effectively serves as a low-level primitive within a VLM-guided planning framework to solve table-cleaning tasks efficiently.
Figures & tables
Fig. 1 : Illustration of a specific table-cleaning task in which all red blocks must be placed in the left staging area, while blue blocks are placed in the right staging area. The numbered annotations indicate one possible sequence of actions considering the feasibility and efficiency.
Fig. 2 : Model Architecture. The input consists of the visual prompt and the latest Tobs steps of image data and robot proprioception. The policy is parameterized by a Diffusion Transformer with alternating self-attention and cross-attention DiT blocks to denoise action tokens A0 into executable trajectories A1.
Fig. 3 : Illustration of the visual prompt. Left: Input prompts consisting of two points: u1 (blue) and u2 (green). Right: Policy-generated action trajectories. The warm and cold colors represent the trajectory of the right and left finger of the gripper.
Fig. 4 : The experimental setup consists of a leader-follower system. The follower (left) is equipped with a wrist-mounted camera and a parallel jaw gripper. The leader device (middle) is used to teleoperate the follower. In the data collection phase, only the red blocks were used, while the embedded figure at the bottom right shows objects used for evaluation.
Method
Displacement
Grouping
Singulation
Goal-Image Policy
60
60
30
Single-Task Policy
75
60
70
Unified Policy (Ours)
85
70
65
TABLE I : Success Rate (%) Comparison with baselines ( 20 trials)
Task Setting
Unified Policy (Ours)
Goal-Image
Displacement ( 3 objs)
70
40
Displacement ( 5 objs)
50
30
Grouping ( 3 objs)
70
40
Grouping ( 5 objs)
70
40
Singulation ( 2 clusters)
50
40
TABLE II : Success Rate (%) in Cluttered Environments ( 10 trials)
Fig. 5 : Overview of the VLM planning framework
Fig. 6 : Snapshots of a table-cleaning task with manual prompting. (a)-(d) Singulation to isolate a red block and perform pick-and-place. (e)-(f) Displacement to rearrange 2 black boxes into target area. (g)-(j) Grouping to group 2 red blocks and simultaneously pick-and-place them to target regions. Note that the annotated dots in (a), (e), and (g) are the visual prompts: the blue dot represents u1 while the other represents u2 .