TiPToP: A Modular Open-Vocabulary Robot Manipulation System That Plans
Authors: William Shen, Nishanth Kumar, Sahit Chintalapudi, Ryan Lindeborg, Jie Wang, Christopher Watson, Edward Hu, Jing Cao, Dinesh Jayaraman, Leslie Pack Kaelbling, Tomás Lozano-Pérez
Organizations: MIT CSAIL · University of Pennsylvania
We present TiPToP, a modular manipulation system that integrates pretrained foundation models with a GPU-accelerated Task and Motion Planner to solve tasks directly from RGB images and natural language. TiPToP composes perception, planning, and execution modules and requires no robot training data. It can be deployed on a standard DROID setup in under an hour and adapted to new embodiments with minimal effort. We evaluate TiPToP against π0.5-DROID, a state-of-the-art VLA fine-tuned on 350 hours of demonstrations, across two real-world DROID setups (one operated by an external team) and simulation, where TiPToP attains a higher average success rate and faster average completion time. We also evaluate on the MolmoSpaces benchmark, where TiPToP ranks first overall on pick and pick-and-place tasks among methods not trained on in-distribution data. We further show that TiPToP's modularity enables us to trace failures to specific components, revealing where to target improvements. We release TiPToP open-source to serve as a reproducible baseline and to enable further research on modular manipulation systems. Project website and code: https://tiptop-robot.github.io
Figures & tables
Fig. 2 : TiPToP System Overview. From a stereo RGB pair and language instruction L , TiPToP outputs joint trajectories with gripper commands. (a) Perception builds an object-centric 3D scene representation via depth estimation, grasp prediction, object detection, and segmentation. (b) Planning runs GPU-parallelized TAMP (cuTAMP) [ 57 ] to find feasible manipulation plans. (c) Execution tracks the plan with a joint impedance controller.
Fig. 3 : Perception Module. (a) Depth map predicted by FoundationStereo with sharp object boundaries. (b) Grasps predicted by M2T2 on the scene point cloud (colors correspond to grasp confidences). (c) Labeled object bounding boxes and symbolic goal G predicted by Gemini ( On(a,b) specifies that object a should be placed on object or surface b ).
Fig. 4 : Shape-Completion Mode. From the observed scene (left), RecGen [ 75 ] faithfully recovers the full geometry of each object (right), including the elephant watering can.
TiPToP
π0.5 -DROID
Scene
SR
TP
SR
TP
Simple
Cube → bowl (sim)
5/10
72.5%
8/10
90%
Can → mug (sim)
9/10
97.5%
2/10
50%
Banana → bin (sim)
0/10
70%
9/10
97.5%
Marker → tray
3/5
80%
5/5
100%
TABLE I: Per-scene performance comparison over 28 scenes. SR = Success Rate, TP = Task Progress. Best results are bolded . Per category and overall, SR is summed and TP averaged. † Evaluated by system designers; unmarked scenes evaluated by the external evaluation team.
π0.5 -DROID
TiPToP
Scene
Time (s)
Time (s)
Plan (s)
Simulation
Cube → bowl
27.4
17.9
9.7
Can → mug
41.0
18.6
9.2
Real-World
Crackers → tray (simple)
32.2
14.9
7.0
TABLE II: Completion time comparison. ‘Time’ reports average time-to-success over successful trials only. ‘Plan’ reports the time required for TiPToP to run the perception and planning modules (included in ‘Time’).
MolmoSpaces
MolmoBot Pick variants
MolmoBot Pick & Place variants
Overall
Method
Pick
PnP
MSProc
Classic
Filam.
RandCam
PnP
PnP-NextTo
PnP-Color
Non-In-Distribution Methods
TiPToP (ours)
68.7
33.2
67.5
50.0
48.5
47.8
29.4
38.0 ⋆
31.5
46.1
WALL-OSS-0.5 (VLA) [ 73 ]
73.8
46.0
78.4
47.9
47.2
30.1
13.8
15.9
18.3
41.3
Cosmos3-Nano (WAM) [ 2 ]
65.5
35.2
66.5
32.3
32.2
25.7
26.0
22.9
34.3
37.8
Psi-R2 (WAM) [ 53 ]
73.1
41.5
50.2
23.4
25.8
27.2
2.6
2.9
1.5
27.6
TABLE III: MolmoSpaces benchmark results (oracle success rate %). Best non-ID method per task in bold . ⋆ ranks first on the full leaderboard. Integration details in Appendix -F .
Fig. 5 : Failure Analysis. Sankey diagram showing outcomes of 173 trials. The most common failure modes are grasping failures (missed or unstable grasps), followed by scene-completion errors, VLM detection errors, then cuTAMP failures.
Fig. 6 : Wiping. We demonstrate that TiPToP can be straightforwardly extended to perform wiping in addition to pick-and-place. Task instruction: “erase the whiteboard and put everything into the bowl”.
Fig. 7 : Object Segmentation. SAM-2 generates eight pixel-level segmentation masks from the bounding boxes in Fig. 3 c.
Scene
Identifier / Language Prompt
Progress Metric
Scene
Identifier / Language Prompt
Progress Metric
Simple
Cube → bowl (sim) “put the cube in the bowl”
25% approach cube, 50% grasp, 75% approach bowl with cube, 100% place
Can → mug (sim) “put the can in the mug”
25% approach can, 50% grasp, 75% approach mug with can, 100% place
Banana → bin (sim) “put banana in the bin”
25% approach banana, 50% grasp, 75% approach bin with banana, 100% place
Marker → tray “put the marker in the tray”
+25% touch marker, +25% grasp, +25% touch tray, +25% place
Crackers → tray † “place the crackers onto the tray”
50% grasp crackers, 100% place
Distractor
Meat can → sugar box (sim) “put the meat can on the sugar box”
25% approach meat can, 50% grasp, 75% approach box with meat can, 100% place
Coffee capsules → plate “put all of the coffee capsules onto the white plate”
+50% per capsule placed, − 20% per distractor
TABLE IV: Evaluation scene details. Each scene shows an image of the task, its identifier (as referenced in Table I ), language prompt, and task progress metric. Scenes are grouped by category: Simple, Distractor, Semantic, and Multi-step. † indicates tasks evaluated by the system designers. Unmarked scenes are evaluated by external evaluators not involved in the development of TiPToP. (sim) denotes tasks evaluated in simulation. Task progress metric numbers are reported in %; a + or − sign indicates that the particular denoted amount is added or subtracted from the overall score, and no sign indicates that the number is the absolute score for achieving that particular condition. Progress metrics may vary by the evaluator and the task. Some metrics penalize manipulating distractors while others do not.