TrySee: Modular Perception-Enhanced VLM Grounding for Closed-Loop Robot Task Planning in Specialized Domains
Authors: Elias Hofele, Thomas Trapp, David Timmermann, Tristan Schnell, Georg Heppner, Arne Roennau
Organizations: FZI Research Center for Information Technology · Machine Intelligence and Robotics Lab (MaiRo), Karlsruhe Institute for Technology (KIT)
Vision Language Models (VLMs) have shown impressive capabilities in high-level reasoning and task planning for robots. However, specialized domains remain a challenge for VLMs' image interpretation and grounding abilities. We introduce TrySee, a modular closed-loop robot task planning pipeline integrating general-purpose VLM reasoning with domain-specific perception and established robotics components. The system consists of an inherently reactive cyclic planning pipeline, augmented with domain-specific modular vision preprocessing, to provide online grounding. Finally, a dynamic preprocessing selection mechanism contributes to the scalability of the pipeline. The system requires neither VLM fine-tuning nor end-to-end demonstrations and adapts through replaceable perception modules and motion planning. The approach is evaluated on planning tasks across 5 widely different domains, including a real-world robot disassembly task. The field tests in a brownfield environment demonstrate the approach's practicality in integrating VLM planning into specialized robotic systems with limited domain-specific training data.
Figures & tables
Fig. 1: Example images used to evaluate TrySee, preprocessed by the pipeline’s Vision Preprocessing Module. The images are labeled A through E, corresponding to the tasks used to evaluate the proposed pipeline (see Table II ). A shows a computer assembly task, B an agricultural task, C a service task, D an aircraft cockpit interaction task, and E an e-waste disassembly task. Additional examples are collected in Fig. 3 .
Fig. 2: TrySee is a cyclic planning pipeline, with the numbers in the top-right corners indicating the flow of operations. In each iteration, the next step of the overall plan is generated, executed and verified. The Vision Preprocessing Module is central to the pipeline, and interacts with every one of the other components. The images shown were not used in the experiments, but purely serve to illustrate the pipeline’s operation.
Action
Parameters
open_gripper
N/A
close_gripper
N/A
move_to
location label
move_next_to
location label & direction
move_towards
direction
Additional task-specific actions…
TABLE I: Toolbox of actions available to the robot.
Task
Objects
Domain
Evaluation
A
PC assembly
PC components
Workshop
Simulation
B
Crop care & harvest
Plant species
Agriculture
Simulation
C
Order fulfillment
Facial recognition
Service
Simulation
D
Cockpit interaction
Button/dial status
Aerospace
Simulation
E
Heat cost allocator disassembly
Allocator parts
E-waste recycling
Hardware
TABLE II: Overview of the tasks (with respective domains & required object recognition capabilities) used to perform the evaluation.
Fig. 3: Example excerpts of input image sequences for the VLM, as used in the evaluation. The sequences are labeled A through E, corresponding to the tasks in Table II .
Fig. 4: The hardware setup used for the real-world evaluation of the pipeline. Note the milling spindle on the right side of the table (circled in green), as well as the cameras mounted on the frame on the top and on the left (circled in red), providing both a top-down and a lateral perspective of the workspace.
Components
Accuracy [%]
Plan.
Verif.
Overall
A Workshop
Baseline
0.0
50.9
0.0
Descriptions D
0.0
52.8
0.0
Vision V
72.7
81.1
27.3
D+V
72.7
81.1
27.3
D+V+ Selection S
- " -
90.6
54.6
TABLE III: Overview of the results of the ablation study, by task.
Components
Accuracy [%]
Plan.
Verif.
Overall
A Workshop (Gemma 3)
Baseline
18.2
60.4
0.0
Descriptions D
27.3
66.0
0.0
Vision V
81.8
81.1
45.5
D+V
81.8
84.9
45.5
D+V+ Selection S
- " -
94.3
63.6
TABLE IV: Cross-model evaluation on task A using Gemma 3.