Organizations: Department of Advanced Robotics Research, Toyota Motor Corporation, 543, Kirigahora, Nishihirose-cho, Toyota, Aichi, Japan · Faculty of Information Science, Aichi Institute of Technology, 1247, Yachigusa, Yakusa-cho, Toyota, Aichi, Japan
To use assistive robots in everyday life, a remote control system with common devices, such as 2D devices, is helpful to control the robots anytime and anywhere as intended. Hand-drawn sketches are one of the intuitive ways to control robots with 2D devices. However, since similar sketches have different intentions from scene to scene, existing work needs additional modalities to set the sketches' semantics. This requires complex operations for users and leads to decreasing usability. In this paper, we propose Sketch-MoMa, a teleoperation system using the user-given hand-drawn sketches as instructions to control a robot. We use Vision-Language Models (VLMs) to understand the user-given sketches superimposed on an observation image and infer drawn shapes and low-level tasks of the robot. We utilize the sketches and the generated shapes for recognition and motion planning of the generated low-level tasks for precise and intuitive operations. We validate our approach using state-of-the-art VLMs with 7 tasks and 5 sketch shapes. We also demonstrate that our approach effectively specifies the detailed motions, such as how to grasp and how much to rotate. Moreover, we show the competitive usability of our approach compared with the existing 2D interface through a user experiment with 14 participants.
Figures & tables
Fig. 1: We propose Sketch-MoMa: easy and intuitive teleoperation for mobile manipulation with hand-drawn sketches by combining task planning with a VLM and motion planning.
Fig. 2: Overview of Sketch-MoMa. Users can interact with a mobile manipulator via a 2D interface by some widgets and drawing sketches on the canvas (left above). We bridge sketch instructions to robot control by implementing task planning with a VLM and motion planning with user-given sketches.
Fig. 3: Key text descriptions to ground a VLM to understand sketches.
Fig. 4: In motion planning, we detect objects and their poses based on tasks, given sketches, and their shapes. We then plan the end-effector trajectory to the target objects. The robot asks for feedback from the user after reaching the target objects.
Circle
U-shape
Arrow
Path
Circle&Arrow
Pick
✓
✓
Place
✓
✓
Move
✓
✓
Pull
✓
✓
✓
✓
Push
✓
✓
Drop
✓
TABLE I: Task-and-Shape Pairs included in Dataset
Fig. 5: Variation of sketches for the instruction of detailed motions. We provide 4 directions for grasping with a U-shape and two rotations with an arrow shape. We set numbers on objects with SoM [ 47 ] for VoxPoser [ 20 ] to make VLMs understand the specified objects visually.
Fig. 6: Environment and task scenes of our user study.
Table 8
Task
Variation
Ours
CaP [ 27 ]
VoxPoser [ 20 ]
VSR 1
TSR 2
VSR
TSR
VSR
TSR
Pick
Right
9/10 3
7/10
0/10
0/10
4/10
0/10
Left
6/10
5/10
2/10
1/10
4/10
0/10
Above
7/10
7/10
1/10
1/10
7/10
0/10
Front
8/10
7/10
2/10
1/10
6/10
0/10
Rotate
+π
9/10
8/10
3/10
1/10
5/10
4/10
TABLE IV: Validation with Sketches Containing Detailed Specification.
Metrics
Ours
Baseline
μ *
s *
μ
s
p *
g *
mw
45.2 †
19.0
53.4
17.0
0.09
0.45
md
2.93
1.00
2.50
1.09
0.16
0.41
mi
2.71
0.99
2.35
1.01
0.13
-0.36
me
1.50
0.51
1.64
0.49
0.34
0.28
TABLE V: Statistical Result of Metrics on User Study.
Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.
the College of Computer Science and Technology, National University of Defense Technology, Changsha, Hunan 410073, China · the School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha, Hunan 410082, China