cs.RODec 26, 2024

Sketch-MoMa: Teleoperation for Mobile Manipulator via Interpretation of Hand-Drawn Sketches

Authors: Kosei Tanada, Yuka Iwanaga, Masayoshi Tsuchinaga, Yuji Nakamura, Takemitsu Mori, Remi Sakai, Takashi Yamamoto

Organizations: Department of Advanced Robotics Research, Toyota Motor Corporation, 543, Kirigahora, Nishihirose-cho, Toyota, Aichi, Japan · Faculty of Information Science, Aichi Institute of Technology, 1247, Yachigusa, Yakusa-cho, Toyota, Aichi, Japan

Abstract

To use assistive robots in everyday life, a remote control system with common devices, such as 2D devices, is helpful to control the robots anytime and anywhere as intended. Hand-drawn sketches are one of the intuitive ways to control robots with 2D devices. However, since similar sketches have different intentions from scene to scene, existing work needs additional modalities to set the sketches' semantics. This requires complex operations for users and leads to decreasing usability. In this paper, we propose Sketch-MoMa, a teleoperation system using the user-given hand-drawn sketches as instructions to control a robot. We use Vision-Language Models (VLMs) to understand the user-given sketches superimposed on an observation image and infer drawn shapes and low-level tasks of the robot. We utilize the sketches and the generated shapes for recognition and motion planning of the generated low-level tasks for precise and intuitive operations. We validate our approach using state-of-the-art VLMs with 7 tasks and 5 sketch shapes. We also demonstrate that our approach effectively specifies the detailed motions, such as how to grasp and how much to rotate. Moreover, we show the competitive usability of our approach compared with the existing 2D interface through a user experiment with 14 participants.

Figures & tables

Explore similar work

CardsList
  1. AnyUser: Translating Sketched User Intent into Domestic Robots

    Apr 6, 2026Songyuan Yang, Huibin Tan, Kailun Yang +2Human-Robot InteractionAssistive Robotics