Organizations: Department of Advanced Robotics Research, Toyota Motor Corporation, 543, Kirigahora, Nishihirose-cho, Toyota, Aichi, Japan · Faculty of Information Science, Aichi Institute of Technology, 1247, Yachigusa, Yakusa-cho, Toyota, Aichi, Japan
To use assistive robots in everyday life, a remote control system with common devices, such as 2D devices, is helpful to control the robots anytime and anywhere as intended. Hand-drawn sketches are one of the intuitive ways to control robots with 2D devices. However, since similar sketches have different intentions from scene to scene, existing work needs additional modalities to set the sketches' semantics. This requires complex operations for users and leads to decreasing usability. In this paper, we propose Sketch-MoMa, a teleoperation system using the user-given hand-drawn sketches as instructions to control a robot. We use Vision-Language Models (VLMs) to understand the user-given sketches superimposed on an observation image and infer drawn shapes and low-level tasks of the robot. We utilize the sketches and the generated shapes for recognition and motion planning of the generated low-level tasks for precise and intuitive operations. We validate our approach using state-of-the-art VLMs with 7 tasks and 5 sketch shapes. We also demonstrate that our approach effectively specifies the detailed motions, such as how to grasp and how much to rotate. Moreover, we show the competitive usability of our approach compared with the existing 2D interface through a user experiment with 14 participants.
Figures & tables
Fig. 1: We propose Sketch-MoMa: easy and intuitive teleoperation for mobile manipulation with hand-drawn sketches by combining task planning with a VLM and motion planning.
Fig. 2: Overview of Sketch-MoMa. Users can interact with a mobile manipulator via a 2D interface by some widgets and drawing sketches on the canvas (left above). We bridge sketch instructions to robot control by implementing task planning with a VLM and motion planning with user-given sketches.
Fig. 3: Key text descriptions to ground a VLM to understand sketches.
Fig. 4: In motion planning, we detect objects and their poses based on tasks, given sketches, and their shapes. We then plan the end-effector trajectory to the target objects. The robot asks for feedback from the user after reaching the target objects.
Circle
U-shape
Arrow
Path
Circle&Arrow
Pick
✓
✓
Place
✓
✓
Move
✓
✓
Pull
✓
✓
✓
✓
Push
✓
✓
Drop
✓
TABLE I: Task-and-Shape Pairs included in Dataset
Fig. 5: Variation of sketches for the instruction of detailed motions. We provide 4 directions for grasping with a U-shape and two rotations with an arrow shape. We set numbers on objects with SoM [ 47 ] for VoxPoser [ 20 ] to make VLMs understand the specified objects visually.
Fig. 6: Environment and task scenes of our user study.
Table 8
Task
Variation
Ours
CaP [ 27 ]
VoxPoser [ 20 ]
VSR 1
TSR 2
VSR
TSR
VSR
TSR
Pick
Right
9/10 3
7/10
0/10
0/10
4/10
0/10
Left
6/10
5/10
2/10
1/10
4/10
0/10
Above
7/10
7/10
1/10
1/10
7/10
0/10
Front
8/10
7/10
2/10
1/10
6/10
0/10
Rotate
+π
9/10
8/10
3/10
1/10
5/10
4/10
TABLE IV: Validation with Sketches Containing Detailed Specification.
Metrics
Ours
Baseline
μ *
s *
μ
s
p *
g *
mw
45.2 †
19.0
53.4
17.0
0.09
0.45
md
2.93
1.00
2.50
1.09
0.16
0.41
mi
2.71
0.99
2.35
1.01
0.13
-0.36
me
1.50
0.51
1.64
0.49
0.34
0.28
TABLE V: Statistical Result of Metrics on User Study.
Robot teleoperation studies how people remotely control robots through interfaces, sensing, feedback, and autonomy, despite distance, delay, and embodiment gaps.
Most agentic robotic systems use vision-language models (VLMs) either to orchestrate predefined skills or to directly generate actions, constraints, and control programs. The former is limited by the available skill library, while the latter requires VLMs to reason about precise geometry and contact. In this paper, we propose WorldSketch, a new paradigm for agentic robotics in which a VLM constructs a world model rather than generating robot actions. The VLM acts as an agent that actively queries perception tools to build and visually validate a minimal kinematic world model, capturing only task-relevant objects, their interaction geometry, and possible motions. A geometric planner then jointly optimizes contact and whole-body motion on the resulting model. Moreover, the world model is self-improving: execution feedback verifies outcomes, updates uncertain hypotheses, and enables the agent to revise the model when its predictions are contradicted. We demonstrate the generality of this paradigm across RoboCasa, LIBERO-PRO, and R2R-CE navigation, achieving zero-shot success rates of 91%, 97%, and 76%, respectively. On RoboCasa, WorldSketch outperforms the next-best zero-shot agent baseline by 58 percentage points while using 10 times fewer VLM input tokens; on LIBERO-PRO, it exceeds CaP-X by 82 percentage points. In navigation, it matches the success rate of step-by-step VLM agents while using 6 times fewer input tokens.
We introduce AnyUser, a unified robotic instruction system for intuitive domestic task instruction via free-form sketches on camera images, optionally with language. AnyUser interprets multimodal inputs (sketch, vision, language) as spatial-semantic primitives to generate executable robot actions requiring no prior maps or models. Novel components include multimodal fusion for understanding and a hierarchical policy for robust action generation. Efficacy is shown via extensive evaluations: (1) Quantitative benchmarks on the large-scale dataset showing high accuracy in interpreting diverse sketch-based commands across various simulated domestic scenes. (2) Real-world validation on two distinct robotic platforms, a statically mounted 7-DoF assistive arm (KUKA LBR iiwa) and a dual-arm mobile manipulator (Realman RMC-AIDAL), performing representative tasks like targeted wiping and area cleaning, confirming the system's ability to ground instructions and execute them reliably in physical environments. (3) A comprehensive user study involving diverse demographics (elderly, simulated non-verbal, low technical literacy) demonstrating significant improvements in usability and task specification efficiency, achieving high task completion rates (85.7%-96.4%) and user satisfaction. AnyUser bridges the gap between advanced robotic capabilities and the need for accessible non-expert interaction, laying the foundation for practical assistive robots adaptable to real-world human environments.
Songyuan Yang, Huibin Tan, Kailun Yang +2
the College of Computer Science and Technology, National University of Defense Technology, Changsha, Hunan 410073, China · the School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha, Hunan 410082, China
Inserting an unseen part requires sub-millimetre precision and the correct orientation of keyed features. Current zero-shot agentic manipulation methods built on vision-language models (VLMs) produce spatial outputs that are far too coarse for this. We propose to let a VLM build a sketch model of the task, i.e., a few geometric primitives and two mate frames, which a VLM agent measures with geometric tools in multi-view depth and which a standard planner and compliant controller execute. Because a sketch describes the parts independently of their placement, the robot needs to build it only once per task. A sketch memory stores each agent run and transfers the stored sketch to later placements by point-cloud registration, so that the VLM is called only for unfamiliar parts. After grasping the part, the robot inspects both mating faces at close range and corrects the sketch by shape matching. On 69 simulated insertion episodes, our method inserts 88% of the parts at the assembled pose in a strict kinematic test, while ReKep and MOKA with the same VLM insert none. On new placements of eight tasks, reusing one stored run inserts as many parts as calling the agent in each episode, and it produces the sketch in 3.4 s instead of about 90 s.