Organizations: the College of Computer Science and Technology, National University of Defense Technology, Changsha, Hunan 410073, China · the School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha, Hunan 410082, China
We introduce AnyUser, a unified robotic instruction system for intuitive domestic task instruction via free-form sketches on camera images, optionally with language. AnyUser interprets multimodal inputs (sketch, vision, language) as spatial-semantic primitives to generate executable robot actions requiring no prior maps or models. Novel components include multimodal fusion for understanding and a hierarchical policy for robust action generation. Efficacy is shown via extensive evaluations: (1) Quantitative benchmarks on the large-scale dataset showing high accuracy in interpreting diverse sketch-based commands across various simulated domestic scenes. (2) Real-world validation on two distinct robotic platforms, a statically mounted 7-DoF assistive arm (KUKA LBR iiwa) and a dual-arm mobile manipulator (Realman RMC-AIDAL), performing representative tasks like targeted wiping and area cleaning, confirming the system's ability to ground instructions and execute them reliably in physical environments. (3) A comprehensive user study involving diverse demographics (elderly, simulated non-verbal, low technical literacy) demonstrating significant improvements in usability and task specification efficiency, achieving high task completion rates (85.7%-96.4%) and user satisfaction. AnyUser bridges the gap between advanced robotic capabilities and the need for accessible non-expert interaction, laying the foundation for practical assistive robots adaptable to real-world human environments.
Figures & tables
Fig. 1 : AnyUser architecture and runtime workflow. The user provides a third-person photograph I , draws sketches S on the image, and may add a short language cue L . The sketch is deterministically segmented into an ordered sequence Sseq and, together with I and L , is encoded by the multimodal model ffuse to yield a runtime representation that conditions the hierarchical policy πHL . For each segment the policy produces a high-level command a′∗k from the discrete set Adisc . During execution, egocentric live perception Pt can be injected as an additional image-channel input for reactive checks such as obstacle presence and under-obstacle clearance. The command translation module gtranslate converts each high-level command into platform-specific multi-DoF control ak∈ADoF , which is executed by the robot’s low-level controllers in a closed loop.
Symbol
Meaning
I,S,L
Environment photo, user sketches, language cue
I=(I,S,L)
Multimodal instruction tuple
Sseq={sk}k=1Nseg
Ordered post-segmentation sketch segments; Nseg is count
xt,Pt
Robot state and live perception at time t
ϕV,ϕS,ϕL
Visual, sketch, and language encoders
ψfuse
Multimodal fusion and spatial grounding module
TABLE I : Main notations used in our paper.
Fig. 2 : Overview of the HouseholdSketch dataset utilized for training and evaluation. Left: A selection of representative images illustrating the visual diversity of indoor environments included. Center: Pie chart depicting the proportional distribution of scene categories within the dataset. Right: Exemplar sketch inputs overlaid on corresponding scene images and shown in isolation.
Fig. 3 : Detailed pipeline of the AnyUser’s architecture. The Input Layer receives Visual ( I ), Sketch ( S ), and Linguistic ( L ). After preprocessing, encoders extract modality features. Fusion ψfuse aligns sketch with image via cross-modal attention and an MLP, and the hierarchical policy πHL predicts a macro-action per segment. Command translation gtranslate converts these macros into platform-specific multi-DoF primitives for execution.
Fig. 4 : Representative robotic platforms relevant to this work. (a) The KUKA LBR iiwa, a 7-DoF collaborative manipulator. (b) The Realman RMC-AIDAL, a dual-arm mobile manipulation platform.
Single-Step
Single-Step Strict Path
Full Task
Full Task Strict Path
Scene
Success Rate (%)
Adherence Rate (%)
Completion Rate (%)
Adherence Rate (%)
Short
Medium
Long
Short
Medium
Long
Short
Medium
Long
Short
Medium
Long
Bedroom
85.3
86.1
85.7
78.1
78.6
78.3
78.0
64.5
50.8
66.2
51.1
36.4
Kitchen
83.8
84.5
84.1
76.9
77.2
77.0
74.3
60.2
46.5
62.7
48.5
33.8
Living room
87.4
87.0
87.2
79.5
79.2
79.4
79.1
65.7
52.3
67.5
52.3
37.5
Bathroom
84.6
85.0
84.8
77.4
77.7
77.5
76.5
62.3
48.9
64.8
49.2
35.1
TABLE II : Quantitative performance evaluation on the HouseholdSketch dataset. Task length categories are defined by the number of detected corners (turning points) in the input sketch: Short ( ≤ 2 corners), Medium (3-5 corners), and Long ( ≥ 6 corners). The table presents Single-Step Success Rate (SSSR) , Single-Step Strict Path Adherence Rate (SSSPAR) , Full Task Completion Rate (FTCR) , and Full Task Strict Path Adherence Rate (FTSPAR) , reported in percentage (%). Results are disaggregated by scene type and task length category.
Fig. 5 : Scene-specific task-level performance comparison. Task length categories are defined by sketch complexity (Short: ≤ 2 corners, Medium: 3-5 corners, Long: ≥ 6 corners). The figure presents (Left) Full Task Completion Rate (FTCR) and (Right) Full Task Strict Path Adherence Rate (FTSPAR) in percentage (%). Results are shown for each scene category from the HouseholdSketch dataset, grouped by task length. Error bars depict simulated standard error, indicating expected performance variability.
Fig. 6 : Aggregate performance comparison across metrics and task lengths. Task length categories (Short: ≤ 2 corners, Medium: 3-5 corners, Long: ≥ 6 corners) are based on the number of turns in the input sketch. Bars represent the average success rates (%) for SSSR, SSSPAR, FTCR, and FTSPAR, computed across all scenes from the HouseholdSketch dataset. Error bars indicate simulated standard error, reflecting anticipated experimental variability.
Fig. 7 : Qualitative illustration of system operation in the iGibson simulation environment. (a) User-provided sketch overlaid on the scene, defining a multi-segment navigation task around furniture and under a table (keypoints highlighted). (b)-(g) Sequence depicting the robot’s execution trace: (b) initial forward motion, (c) approaching the table triggering under-obstacle check, (d) simulated perception check under the table, (e)-(f) navigating turns and proceeding under the table, (g) final segment execution. The sequence demonstrates task decomposition, path following, and integration of adaptive behaviors like obstacle checking.
Setting
SSSR
SSSPAR
FTCR
FTSPAR
I+S+L+Pt (full)
84.6
77.0
60.9
48.4
I+S+L (no Pt )
84.0 (-0.6)
76.0 (-1.0)
58.7 (-2.2)
47.2 (-1.2)
I+S (no user L )
83.8 (-0.8)
76.4 (-0.6)
59.4 (-1.5)
47.1 (-1.3)
S+L (no I )
78.1 (-6.5)
70.2 (-6.8)
41.3 (-19.6)
28.4 (-20.0)
I+L (no S )
68.5 (-16.1)
55.3 (-21.7)
32.1 (-28.8)
20.2 (-28.2)
TABLE III : Ablation on input modalities and live perception on HouseholdSketch (overall across scenes and task lengths). Numbers are percentages. In parentheses are absolute changes relative to the full model.
Fig. 8 : Qualitative illustration of system operation with the KUKA LBR iiwa robot executing a table wiping task. (a) User-provided sketch overlaid on the camera view, indicating a path (arrow) and a target area (rectangle). (b)-(f) Sequence depicting the robot’s execution: (b) Initial state. (c)-(d) Arm approaches the target region. (e) Arm executes wiping motion along the sketched path. (f) Arm interacts with the designated rectangular area. The sequence demonstrates successful 3D grounding, trajectory execution, and task completion based on the sketch.
Fig. 9 : Qualitative illustration of dual-arm cover-area task on the Realman RMC-AIDAL mobile manipulator driven by sketch-based commands. (a) User sketch overlaid on the onboard camera image: a single curved arrow on the tabletop specifies the region to be traversed and the desired sweeping direction. After sketch-guided navigation brings the base to a pose in front of the table, (b)–(d) the hierarchical policy instantiates a cover_area macro that converts the arrow into synchronized, collision-aware trajectories for both RM65-B arms: the right arm performs the primary sweeping motion, while the left arm adopts a reactive tucked posture to maintain clearance in the cluttered workspace.
Metric
Elderly (n=10)
Sim. Non-Verbal (n=8)
Low Tech Literacy (n=7)
Control (n=7)
Specification Time (s)
62.5 (±14.8)
55.1 (±10.2)
68.3 (±16.5)
49.8 (±9.0)
Attempts per Task
1.5 (±0.7)
1.3 (±0.5)
1.7 (±0.8)
1.2 (±0.4)
Full Task Completion Rate (FTCR) (%)
90.0%
93.8%
85.7%
96.4%
Expert Fidelity Rating (EFR) (1-5)
4.0 (±0.8)
4.2 (±0.6)
3.8 (±0.9)
4.4 (±0.5)
Subjective Ease of Use (1-5)
4.3 (±0.7)
4.5 (±0.5)
3.9 (±1.0)
4.6 (±0.4)
Subjective Confidence (1-5)
4.1 (±0.8)
4.4 (±0.6)
3.7 (±1.1)
4.5 (±0.5)
TABLE IV : User Study Performance Metrics Across Diverse Demographic Groups. Values represent mean (± standard deviation) where applicable. TCSR is presented as a percentage. Subjective ratings and EFR are on a 1-5 scale (higher is better).
Failure source
Share %
Perception-driven mismatches in grounding with I or Pt
29.1
Sketch-to-scene misalignment in fusion ffuse
23.7
Macro-action misclassification in πHL
20.8
Execution drift in gtranslate and low-level control
19.6
Timeouts or conservative halts from safety checks
6.8
TABLE V : Primary failure sources on Long tasks in HouseholdSketch (audited subset, percentages sum to 100).
Setting
SSSR
SSSPAR
FTCR
FTSPAR
Baseline: ±45∘,±90∘ , 0.05m
84.4
76.9
46.7
33.5
Coarser: ±90∘ , 0.10m
83.9
71.8 ( −5.1 )
42.3 ( −4.4 )
28.1 ( −5.4 )
Finer: ±22.5∘,±45∘,±90∘ , 0.025m
84.1
79.3 (+2.4)
45.6 ( −1.1 )
35.4 (+1.9)
Continuous yaw regression, 0.05m
84.0
78.4 (+1.5)
44.1 ( −2.6 )
34.2 (+0.7)
TABLE VI : Effect of action-space resolution on Long tasks in HouseholdSketch. Percentages reported; deltas are absolute changes relative to the baseline.
Setting
Macro Acc.
SSSR
FTCR
FTSPAR
L0ZS (zero-shot)
88.4
84.0
74.2
62.0
L01S (one-shot)
89.8
85.1
75.2
63.0
TABLE VII : Zero-shot vs. one-shot prompting on HouseholdSketch. Values are percentages. One-shot adds a single segment-matched exemplar to L0 .
SSSR (%)
Short
Medium
Long
Overall
L0ZS
84.7
84.2
83.8
84.0
L01S
85.5
85.0
84.8
85.1
FTCR (%)
Short
Medium
Long
Overall
L0ZS
74.8
60.0
46.4
74.2
TABLE VIII : Length-conditioned comparison for zero-shot and one-shot prompting.
Setting
SSSR
SSSPAR
FTCR
UOMS
(A) (I,S,L)
84.1
76.7
58.2
62.5
(B) (I,S,L)+Pt late-fusion gated
84.3
77.5
64.9
78.4
(C) (I,S,L)+Pt naive early fusion
83.9
75.9
60.1
70.2
(D) (I,S,L)+Ptdepth (replicated)
84.0
76.1
59.0
67.3
TABLE IX : Live perception ablation on the obstacle subset (1,200 trials). Percentages are averages across scenes and sketch-length strata.
As collaborative robots have entered domains such as manufacturing, agriculture, and healthcare, programming or adapting robot behavior typically requires robotic expertise that most end users lack. Natural language lowers this barrier. Recent advancements in large language models (LLMs) have made it feasible to translate natural language into robot task plans. However, language-based task specification suffers from semantic ambiguity, and generative models lack transparency for how language instructions become robot actions, making it difficult for users to validate the plan before execution. To address these issues, we introduce SHRIMP, a system that allows users to automatically generate a hierarchical robot primitive plan using natural language and iteratively revise their plan through re-prompting and explicit correction. At each revision, SHRIMP allows users to validate their plan in simulation, and once satisfied, execute it on the physical robot. Through a user study involving participants planning tabletop kitchen tasks (n=35), we validate that SHRIMP improves perceived control and enhances robot transparency. System videos and source code are available at https://wisc-hci.github.io/SHRIMP.
Mya Schroder, Yuna Hwang, Callie Y. Kim +5
Department of Computer Sciences, University of Wisconsin–Madison, Madison, Wisconsin, USA
Natural-language interfaces can lower the barrier to programming robots, but existing systems struggle when users request complex tasks. While large language models (LLMs) perform well with simple commands, they often struggle to generate code for multi-step tasks, decompose high-level instructions, or reuse prior solutions. We present SkillComposer, an interactive natural-language robot programming system for simulation environments that continually learns reusable program abstractions. SkillComposer uses a generate-test architecture in which an LLM iteratively generates and revises robot programs before execution. Successful programs are stored and processed by an online library-learning algorithm that compresses recurring function sequences into reusable macro skills for future tasks. We evaluate SkillComposer through ablation experiments and a user study with 12 participants to determine its effectiveness on manipulation and robot caregiving tasks. The results show that evaluator-guided generation and learned abstractions improve success rates and usability while reducing user effort in natural-language robot programming.
John Woods, Hasti Seifi
School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ
We present a hierarchical language-driven framework for robotic task and motion planning to improve natural, intuitive human-robot interaction in service and assistance scenarios. The proposed system employs two large language model (LLM) modules: a high-level planning agent and a low-level spatial reasoning sub-module. The primary agent processes natural language commands and generates action sequences using a ReAct-style prompt, interacting with tools for object perception and manipulation (e.g., pick, place, release). For precise spatial placement, such as interpreting "place the mug next to the plate", a separate sub-prompting module handles 3D reasoning based on object geometry and scene layout. The system integrates YOLOX-GDRNet for object detection and pose estimation, along with a motion execution stub. We evaluated the system in 24 test scenarios, ranging from simple spatial commands to high-level instructions and infeasible requests. The system achieved an overall task success rate of 86%.
Karolina Źróbek, Tessa Pulli, Paweł Gajewski +2
IBM, Krakow, Poland · TU Wien, Vienna, Austria · Jagiellonian University, Krakow, Poland +1