AnyUser: Translating Sketched User Intent into Domestic Robots

Songyuan Yang; Huibin Tan; Kailun Yang; Wenjing Yang; Shaowu Yang

cs.ROApr 6, 2026

AnyUser: Translating Sketched User Intent into Domestic Robots

Authors: Songyuan Yang, Huibin Tan, Kailun Yang, Wenjing Yang, Shaowu Yang

Organizations: the College of Computer Science and Technology, National University of Defense Technology, Changsha, Hunan 410073, China · the School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha, Hunan 410082, China

Abstract

We introduce AnyUser, a unified robotic instruction system for intuitive domestic task instruction via free-form sketches on camera images, optionally with language. AnyUser interprets multimodal inputs (sketch, vision, language) as spatial-semantic primitives to generate executable robot actions requiring no prior maps or models. Novel components include multimodal fusion for understanding and a hierarchical policy for robust action generation. Efficacy is shown via extensive evaluations: (1) Quantitative benchmarks on the large-scale dataset showing high accuracy in interpreting diverse sketch-based commands across various simulated domestic scenes. (2) Real-world validation on two distinct robotic platforms, a statically mounted 7-DoF assistive arm (KUKA LBR iiwa) and a dual-arm mobile manipulator (Realman RMC-AIDAL), performing representative tasks like targeted wiping and area cleaning, confirming the system's ability to ground instructions and execute them reliably in physical environments. (3) A comprehensive user study involving diverse demographics (elderly, simulated non-verbal, low technical literacy) demonstrating significant improvements in usability and task specification efficiency, achieving high task completion rates (85.7%-96.4%) and user satisfaction. AnyUser bridges the gap between advanced robotic capabilities and the need for accessible non-expert interaction, laying the foundation for practical assistive robots adaptable to real-world human environments.

Figures & tables

Fig. 1 : AnyUser architecture and runtime workflow. The user provides a third-person photograph $I$ , draws sketches $S$ on the image, and may add a short language cue $L$ . The sketch is deterministically segmented into an ordered sequence $S_{seq}$ and, together with $I$ and $L$ , is encoded by the multimodal model $f_{fuse}$ to yield a runtime representation that conditions the hierarchical policy $\pi_{HL}$ . For each segment the policy produces a high-level command $a^{\prime}*k$ from the discrete set $A_{disc}$ . During execution, egocentric live perception $P_{t}$ can be injected as an additional image-channel input for reactive checks such as obstacle presence and under-obstacle clearance. The command translation module $g_{translate}$ converts each high-level command into platform-specific multi-DoF control $a_{k}\in A_{DoF}$ , which is executed by the robot’s low-level controllers in a closed loop.

Symbol	Meaning
$I,\,S,\,L$	Environment photo, user sketches, language cue
$\mathcal{I}=(I,S,L)$	Multimodal instruction tuple
$\mathcal{S}_{\text{seq}}=\{s_{k}\}_{k=1}^{N_{\text{seg}}}$	Ordered post-segmentation sketch segments; $N_{\text{seg}}$ is count
$x_{t},\,P_{t}$	Robot state and live perception at time $t$
$\phi_{V},\,\phi_{S},\,\phi_{L}$	Visual, sketch, and language encoders
$\psi_{\text{fuse}}$	Multimodal fusion and spatial grounding module

TABLE I : Main notations used in our paper.

Fig. 2 : Overview of the HouseholdSketch dataset utilized for training and evaluation. Left: A selection of representative images illustrating the visual diversity of indoor environments included. Center: Pie chart depicting the proportional distribution of scene categories within the dataset. Right: Exemplar sketch inputs overlaid on corresponding scene images and shown in isolation.

Fig. 3 : Detailed pipeline of the AnyUser’s architecture. The Input Layer receives Visual (

I

), Sketch (

S

), and Linguistic (

L

). After preprocessing, encoders extract modality features. Fusion

\psi_{\text{fuse}}

aligns sketch with image via cross-modal attention and an MLP, and the hierarchical policy

\pi_{\text{HL}}

predicts a macro-action per segment. Command translation

g_{\text{translate}}

converts these macros into platform-specific multi-DoF primitives for execution.

Fig. 4 : Representative robotic platforms relevant to this work. (a) The KUKA LBR iiwa, a 7-DoF collaborative manipulator. (b) The Realman RMC-AIDAL, a dual-arm mobile manipulation platform.

	Single-Step			Single-Step Strict Path			Full Task			Full Task Strict Path
Scene	Success Rate (%)			Adherence Rate (%)			Completion Rate (%)			Adherence Rate (%)
	Short	Medium	Long	Short	Medium	Long	Short	Medium	Long	Short	Medium	Long
Bedroom	85.3	86.1	85.7	78.1	78.6	78.3	78.0	64.5	50.8	66.2	51.1	36.4
Kitchen	83.8	84.5	84.1	76.9	77.2	77.0	74.3	60.2	46.5	62.7	48.5	33.8
Living room	87.4	87.0	87.2	79.5	79.2	79.4	79.1	65.7	52.3	67.5	52.3	37.5
Bathroom	84.6	85.0	84.8	77.4	77.7	77.5	76.5	62.3	48.9	64.8	49.2	35.1

TABLE II : Quantitative performance evaluation on the HouseholdSketch dataset. Task length categories are defined by the number of detected corners (turning points) in the input sketch: Short (

\leq

2 corners), Medium (3-5 corners), and Long (

\geq

6 corners). The table presents Single-Step Success Rate (SSSR) , Single-Step Strict Path Adherence Rate (SSSPAR) , Full Task Completion Rate (FTCR) , and Full Task Strict Path Adherence Rate (FTSPAR) , reported in percentage (%). Results are disaggregated by scene type and task length category.

Fig. 5 : Scene-specific task-level performance comparison. Task length categories are defined by sketch complexity (Short: $\leq$ 2 corners, Medium: 3-5 corners, Long: $\geq$ 6 corners). The figure presents (Left) Full Task Completion Rate (FTCR) and (Right) Full Task Strict Path Adherence Rate (FTSPAR) in percentage (%). Results are shown for each scene category from the HouseholdSketch dataset, grouped by task length. Error bars depict simulated standard error, indicating expected performance variability.

Fig. 6 : Aggregate performance comparison across metrics and task lengths. Task length categories (Short: $\leq$ 2 corners, Medium: 3-5 corners, Long: $\geq$ 6 corners) are based on the number of turns in the input sketch. Bars represent the average success rates (%) for SSSR, SSSPAR, FTCR, and FTSPAR, computed across all scenes from the HouseholdSketch dataset. Error bars indicate simulated standard error, reflecting anticipated experimental variability.

Fig. 7 : Qualitative illustration of system operation in the iGibson simulation environment. (a) User-provided sketch overlaid on the scene, defining a multi-segment navigation task around furniture and under a table (keypoints highlighted). (b)-(g) Sequence depicting the robot’s execution trace: (b) initial forward motion, (c) approaching the table triggering under-obstacle check, (d) simulated perception check under the table, (e)-(f) navigating turns and proceeding under the table, (g) final segment execution. The sequence demonstrates task decomposition, path following, and integration of adaptive behaviors like obstacle checking.

Setting	SSSR	SSSPAR	FTCR	FTSPAR
$I{+}S{+}L{+}P_{t}$ (full)	84.6	77.0	60.9	48.4
$I{+}S{+}L$ (no $P_{t}$ )	84.0 (-0.6)	76.0 (-1.0)	58.7 (-2.2)	47.2 (-1.2)
$I{+}S$ (no user $L$ )	83.8 (-0.8)	76.4 (-0.6)	59.4 (-1.5)	47.1 (-1.3)
$S{+}L$ (no $I$ )	78.1 (-6.5)	70.2 (-6.8)	41.3 (-19.6)	28.4 (-20.0)
$I{+}L$ (no $S$ )	68.5 (-16.1)	55.3 (-21.7)	32.1 (-28.8)	20.2 (-28.2)

TABLE III : Ablation on input modalities and live perception on HouseholdSketch (overall across scenes and task lengths). Numbers are percentages. In parentheses are absolute changes relative to the full model.

Fig. 8 : Qualitative illustration of system operation with the KUKA LBR iiwa robot executing a table wiping task. (a) User-provided sketch overlaid on the camera view, indicating a path (arrow) and a target area (rectangle). (b)-(f) Sequence depicting the robot’s execution: (b) Initial state. (c)-(d) Arm approaches the target region. (e) Arm executes wiping motion along the sketched path. (f) Arm interacts with the designated rectangular area. The sequence demonstrates successful 3D grounding, trajectory execution, and task completion based on the sketch.

Fig. 9 : Qualitative illustration of dual-arm cover-area task on the Realman RMC-AIDAL mobile manipulator driven by sketch-based commands. (a) User sketch overlaid on the onboard camera image: a single curved arrow on the tabletop specifies the region to be traversed and the desired sweeping direction. After sketch-guided navigation brings the base to a pose in front of the table, (b)–(d) the hierarchical policy instantiates a cover_area macro that converts the arrow into synchronized, collision-aware trajectories for both RM65-B arms: the right arm performs the primary sweeping motion, while the left arm adopts a reactive tucked posture to maintain clearance in the cluttered workspace.

Metric	Elderly (n=10)	Sim. Non-Verbal (n=8)	Low Tech Literacy (n=7)	Control (n=7)
Specification Time (s)	62.5 (±14.8)	55.1 (±10.2)	68.3 (±16.5)	49.8 (±9.0)
Attempts per Task	1.5 (±0.7)	1.3 (±0.5)	1.7 (±0.8)	1.2 (±0.4)
Full Task Completion Rate (FTCR) (%)	90.0%	93.8%	85.7%	96.4%
Expert Fidelity Rating (EFR) (1-5)	4.0 (±0.8)	4.2 (±0.6)	3.8 (±0.9)	4.4 (±0.5)
Subjective Ease of Use (1-5)	4.3 (±0.7)	4.5 (±0.5)	3.9 (±1.0)	4.6 (±0.4)
Subjective Confidence (1-5)	4.1 (±0.8)	4.4 (±0.6)	3.7 (±1.1)	4.5 (±0.5)

TABLE IV : User Study Performance Metrics Across Diverse Demographic Groups. Values represent mean (± standard deviation) where applicable. TCSR is presented as a percentage. Subjective ratings and EFR are on a 1-5 scale (higher is better).

Failure source	Share %
Perception-driven mismatches in grounding with $I$ or $P_{t}$	29.1
Sketch-to-scene misalignment in fusion $f_{\text{fuse}}$	23.7
Macro-action misclassification in $\pi_{\text{HL}}$	20.8
Execution drift in $g_{\text{translate}}$ and low-level control	19.6
Timeouts or conservative halts from safety checks	6.8

TABLE V : Primary failure sources on Long tasks in HouseholdSketch (audited subset, percentages sum to 100).

Setting	SSSR	SSSPAR	FTCR	FTSPAR
Baseline: $\pm 45^{\circ},\pm 90^{\circ}$ , $0.05\,\mathrm{m}$	84.4	76.9	46.7	33.5
Coarser: $\pm 90^{\circ}$ , $0.10\,\mathrm{m}$	83.9	71.8 ( $-5.1$ )	42.3 ( $-4.4$ )	28.1 ( $-5.4$ )
Finer: $\pm 22.5^{\circ},\pm 45^{\circ},\pm 90^{\circ}$ , $0.025\,\mathrm{m}$	84.1	79.3 (+2.4)	45.6 ( $-1.1$ )	35.4 (+1.9)
Continuous yaw regression, $0.05\,\mathrm{m}$	84.0	78.4 (+1.5)	44.1 ( $-2.6$ )	34.2 (+0.7)

TABLE VI : Effect of action-space resolution on Long tasks in HouseholdSketch. Percentages reported; deltas are absolute changes relative to the baseline.

Setting	Macro Acc.	SSSR	FTCR	FTSPAR
$L_{0}^{\mathrm{ZS}}$ (zero-shot)	88.4	84.0	74.2	62.0
$L_{0}^{\mathrm{1S}}$ (one-shot)	89.8	85.1	75.2	63.0

TABLE VII : Zero-shot vs. one-shot prompting on HouseholdSketch. Values are percentages. One-shot adds a single segment-matched exemplar to

L_{0}

SSSR (%)
	Short	Medium	Long	Overall
$L_{0}^{\mathrm{ZS}}$	84.7	84.2	83.8	84.0
$L_{0}^{\mathrm{1S}}$	85.5	85.0	84.8	85.1
FTCR (%)
	Short	Medium	Long	Overall
$L_{0}^{\mathrm{ZS}}$	74.8	60.0	46.4	74.2

TABLE VIII : Length-conditioned comparison for zero-shot and one-shot prompting.

Setting	SSSR	SSSPAR	FTCR	UOMS
(A) $(I,S,L)$	84.1	76.7	58.2	62.5
(B) $(I,S,L)+P_{t}$ late-fusion gated	84.3	77.5	64.9	78.4
(C) $(I,S,L)+P_{t}$ naive early fusion	83.9	75.9	60.1	70.2
(D) $(I,S,L)+P_{t}^{\text{depth}}$ (replicated)	84.0	76.1	59.0	67.3

TABLE IX : Live perception ablation on the obstacle subset (1,200 trials). Percentages are averages across scenes and sketch-length strata.

Human-Robot Interaction

Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.

See all 452 papers

SHRIMP: Iterative Refinement of Robot Task Plans
Aug 9, 2026Mya Schroder, Yuna Hwang, Callie Y. Kim +5Large Language Model-Based Robot Planning Robot Task Planning
Department of Computer Sciences, University of Wisconsin–Madison, Madison, Wisconsin, USA
SkillComposer: Learning Reusable Skills for Natural-Language Robot Programming
Aug 14, 2026John Woods, Hasti SeifiCode Generation for Robotics Robot Skill Learning
School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ
Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning
May 8, 2026Karolina Źróbek, Tessa Pulli, Paweł Gajewski +2Large Language Model-Based Robot Planning LLM Prompting
IBM, Krakow, Poland · TU Wien, Vienna, Austria · Jagiellonian University, Krakow, Poland +1

Aug 9, 2026J/K move · Enter open · S save

SHRIMP: Iterative Refinement of Robot Task Plans

Mya Schroder, Yuna Hwang, Callie Y. Kim, Leqian Cheng+4

Department of Computer Sciences, University of Wisconsin–Madison, Madison, Wisconsin, USA

Large Language Model-Based Robot Planning Robot Task Planning Human-Robot Interaction

Open paper AskPDF arXiv

As collaborative robots have entered domains such as manufacturing, agriculture, and healthcare, programming or adapting robot behavior typically requires robotic expertise that most end users lack. Natural language lowers this barrier. Recent advancements in large language models (LLMs) have made it feasible to translate natural language into robot task plans. However, language-based task specification suffers from semantic ambiguity, and generative models lack transparency for how language instructions become robot actions, making it difficult for users to validate the plan before execution. To address these issues, we introduce SHRIMP, a system that allows users to automatically generate a hierarchical robot primitive plan using natural language and iteratively revise their plan through re-prompting and explicit correction. At each revision, SHRIMP allows users to validate their plan in simulation, and once satisfied, execute it on the physical robot. Through a user study involving participants planning tabletop kitchen tasks (n=35), we validate that SHRIMP improves perceived control and enhances robot transparency. System videos and source code are available at https://wisc-hci.github.io/SHRIMP.

Abstract

Human-Robot Interaction

Explore similar work

SHRIMP: Iterative Refinement of Robot Task Plans

SkillComposer: Learning Reusable Skills for Natural-Language Robot Programming

Hierarchical Prompting with Dual LLM Modules for Robotic Task and Motion Planning