In this study, we focus on service robots that retrieve and transport everyday objects based on free-form instructions that may refer to visual features and scene text. This is challenging because it requires a multimodal understanding of the visual features and spatial relationships of objects, as well as scene text embedded in the environment. To address this, we propose Scene Text-Aware multimodal Retrieval for Everyday objects (STARE), which leverages visual and textual cues in response to free-form instructions. STARE introduces Crosslingual Visual Prompt to enhance scene text interpretation, and models the complex relationships between named entities in the instructions and their corresponding objects. We evaluate STARE using two novel benchmarks, GoGetIt and TextCaps-test, both of which consist of real-world indoor/outdoor images and instructions involving object manipulation and navigation. The results show that STARE outperforms all baseline methods on standard multimodal retrieval metrics. Furthermore, in physical experiments conducted under a zero-shot transfer setting, STARE achieves an overall success rate of 80%. Our project page and code are provided in the Supplementary Materials.
Figures & tables
Figure 1: Typical scene of our task . First, environmental images are collected in a pre-exploration phase. Next, STARE performs multimodal retrieval in response to a free-form instruction. Finally, the robot navigates to and grasps the specified object.
Figure 2: Architecture of STARE . STARE includes a narrative representation that integrates Crosslingual Visual Prompt (CVP) and scene text, and reranks based on lexical matches between named entities in the instruction and scene texts. SSM represents a Mamba encoder.
Optimizer
AdamW
β1 , β2
0.9, 0.98
Learning rate
1×10−4
Batch size
128
Number of epochs
20
Lst , Hst , Ast
5, 2,048, 1
Lnp , Hnp , Anp
4, 2,304, 8
Table 1: Experimental settings for our method.
[%]
FT
GoGetIt (RefText)
GoGetIt (Instruction)
TextCaps-test
LTRRIE
Method
R@5 ↑
R@10 ↑
R@5 ↑
R@10 ↑
R@5 ↑
R@10 ↑
R@5 ↑
R@10 ↑
CLIP [ 29 ]
63.2
72.8
74.3
83.3
79.8
86.3
56.1
71.0
CLIP [ 29 ]
63.5 ± 0.3
74.1 ± 0.3
73.9 ± 0.7
83.9 ± 0.3
82.0 ± 0.6
90.1 ± 0.4
56.8 ± 0.7
72.3 ± 0.4
NLMap* [ 8 ]
50.9
60.1
61.0
73.5
70.0
78.8
50.9
66.4
BEiT-3 [ 35 ]
54.4
65.3
63.7
79.5
76.5
84.8
59.9
76.6
BLIP-2 [ 17 ]
-
-
-
-
76.5
86.3
66.6
81.9
Table 2: Quantitative comparison on GoGetIt-RefText, GoGetIt-Instruction, TextCaps-test, and LTRRIE. Bold, underline, ‘*’, and ‘FT’ denote the best score, second-best score, reproduced results, and fine-tuning, respectively. BLIP-2, Long-CLIP, and RelaX-Former scores on GoGetIt are omitted because of training–test data leakage (Section 4.1 ).
Figure 3: Qualitative results for STARE and Meta CLIP 2 [ 10 ] on (a) GoGetIt-Instruction and (b) GoGetIt-RefText. The instructions are (a) “Pass me the red container of Sun-Maid raisins on the kitchen counter.” and (b) “A long box of non-stick baking paper.” Green borders mark target images in the retrieval lists.
[%]
STRR
CVP
Patchification
GoGetIt(RefText)
GoGetIt(Instruction)
TextCaps-test
LTRRIE
Model
R@5 ↑
R@10 ↑
R@5 ↑
R@10 ↑
R@5 ↑
R@10 ↑
R@5 ↑
R@10 ↑
(a)
TNE
K
overlap
90.2 ± 0.4
94.4 ± 0.4
89.8 ± 0.6
92.8 ± 0.6
92.4 ± 1.3
95.4 ± 0.6
72.5 ± 0.8
86.6 ± 0.9
(b)
-
K
overlap
76.4 ± 0.8
85.0 ± 0.7
78.2 ± 1.2
85.9 ± 0.5
83.6 ± 1.1
89.5 ± 1.9
71.1 ± 1.0
85.1 ± 0.6
(c)
Ttxt
K
overlap
81.4 ± 0.8
89.2 ± 0.6
79.7 ± 1.1
87.7 ± 1.1
82.1 ± 0.6
89.4 ± 1.0
70.6 ± 1.3
85.1 ± 0.8
(d)
TNE
AN
overlap
90.2 ± 0.6
94.3 ± 0.2
88.4 ± 0.5
92.3 ± 0.8
92.3 ± 1.0
95.3 ± 0.4
71.3 ± 1.0
85.9 ± 0.3
(e)
TNE
E
overlap
90.1 ± 0.9
94.3 ± 0.4
89.3 ± 0.5
92.2 ± 0.6
91.3 ± 0.6
95.3 ± 0.6
71.6 ± 0.9
85.7 ± 1.3
Table 3: Ablation results . Mean and standard deviation over five trials are reported; bold indicates the highest value. Models (a)–(i), (k), and (l) are shown. K, AN, E, G, and BB denote Katakana, Arabic numerals, English, Greek, and bounding box, respectively.
Method
FT
R@5 ↑
SR ↑
CLIP [ 29 ]
71
64
CLIP [ 29 ]
74
65
NLMap* [ 8 ]
71
65
BEiT-3 [ 35 ]
66
62
BLIP-2 [ 17 ]
72
66
Long-CLIP [ 41 ]
70
66
Table 4: Quantitative hardware results. FT: fine-tuned; scores are percentages.
Figure 4: Qualitative results for hardware experiments. For the samples, xtxt values were as follows: (a) “Pass me the Maxwell can.”; (b) “Hand me the SESAME MILL.” Each example shows the target object and the top-3 retrieved images. The green box indicates the target object.
Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim
Mengao Zhao, Ziang Li, Chaodong Huang +15
Horizon Robotics · Southeast University · Huazhong University of Science and Technology +1
To serve a request in a human environment, a robot system must find an object that satisfies a functional requirement and a robot that can feasibly reach it. Scene graphs record which objects exist and where, but they often omit capabilities documented only in product manuals. Retrieval-augmented generation methods can read such manuals, yet they rarely reason jointly over object capabilities, physical instances, robot states, and dynamic connectivity. We introduce E2W-Bench, the Exists-to-Works Benchmark, which links dynamic hybrid scene graphs with real product manuals to evaluate capability-grounded robot--object assignment under availability, mobility, vertical-reach, and connector constraints. We also present Embodied RAG++, a spatially grounded retrieval framework that scopes retrieval to the manuals linked to candidate objects, caches extracted capabilities per product model, and resolves robot--object feasibility through deterministic spatial reasoning. On E2W-Bench, Embodied RAG++ reaches a pair accuracy of 0.90 with Qwen2.5-32B and 1.00 with Llama3.3-70B, compared with at most 0.30 and 0.55 for the baselines, while requiring the shortest preprocessing time among the retrieval-based methods and tying for the lowest total token count.
We present a hierarchical language-driven framework for robotic task and motion planning to improve natural, intuitive human-robot interaction in service and assistance scenarios. The proposed system employs two large language model (LLM) modules: a high-level planning agent and a low-level spatial reasoning sub-module. The primary agent processes natural language commands and generates action sequences using a ReAct-style prompt, interacting with tools for object perception and manipulation (e.g., pick, place, release). For precise spatial placement, such as interpreting "place the mug next to the plate", a separate sub-prompting module handles 3D reasoning based on object geometry and scene layout. The system integrates YOLOX-GDRNet for object detection and pose estimation, along with a motion execution stub. We evaluated the system in 24 test scenarios, ranging from simple spatial commands to high-level instructions and infeasible requests. The system achieved an overall task success rate of 86%.
Karolina Źróbek, Tessa Pulli, Paweł Gajewski +2
IBM, Krakow, Poland · TU Wien, Vienna, Austria · Jagiellonian University, Krakow, Poland +1