In this study, we focus on service robots that retrieve and transport everyday objects based on free-form instructions that may refer to visual features and scene text. This is challenging because it requires a multimodal understanding of the visual features and spatial relationships of objects, as well as scene text embedded in the environment. To address this, we propose Scene Text-Aware multimodal Retrieval for Everyday objects (STARE), which leverages visual and textual cues in response to free-form instructions. STARE introduces Crosslingual Visual Prompt to enhance scene text interpretation, and models the complex relationships between named entities in the instructions and their corresponding objects. We evaluate STARE using two novel benchmarks, GoGetIt and TextCaps-test, both of which consist of real-world indoor/outdoor images and instructions involving object manipulation and navigation. The results show that STARE outperforms all baseline methods on standard multimodal retrieval metrics. Furthermore, in physical experiments conducted under a zero-shot transfer setting, STARE achieves an overall success rate of 80%. Our project page and code are provided in the Supplementary Materials.
Figures & tables
Figure 1: Typical scene of our task . First, environmental images are collected in a pre-exploration phase. Next, STARE performs multimodal retrieval in response to a free-form instruction. Finally, the robot navigates to and grasps the specified object.
Figure 2: Architecture of STARE . STARE includes a narrative representation that integrates Crosslingual Visual Prompt (CVP) and scene text, and reranks based on lexical matches between named entities in the instruction and scene texts. SSM represents a Mamba encoder.
Optimizer
AdamW
β1 , β2
0.9, 0.98
Learning rate
1×10−4
Batch size
128
Number of epochs
20
Lst , Hst , Ast
5, 2,048, 1
Lnp , Hnp , Anp
4, 2,304, 8
Table 1: Experimental settings for our method.
[%]
FT
GoGetIt (RefText)
GoGetIt (Instruction)
TextCaps-test
LTRRIE
Method
R@5 ↑
R@10 ↑
R@5 ↑
R@10 ↑
R@5 ↑
R@10 ↑
R@5 ↑
R@10 ↑
CLIP [ 29 ]
63.2
72.8
74.3
83.3
79.8
86.3
56.1
71.0
CLIP [ 29 ]
63.5 ± 0.3
74.1 ± 0.3
73.9 ± 0.7
83.9 ± 0.3
82.0 ± 0.6
90.1 ± 0.4
56.8 ± 0.7
72.3 ± 0.4
NLMap* [ 8 ]
50.9
60.1
61.0
73.5
70.0
78.8
50.9
66.4
BEiT-3 [ 35 ]
54.4
65.3
63.7
79.5
76.5
84.8
59.9
76.6
BLIP-2 [ 17 ]
-
-
-
-
76.5
86.3
66.6
81.9
Table 2: Quantitative comparison on GoGetIt-RefText, GoGetIt-Instruction, TextCaps-test, and LTRRIE. Bold, underline, ‘*’, and ‘FT’ denote the best score, second-best score, reproduced results, and fine-tuning, respectively. BLIP-2, Long-CLIP, and RelaX-Former scores on GoGetIt are omitted because of training–test data leakage (Section 4.1 ).
Figure 3: Qualitative results for STARE and Meta CLIP 2 [ 10 ] on (a) GoGetIt-Instruction and (b) GoGetIt-RefText. The instructions are (a) “Pass me the red container of Sun-Maid raisins on the kitchen counter.” and (b) “A long box of non-stick baking paper.” Green borders mark target images in the retrieval lists.
[%]
STRR
CVP
Patchification
GoGetIt(RefText)
GoGetIt(Instruction)
TextCaps-test
LTRRIE
Model
R@5 ↑
R@10 ↑
R@5 ↑
R@10 ↑
R@5 ↑
R@10 ↑
R@5 ↑
R@10 ↑
(a)
TNE
K
overlap
90.2 ± 0.4
94.4 ± 0.4
89.8 ± 0.6
92.8 ± 0.6
92.4 ± 1.3
95.4 ± 0.6
72.5 ± 0.8
86.6 ± 0.9
(b)
-
K
overlap
76.4 ± 0.8
85.0 ± 0.7
78.2 ± 1.2
85.9 ± 0.5
83.6 ± 1.1
89.5 ± 1.9
71.1 ± 1.0
85.1 ± 0.6
(c)
Ttxt
K
overlap
81.4 ± 0.8
89.2 ± 0.6
79.7 ± 1.1
87.7 ± 1.1
82.1 ± 0.6
89.4 ± 1.0
70.6 ± 1.3
85.1 ± 0.8
(d)
TNE
AN
overlap
90.2 ± 0.6
94.3 ± 0.2
88.4 ± 0.5
92.3 ± 0.8
92.3 ± 1.0
95.3 ± 0.4
71.3 ± 1.0
85.9 ± 0.3
(e)
TNE
E
overlap
90.1 ± 0.9
94.3 ± 0.4
89.3 ± 0.5
92.2 ± 0.6
91.3 ± 0.6
95.3 ± 0.6
71.6 ± 0.9
85.7 ± 1.3
Table 3: Ablation results . Mean and standard deviation over five trials are reported; bold indicates the highest value. Models (a)–(i), (k), and (l) are shown. K, AN, E, G, and BB denote Katakana, Arabic numerals, English, Greek, and bounding box, respectively.
Method
FT
R@5 ↑
SR ↑
CLIP [ 29 ]
71
64
CLIP [ 29 ]
74
65
NLMap* [ 8 ]
71
65
BEiT-3 [ 35 ]
66
62
BLIP-2 [ 17 ]
72
66
Long-CLIP [ 41 ]
70
66
Table 4: Quantitative hardware results. FT: fine-tuned; scores are percentages.
Figure 4: Qualitative results for hardware experiments. For the samples, xtxt values were as follows: (a) “Pass me the Maxwell can.”; (b) “Hand me the SESAME MILL.” Each example shows the target object and the top-3 retrieved images. The green box indicates the target object.