Reference Wording Shapes Retrieval of Robot Task Instructions

Jahnavi Yelamanchi

cs.AIOct 13, 2026

Reference Wording Shapes Retrieval of Robot Task Instructions

Authors: Jahnavi Yelamanchi

Abstract

Identifying a robot task from a short instruction is sensitive to which descriptions are available for retrieval. We investigate this effect in the natural-language instructions included with the RLBench manipulation benchmark. We extract 296 fixed descriptions from 62 publicly available task files and compare five training-free lexical rankers over 80 random choices of one indexed description per task. Exact strings assigned to conflicting task labels are excluded from the primary score. Character n-gram TF-IDF achieves 61.3% mean top-1 accuracy on the resulting held-out instructions, compared with 55.6% for BM25 and 54.9% for word TF-IDF. Across gallery choices, character TF-IDF ranges from 55.2% to 66.8%; its 5th-95th percentile range is 56.9%-64.8%. Adding a second indexed description improves character TF-IDF by 7.0 percentage points on matched queries. These results quantify the influence of reference wording and representation choice on instruction-to-task retrieval under a consistent evaluation protocol.

Figures & tables

Figure 1: Evaluation design. Existing RLBench instructions form task-wise reference galleries and held-out queries; the reference wording is resampled across trials.

Design	Tasks	Gallery	Query candidates
One-description, 80 trials	62	62	234 per trial
Two-description matched comparison, 30 trials	62	124	descriptions at position $\geq 3$
First-listed reference, audit only	62	62	234

Table 1: Data accounting and comparison designs. Exclusion applies only to exact descriptions occurring under multiple task labels.

Figure 2: Mean top-1 accuracy (dots) and 5th–95th percentile range (thick lines) over 80 random galleries. The whiskers show gallery-choice variability, not confidence bounds.

Ranker	Mean	SD	5th–95th	Min–max
BM25	55.6	2.5	52.1–60.8	51.1–62.1
Word TF–IDF	54.9	3.0	50.0–59.1	47.8–61.6
Char TF–IDF	61.3	2.6	56.9–64.8	55.2–66.8
Word+Char	60.9	3.0	55.6–65.5	53.8–67.7
LSA-32	52.8	2.9	48.3–57.3	47.0–60.8

Table 2: One-description results over 80 gallery choices. Scores are percentages on unambiguous query texts. SD is across gallery draws; min–max is the observed range.

Figure 3: Distribution of trial accuracies for the same five rankers. Boxes summarize variation attributable to which task descriptions were selected as references.

Figure 4: Matched one- versus two-description galleries (30 trials). The candidate queries are unchanged between conditions within each trial.

Ranker	One example	Two examples	Difference (pp)
BM25	54.0%	64.2%	+10.2
Word TF–IDF	53.3%	60.7%	+7.4
Char TF–IDF	59.3%	66.4%	+7.0
Word+Char	59.2%	65.1%	+5.9
LSA-32	52.0%	56.2%	+4.3

Table 3: Average top-1 accuracy in the matched setting. The change is expressed in percentage points (pp).

Ranker	Strict	Either label	Duplicates removed
BM25	55.4%	55.6%	55.6%
Char TF–IDF	60.9%	61.0%	61.3%
Word TF–IDF	54.7%	55.1%	54.9%

Table 4: Scoring audit: means over the same 80 one-description gallery draws. “Either label” accepts any original label for a repeated source string.

Lexical Retrieval

Lexical retrieval ranks documents by query-term overlap using methods such as TF-IDF and BM25.

See all 27 papers

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation
Aug 24, 2026Mengao Zhao, Ziang Li, Chaodong Huang +15Robot Manipulation Benchmarks Language-Conditioned Robot Manipulation
Horizon Robotics · Southeast University · Huazhong University of Science and Technology +1
One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning
Sep 4, 2026Yiwei Liu, Luwei Yang, Shunbo LeiLanguage-Conditioned Robot Control Embodied Reasoning
School of Science and Engineering, The Chinese University of Hong Kong, Shenzhen, Guangdong 518172, China · Shenzhen Research Institute of Big Data (SRIBD), Shenzhen 518172, China
EarlyCommit: Calibrated Word-by-Word Commitment for Robots Acting Before You Finish Speaking
Oct 14, 2026Muhammad Haris KhanLanguage-Conditioned Robot Manipulation Spoken Language Understanding
University of Copenhagen, Copenhagen, Denmark

Aug 24, 2026J/K move · Enter open · S save

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

Mengao Zhao, Ziang Li, Chaodong Huang, Mengchen Ma+14

Horizon Robotics · Southeast University · Huazhong University of Science and Technology+1

Robot Manipulation Benchmarks Language-Conditioned Robot Manipulation Robotic Manipulation Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-language instructions. A key test of such generality is whether policies actually follow language instructions. Yet many manipulation benchmarks leave this ability underdetermined: the intended object or destination is often visually salient or uniquely feasible, allowing policies to succeed without grounding the instruction. We argue that instruction-following evaluation should be text-indispensable: multiple actions should be visually and physically plausible, while only one should be consistent with the language instruction. We introduce InstructMove, a text-indispensable benchmark for instruction-following manipulation. InstructMove instantiates this principle in pick-and-place scenes with semantic distractors, decomposing instruction following into category identification, attribute discrimination, spatial reasoning, and compositional pick-and-place. InstructMove supports a train-eval protocol with InstructMove training data and held-out evaluation tasks, with additional diagnostics for language dependence. Experiments with representative VLA policies show that InstructMove provides a controlled testbed for diagnosing visual shortcuts and that InstructMove simulation data can improve real-world instruction-following manipulation performance. Code: https://github.com/HorizonRobotics/RoboOrchardSim

Abstract

Lexical Retrieval

Explore similar work

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

One Word, Different Action: A Real-Robot Benchmark for Language-Conditioned Embodied Reasoning

EarlyCommit: Calibrated Word-by-Word Commitment for Robots Acting Before You Finish Speaking