Assistive robotics is an important subarea of robotics that focuses on the well-being of people with disabilities. A robotic guide dog is an assistive quadruped robot that helps visually impaired people in obstacle avoidance and navigation. Enabling language capabilities for robotic guide dogs goes beyond naively adding an existing dialog system onto a mobile robot. The novel challenges include grounding language in the dynamically changing environment and improving spatial awareness for the human handler. To address those challenges, we develop a novel dialog system for robotic guide dogs that uses LLMs to verbalize both navigational plans and scenes. The goal is to enable verbal communication for collaborative decision-making within the handler-robot team. In experiments, we conducted a human study to evaluate different verbalization strategies and a simulation study to assess the efficiency and accuracy in navigation tasks.
Figures & tables
Figure 1: A legally blind person walking with our robotic guide dog system during a participant study.
Figure 2: Our system uses human-robot dialog to define a formal service task. An LLM first determines relevant navigable locations, and a task planner generates multiple action sequences (plans) for each candidate. These plans are summarized for the human via plan verbalization , detailing metrics like navigation cost and door openings. After human selection, the robot executes the chosen plan, providing navigation guidance while providing scene verbalization to describe the surroundings.
Figure 3: LLM prompt defining the role of our robot guide dog dialog system. Given possibly ambiguous human service tasks in natural language, the LLM must conduct a dialog and select the location that best satisfies the task.
Age
Gender
Impairment
Guide Dog
Duration (yrs)
Experience
68
Female
68
None
68
Male
60
None
59
Male
51
8 yrs, 2 dogs
40
Male
40
None
64
Male
35
18 yrs, 3 dogs
Table 1: Participant Demographics.
Q1 Utility
The natural language interface is a useful component of the system.
Q2 Helpfulness
The robot is helpful for visually impaired navigation when compared to having no tools for visually impaired navigation.
Q3 Helpfulness
The robot is helpful for visually impaired navigation when compared to having other tools for visually impaired navigation.
Q4 Safety
The robot is capable of safely guiding me to a navigation goal.
Q5 Ease of comm.
I can easily communicate with the robot.
Q6 Preference
Given the same navigation task we performed today I prefer a guide dog to guide me instead of the robot.
Q7 Feedback
If you have any questions, comments, or concerns please leave them here.
Table 2: The list of questions in the questionnaire. The items assessed interface utility, helpfulness, safety, ease of communication, and preference relative to a real guide dog.
Condition
Q1
Q2
Q3
Q4
Q5
Q6
Mini. Verb.
4.33
4.67
3.50
4.00
4.33
3.17
Scene Verb.
4.67
4.83
4.00
4.00
4.00
3.17
Ours
4.83
4.83
4.33
3.83
4.50
3.67
Table 3: Average survey ratings for each verbalization condition. Ratings are on a 5-point Likert scale (1: Strongly Disagree, 5: Strongly Agree).
Figure 4: An illustrative example of our robotic guide dog assisting a legally blind participant to navigate to a conference room. The sequence shows: (a) The robot verbalizes the generated navigation plans for a user request, and starts navigation after the user chooses one of the plans. (b, c) During navigation, the system provides real-time scene verbalization, such as passing a lobby and entering a corridor. (d) The robot announces that they have arrived at the destination.
Table 4: Samples in our collected service task dataset: locations in each category considered functionally equivalent.
Figure 5: An example simulated conversation. The handler implicitly states a purpose sampled from <location, purpose> pair. The dialog system suggests relevant locations. After confirming the handler is looking for a bench, the system generates a formal task and concludes the task specification conversation.
Figure 6: Accuracy vs. Average Dialog Length without noise. Our approach provides the best trade-off between high accuracy and efficient conversation.
Figure 7: Accuracy vs. Dialog Length with perturbed (noisy) inputs. Our approach remains highly accurate, while the keyword-based method fails.
Location
Purpose
elevator
“I want to go to the 2nd floor”
water fountain
“I want something to drink”
waiting room
“I want to sit down”
Table 5: The three location-purpose pairs used for planner-informed evaluation.
Method
Dialog Len.
Nav. Cost
Total Time
w/ Plan Info
115.67
55.17
230.15
w/o Plan Info
84.83
73.00
277.27
Table 6: Effect of Plan Information on Performance Metrics