cs.HCMar 10, 2026
SaveUnderstanding the Use of a Large Language Model-Powered Guide to Make Virtual Reality Accessible for Blind and Low Vision People
Organizations: Cornell University New York, New York, USA · Cornell Tech New York, New York, USA · Cornell University Ithaca, New York, USA
Abstract
As social virtual reality (VR) grows more popular, addressing accessibility for blind and low vision (BLV) users is increasingly critical. Researchers have proposed an AI "sighted guide" to help users navigate VR and answer their questions, but it has not been studied with users. To address this gap, we developed a large language model (LLM)-powered guide and studied its use with 16 BLV participants in virtual environments with confederates posing as other users. We found that when alone, participants treated the guide as a tool, but treated it companionably around others, giving it nicknames, rationalizing its mistakes with its appearance, and encouraging confederate-guide interaction. Our work furthers understanding of guides as a versatile method for VR accessibility and presents design recommendations for future guides.
Figures & tables
Figure 1. A screenshot of a participant utilizing the AI guide in its dog form, traveling through a virtual park together. A screenshot of a virtual park environment. A female avatar and a German Shepherd breed dog with a bright red guide dog harness stand in the center of the image. A wooden dock is behind them at the edge of a river cutting through the park. A bridge crosses the river to the woman's right. In the distance, there is a line of green trees.
Figure 2. The three virtual environments used in the study. From left to right, the Tutorial contains various simplistic 3D objects such as yellow and green cubes, Park 1 contains a large river and several colorful gazebos, and Park 2 contains colorful flower beds and a series of benches for resting throughout the park. A table with three images of the virtual environments used in the study. On the far left is an image of the tutorial room, a simple white room with various geometrical shapes, including a yellow, horizontal cube acting as a table with three interactable shapes resting on it. In the middle is a screenshot of the first park with trees scattered about and a river cutting through, populated by stone walkways and bridges, two water fountains, three gazebos, and three groups of avatars. On the right is a screenshot of the second park, a brightly-lit park with stone tile walkways around a few square patches of grass, two gazebos, and a group of avatars gathered for a dance party.
| Pseudonym | Age | Gender | Visual Condition | Etiology | Visual Acuity | Visual Field | Navigation Tools Used |
|---|---|---|---|---|---|---|---|
| Niya | 45 | F | Blind; No Light Perception | NR | NR | NR | White Cane; Human Guide; Navigation Apps; Tactile Maps |
| Anika | 46 | F | Low Vision | Optic Atrophy | R: 20/400, L: 20/200 | Vision in upper field of left eye, right eye blurry | White Cane; Navigation Apps |
| Teo | 51 | M | Low Vision | Cortical Impairment | R: 20/400, L: 20/400 | NR | White Cane; Navigation Apps; GPS Devices |
| Nikolai | 34 | M | Partially Sighted; Legally Blind; Night Blindness; Central Vision Loss | NR | NR | NR | Navigation Apps |
| Sara | 22 | F | Legally Blind | Dual Optic Coloboma | NR | NR | White Cane; Navigation Apps |
| Xiran | 38 | F | Legally Blind; Night Blindness; Peripheral Vision Loss | NR | NR | NR | White Cane; Human Guide; Navigation Apps |
Table 1. Participant demographics. F = female, M = male. Participant information is self-reported. NR indicates that participants chose not to report that information. Information about our participant demographics. Contains all textual content.
| Group | Phase | Task Type | Duration | Park | Persona |
|---|---|---|---|---|---|
| (N = 4 participants per group) | Assignment | Assignment | |||
| 1 | Round 1 | Individual Task (Exploration) | 10 minutes | Park 1 | Dog |
| Social Task (Tour) | 15 minutes | ||||
| Round 2 | Individual Task (Exploration) | 10 minutes | Park 2 | Robot | |
| Social Task (Tour) | 15 minutes | ||||
| 2 | Round 1 | Individual Task (Exploration) | 10 minutes | Park 1 | Dog |
Table 2. The counterbalanced study conditions (e.g., duration, park, persona) across all groups of participants. Group 1 = Niya, Anika, Teo, Nikolai; Group 2 = Sara, Xiran, Yuki, Maritza; Group 3 = Aiden, Len, Siobhan, Isaiah; Group 4 = Li, Jamal, Yadriel, Jiwoo. Participants completed two tasks per study phase (i.e., one individual and one social task).
| Persona | Guide Avatar | Guide Description | Guide Movement | Voice + Speech Characteristics | Personality Prompt | Speech Sample |
| Human | An androgynous human with short black hair and glasses, wearing a gray hoodie and black pants. | Follows behind the user from an arm’s length away. Footstep sound effects indicate movement. | Alloy, androgynous voice, medium formality (some jokes, casual speech), averages 101.2 words per response | Warm, friendly, but still professional sighted guide | The scene around you is a simple, geometric landscape composed of distinctive objects. To your right stands the Tall Building, a towering yellow cube that rises high above the rest… | |
| Guide Dog | German-shepherd with black and brown fur, wearing a bright-red guide dog harness. | Walks beside the user on their left side. The sounds of panting and paws scratching the floor indicate movement. | Nova, feminine voice, low formality (uses slang, jokes, and casual speech), averages 185.4 words per response | Very friendly, excited companion who is eager to please who you’re talking to | What an exciting world we’re in! To your right stands the impressive Tall Building, reaching up high with its yellow facade… | |
| Robot | A shiny, silver robot that is humanoid in stature, with a masculine frame. | Follows behind the user from an arm’s length away. Creaking machinery sound effects indicate movement. | Onyx, masculine voice, high formality (speaks entirely in proper sentences, no casual speech), averages 91.2 words per response | Formal and assertive assistant who talks like a robot | Affirmative. The area around you is composed of geometric shapes and structures. Directly to your right lies the Tall Building… |
Table 3. Our guide’s three personas: the human, the guide dog, and the robot. The voices mentioned for each persona (Alloy, Nova, and Onyx) are OpenAI’s default TTS voices ( OpenAI, ) . Audio samples of each voice can be found at the cited link. The personality prompts were sent to GPT-4 to shape the guide’s responses (see supplemental materials). All columns are based on our previous designs from Collins et al. ( Collins et al., 2024 ) , except for Speech Sample, which was produced by our current implementation of the AI guide. Information about three different guide personas. Contains all textual content, aside from the three images of the persona's avatars in the column "Guide Avatar." Each avatar is described in the following column "Guide Description."
Figure 3. Types of requests made to the guide. Participants made a total of 799 individual requests within 467 user queries, as many queries encompassed multiple request types. For instance, the query “Describe where that sound is coming from” includes both a visual description and an auditory description. A bar chart depicting the types of requests users made to the guide. \parThe bar chart is titled "Types of User Requests Made to the Guide." It shows seven categories on the vertical axis labeled "Request Types", and runs from 0 to 500 on the horizontal axis labeled "Request Counts." In order from top to bottom, and greatest to least, the bar chart shows bars for: "Navigation" (477 requests), "Visual Description" (248 requests), "Clarification" (23 requests), "Audio Beacon" (19 requests), "Social Interaction" (10 requests), "Confirmation" (8 requests), and "Auditory Description" (4 requests).
Figure 4. Tones of requests made to the guide. Participants made 476 requests total, one for each query to the guide, as each query counted as a separate utterance with its own tone. A bar chart depicting the different tones of requests users made to the guide. \parThe bar chart is titled "Tones of Requests Made to the Guide." It shows three categories on the vertical axis labeled "Tone Types", and runs from 0 to 400 on the horizontal axis labeled "Tone Counts." In order from top to bottom, and greatest to least, the bar chart shows bars for: "Utilitarian" (355 requests), "Polite" (75 requests), and "Friendly" (46 requests).
Figure 5. Likert scale responses from participants rating the guide, collected after participants experienced both parks and guide personas. Scores were given on a scale of 1 to 5, where 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree. Note: In the statement on Social Comfort, “others” refers to the confederates participants met during the study. We only include scores from thirteen participants who used the guide. A histogram containing the Likert-scale response data from participants rating the guide on various qualities. The scale runs from 1 to 5, with numbers in columns below depicting how many responses from participants fell under each number on the scale for a given statement and factor. They are colored in various shades of blue, with the darker colors representing more responses falling under a particular number on the scale and the lighter colors representing fewer. \parFor the statement: [Usability] It was easy to work with the guide to complete tasks. The histogram reports 1 user responded 1, 2 users responded 2, 6 users responded 3, 2 users responded 4, and 2 users responded 5. The mean is 3.2, standard deviation 1.1. \parFor the statement: [Usefulness] The guide was able to effectively address my commands and questions. The histogram reports 0 users responded 1, 3 users responded 1, 1 user responded 3, 9 users responded 4, and 0 users responded 5. The mean is 3.5, standard deviation 0.8. \parFor the statement: [Joy of Use] The guide was an enjoyable part of the park experience. The histogram reports 0 users responded 1, 1 user responded 2, 3 users responded 3, 3 users responded 4, and 6 users responded 5. The mean is 4.1, standard deviation 1.0. \parFor the statement: [Social Comfort] I felt comfortable using the guide in front of others. The histogram reports 1 user responded 1, 2 users responded 2, 1 user responded 3, 5 users responded 4, and 4 users responded 5. The mean is 3.7, standard deviation 1.3. \parFor the statement: [Scene Understanding] I felt like I had a full understanding of the contents of the virtual parks. The histogram reports 1 user responded 1, 3 users responded 2, 0 users responded 3, 5 users responded 4, and 4 users responded 5. The mean is 3.6, standard deviation 1.1. \parFor the statement: [Object Perception] It was easy to identify the contents of the virtual parks, such as specific objects. The histogram reports 1 user responded 1, 1 user responded 2, 2 users responded 3, 7 users responded 4, and 2 users responded 5. The mean is 3.6, standard deviation 1.1. \parFor the statement: [Navigation] It was easy to move around the contents of the virtual parks. The histogram reports 2 users responded 1, 2 users responded 2, 4 users responded 3, 3 users responded 4, and 2 users responded 5. The mean is 3.1, standard deviation 1.3. \par