cs.HCMar 10, 2026

Understanding the Use of a Large Language Model-Powered Guide to Make Virtual Reality Accessible for Blind and Low Vision People

Authors: Jazmin Collins, Sharon Y Lin, Tianqi Liu, Andrea Stevenson Won, Shiri Azenkot

Organizations: Cornell University New York, New York, USA · Cornell Tech New York, New York, USA · Cornell University Ithaca, New York, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

As social virtual reality (VR) grows more popular, addressing accessibility for blind and low vision (BLV) users is increasingly critical. Researchers have proposed an AI "sighted guide" to help users navigate VR and answer their questions, but it has not been studied with users. To address this gap, we developed a large language model (LLM)-powered guide and studied its use with 16 BLV participants in virtual environments with confederates posing as other users. We found that when alone, participants treated the guide as a tool, but treated it companionably around others, giving it nicknames, rationalizing its mistakes with its appearance, and encouraging confederate-guide interaction. Our work furthers understanding of guides as a versatile method for VR accessibility and presents design recommendations for future guides.

Figures & tables

Figure 1. A screenshot of a participant utilizing the AI guide in its dog form, traveling through a virtual park together. A screenshot of a virtual park environment. A female avatar and a German Shepherd breed dog with a bright red guide dog harness stand in the center of the image. A wooden dock is behind them at the edge of a river cutting through the park. A bridge crosses the river to the woman's right. In the distance, there is a line of green trees.
Figure 2. The three virtual environments used in the study. From left to right, the Tutorial contains various simplistic 3D objects such as yellow and green cubes, Park 1 contains a large river and several colorful gazebos, and Park 2 contains colorful flower beds and a series of benches for resting throughout the park. A table with three images of the virtual environments used in the study. On the far left is an image of the tutorial room, a simple white room with various geometrical shapes, including a yellow, horizontal cube acting as a table with three interactable shapes resting on it. In the middle is a screenshot of the first park with trees scattered about and a river cutting through, populated by stone walkways and bridges, two water fountains, three gazebos, and three groups of avatars. On the right is a screenshot of the second park, a brightly-lit park with stone tile walkways around a few square patches of grass, two gazebos, and a group of avatars gathered for a dance party.
PseudonymAgeGenderVisual ConditionEtiologyVisual AcuityVisual FieldNavigation Tools Used
Niya45FBlind; No Light PerceptionNRNRNRWhite Cane; Human Guide; Navigation Apps; Tactile Maps
Anika46FLow VisionOptic AtrophyR: 20/400, L: 20/200Vision in upper field of left eye, right eye blurryWhite Cane; Navigation Apps
Teo51MLow VisionCortical ImpairmentR: 20/400, L: 20/400NRWhite Cane; Navigation Apps; GPS Devices
Nikolai34MPartially Sighted; Legally Blind; Night Blindness; Central Vision LossNRNRNRNavigation Apps
Sara22FLegally BlindDual Optic ColobomaNRNRWhite Cane; Navigation Apps
Xiran38FLegally Blind; Night Blindness; Peripheral Vision LossNRNRNRWhite Cane; Human Guide; Navigation Apps
Table 1. Participant demographics. F = female, M = male. Participant information is self-reported. NR indicates that participants chose not to report that information. Information about our participant demographics. Contains all textual content.
GroupPhaseTask TypeDurationParkPersona
(N = 4 participants per group)   AssignmentAssignment
1Round 1Individual Task (Exploration)10 minutesPark 1Dog
Social Task (Tour)15 minutes
Round 2Individual Task (Exploration)10 minutesPark 2Robot
Social Task (Tour)15 minutes
2Round 1Individual Task (Exploration)10 minutesPark 1Dog
Table 2. The counterbalanced study conditions (e.g., duration, park, persona) across all groups of participants. Group 1 = Niya, Anika, Teo, Nikolai; Group 2 = Sara, Xiran, Yuki, Maritza; Group 3 = Aiden, Len, Siobhan, Isaiah; Group 4 = Li, Jamal, Yadriel, Jiwoo. Participants completed two tasks per study phase (i.e., one individual and one social task).
PersonaGuide AvatarGuide DescriptionGuide MovementVoice + Speech CharacteristicsPersonality PromptSpeech Sample
Human An androgynous human with short black hair and glasses, wearing a gray hoodie and black pants.Follows behind the user from an arm’s length away. Footstep sound effects indicate movement.Alloy, androgynous voice, medium formality (some jokes, casual speech), averages 101.2 words per responseWarm, friendly, but still professional sighted guideThe scene around you is a simple, geometric landscape composed of distinctive objects. To your right stands the Tall Building, a towering yellow cube that rises high above the rest…
Guide Dog German-shepherd with black and brown fur, wearing a bright-red guide dog harness.Walks beside the user on their left side. The sounds of panting and paws scratching the floor indicate movement.Nova, feminine voice, low formality (uses slang, jokes, and casual speech), averages 185.4 words per responseVery friendly, excited companion who is eager to please who you’re talking toWhat an exciting world we’re in! To your right stands the impressive Tall Building, reaching up high with its yellow facade…
Robot A shiny, silver robot that is humanoid in stature, with a masculine frame.Follows behind the user from an arm’s length away. Creaking machinery sound effects indicate movement.Onyx, masculine voice, high formality (speaks entirely in proper sentences, no casual speech), averages 91.2 words per responseFormal and assertive assistant who talks like a robotAffirmative. The area around you is composed of geometric shapes and structures. Directly to your right lies the Tall Building…
Table 3. Our guide’s three personas: the human, the guide dog, and the robot. The voices mentioned for each persona (Alloy, Nova, and Onyx) are OpenAI’s default TTS voices ( OpenAI, ) . Audio samples of each voice can be found at the cited link. The personality prompts were sent to GPT-4 to shape the guide’s responses (see supplemental materials). All columns are based on our previous designs from Collins et al. ( Collins et al., 2024 ) , except for Speech Sample, which was produced by our current implementation of the AI guide. Information about three different guide personas. Contains all textual content, aside from the three images of the persona's avatars in the column "Guide Avatar." Each avatar is described in the following column "Guide Description."
Figure 3. Types of requests made to the guide. Participants made a total of 799 individual requests within 467 user queries, as many queries encompassed multiple request types. For instance, the query “Describe where that sound is coming from” includes both a visual description and an auditory description. A bar chart depicting the types of requests users made to the guide. \parThe bar chart is titled "Types of User Requests Made to the Guide." It shows seven categories on the vertical axis labeled "Request Types", and runs from 0 to 500 on the horizontal axis labeled "Request Counts." In order from top to bottom, and greatest to least, the bar chart shows bars for: "Navigation" (477 requests), "Visual Description" (248 requests), "Clarification" (23 requests), "Audio Beacon" (19 requests), "Social Interaction" (10 requests), "Confirmation" (8 requests), and "Auditory Description" (4 requests).
Figure 4. Tones of requests made to the guide. Participants made 476 requests total, one for each query to the guide, as each query counted as a separate utterance with its own tone. A bar chart depicting the different tones of requests users made to the guide. \parThe bar chart is titled "Tones of Requests Made to the Guide." It shows three categories on the vertical axis labeled "Tone Types", and runs from 0 to 400 on the horizontal axis labeled "Tone Counts." In order from top to bottom, and greatest to least, the bar chart shows bars for: "Utilitarian" (355 requests), "Polite" (75 requests), and "Friendly" (46 requests).
Figure 5. Likert scale responses from participants rating the guide, collected after participants experienced both parks and guide personas. Scores were given on a scale of 1 to 5, where 1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree. Note: In the statement on Social Comfort, “others” refers to the confederates participants met during the study. We only include scores from thirteen participants who used the guide. A histogram containing the Likert-scale response data from participants rating the guide on various qualities. The scale runs from 1 to 5, with numbers in columns below depicting how many responses from participants fell under each number on the scale for a given statement and factor. They are colored in various shades of blue, with the darker colors representing more responses falling under a particular number on the scale and the lighter colors representing fewer. \parFor the statement: [Usability] It was easy to work with the guide to complete tasks. The histogram reports 1 user responded 1, 2 users responded 2, 6 users responded 3, 2 users responded 4, and 2 users responded 5. The mean is 3.2, standard deviation 1.1. \parFor the statement: [Usefulness] The guide was able to effectively address my commands and questions. The histogram reports 0 users responded 1, 3 users responded 1, 1 user responded 3, 9 users responded 4, and 0 users responded 5. The mean is 3.5, standard deviation 0.8. \parFor the statement: [Joy of Use] The guide was an enjoyable part of the park experience. The histogram reports 0 users responded 1, 1 user responded 2, 3 users responded 3, 3 users responded 4, and 6 users responded 5. The mean is 4.1, standard deviation 1.0. \parFor the statement: [Social Comfort] I felt comfortable using the guide in front of others. The histogram reports 1 user responded 1, 2 users responded 2, 1 user responded 3, 5 users responded 4, and 4 users responded 5. The mean is 3.7, standard deviation 1.3. \parFor the statement: [Scene Understanding] I felt like I had a full understanding of the contents of the virtual parks. The histogram reports 1 user responded 1, 3 users responded 2, 0 users responded 3, 5 users responded 4, and 4 users responded 5. The mean is 3.6, standard deviation 1.1. \parFor the statement: [Object Perception] It was easy to identify the contents of the virtual parks, such as specific objects. The histogram reports 1 user responded 1, 1 user responded 2, 2 users responded 3, 7 users responded 4, and 2 users responded 5. The mean is 3.6, standard deviation 1.1. \parFor the statement: [Navigation] It was easy to move around the contents of the virtual parks. The histogram reports 2 users responded 1, 2 users responded 2, 4 users responded 3, 3 users responded 4, and 2 users responded 5. The mean is 3.1, standard deviation 1.3. \par
More in this topic

Assistive Technology for Blind and Low-Vision People

See all 33 papers

Explore similar work

CardsList
Sep 14, 2026

More Than Just Access: Generative AI as Communication Intermediary for Blind and Low-Vision Users

Generative AI (GenAI) tools are increasingly woven into how blind and low-vision (BLV) people communicate, not only with digital information, but with the physical world and with other people. Tools such as ChatGPT, Google Gemini, Be My AI, and Seeing AI translate visual and textual content into accessible form, and are beginning to substitute for interpersonal requests for help, such as asking a family member to read a label or describe a scene. Drawing on semi-structured interviews with 19 BLV participants, we examine GenAI as a communication intermediary and how it succeeds and fails as an alternative for reading, describing, and even asking another person for help. We also investigated what BLV users gain and risk when these tools take over that role. We conclude with design and policy implications for GenAI systems that communicate uncertainty honestly, protect information, and support BLV users' independence rather than substitute for it unsafely.

Protik Dey, Mohd Saifuzzaman, Taslima Akter

University of Texas at San Antonio

Assistive Technology for Blind and Low-Vision PeopleHuman-AI InteractionHuman-Centered AI
AskPDF
Sep 13, 2026

Beyond Scene Description: Multi-Agent Orchestration for Non-visual Access to Virtual Worlds

Virtual worlds now host classrooms, meetings, conferences, shops, and social venues, and nearly every interaction they expose assumes a user who can scan a three-dimensional scene, follow avatars, and read floating panels. Blind and visually impaired (BVI) users are left with assistive tools that each solve one task in isolation: naming an object, reading text, describing a scene, or planning a route. A live virtual room defeats that model: obstacles, speakers, gestures, chat, slides, and notifications arrive together, and a tool that narrates all of them trades a visual barrier for an auditory one. This paper presents MetaBlind, an architecture that distributes nonvisual access across eight specialized agents, spanning perception, navigation, social and object interaction, communication, safety and trust, memory, and personalization, and that places an Accessibility Orchestrator between those agents and the user. Agents publish candidate information into a shared accessibility context instead of speaking to the user directly. The orchestrator scores each candidate on safety relevance, goal relevance, urgency, confidence, user relevance, and estimated listening load, then releases only the items it judges relevant at that moment through speech, structured audio, or haptic output. We give the selection step a formal statement, specify the orchestration cycle as an algorithm, and define an evaluation protocol against a single-agent assistant. MetaBlind is reported at the design stage, with no prototype measurement or user study, and the protocol states which outcomes would support the design and which would refute it.

Toqeer Ali Syed, Ali Akarma, Adeel Ahmad +1

AI Center, Faculty of Computer and Information Systems, Islamic University of Madinah, Madinah, Saudi Arabia

Assistive Technology for Blind and Low-Vision PeopleMulti-Agent OrchestrationMultimodal AgentsVirtual Reality
AskPDF
May 10, 2026

Insight: Enhancing Mobile Accessibility for Blind and Visually Impaired Users with LLMs

This research paper addresses the limitations of current mobile accessibility services like TalkBack, which provide manual gesture-based sequential feedback to BVI users. Motivated by the promise of large language models (LLMs), this paper introduces Insight, an Android accessibility service that provides natural language interaction and real-time summarization of the screen. The paper performs a within-subject experimental study with users to compare Insight and TalkBack on usability factors. Results show Insight reduced mental effort and task time, and was preferred because of its dialogue interface, but users felt the need for interruption management. Results show LLM-based interfaces can significantly improve mobile accessibility, and describe the potential of hybrid solutions combining gesture and dialogue modalities towards more inclusive design.

Joshua Owusu Ansah, Anuj Kapoor, Ayush Khanna +3

Arizona State University · Tempe, Arizona, USA

Assistive Technology for Blind and Low-Vision People
AskPDF