cs.HCSep 18, 2026

Touvigation: Embodied Adaptive Object Acquisition for Blind and Low-Vision Users in Unfamiliar Indoor Environments

Authors: George Xi Wang, Xiangyu Li, Shaoyue Wen, Jiaqian Hu, Junan Xie, Yupeng Wang, Ziyue Shi, Qijun Chen, +3 more

Organizations: Stony Brook University, United States · New York University, United States · Brown University, United States · Imperial College London, United Kingdom · Middlebury Institute of International Studies at Monterey, United States · The Hong Kong University of Science and Technology (Guangzhou), China · Tongji University, China · Shanghai Qibao Dwight High School, China · The Chinese University of Hong Kong, Shenzhen, China

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Blind and low-vision users often face challenges when locating and physically acquiring objects in unfamiliar indoor environments. Existing vision-language-model-based assistants can provide semantic descriptions but may introduce latency, hallucinations, and guidance that is poorly aligned with embodied action. We present Touvigation, a hands-free object acquisition system that combines vision-language understanding with persistent local spatial modeling to provide low-latency, body-relative guidance. Drawing on formative interviews with eight blind and low-vision participants, we design a multi-stage guidance framework that adapts spatial references as users transition from orienting, to walking, to reaching and tactile verification. We evaluated Touvigation with 12 blind and low-vision participants against a multimodal large-language-model assistant and unassisted search. Touvigation achieved 100% task success, compared with 58% for the multimodal assistant and 85% for unassisted search, while reducing completion time and cognitive workload. Our findings demonstrate how persistent spatial grounding and adaptive embodied guidance can improve object acquisition for blind and low-vision users.

Figures & tables

Figure 1. Touvigation supports hands-free object acquisition through a chest-mounted, LiDAR-equipped iPhone. The illustrated interaction progresses through object locating, adaptive navigation, and tactile verification. Spoken guidance uses body-relative clock directions, personalized step counts, and descriptions of target height and tactile features to help users locate, reach, and identify the requested object. The figure illustrates Touvigation’s three-stage process for helping a blind or low-vision user locate and physically acquire an object. On the left, a simplified diagram shows a person surrounded by three body-centered guidance regions: a large lavender area representing orientation to the surrounding space, a green region around the feet representing walking guidance, and an orange region around the extended hand representing reaching guidance. On the right, three columns show the process in the same indoor room, with a yellow water bottle on a counter as the target. Phase 1, Object Locating: the system identifies the target and gives broad spatial guidance. A chest-mounted phone user receives the instructions, “Find the yellow water bottle” and “On the table, around 6 steps away.” Phase 2, Adaptive Navigation: guidance becomes movement-specific as the user approaches the target. Instructions include “Turn to 2 o’clock. Walk 4 steps,” followed by a corrective instruction, “Turn back to 4 o’clock,” illustrating that guidance is updated when the user changes direction. Phase 3, Tactile Acquisition: once the user is within reach, the system shifts from walking guidance to hand-level guidance. It says, “In front of you, at waist height. Tall cylinder. Smooth, cool metal surface, with a slightly textured plastic lid.” The final first-person image shows both hands reaching toward the yellow bottle on the counter.
SystemHands-freeObstacle avoidanceTarget persists outside FOVHand / reach guidanceTactile / material informationLow-latency real-timeAdapts to user’s sensing strategy
Be My Eyes / Be My AI ( Be My Eyes, 2026 ; Be My Eyes, 2023 )–✓–✓–––
Google Lookout ( Google LLC, 2025 )–✓–––✓–
Doubao ( ByteDance, 2026 )–✓–––––
Shike ( Shike, 2026 )✓––––✓–
Take My Hand ( Rahman et al., 2023 )✓––✓–-–
NaviSense ( Sridhar et al., 2025 )––✓✓–––
Table 1. Comparison of Touvigation with existing visual-assistance, navigation, and object-search systems for BLV users. ✓ = supported; – = not supported.
ParticipantGenderAgeVision Impairment LevelOnset of ImpairmentOccupation
FP1Male301Since age 4Student
FP2Female301Since 2021N/A
FP3Female274Since birthTeacher
FP4Male194Since age 9Student
FP5Female251Since 2023N/A
FP6Male231Since age 6Massage Therapist
Table 2. Participant demographics from the formative interview study ( N=8N=8 ). Vision impairment levels follow the Chinese national visual-disability classification standard ( Standardization Administration of China, 2010 ) , where Level 1 indicates more severe vision loss and Level 4 indicates lower severity.
Figure 2. Five challenges with current LLM-powered visual assistance identified through our formative study: (1) inconsistent spatial reference frames, (2) distance descriptions that are difficult to translate into movement, (3) missing tactile attributes for distinguishing objects, (4) delayed feedback during movement, and (5) excessive information unrelated to the current action. The figure summarizes five challenges with current LLM-powered visual assistance identified in the formative study. A confused stick figure appears on the left, while five numbered panels on the right illustrate the challenges. (1) Inconsistent References: a compass, left/right arrows around a person, and a camera frame with a location pin represent conflicting spatial reference frames, such as world-centered, body-relative, and camera-relative directions. (2) Ambiguous Distances: examples including 2 m, 2 steps, near, and not far show that distance descriptions may be inconsistent or difficult to translate into movement. (3) Missing Attributes: two visually distinguishable mugs and a reaching hand illustrate that visual descriptions may omit tactile properties needed to identify an object by touch. (4) Delayed Feedback: a person speaks to an AI and waits for a response, while a clock and turtle symbolize the latency of current LLMs and guidance that may arrive too slowly during movement. (5) Excessive Information: many overlapping speech bubbles surrounding a confused user illustrate descriptions that provide more information than is useful for the user's immediate action.
Figure 3. The design rationale of Touvigation. D1: embodied guidance uses torso, foot, and hand envelopes for direction, distance, and final reach. D2: continuous guidance updates instructions and corrects errors during movement. D3: situated guidance progressively increases detail as users approach the target. The figure presents three design principles of Touvigation in a hand-drawn, three-row illustration using lavender, green, and orange panels. Stick figures, arrows, footprints, clock faces, and colored regions visualize how guidance changes around the user's body and during movement. \parD1, Embodied Guidance, uses three body-centered envelopes. The torso envelope places the user at the center of a clock face with furniture around them to show body-relative directions. The foot envelope uses walking direction and step counts, while the hand envelope shows a reaching region over nearby objects on a table. \parD2, Continuous Guidance, shows guidance updating during movement. Step counts decrease as the user approaches the target, a wrong turn is corrected with a curved arrow and new clock direction, and an obstacle is marked with an exclamation point while a dashed path redirects the user around it. \parD3, Situated Guidance, progresses from Far to Near to Within Reach. Detail increases from coarse step counts, to step counts plus clock directions, and finally to close-range spatial and tactile descriptions such as chest height, tall cylinder, hanging ring, and flip-top lid.
Figure 4. System overview of Touvigation. (a) Semantic-spatial grounding binds visual detections to a persistent 3D target. (b) Continuous locomotion guidance updates the route using clock directions and personalized step counts. (c) Hand acquisition combines target semantics with hand tracking to guide reaching and confirmation. (d) Cloud models support object verification and semantic descriptions, with guidance delivered through voice. A flowchart. On the left, a BLV user with RGB camera, LiDAR, IMU, microphone, and a body profile provides input and receives voice guidance. Stage 1 turns YOLO26 detections ~\cite{jocher2026ultralytics} into a persistent SLAM-locked 3D target through 2D-to-3D binding and multi-view evidence voting. Stage 2 runs voxel fusion and route planning, producing clock-bearing and low-latency instructions using step length and orientation, until the user is within 0.75 meters. Stage 3 adjusts orientation, sends the image stream and target to the cloud for semantics, and fuses the result with hand tracking until the target is acquired. A cloud section at the top holds GPT-5.2 and GPT realtime.
IDGenderAgeResidual perceptionOnsetAI/assistive tools used
P1M43NoneCongenitalDoubao, Shike
P2F49LightAcquired (25)Doubao, VoiceOver, Shike
P3M52Light + colorAcquired (16)Doubao
P4F53LightCongenitalDoubao
P5M46Light + slight colorAcquired (30)Doubao, Be My Eyes
P6F55LightCongenitalDoubao
Table 3. Main-study participants (P1–P12). All were blind at Level 1 of the Chinese national visual-disability classification (see text). Residual perception: None = no light perception; Light = light perception only; “+ color” / “+ slight color” = additional partial color discrimination. Shike ( Shike, 2026 ) is a Chinese navigation app for blind users; Tencent, Baidu, and Gaode Maps are mainstream Chinese map apps.
Figure 5. Participants P1–P12 during the study, each wearing the chest-mounted phone in one of the two rooms. Faces and selected content are pixelated for privacy. A four-by-three grid of photographs labeled P1 to P12. Each shows one participant standing or walking in the conference room or the activity room with a phone mounted on a chest harness; faces and any whiteboard writing are pixelated.
Figure 6. The two study rooms and their target positions. (a) L1, a conference room. (b) L2, an activity room. The lower pictures show the six predefined target positions in each room, each marked with a pink note. Both rooms were unfamiliar to all participants. Two color photographs of the study rooms above two strips of small photographs. Photograph (a) shows L1, a conference room with purple curtains, a light wood floor, a white metal storage rack, small tables with folding chairs, and a whiteboard. Photograph (b) shows L2, a more open activity room with gray concrete walls, a green billiard table near the center, black leather sofas, and a corner shelf. The strip labeled L1 shows five close-up photographs of the six target positions in the conference room (positions four and five in one photo), each a surface such as a table, the floor, or a shelf with a small object and a pink marker note. The strip labeled L2 shows six close-up photographs of the target positions in the activity room, including a sofa, a chair, the billiard table, and a corner shelf, each with a small object and a pink marker note.
RoomConditionTrials per participant
L1 (conference room)C1 Touvigation2
 C2 Doubao2
 C3 Unassisted2
L2 (activity room)C1 Touvigation2
 C2 Doubao2
 C3 Unassisted2
Table 4. Study design. Each participant completed 2 trials in every room–condition cell: 3 conditions ×\times 2 rooms ×\times 2 trials == 12 trials per participant, and 144 trials across the 12 participants. Condition order and room order were counterbalanced (see text).
 C1 TouvigationC2 DoubaoC3 Unassisted
IDSuccessTimeSuccessTime ∗Time †SuccessTime ∗Time †
P14/41011/42061062/414041
P24/41052/41771153/4138104
P34/4621/42302003/413398
P44/41152/41881373/412587
P54/4754/41191194/49494
Table 5. Per-participant successes and mean completion time (s). Time ∗ scores failures at 240 s; Time † counts successful trials only.
Figure 7. Mean straight-line distance to the target over time for successful trials with trajectory data, by condition. Shading shows ±\pm 1 SE; within-reach zone ( <0.75<0.75 m) ( Qian et al., 2019 ; Drillis and Contini, 1966 ) . Line chart of mean distance to target against time for the three conditions, successful trials only. Touvigation falls into the within-reach zone by about one minute and unassisted search by about 100 seconds, overlapping thereafter; Doubao descends more slowly and enters the zone at about 140 seconds.
Figure 8. Participant-level distributions of overall workload and individual ratings from the NASA-TLX questionnaire across Touvigation, Doubao, and unassisted search (N = 12; 0–100 scale). Horizontal boxplots compare the three conditions on overall workload, mental, physical, and temporal demand, effort, and self-rated performance, on a 0 to 100 scale. Touvigation, in purple, has the lowest boxes on every workload row and the highest box for performance. Unassisted search is highest on mental and physical demand while Doubao is highest on temporal demand and effort.
Figure 9. Experience items by condition: mean (SD) agreement on a 0–5 scale over the 12 participants, the Friedman test across the three conditions (Kendall’s WW as effect size), and Wilcoxon signed-rank pairwise comparisons with Holm-corrected pp (T = Touvigation, D = Doubao, U = unassisted). Shaded cells are significant at p<.05p<.05 . A table of nine questionnaire items grouped into spatial awareness, trust, perceived safety, and experience. For every item, Touvigation has the highest mean and its comparisons against Doubao and against unassisted search are significant, except for willingness to use the method in everyday settings. No comparison between Doubao and unassisted search is significant.
Figure 10. Four example unassisted-search trajectories overlaid on reconstructed room maps. Blue stars mark targets, circles mark starts, and squares mark endpoints. Path color encodes elapsed time normalized within each trial, from yellow at the start to red at the end. (a) P3 searched the activity-room perimeter and around the billiard table without confirming the target. (b) P9 passed near the target before finding it at 134 s. (c) P10 searched across the conference room before finding the target at 184 s. (d) P11 followed a more compact route and found the target at 97 s. Four top-down room maps, two of the activity room and two of the conference room, each with a search path colored from yellow to red, a start circle, an end square, and a blue star marking the target.

Explore similar work

CardsList
Sep 17, 2026cs.RO

RoboFind: Multi-Agent Personalized Object Search for People Who Are Blind or Have Low Vision

Blind and low-vision users often need to locate a specific personal object rather than an arbitrary instance of the same category. The task calls for a robot that can move through the space and reach viewpoints the user cannot, and for an accessible interface where the user says which object is meant and learns whether the right one was found. We present RoboFind, a multi-agent framework in which a smartphone teaches the target and a quadruped robot carries out the search. A Target Teaching Agent converts guided smartphone recordings into a semantic target profile and a reusable multi-view reference bank through an accessible capture flow with AR guidance, speech and haptic feedback, and screen-reader support, so later missions refer to a stored object without repeating the teaching process. At runtime, a Navigation Agent explores the environment and proposes candidate targets, a Verification Agent checks each candidate against the stored references, and a Coordination and Recovery Agent completes the mission or triggers recovery and continued search. Across 32 real-robot missions, RoboFind reaches 85.0% success against 25.0% for a reconstructed sequential first-stop baseline over 20 trials with ten targets, and reduces false success from 75.0% to 5.0%. On six shared targets it succeeds in 10/12 trials, against 5/12 for 12 independently executed GPT-6 Astra-only trials. These results show that the multi-agent design fits the demands of personalized object search, where verifying object identity before declaring completion is what makes the outcome something a user can rely on.

Ruiping Liu, Shaofang Quan, Qian Yin +10

The authors are with the Institute for Anthropomatics and Robotics, Karlsruhe Institute of Technology, Karlsruhe 76131, Germany. · The author is with the School of Artificial Intelligence and Robotics and the National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, Changsha 410012, China. · The author is with Mercedes-Benz.

Assistive RoboticsMulti-Agent SystemsVisual Search
AskPDF
Sep 11, 2026cs.AI

Assisted Spatial Cognition Through Vision-Language Models

Multimodal AI, powered by Large Language Models (LLMs) and Vision-Language Models (VLMs), is transforming assistive technologies by enabling simultaneous processing of visual and textual data. This advancement holds significant promise for over 43 million visually impaired and neuro-divergent individuals worldwide who face persistent challenges in navigating indoor and outdoor environments due to limited spatial awareness and insufficient environmental cues. Existing navigation aids often lack comprehensive 3D scene understanding, relying on constrained route-based strategies that hinder user autonomy. In this paper, we introduce a novel end-to-end framework that integrates LLMs, VLMs and digital twin technologies to deliver a spatially cognitive navigation support for visually impaired and neuro-divergent users. Our system captures video input via standard mobile phone cameras, and employs SLAM3R to generate dense 3D point clouds from monocular RGB sequences in real-time. Our custom post-processing algorithm ensures accurate point cloud alignment across multiple viewpoints without requiring predefined reference points. This enhances the capabilities of SpatialLM to produce structured 3D representations, including architectural elements and oriented object bounding boxes. The enriched spatial data is then processed by a locally deployed LLM, which interprets 3D contexts to generate detailed scene descriptions and precise distance measurements between users and surrounding objects. We evaluated our approach across diverse video scenarios featuring various perspectives, looped walking views and captured in multiple environments. The evaluation results demonstrate consistent accuracy in 3D scene interpretation and object localisation, underscoring the potential of our system as a transformative assistive navigation solution that combines advanced visual perception with spatial reasoning

H. Riaz, J. B. Fernandez, I. Mills +3

School of Electronic Engineering, Dublin City University, Dublin, Ireland · Insight Research Ireland Centre for Data Analytics, Dublin, Ireland · South East Technological University, Waterford, Ireland

Visual Spatial ReasoningVision-Language Navigation3D Scene Understanding3D Visual Grounding
AskPDF
Sep 13, 2026cs.AI

Beyond Scene Description: Multi-Agent Orchestration for Non-visual Access to Virtual Worlds

Virtual worlds now host classrooms, meetings, conferences, shops, and social venues, and nearly every interaction they expose assumes a user who can scan a three-dimensional scene, follow avatars, and read floating panels. Blind and visually impaired (BVI) users are left with assistive tools that each solve one task in isolation: naming an object, reading text, describing a scene, or planning a route. A live virtual room defeats that model: obstacles, speakers, gestures, chat, slides, and notifications arrive together, and a tool that narrates all of them trades a visual barrier for an auditory one. This paper presents MetaBlind, an architecture that distributes nonvisual access across eight specialized agents, spanning perception, navigation, social and object interaction, communication, safety and trust, memory, and personalization, and that places an Accessibility Orchestrator between those agents and the user. Agents publish candidate information into a shared accessibility context instead of speaking to the user directly. The orchestrator scores each candidate on safety relevance, goal relevance, urgency, confidence, user relevance, and estimated listening load, then releases only the items it judges relevant at that moment through speech, structured audio, or haptic output. We give the selection step a formal statement, specify the orchestration cycle as an algorithm, and define an evaluation protocol against a single-agent assistant. MetaBlind is reported at the design stage, with no prototype measurement or user study, and the protocol states which outcomes would support the design and which would refute it.

Toqeer Ali Syed, Ali Akarma, Adeel Ahmad +1

AI Center, Faculty of Computer and Information Systems, Islamic University of Madinah, Madinah, Saudi Arabia

Multi-Agent OrchestrationMultimodal AgentsVirtual Reality
AskPDF