cs.HCSep 18, 2026
SaveTouvigation: Embodied Adaptive Object Acquisition for Blind and Low-Vision Users in Unfamiliar Indoor Environments
Organizations: Stony Brook University, United States · New York University, United States · Brown University, United States · Imperial College London, United Kingdom · Middlebury Institute of International Studies at Monterey, United States · The Hong Kong University of Science and Technology (Guangzhou), China · Tongji University, China · Shanghai Qibao Dwight High School, China · The Chinese University of Hong Kong, Shenzhen, China
Abstract
Blind and low-vision users often face challenges when locating and physically acquiring objects in unfamiliar indoor environments. Existing vision-language-model-based assistants can provide semantic descriptions but may introduce latency, hallucinations, and guidance that is poorly aligned with embodied action. We present Touvigation, a hands-free object acquisition system that combines vision-language understanding with persistent local spatial modeling to provide low-latency, body-relative guidance. Drawing on formative interviews with eight blind and low-vision participants, we design a multi-stage guidance framework that adapts spatial references as users transition from orienting, to walking, to reaching and tactile verification. We evaluated Touvigation with 12 blind and low-vision participants against a multimodal large-language-model assistant and unassisted search. Touvigation achieved 100% task success, compared with 58% for the multimodal assistant and 85% for unassisted search, while reducing completion time and cognitive workload. Our findings demonstrate how persistent spatial grounding and adaptive embodied guidance can improve object acquisition for blind and low-vision users.
Figures & tables
Figure 1. Touvigation supports hands-free object acquisition through a chest-mounted, LiDAR-equipped iPhone. The illustrated interaction progresses through object locating, adaptive navigation, and tactile verification. Spoken guidance uses body-relative clock directions, personalized step counts, and descriptions of target height and tactile features to help users locate, reach, and identify the requested object. The figure illustrates Touvigation’s three-stage process for helping a blind or low-vision user locate and physically acquire an object. On the left, a simplified diagram shows a person surrounded by three body-centered guidance regions: a large lavender area representing orientation to the surrounding space, a green region around the feet representing walking guidance, and an orange region around the extended hand representing reaching guidance. On the right, three columns show the process in the same indoor room, with a yellow water bottle on a counter as the target. Phase 1, Object Locating: the system identifies the target and gives broad spatial guidance. A chest-mounted phone user receives the instructions, “Find the yellow water bottle” and “On the table, around 6 steps away.” Phase 2, Adaptive Navigation: guidance becomes movement-specific as the user approaches the target. Instructions include “Turn to 2 o’clock. Walk 4 steps,” followed by a corrective instruction, “Turn back to 4 o’clock,” illustrating that guidance is updated when the user changes direction. Phase 3, Tactile Acquisition: once the user is within reach, the system shifts from walking guidance to hand-level guidance. It says, “In front of you, at waist height. Tall cylinder. Smooth, cool metal surface, with a slightly textured plastic lid.” The final first-person image shows both hands reaching toward the yellow bottle on the counter.
| System | Hands-free | Obstacle avoidance | Target persists outside FOV | Hand / reach guidance | Tactile / material information | Low-latency real-time | Adapts to user’s sensing strategy |
| Be My Eyes / Be My AI ( Be My Eyes, 2026 ; Be My Eyes, 2023 ) | – | ✓ | – | ✓ | – | – | – |
| Google Lookout ( Google LLC, 2025 ) | – | ✓ | – | – | – | ✓ | – |
| Doubao ( ByteDance, 2026 ) | – | ✓ | – | – | – | – | – |
| Shike ( Shike, 2026 ) | ✓ | – | – | – | – | ✓ | – |
| Take My Hand ( Rahman et al., 2023 ) | ✓ | – | – | ✓ | – | - | – |
| NaviSense ( Sridhar et al., 2025 ) | – | – | ✓ | ✓ | – | – | – |
Table 1. Comparison of Touvigation with existing visual-assistance, navigation, and object-search systems for BLV users. ✓ = supported; – = not supported.
| Participant | Gender | Age | Vision Impairment Level | Onset of Impairment | Occupation |
|---|---|---|---|---|---|
| FP1 | Male | 30 | 1 | Since age 4 | Student |
| FP2 | Female | 30 | 1 | Since 2021 | N/A |
| FP3 | Female | 27 | 4 | Since birth | Teacher |
| FP4 | Male | 19 | 4 | Since age 9 | Student |
| FP5 | Female | 25 | 1 | Since 2023 | N/A |
| FP6 | Male | 23 | 1 | Since age 6 | Massage Therapist |
Table 2. Participant demographics from the formative interview study ( ). Vision impairment levels follow the Chinese national visual-disability classification standard ( Standardization Administration of China, 2010 ) , where Level 1 indicates more severe vision loss and Level 4 indicates lower severity.
Figure 2. Five challenges with current LLM-powered visual assistance identified through our formative study: (1) inconsistent spatial reference frames, (2) distance descriptions that are difficult to translate into movement, (3) missing tactile attributes for distinguishing objects, (4) delayed feedback during movement, and (5) excessive information unrelated to the current action. The figure summarizes five challenges with current LLM-powered visual assistance identified in the formative study. A confused stick figure appears on the left, while five numbered panels on the right illustrate the challenges. (1) Inconsistent References: a compass, left/right arrows around a person, and a camera frame with a location pin represent conflicting spatial reference frames, such as world-centered, body-relative, and camera-relative directions. (2) Ambiguous Distances: examples including 2 m, 2 steps, near, and not far show that distance descriptions may be inconsistent or difficult to translate into movement. (3) Missing Attributes: two visually distinguishable mugs and a reaching hand illustrate that visual descriptions may omit tactile properties needed to identify an object by touch. (4) Delayed Feedback: a person speaks to an AI and waits for a response, while a clock and turtle symbolize the latency of current LLMs and guidance that may arrive too slowly during movement. (5) Excessive Information: many overlapping speech bubbles surrounding a confused user illustrate descriptions that provide more information than is useful for the user's immediate action.
Figure 3. The design rationale of Touvigation. D1: embodied guidance uses torso, foot, and hand envelopes for direction, distance, and final reach. D2: continuous guidance updates instructions and corrects errors during movement. D3: situated guidance progressively increases detail as users approach the target. The figure presents three design principles of Touvigation in a hand-drawn, three-row illustration using lavender, green, and orange panels. Stick figures, arrows, footprints, clock faces, and colored regions visualize how guidance changes around the user's body and during movement. \parD1, Embodied Guidance, uses three body-centered envelopes. The torso envelope places the user at the center of a clock face with furniture around them to show body-relative directions. The foot envelope uses walking direction and step counts, while the hand envelope shows a reaching region over nearby objects on a table. \parD2, Continuous Guidance, shows guidance updating during movement. Step counts decrease as the user approaches the target, a wrong turn is corrected with a curved arrow and new clock direction, and an obstacle is marked with an exclamation point while a dashed path redirects the user around it. \parD3, Situated Guidance, progresses from Far to Near to Within Reach. Detail increases from coarse step counts, to step counts plus clock directions, and finally to close-range spatial and tactile descriptions such as chest height, tall cylinder, hanging ring, and flip-top lid.
Figure 4. System overview of Touvigation. (a) Semantic-spatial grounding binds visual detections to a persistent 3D target. (b) Continuous locomotion guidance updates the route using clock directions and personalized step counts. (c) Hand acquisition combines target semantics with hand tracking to guide reaching and confirmation. (d) Cloud models support object verification and semantic descriptions, with guidance delivered through voice. A flowchart. On the left, a BLV user with RGB camera, LiDAR, IMU, microphone, and a body profile provides input and receives voice guidance. Stage 1 turns YOLO26 detections ~\cite{jocher2026ultralytics} into a persistent SLAM-locked 3D target through 2D-to-3D binding and multi-view evidence voting. Stage 2 runs voxel fusion and route planning, producing clock-bearing and low-latency instructions using step length and orientation, until the user is within 0.75 meters. Stage 3 adjusts orientation, sends the image stream and target to the cloud for semantics, and fuses the result with hand tracking until the target is acquired. A cloud section at the top holds GPT-5.2 and GPT realtime.
| ID | Gender | Age | Residual perception | Onset | AI/assistive tools used |
|---|---|---|---|---|---|
| P1 | M | 43 | None | Congenital | Doubao, Shike |
| P2 | F | 49 | Light | Acquired (25) | Doubao, VoiceOver, Shike |
| P3 | M | 52 | Light + color | Acquired (16) | Doubao |
| P4 | F | 53 | Light | Congenital | Doubao |
| P5 | M | 46 | Light + slight color | Acquired (30) | Doubao, Be My Eyes |
| P6 | F | 55 | Light | Congenital | Doubao |
Table 3. Main-study participants (P1–P12). All were blind at Level 1 of the Chinese national visual-disability classification (see text). Residual perception: None = no light perception; Light = light perception only; “+ color” / “+ slight color” = additional partial color discrimination. Shike ( Shike, 2026 ) is a Chinese navigation app for blind users; Tencent, Baidu, and Gaode Maps are mainstream Chinese map apps.
Figure 5. Participants P1–P12 during the study, each wearing the chest-mounted phone in one of the two rooms. Faces and selected content are pixelated for privacy. A four-by-three grid of photographs labeled P1 to P12. Each shows one participant standing or walking in the conference room or the activity room with a phone mounted on a chest harness; faces and any whiteboard writing are pixelated.
Figure 6. The two study rooms and their target positions. (a) L1, a conference room. (b) L2, an activity room. The lower pictures show the six predefined target positions in each room, each marked with a pink note. Both rooms were unfamiliar to all participants. Two color photographs of the study rooms above two strips of small photographs. Photograph (a) shows L1, a conference room with purple curtains, a light wood floor, a white metal storage rack, small tables with folding chairs, and a whiteboard. Photograph (b) shows L2, a more open activity room with gray concrete walls, a green billiard table near the center, black leather sofas, and a corner shelf. The strip labeled L1 shows five close-up photographs of the six target positions in the conference room (positions four and five in one photo), each a surface such as a table, the floor, or a shelf with a small object and a pink marker note. The strip labeled L2 shows six close-up photographs of the target positions in the activity room, including a sofa, a chair, the billiard table, and a corner shelf, each with a small object and a pink marker note.
| Room | Condition | Trials per participant |
|---|---|---|
| L1 (conference room) | C1 Touvigation | 2 |
| C2 Doubao | 2 | |
| C3 Unassisted | 2 | |
| L2 (activity room) | C1 Touvigation | 2 |
| C2 Doubao | 2 | |
| C3 Unassisted | 2 |
Table 4. Study design. Each participant completed 2 trials in every room–condition cell: 3 conditions 2 rooms 2 trials 12 trials per participant, and 144 trials across the 12 participants. Condition order and room order were counterbalanced (see text).
| C1 Touvigation | C2 Doubao | C3 Unassisted | ||||||
|---|---|---|---|---|---|---|---|---|
| ID | Success | Time | Success | Time ∗ | Time † | Success | Time ∗ | Time † |
| P1 | 4/4 | 101 | 1/4 | 206 | 106 | 2/4 | 140 | 41 |
| P2 | 4/4 | 105 | 2/4 | 177 | 115 | 3/4 | 138 | 104 |
| P3 | 4/4 | 62 | 1/4 | 230 | 200 | 3/4 | 133 | 98 |
| P4 | 4/4 | 115 | 2/4 | 188 | 137 | 3/4 | 125 | 87 |
| P5 | 4/4 | 75 | 4/4 | 119 | 119 | 4/4 | 94 | 94 |
Table 5. Per-participant successes and mean completion time (s). Time ∗ scores failures at 240 s; Time † counts successful trials only.
Figure 7. Mean straight-line distance to the target over time for successful trials with trajectory data, by condition. Shading shows 1 SE; within-reach zone ( m) ( Qian et al., 2019 ; Drillis and Contini, 1966 ) . Line chart of mean distance to target against time for the three conditions, successful trials only. Touvigation falls into the within-reach zone by about one minute and unassisted search by about 100 seconds, overlapping thereafter; Doubao descends more slowly and enters the zone at about 140 seconds.
Figure 8. Participant-level distributions of overall workload and individual ratings from the NASA-TLX questionnaire across Touvigation, Doubao, and unassisted search (N = 12; 0–100 scale). Horizontal boxplots compare the three conditions on overall workload, mental, physical, and temporal demand, effort, and self-rated performance, on a 0 to 100 scale. Touvigation, in purple, has the lowest boxes on every workload row and the highest box for performance. Unassisted search is highest on mental and physical demand while Doubao is highest on temporal demand and effort.
Figure 9. Experience items by condition: mean (SD) agreement on a 0–5 scale over the 12 participants, the Friedman test across the three conditions (Kendall’s as effect size), and Wilcoxon signed-rank pairwise comparisons with Holm-corrected (T = Touvigation, D = Doubao, U = unassisted). Shaded cells are significant at . A table of nine questionnaire items grouped into spatial awareness, trust, perceived safety, and experience. For every item, Touvigation has the highest mean and its comparisons against Doubao and against unassisted search are significant, except for willingness to use the method in everyday settings. No comparison between Doubao and unassisted search is significant.
Figure 10. Four example unassisted-search trajectories overlaid on reconstructed room maps. Blue stars mark targets, circles mark starts, and squares mark endpoints. Path color encodes elapsed time normalized within each trial, from yellow at the start to red at the end. (a) P3 searched the activity-room perimeter and around the billiard table without confirming the target. (b) P9 passed near the target before finding it at 134 s. (c) P10 searched across the conference room before finding the target at 184 s. (d) P11 followed a more compact route and found the target at 97 s. Four top-down room maps, two of the activity room and two of the conference room, each with a search path colored from yellow to red, a start circle, an end square, and a blue star marking the target.