cs.CVOct 16, 2026
SaveVISTA: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users
Organizations: Harvard AI and Robotics Lab, Harvard University · Télécom Paris, Institut Polytechnique de Paris · Texas A&M University · University of Toronto ·
Abstract
Blindness and low vision limit many people's ability to complete daily tasks, motivating AI assistants that provide useful guidance. Yet, existing benchmarks do not adequately test whether vision models can provide effective everyday assistance. In particular, i) existing benchmarks primarily rely on visual and textual inputs, with limited support for other modalities that are increasingly available in egocentric wearable devices such as gaze, inertial measurements, spatial trajectories, and others; and ii) existing evaluations emphasize scene understanding and question answering, with limited coverage of goal-conditioned guidance for blind and low-vision users across everyday tasks. To bridge this gap, we introduce VISTA (Visually Impaired Scene and Task Annotations), a multimodal egocentric benchmark for goal-oriented assistance for blind and low-vision users. VISTA contains 1,000 egocentric recordings captured with Project Aria glasses; each recording provides synchronized sensor streams, six of which serve as model inputs, and three different types of annotation, covering ten task categories (TC), from object localization to hazard detection. We evaluate 14 representative vision-language models zero-shot on VISTA and design a lightweight multimodal model baseline that integrates all six modalities. Experimental results show that, although existing models can often describe egocentric scenes, they remain limited in generating goal-directed assistive guidance. VISTA thus provides a benchmark for evaluating multimodal models that could improve everyday accessibility.
Figures & tables
Figure 1: Sensor modalities of the Aria glasses. VISTA records synchronized RGB video, stereo SLAM and eye-tracking cameras, audio, IMU, barometer, and magnetometer. See Appendix A.2 (Figure 6 ) for an expanded visualization.
Figure 2: Examples of the ten task categories. Each strip shows three clips, with the RGB frame on top and the SLAM and eye-tracking frames below. Each category has about 100 clips (997 in total); definitions are in Table 4 .
Figure 3: Simulated vision conditions (conceptual illustration). Top: The same scene under blindness, central vision loss, peripheral vision loss, and high myopia. Bottom: Regions that may be most useful to describe for each condition. The regions are illustrative and are not evaluated in this work.
Figure 4: Annotation pipeline. Recorded clips are annotated by multiple annotators with scene descriptions, goal–action pairs, and QA pairs, and then verified.
| Type | Dataset | # Videos | # Queries | Avg. len. (s) | Setting | Modalities | Annotation | QA format | Supported tasks | Human eval. | |||||||||
| BLV orient. | Smart glasses | RGB | Audio | Gaze | IMU | SLAM/ depth | Multi- Ans. | Ques.- Cat. | OE | MC | Scene desc. | QA | Goal instr. | ||||||
| General purpose | EgoTaskQA ( Jia et al., 2022 ) | 2K | 40K | 25.0 | ✗ | ✗ | ✓ | – | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | – |
| VidEgoThink ( Cheng et al., 2024 ) | 217 | 600 | 23.4 | ✗ | ✗ | ✓ | – | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | – | |
| EgoSchema ( Mangalam et al., 2023 ) | 5K | 5K | 180.0 | ✗ | ✗ | ✓ | – | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | – | |
| EgoMemoria ( Ye et al., 2025 ) | 629 | 7K | 858.5 | ✗ | ✗ | ✓ | – | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✓ | ✗ | – | |
| Assistive | QAEgo4D ( Bärmann and Waibel, 2022 ) | 1.3K | 14.5K | 495.1 | ✓ | ✗ | ✓ | – | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | – |
Table 1: Comparison with egocentric and assistive VQA datasets. BLV: blind and low-vision users; Multi-Ans.: multiple reference answers; Ques.-Cat.: question-category labels; OE/MC: open-ended/multiple-choice QA. ✓/✗: supported/not supported; –: not reported. Human eval.: human assessment reported; for VISTA, a BLV community reviewer assessed 34 annotations from 27 clips (model outputs were not human-evaluated). VISTA (highlighted) is the only dataset among those compared that supports scene description, QA, and goal-driven instruction together.
Figure 5: Multimodal baseline. RGB frames are encoded by the frozen vision encoder of the VLM. The five sensor streams used as inputs (IMU, magnetometer/barometer, eye gaze, hand tracking, 6DoF trajectory) pass through temporal adapters, question-conditioned reliability gates, and bottleneck exchange, giving 8 (stage 1) or 28 (stage 2) sensor tokens that are inserted into the prompt. Only the LoRA adapters and the sensor-fusion network are trained.
| Model | Size | Score | Rubric components | Response type | |||||
| EG | AC | TC | ACom | Scene | Q&A | Guidance | |||
| Open-source models (frames text) | |||||||||
| InternVL3 | 1B | 2.39 | 0.82 | 0.79 | 0.47 | 0.33 | 3.49 | 3.09 | 1.42 |
| 2B | 2.75 | 0.96 | 0.91 | 0.54 | 0.35 | 3.77 | 3.19 | 1.96 | |
| 8B | 3.16 | 1.09 | 1.07 | 0.64 | 0.38 | 3.72 | 3.48 | 2.69 | |
| 9B | 3.13 | 1.08 | 1.06 | 0.63 | 0.38 | 3.63 | 3.32 | 2.76 | |
Table 2: Zero-shot results on VISTA. Judge scores (0–5). EG: evidence grounding (0–1.75); AC: answer correctness (0–1.75); TC: task completion (0–1); ACom: accessible communication (0–0.5); Scene, Q&A, Guidance: mean score per annotation type. Bold / underline : best/second-best per column within each model group; –: size not disclosed.
| Model | Size | Method | Score | Rubric components | Response type | |||||
| EG | AC | TC | ACom | Scene | Q&A | Guidance | ||||
| Pretrained | 2.75 | 0.96 | 0.91 | 0.54 | 0.35 | 3.77 | 3.19 | 1.96 | ||
| 2B | VISTA (ours) | 3.54 (+28.7%) | 1.24 | 1.20 | 0.68 | 0.42 | 3.82 | 3.78 | 3.28 | |
| Pretrained | 3.16 | 1.09 | 1.07 | 0.64 | 0.38 | 3.72 | 3.48 | 2.69 | ||
| InternVL3 | 8B | VISTA (ours) | 3.72 (+17.7%) | 1.29 | 1.27 | 0.73 | 0.44 | 4.03 | 4.07 | 3.39 |
| Pretrained | 3.22 | 1.11 | 1.10 | 0.68 | 0.38 | 3.78 | 3.69 | 2.65 | ||
Table 3: VISTA versus pretrained backbones. Columns as in Table 2 ; VISTA rows are shaded, with the gain over the pretrained model in green. Bold / underline : best/second-best per column.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| ID | Category | #Clips | Question–answer example | Goal–instruction example |
| TC01 | Object Localization (LOC) | 100 | Q: Where is the bench? A: The bench is about 2 meters in front of you on the brick patio. It is positioned before the black metal fence that separates you from the street. | Goal: Reach the bench. Instruction: Walk forward about three steps toward the brick patio area. The bench is directly ahead of you and slightly to your left, positioned parallel to the black metal fence. |
| TC02 | Hazard Detection (HAZ) | 99 | Q: Are there any hazards here? A: This is a construction area that does not have any construction tape. There are potential hazards with not fully paved roads and sidewalks. There can also be debris falling from the sky. | Goal: Get past the building walking straight. Instruction: Look above you as you are walking past the building. Walk directly forward and at a faster pace than normal walking. Look to the right and left of you as well. |
| TC03 | Scene Awareness (SCN) | 102 | Q: How crowded is the store? A: There are approximately four to five people visible in the main checkout and aisle area ahead of you, but there is plenty of open floor space to navigate comfortably. | Goal: Navigate to checkout. Instruction: Walk straight forward for approximately five meters toward the counter area where the other customers are standing. |
| TC04 | Navigation (NAV) | 101 | Q: How do I get to the crosswalk? A: Walk straight ahead and turn right toward the crosswalk. | Goal: Reach the nearest pedestrian crossing. Instruction: Walk straight ahead, pass the flower display, and turn right toward the marked crossing. |
| TC05 | Social Cues (SOC) | 97 | Q: Describe this person’s facial expression in front of me. A: The person has a calm expression with a slight, closed-mouth smile | Goal: Locate the person Instruction: Look straight ahead at the man sitting across the table. |
| TC06 | Distance Estimation (DST) | 99 | Q: How far is the black car? A: The black car is about 5 meters in front of you. | Goal: Reach the door of the black car. Instruction: Walk straight about 5 meters and reach your hand up. |
Table 4: Task categories. Clip counts and one annotated example per category, quoted from the annotations. ID: category identifier; code in parentheses: short name; #Clips: clips in the category (997 in total).
Figure 6: Recorded streams and derived tracking signals. Rows (a–c) show object localization, hazard detection, and navigation. Blue/red/green denote for IMU and magnetometer, and roll/pitch/yaw for orientation; blue/red denote gaze yaw/pitch (degrees). Trajectories show relative position (m), with omitted and circles/triangles marking start/end. Orientation changes (degrees) are relative to the initial device frame. Trajectory, orientation, and environmental scales vary by clip.
| Frame | Scene description | Question–answer pair | Goal–instruction pair |
| TC01_001 | You are in a supermarket aisle. On the shelves to your left, oranges and lemons are piled up. Bananas are hanging from the shelves on your right. | Q: Where is the oranges? A: The oranges is to your left. | Goal: Go to get the oranges. Instruction: Walk straightly to your left and the oranges is on the shelves. |
| TC02_001 | You are standing in an open paved plaza with a curved glass building on your right. Gray bollards with chains form a barrier directly ahead. A black vehicle and orange cones are to your right, with a silver car farther ahead. | Q: Are there any hazards in front of me? A: You are facing short gray bollards connected by white ropes directly ahead, creating a tripping hazard. Orange traffic cones and a parked vehicle are to your right, and cars may pass through the driveway nearby. | Goal: Navigate safely around the barrier and cross the road. Instruction: Turn 45 degrees left. Walk about 5 steps to the end of the bollards, keeping clear of the rope and posts. Pause and check for cars, then face the orange cone and walk about 10 steps forward across the driveway. |
| TC04_042 | You are looking at an outdoor plaza area with wide concrete paths and multiple stairways leading upward. There are handrails along the steps, landscaped grassy areas on both sides, and tall buildings in the background. | Q: How do I get to the park? A: Walk up the stairs in front of you to get to the park. | Goal: Walk up the stairs safely. Instruction: Move toward the steps, place your foot firmly on each stair, and use the handrail for balance as you make your way to the upper level. |
Table 5: Annotation examples. Scene description, question–answer pair, and goal–instruction pair for the three clips in Figure 6 , quoted verbatim from the test split. Below each frame: clip ID and preview/total duration .
| Field | Count | Mean | Min | Max |
| Scene descriptions | 1,063 | 230.1 | 74 | 445 |
| Questions | 1,191 | 31.6 | 11 | 85 |
| Answers | 1,191 | 81.8 | 1 | 352 |
| Goals | 2,067 | 30.5 | 8 | 100 |
| Instructions | 2,067 | 121.1 | 29 | 319 |
Table 6: Annotation field statistics. Count: number of annotations; Mean, Min, Max: length in characters. Counts pool the train, validation, and test splits.
Figure 7: Annotation statistics by split. (a–e) Text-length distributions in characters; dashed lines mark means, and headers give counts, means, and medians. Bin widths are 20, 5, 20, 5, and 20 characters. (f) Task-item counts per split. Scene lengths include all 1,063 references, while task counts use one scene-description item per clip (997 in total).
| Model | Params | Frames | Resolution | Max tokens |
| InternVL3-1B | 1B | 8 | 4,096 | |
| InternVL3-2B | 2B | 8 | 4,096 | |
| InternVL3-8B | 8B | 8 | 4,096 | |
| InternVL3-9B | 9B | 8 | 4,096 | |
| InternVL3-14B | 14B | 8 | 4,096 | |
| Qwen3-VL-2B | 2B | 8 | dynamic ∗ | 4,096 |
Table 7: Zero-shot inference settings. Params: number of parameters; Frames: frames per item; Resolution: input resolution after preprocessing (pixels); Max tokens: maximum output length. ∗ Computed automatically by the Qwen-VL preprocessing pipeline.
| Model | Response type | Score | EG | AC | TC | ACom | |
| InternVL3-1B | Scene description | 3.49 | 1.16 | 1.18 | 0.76 | 0.43 | 450 |
| Q&A | 3.09 | 1.05 | 1.01 | 0.59 | 0.44 | 510 | |
| Guidance | 1.42 | 0.51 | 0.45 | 0.25 | 0.21 | 868 | |
| InternVL3-2B | Scene description | 3.77 | 1.28 | 1.30 | 0.79 | 0.44 | 450 |
| Q&A | 3.19 | 1.11 | 1.03 | 0.60 | 0.44 | 510 | |
| Guidance | 1.96 | 0.70 | 0.64 | 0.37 | 0.25 | 868 |
Table 8: Zero-shot VISTA results by response type. Score: judge score (0–5); EG: evidence grounding (0–1.75); AC: answer correctness (0–1.75); TC: task completion (0–1); ACom: accessible communication (0–0.5); : number of test items. Bold and underline mark the best and second-best open-source result per column within each response type.
| ID | Category | InternVL3-1B | InternVL3-2B | InternVL3-8B | |||||||||||||
| S | EG | AC | TC | ACom | S | EG | AC | TC | ACom | S | EG | AC | TC | ACom | |||
| TC01 | Object Localization | 45 | 3.47 | 1.14 | 1.17 | 0.76 | 0.43 | 3.83 | 1.28 | 1.30 | 0.80 | 0.44 | 3.72 | 1.24 | 1.29 | 0.80 | 0.42 |
| TC02 | Hazard Detection | 45 | 3.44 | 1.16 | 1.17 | 0.74 | 0.42 | 3.67 | 1.29 | 1.29 | 0.78 | 0.44 | 3.86 | 1.31 | 1.34 | 0.82 | 0.43 |
| TC03 | Scene Awareness | 46 | 3.54 | 1.18 | 1.22 | 0.78 | 0.42 | 3.69 | 1.27 | 1.29 | 0.79 | 0.43 | 3.83 | 1.27 | 1.32 | 0.81 | 0.42 |
| TC04 | Navigation | 45 | 3.66 | 1.22 | 1.24 | 0.78 | 0.44 | 3.86 | 1.31 | 1.33 | 0.79 | 0.45 | 3.76 | 1.25 | 1.29 | 0.79 | 0.43 |
| TC05 | Social Cues | 44 | 3.64 | 1.20 | 1.23 | 0.78 | 0.44 | 3.81 | 1.29 | 1.32 | 0.79 | 0.45 | 3.56 | 1.19 | 1.22 | 0.74 | 0.40 |
Table 9: Scene-description scores per task category. The first four blocks cover the twelve open-source models and the last block the two proprietary models. ID: task category; : test items per category; S: judge score (0–5); EG, AC, TC, ACom: rubric components as in Table 8 ; Avg.: mean S over the twelve open-source models, in bold . The lowest averages are TC07 (3.39), TC05 (3.41), and TC10 (3.45).
| ID | Category | InternVL3-1B | InternVL3-2B | InternVL3-8B | |||||||||||||
| S | EG | AC | TC | ACom | S | EG | AC | TC | ACom | S | EG | AC | TC | ACom | |||
| TC01 | Object Localization | 54 | 3.63 | 1.26 | 1.23 | 0.70 | 0.43 | 3.51 | 1.20 | 1.17 | 0.69 | 0.44 | 3.46 | 1.17 | 1.16 | 0.69 | 0.44 |
| TC02 | Hazard Detection | 48 | 3.16 | 1.05 | 1.07 | 0.62 | 0.42 | 2.63 | 0.88 | 0.85 | 0.50 | 0.40 | 3.90 | 1.34 | 1.34 | 0.78 | 0.45 |
| TC03 | Scene Awareness | 51 | 3.60 | 1.25 | 1.20 | 0.68 | 0.46 | 3.62 | 1.25 | 1.20 | 0.70 | 0.46 | 4.11 | 1.41 | 1.40 | 0.82 | 0.47 |
| TC04 | Navigation | 48 | 2.26 | 0.77 | 0.71 | 0.42 | 0.37 | 2.71 | 0.93 | 0.88 | 0.53 | 0.38 | 3.06 | 1.07 | 1.01 | 0.60 | 0.40 |
| TC05 | Social Cues | 45 | 3.55 | 1.20 | 1.17 | 0.70 | 0.48 | 3.96 | 1.38 | 1.33 | 0.78 | 0.48 | 3.75 | 1.33 | 1.23 | 0.70 | 0.49 |
Table 10: Question-answering scores per task category. Same format as Table 9 . The lowest-scoring categories are TC07 (2.72), TC06 (2.75), and TC04 (2.80).
| ID | Category | InternVL3-1B | InternVL3-2B | InternVL3-8B | |||||||||||||
| S | EG | AC | TC | ACom | S | EG | AC | TC | ACom | S | EG | AC | TC | ACom | |||
| TC01 | Object Localization | 82 | 1.45 | 0.51 | 0.45 | 0.27 | 0.23 | 1.94 | 0.69 | 0.64 | 0.38 | 0.25 | 2.98 | 1.02 | 1.04 | 0.62 | 0.33 |
| TC02 | Hazard Detection | 93 | 1.43 | 0.52 | 0.46 | 0.26 | 0.20 | 1.99 | 0.73 | 0.64 | 0.37 | 0.25 | 2.54 | 0.92 | 0.86 | 0.51 | 0.31 |
| TC03 | Scene Awareness | 95 | 1.47 | 0.54 | 0.48 | 0.26 | 0.21 | 2.04 | 0.72 | 0.69 | 0.40 | 0.24 | 2.77 | 0.94 | 0.96 | 0.58 | 0.32 |
| TC04 | Navigation | 88 | 1.43 | 0.51 | 0.47 | 0.25 | 0.21 | 1.91 | 0.68 | 0.65 | 0.37 | 0.24 | 2.58 | 0.91 | 0.88 | 0.53 | 0.31 |
| TC05 | Social Cues | 75 | 1.50 | 0.52 | 0.50 | 0.27 | 0.23 | 1.98 | 0.69 | 0.66 | 0.37 | 0.26 | 2.59 | 0.90 | 0.90 | 0.53 | 0.33 |
Table 11: Guidance scores per task category. Same format as Table 9 . The lowest-scoring categories are TC06 (2.08), TC10 (2.15), and TC04 (2.17).
| Model | Size | Score | EG | AC | TC | ACom |
| Open-source models (frames text) | ||||||
| InternVL3 | 1B | 3.14 | 1.05 | 1.04 | 0.61 | 0.45 |
| 2B | 3.29 | 1.11 | 1.09 | 0.64 | 0.45 | |
| 8B | 3.46 | 1.17 | 1.16 | 0.68 | 0.45 | |
| VideoLLaMA3 | 7B | 3.11 | 1.06 | 1.03 | 0.59 | 0.43 |
| Qwen3-VL | 2B | 3.11 | 1.06 | 1.03 | 0.59 | 0.42 |
Table 12: Zero-shot results on EgoBlind. Judge scores (0–5) on items with the same judge and rubric as VISTA. EG, AC, TC, ACom: rubric components as in Table 8 ; –: size not disclosed. Bold and underline mark the best and second-best open-source result per column. Absolute scores are not directly comparable with VISTA, since EgoBlind provides more references per item (2.24 vs. 1.01 on average).
Table 13: Failure statistics on VISTA (RGB and text input). Crit.: answers flagged by the judge as containing an error that could cause immediate harm; Corr.: answers not judged meaningfully correct; 2/5: answers scoring below 2/5; Univ.: items on which no model reaches 3/5; Conflict: answers contradicting the reference but supported by the frames; Unver.: answers with a claim the eight frames cannot verify. Score: mean judge score (0–5). Items: test items per row. Panels (b) and (c) pool all fourteen models.
| Clip: TC02_018 | ||
| Question: Am I walking on the correct path with no obstacles? | ||
| Reference: No. You are not on the sidewalk, and there are gates in front of you that block your path. | ||
| Model | Score | Response |
| Qwen3-VL-2B | 2.10 | Based on the visual information provided, it appears that you are walking on a path that is not entirely clear of obstacles. The path you are on is […] |
| Gemini 3.1 Flash-Lite † | 1.60 | Based on the images provided, it appears you are standing on a sidewalk or a paved area with a small, raised metal grate structure immediately to […] |
| InternVL3-14B † | 1.25 | Yes, you seem to be walking on a path with no immediate obstacles. The path appears clear and is bordered by a fence on one side. Just be cautious of […] |
Table 14: Failure case (a): egocentric body orientation. All fourteen responses, ordered by score and truncated to 150 characters. † : flagged by the judge as an error that could cause immediate harm (13 of 14 responses).
| Clip: TC10_003 | ||
| Goal: Retrieve or play with the soccer ball. | ||
| Reference: The orange soccer ball is on behind you on your left. Turn around 180 degrees and locate the orange soccer ball. Walk towards it and reach down to retrieve the soccer ball. | ||
| Model | Score | Response |
| GPT-Realtime Mini | 2.05 | All right! Let’s work together step-by-step. Let’s get started. First, let’s locate your position. You’re in a grassy area with large trees […] |
| Qwen3-VL-2B | 2.00 | Based on the provided images, it appears that the person is in a park, likely in Boston, USA, given the architecture and the presence of a large […] |
| Qwen3-VL-4B | 1.95 | Based on the images provided, here are clear step-by-step instructions to help the visually impaired person retrieve or play with the soccer ball: 1. […] |
Table 15: Failure case (b): objects behind the wearer. All fourteen responses, ordered by score and truncated to 150 characters. † : flagged by the judge as an error that could cause immediate harm.
| Clip: TC10_079 | ||
| Question: Am I moving? | ||
| Reference: No, you are not moving. | ||
| Model | Score | Response |
| Qwen3-VL-2B | 1.10 | Based on the visual evidence provided in the series of images, we can analyze the person’s movement. - In the second image, the person is seen […] |
| InternVL3-9B | 1.05 | Yes, it appears you are moving. The changing perspective and the position of other people in relation to you suggest movement. |
| InternVL3-14B | 1.05 | Based on the images, it looks like you are moving. The changing positions of people and objects suggest movement through the scene. |
Table 16: Failure case (c): dynamic and temporal reasoning. All fourteen responses, ordered by score and truncated to 150 characters. No response is flagged as a critical error, but every model answers yes where the reference answer is no .
| Clip: TC06_094 | ||
| Question: How far is the trash can? | ||
| Reference: The trash can is 10 meters directly in front of you. | ||
| Model | Score | Response |
| Gemma 3-4B | 2.30 | Based on the images, the trash can appears to be approximately 10-15 feet away from the center of the frame. It’s positioned roughly halfway between […] |
| Gemma 3-12B | 2.25 | Based on the images, it appears the trash can is approximately 15-20 feet away from the viewer. It’s positioned across the pedestrian area, closer to […] |
| Gemma 3-27B | 2.10 | Based on the images, it appears the trash can is approximately 15-20 feet away. Here’s how I estimate that: It’s positioned within the […] |
Table 17: Failure case (d): metric distance estimation. All fourteen responses, ordered by score and truncated to 150 characters. Every response is flagged by the judge as containing a claim the eight frames cannot verify.