VISTA: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users

Noé Mosca; Mohamed Rayan Barhdadi; Muhammad Hamza; Joshua Yeh; Suryodaya Bikram Shahi; Zhiyan Li; Selim-Antoine Lali; Weiyue Li

cs.CVOct 16, 2026

VISTA: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users

Authors: Noé Mosca, Mohamed Rayan Barhdadi, Muhammad Hamza, Joshua Yeh, Suryodaya Bikram Shahi, Zhiyan Li, Selim-Antoine Lali, Weiyue Li,

Organizations: Harvard AI and Robotics Lab, Harvard University · Télécom Paris, Institut Polytechnique de Paris · Texas A&M University · University of Toronto ·

Abstract

Blindness and low vision limit many people's ability to complete daily tasks, motivating AI assistants that provide useful guidance. Yet, existing benchmarks do not adequately test whether vision models can provide effective everyday assistance. In particular, i) existing benchmarks primarily rely on visual and textual inputs, with limited support for other modalities that are increasingly available in egocentric wearable devices such as gaze, inertial measurements, spatial trajectories, and others; and ii) existing evaluations emphasize scene understanding and question answering, with limited coverage of goal-conditioned guidance for blind and low-vision users across everyday tasks. To bridge this gap, we introduce VISTA (Visually Impaired Scene and Task Annotations), a multimodal egocentric benchmark for goal-oriented assistance for blind and low-vision users. VISTA contains $\sim$ 1,000 egocentric recordings captured with Project Aria glasses; each recording provides synchronized sensor streams, six of which serve as model inputs, and three different types of annotation, covering ten task categories (TC), from object localization to hazard detection. We evaluate 14 representative vision-language models zero-shot on VISTA and design a lightweight multimodal model baseline that integrates all six modalities. Experimental results show that, although existing models can often describe egocentric scenes, they remain limited in generating goal-directed assistive guidance. VISTA thus provides a benchmark for evaluating multimodal models that could improve everyday accessibility.

Figures & tables

Figure 1: Sensor modalities of the Aria glasses. VISTA records synchronized RGB video, stereo SLAM and eye-tracking cameras, audio, IMU, barometer, and magnetometer. See Appendix A.2 (Figure 6 ) for an expanded visualization.

Figure 2: Examples of the ten task categories. Each strip shows three clips, with the RGB frame on top and the SLAM and eye-tracking frames below. Each category has about 100 clips (997 in total); definitions are in Table 4 .

Figure 3: Simulated vision conditions (conceptual illustration). Top: The same scene under blindness, central vision loss, peripheral vision loss, and high myopia. Bottom: Regions that may be most useful to describe for each condition. The regions are illustrative and are not evaluated in this work.

Figure 4: Annotation pipeline. Recorded clips are annotated by multiple annotators with scene descriptions, goal–action pairs, and QA pairs, and then verified.

Type

Dataset

# Videos

# Queries

Avg. len. (s)

Setting

Modalities

Annotation

QA format

Supported tasks

Human eval.

BLV orient.

Smart glasses

RGB

Audio

Gaze

IMU

SLAM/ depth

Multi- Ans.

Ques.- Cat.

Scene desc.

Goal instr.

General purpose

EgoTaskQA ( Jia et al., 2022 )

40K

25.0

✗

✓

–

✗

✓

✗

✓

✗

–

VidEgoThink ( Cheng et al., 2024 )

217

600

23.4

✗

✓

–

✗

✓

✗

✓

✗

–

EgoSchema ( Mangalam et al., 2023 )

180.0

✗

✓

–

✗

✓

✗

✓

✗

–

EgoMemoria ( Ye et al., 2025 )

629

858.5

✗

✓

–

✗

✓

✗

✓

✗

–

Assistive

QAEgo4D ( Bärmann and Waibel, 2022 )

1.3K

14.5K

495.1

✓

✗

✓

–

✗

✓

✗

✓

✗

–

Table 1: Comparison with egocentric and assistive VQA datasets. BLV: blind and low-vision users; Multi-Ans.: multiple reference answers; Ques.-Cat.: question-category labels; OE/MC: open-ended/multiple-choice QA. ✓/✗: supported/not supported; –: not reported. Human eval.: human assessment reported; for VISTA, a BLV community reviewer assessed 34 annotations from 27 clips (model outputs were not human-evaluated). VISTA (highlighted) is the only dataset among those compared that supports scene description, QA, and goal-driven instruction together.

Figure 5: Multimodal baseline. RGB frames are encoded by the frozen vision encoder of the VLM. The five sensor streams used as inputs (IMU, magnetometer/barometer, eye gaze, hand tracking, 6DoF trajectory) pass through temporal adapters, question-conditioned reliability gates, and bottleneck exchange, giving 8 (stage 1) or 28 (stage 2) sensor tokens that are inserted into the prompt. Only the LoRA adapters and the sensor-fusion network are trained.

Model	Size	Score	Rubric components				Response type
Model	Size		EG	AC	TC	ACom	Scene	Q&A	Guidance
Open-source models (frames $+$ text)
InternVL3	1B	2.39	0.82	0.79	0.47	0.33	3.49	3.09	1.42
	2B	2.75	0.96	0.91	0.54	0.35	3.77	3.19	1.96
	8B	3.16	1.09	1.07	0.64	0.38	3.72	3.48	2.69
	9B	3.13	1.08	1.06	0.63	0.38	3.63	3.32	2.76

Table 2: Zero-shot results on VISTA. Judge scores (0–5). EG: evidence grounding (0–1.75); AC: answer correctness (0–1.75); TC: task completion (0–1); ACom: accessible communication (0–0.5); Scene, Q&A, Guidance: mean score per annotation type. Bold / underline : best/second-best per column within each model group; –: size not disclosed.

Model	Size	Method	Score	Rubric components				Response type
Model	Size	Method		EG	AC	TC	ACom	Scene	Q&A	Guidance
		Pretrained	2.75	0.96	0.91	0.54	0.35	3.77	3.19	1.96
	2B	VISTA (ours)	3.54 (+28.7%)	1.24	1.20	0.68	0.42	3.82	3.78	3.28
		Pretrained	3.16	1.09	1.07	0.64	0.38	3.72	3.48	2.69
InternVL3	8B	VISTA (ours)	3.72 (+17.7%)	1.29	1.27	0.73	0.44	4.03	4.07	3.39
		Pretrained	3.22	1.11	1.10	0.68	0.38	3.78	3.69	2.65

Table 3: VISTA versus pretrained backbones. Columns as in Table 2 ; VISTA rows are shaded, with the gain over the pretrained model in green. Bold / underline : best/second-best per column.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Category	#Clips	Question–answer example	Goal–instruction example
TC01	Object Localization (LOC)	100	Q: Where is the bench? A: The bench is about 2 meters in front of you on the brick patio. It is positioned before the black metal fence that separates you from the street.	Goal: Reach the bench. Instruction: Walk forward about three steps toward the brick patio area. The bench is directly ahead of you and slightly to your left, positioned parallel to the black metal fence.
TC02	Hazard Detection (HAZ)	99	Q: Are there any hazards here? A: This is a construction area that does not have any construction tape. There are potential hazards with not fully paved roads and sidewalks. There can also be debris falling from the sky.	Goal: Get past the building walking straight. Instruction: Look above you as you are walking past the building. Walk directly forward and at a faster pace than normal walking. Look to the right and left of you as well.
TC03	Scene Awareness (SCN)	102	Q: How crowded is the store? A: There are approximately four to five people visible in the main checkout and aisle area ahead of you, but there is plenty of open floor space to navigate comfortably.	Goal: Navigate to checkout. Instruction: Walk straight forward for approximately five meters toward the counter area where the other customers are standing.
TC04	Navigation (NAV)	101	Q: How do I get to the crosswalk? A: Walk straight ahead and turn right toward the crosswalk.	Goal: Reach the nearest pedestrian crossing. Instruction: Walk straight ahead, pass the flower display, and turn right toward the marked crossing.
TC05	Social Cues (SOC)	97	Q: Describe this person’s facial expression in front of me. A: The person has a calm expression with a slight, closed-mouth smile	Goal: Locate the person Instruction: Look straight ahead at the man sitting across the table.
TC06	Distance Estimation (DST)	99	Q: How far is the black car? A: The black car is about 5 meters in front of you.	Goal: Reach the door of the black car. Instruction: Walk straight about 5 meters and reach your hand up.

Appendix

Table 4: Task categories. Clip counts and one annotated example per category, quoted from the annotations. ID: category identifier; code in parentheses: short name; #Clips: clips in the category (997 in total).

Figure 6: Recorded streams and derived tracking signals. Rows (a–c) show object localization, hazard detection, and navigation. Blue/red/green denote $x/y/z$ for IMU and magnetometer, and roll/pitch/yaw for orientation; blue/red denote gaze yaw/pitch (degrees). Trajectories show relative $xy$ position (m), with $z$ omitted and circles/triangles marking start/end. Orientation changes (degrees) are relative to the initial device frame. Trajectory, orientation, and environmental scales vary by clip.

Frame	Scene description	Question–answer pair	Goal–instruction pair
TC01_001 $t=5.5/11.0\,\mathrm{s}$	You are in a supermarket aisle. On the shelves to your left, oranges and lemons are piled up. Bananas are hanging from the shelves on your right.	Q: Where is the oranges? A: The oranges is to your left.	Goal: Go to get the oranges. Instruction: Walk straightly to your left and the oranges is on the shelves.
TC02_001 $t=5.8/11.5\,\mathrm{s}$	You are standing in an open paved plaza with a curved glass building on your right. Gray bollards with chains form a barrier directly ahead. A black vehicle and orange cones are to your right, with a silver car farther ahead.	Q: Are there any hazards in front of me? A: You are facing short gray bollards connected by white ropes directly ahead, creating a tripping hazard. Orange traffic cones and a parked vehicle are to your right, and cars may pass through the driveway nearby.	Goal: Navigate safely around the barrier and cross the road. Instruction: Turn 45 degrees left. Walk about 5 steps to the end of the bollards, keeping clear of the rope and posts. Pause and check for cars, then face the orange cone and walk about 10 steps forward across the driveway.
TC04_042 $t=4.6/9.2\,\mathrm{s}$	You are looking at an outdoor plaza area with wide concrete paths and multiple stairways leading upward. There are handrails along the steps, landscaped grassy areas on both sides, and tall buildings in the background.	Q: How do I get to the park? A: Walk up the stairs in front of you to get to the park.	Goal: Walk up the stairs safely. Instruction: Move toward the steps, place your foot firmly on each stair, and use the handrail for balance as you make your way to the upper level.

Appendix

Table 5: Annotation examples. Scene description, question–answer pair, and goal–instruction pair for the three clips in Figure 6 , quoted verbatim from the test split. Below each frame: clip ID and preview/total duration

t

Field	Count	Mean	Min	Max
Scene descriptions	1,063	230.1	74	445
Questions	1,191	31.6	11	85
Answers	1,191	81.8	1	352
Goals	2,067	30.5	8	100
Instructions	2,067	121.1	29	319

Appendix

Table 6: Annotation field statistics. Count: number of annotations; Mean, Min, Max: length in characters. Counts pool the train, validation, and test splits.

Panel of Figure 7 — Figure 7: Annotation statistics by split. (a–e) Text-length distributions in characters; dashed lines mark means, and headers give counts, means, and medians. Bin widths are 20, 5, 20, 5, and 20 characters. (f) Task-item counts per split. Scene lengths include all 1,063 references, while task counts use one scene-description item per clip (997 in total).

Model	Params	Frames	Resolution	Max tokens
InternVL3-1B	1B	8	$448\times 448$	4,096
InternVL3-2B	2B	8	$448\times 448$	4,096
InternVL3-8B	8B	8	$448\times 448$	4,096
InternVL3-9B	9B	8	$448\times 448$	4,096
InternVL3-14B	14B	8	$448\times 448$	4,096
Qwen3-VL-2B	2B	8	dynamic ∗	4,096

Appendix

Table 7: Zero-shot inference settings. Params: number of parameters; Frames: frames per item; Resolution: input resolution after preprocessing (pixels); Max tokens: maximum output length. ∗ Computed automatically by the Qwen-VL preprocessing pipeline.

Model	Response type	Score	EG	AC	TC	ACom	$n$
InternVL3-1B	Scene description	3.49	1.16	1.18	0.76	0.43	450
	Q&A	3.09	1.05	1.01	0.59	0.44	510
	Guidance	1.42	0.51	0.45	0.25	0.21	868
InternVL3-2B	Scene description	3.77	1.28	1.30	0.79	0.44	450
	Q&A	3.19	1.11	1.03	0.60	0.44	510
	Guidance	1.96	0.70	0.64	0.37	0.25	868

Appendix

Table 8: Zero-shot VISTA results by response type. Score: judge score (0–5); EG: evidence grounding (0–1.75); AC: answer correctness (0–1.75); TC: task completion (0–1); ACom: accessible communication (0–0.5);

n

: number of test items. Bold and underline mark the best and second-best open-source result per column within each response type.

ID	Category	$n$	InternVL3-1B					InternVL3-2B					InternVL3-8B
ID	Category	$n$	S	EG	AC	TC	ACom	S	EG	AC	TC	ACom	S	EG	AC	TC	ACom
TC01	Object Localization	45	3.47	1.14	1.17	0.76	0.43	3.83	1.28	1.30	0.80	0.44	3.72	1.24	1.29	0.80	0.42
TC02	Hazard Detection	45	3.44	1.16	1.17	0.74	0.42	3.67	1.29	1.29	0.78	0.44	3.86	1.31	1.34	0.82	0.43
TC03	Scene Awareness	46	3.54	1.18	1.22	0.78	0.42	3.69	1.27	1.29	0.79	0.43	3.83	1.27	1.32	0.81	0.42
TC04	Navigation	45	3.66	1.22	1.24	0.78	0.44	3.86	1.31	1.33	0.79	0.45	3.76	1.25	1.29	0.79	0.43
TC05	Social Cues	44	3.64	1.20	1.23	0.78	0.44	3.81	1.29	1.32	0.79	0.45	3.56	1.19	1.22	0.74	0.40

Appendix

Table 9: Scene-description scores per task category. The first four blocks cover the twelve open-source models and the last block the two proprietary models. ID: task category;

n

: test items per category; S: judge score (0–5); EG, AC, TC, ACom: rubric components as in Table 8 ; Avg.: mean S over the twelve open-source models, in bold . The lowest averages are TC07 (3.39), TC05 (3.41), and TC10 (3.45).

ID	Category	$n$	InternVL3-1B					InternVL3-2B					InternVL3-8B
ID	Category	$n$	S	EG	AC	TC	ACom	S	EG	AC	TC	ACom	S	EG	AC	TC	ACom
TC01	Object Localization	54	3.63	1.26	1.23	0.70	0.43	3.51	1.20	1.17	0.69	0.44	3.46	1.17	1.16	0.69	0.44
TC02	Hazard Detection	48	3.16	1.05	1.07	0.62	0.42	2.63	0.88	0.85	0.50	0.40	3.90	1.34	1.34	0.78	0.45
TC03	Scene Awareness	51	3.60	1.25	1.20	0.68	0.46	3.62	1.25	1.20	0.70	0.46	4.11	1.41	1.40	0.82	0.47
TC04	Navigation	48	2.26	0.77	0.71	0.42	0.37	2.71	0.93	0.88	0.53	0.38	3.06	1.07	1.01	0.60	0.40
TC05	Social Cues	45	3.55	1.20	1.17	0.70	0.48	3.96	1.38	1.33	0.78	0.48	3.75	1.33	1.23	0.70	0.49

Appendix

Table 10: Question-answering scores per task category. Same format as Table 9 . The lowest-scoring categories are TC07 (2.72), TC06 (2.75), and TC04 (2.80).

ID	Category	$n$	InternVL3-1B					InternVL3-2B					InternVL3-8B
ID	Category	$n$	S	EG	AC	TC	ACom	S	EG	AC	TC	ACom	S	EG	AC	TC	ACom
TC01	Object Localization	82	1.45	0.51	0.45	0.27	0.23	1.94	0.69	0.64	0.38	0.25	2.98	1.02	1.04	0.62	0.33
TC02	Hazard Detection	93	1.43	0.52	0.46	0.26	0.20	1.99	0.73	0.64	0.37	0.25	2.54	0.92	0.86	0.51	0.31
TC03	Scene Awareness	95	1.47	0.54	0.48	0.26	0.21	2.04	0.72	0.69	0.40	0.24	2.77	0.94	0.96	0.58	0.32
TC04	Navigation	88	1.43	0.51	0.47	0.25	0.21	1.91	0.68	0.65	0.37	0.24	2.58	0.91	0.88	0.53	0.31
TC05	Social Cues	75	1.50	0.52	0.50	0.27	0.23	1.98	0.69	0.66	0.37	0.26	2.59	0.90	0.90	0.53	0.33

Appendix

Table 11: Guidance scores per task category. Same format as Table 9 . The lowest-scoring categories are TC06 (2.08), TC10 (2.15), and TC04 (2.17).

Model	Size	Score	EG	AC	TC	ACom
Open-source models (frames $+$ text)
InternVL3	1B	3.14	1.05	1.04	0.61	0.45
	2B	3.29	1.11	1.09	0.64	0.45
	8B	3.46	1.17	1.16	0.68	0.45
VideoLLaMA3	7B	3.11	1.06	1.03	0.59	0.43
Qwen3-VL	2B	3.11	1.06	1.03	0.59	0.42

Appendix

Table 12: Zero-shot results on EgoBlind. Judge scores (0–5) on

n=2{,}565

items with the same judge and rubric as VISTA. EG, AC, TC, ACom: rubric components as in Table 8 ; –: size not disclosed. Bold and underline mark the best and second-best open-source result per column. Absolute scores are not directly comparable with VISTA, since EgoBlind provides more references per item (2.24 vs. 1.01 on average).

Table 13: Failure statistics on VISTA (RGB and text input). Crit.: answers flagged by the judge as containing an error that could cause immediate harm;

\neg

Corr.: answers not judged meaningfully correct;

<

2/5: answers scoring below 2/5; Univ.: items on which no model reaches 3/5; Conflict: answers contradicting the reference but supported by the frames; Unver.: answers with a claim the eight frames cannot verify. Score: mean judge score (0–5). Items: test items per row. Panels (b) and (c) pool all fourteen models.

Clip: TC02_018
Question: Am I walking on the correct path with no obstacles?
Reference: No. You are not on the sidewalk, and there are gates in front of you that block your path.
Model	Score	Response
Qwen3-VL-2B	2.10	Based on the visual information provided, it appears that you are walking on a path that is not entirely clear of obstacles. The path you are on is […]
Gemini 3.1 Flash-Lite †	1.60	Based on the images provided, it appears you are standing on a sidewalk or a paved area with a small, raised metal grate structure immediately to […]
InternVL3-14B †	1.25	Yes, you seem to be walking on a path with no immediate obstacles. The path appears clear and is bordered by a fence on one side. Just be cautious of […]

Appendix

Table 14: Failure case (a): egocentric body orientation. All fourteen responses, ordered by score and truncated to 150 characters. † : flagged by the judge as an error that could cause immediate harm (13 of 14 responses).

Clip: TC10_003
Goal: Retrieve or play with the soccer ball.
Reference: The orange soccer ball is on behind you on your left. Turn around 180 degrees and locate the orange soccer ball. Walk towards it and reach down to retrieve the soccer ball.
Model	Score	Response
GPT-Realtime Mini	2.05	All right! Let’s work together step-by-step. Let’s get started. First, let’s locate your position. You’re in a grassy area with large trees […]
Qwen3-VL-2B	2.00	Based on the provided images, it appears that the person is in a park, likely in Boston, USA, given the architecture and the presence of a large […]
Qwen3-VL-4B	1.95	Based on the images provided, here are clear step-by-step instructions to help the visually impaired person retrieve or play with the soccer ball: 1. […]

Appendix

Table 15: Failure case (b): objects behind the wearer. All fourteen responses, ordered by score and truncated to 150 characters. † : flagged by the judge as an error that could cause immediate harm.

Clip: TC10_079
Question: Am I moving?
Reference: No, you are not moving.
Model	Score	Response
Qwen3-VL-2B	1.10	Based on the visual evidence provided in the series of images, we can analyze the person’s movement. - In the second image, the person is seen […]
InternVL3-9B	1.05	Yes, it appears you are moving. The changing perspective and the position of other people in relation to you suggest movement.
InternVL3-14B	1.05	Based on the images, it looks like you are moving. The changing positions of people and objects suggest movement through the scene.

Appendix

Table 16: Failure case (c): dynamic and temporal reasoning. All fourteen responses, ordered by score and truncated to 150 characters. No response is flagged as a critical error, but every model answers yes where the reference answer is no .

Clip: TC06_094
Question: How far is the trash can?
Reference: The trash can is 10 meters directly in front of you.
Model	Score	Response
Gemma 3-4B	2.30	Based on the images, the trash can appears to be approximately 10-15 feet away from the center of the frame. It’s positioned roughly halfway between […]
Gemma 3-12B	2.25	Based on the images, it appears the trash can is approximately 15-20 feet away from the viewer. It’s positioned across the pedestrian area, closer to […]
Gemma 3-27B	2.10	Based on the images, it appears the trash can is approximately 15-20 feet away. Here’s how I estimate that: It’s positioned within the […]

Appendix

Table 17: Failure case (d): metric distance estimation. All fourteen responses, ordered by score and truncated to 150 characters. Every response is flagged by the judge as containing a claim the eight frames cannot verify.

Assistive Technology for Blind and Low-Vision People

See all 33 papers

Jul 16, 2026

VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance

Visually impaired individuals (VIIs) encounter significant daily challenges due to limited access to visual information. Although Multimodal Large Language Models (MLLMs) have achieved impressive results on general vision and language tasks, their practical utility in real-world blind assistance still remains largely underexplored. To fill this gap, we introduce VIABench, a comprehensive video benchmark specifically designed to evaluate MLLMs in Visually Impaired Assistance scenarios using first-person videos recorded or shared by VIIs themselves. VIABench defines three core tasks, each targeting a distinct requirement in visual assistance. Proactive Reminder: Assesses the model's ability to interpret ongoing video content while proactively anticipating and verbally describing upcoming navigation-critical events; Visual Question Answering (VQA): Evaluates the model's capacity to answer user-posed questions about the environment or objects within the video; Vision-Guided Interaction: Tests context-aware reasoning to accomplish intentional interactions between user and environment. To ensure a robust and fair evaluation, we propose a rigorous benchmarking pipeline that supports both online (real-time) and offline settings. Our experiments demonstrate that current MLLMs still struggle to deliver comprehensive support for VIIs, especially in the Proactive Reminder task, which demands accurate anticipation and real-time responsiveness. We hope VIABench will drive future research toward developing customized MLLMs for real-world assistance, ultimately improving navigation and interaction experiences for visually impaired individuals. Code and data will be released at https://github.com/MCG-NJU/VIABench.

Yunfeng Liu, Yuandong Yang, Jiarui Han +5

1Nanjing University · 2Shanghai AI Laboratory

VLM Evaluation Video Understanding Large Vision-Language Models Assistive Technology for Blind and Low-Vision People Visually Impaired Navigation Assistance

Ask PDF

Sep 18, 2026

Touvigation: Embodied Adaptive Object Acquisition for Blind and Low-Vision Users in Unfamiliar Indoor Environments

Blind and low-vision users often face challenges when locating and physically acquiring objects in unfamiliar indoor environments. Existing vision-language-model-based assistants can provide semantic descriptions but may introduce latency, hallucinations, and guidance that is poorly aligned with embodied action. We present Touvigation, a hands-free object acquisition system that combines vision-language understanding with persistent local spatial modeling to provide low-latency, body-relative guidance. Drawing on formative interviews with eight blind and low-vision participants, we design a multi-stage guidance framework that adapts spatial references as users transition from orienting, to walking, to reaching and tactile verification. We evaluated Touvigation with 12 blind and low-vision participants against a multimodal large-language-model assistant and unassisted search. Touvigation achieved 100% task success, compared with 58% for the multimodal assistant and 85% for unassisted search, while reducing completion time and cognitive workload. Our findings demonstrate how persistent spatial grounding and adaptive embodied guidance can improve object acquisition for blind and low-vision users.

George Xi Wang, Xiangyu Li, Shaoyue Wen +8

Stony Brook University, United States · New York University, United States · Brown University, United States +6

Assistive Technology for Blind and Low-Vision People Visually Impaired Navigation Assistance Embodied Navigation Assistive Robotics

Ask PDF

May 29, 2026

A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation

AI-based Visually Impaired Assistance (VIA) remains challenging, largely due to the high cost of human evaluation. The VLM-as-a-Judge paradigm may offer a promising alternative, although it has mostly been studied in general domains. We therefore ask whether such judges can be trusted for VIA tasks. To investigate this question, we introduce VIABLE (Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation), the first benchmark for VLM-as-a-Judge evaluation in VIA. VIABLE contains over 300K judgment samples across three scenarios and introduces an Effectiveness--Impartiality--Stability framework with a 12-mode failure taxonomy. Based on VIABLE, our systematic study of seven judges across different model scales shows that existing models are largely unreliable across all evaluation axes. The strongest judge, GPT-5.4, achieves only 52.6% single-failure diagnostic accuracy, yet exhibits the highest self-preference rate at 94.2%; while open-source judges are strongly biased and adversarially fragile. To address these issues, we propose VIA-Judge-Agent, a model-agnostic inference-time harness that augments judges with visual evidence extraction and a taxonomy-guided workflow. It enables positive improvements in diagnostic accuracy and downstream VIA responses more preferred by BLV users. Data and code are available at: https://github.com/YiyiyiZhao/VIABLE

Yi Zhao, Siqi Wang, Zhe Hu +2

Department of Computing, The Hong Kong Polytechnic University

VLM Evaluation LLM-as-a-Judge Language Model Bias Evaluation Vision-Language Model-as-a-Judge Visually Impaired Navigation Assistance Assistive Technology for Blind and Low-Vision People

Ask PDF

Abstract

Assistive Technology for Blind and Low-Vision People

Explore similar work

VIABench: A Comprehensive Video Benchmark Collected from Blind Individuals for Visual Impairment Assistance

Touvigation: Embodied Adaptive Object Acquisition for Blind and Low-Vision Users in Unfamiliar Indoor Environments

A Visually Impaired Assistance Benchmark for VLM-as-a-Judge Evaluation