MAP: A Benchmark on Multimodal Accessibility Planning for Real World Places

Jason Armitage; Ioannis Tsochantaridis; Linda Mazzone; Chuqiao Yan; Srini Narayanan; Sarah Ebling

cs.AIAug 28, 2026

MAP: A Benchmark on Multimodal Accessibility Planning for Real World Places

Authors: Jason Armitage, Ioannis Tsochantaridis, Linda Mazzone, Chuqiao Yan, Srini Narayanan, Sarah Ebling

Organizations: University of Zurich Switzerland · Google DeepMind

Abstract

We introduce MAP, the first benchmark to evaluate multimodal AI systems as assistants for users with accessibility requirements when planning visits to places in the real world. In our evaluation, systems are presented with requests to verify or recommend a point of interest meeting an accessibility requirement. MAP contains two novel assessments: Claim verification for accessibility planning assesses if information on places and stated accessibility features is supported and identifies places that satisfy requested accessibility features. Visual evidence retrieval for accessibility planning checks if a multimodal AI system can select visual evidence for the requested place and accessibility feature. Our methodology supports comparison of AI systems in a setting where place information and accessibility information can change over time by evaluating systems and refreshing ground truth data at scheduled times. The benchmark is based on automatic rating and human rating for a proportion of responses.

Figures & tables

Figure 1: Overview of the two evaluation tasks in MAP: (1) Claim verification evaluates the correctness of responses on accessibility-related claims and provides a justified assessment. (2) Visual evidence retrieval evaluates whether a model can find relevant visual evidence that supports an accessibility query. The two tasks are detailed in Subsection 4.1 and Subsection 4.2 .

Figure 2: Inputs for the visual evidence retrieval task described in Subsection 4.2 . Each prompt is paired with a set of $n=20$ candidate images and the evaluated model must select one candidate ID or None . The four key image candidate types are the GT sample in green and three controlled distractors in red, gold, and blue.

	No Search				Search
	Safe	Unverified	Contradictory	NU	Safe	Unverified	Contradictory	NU
Claude Sonnet 4.6	1	898	1	0	69	742	85	4
Claude Sonnet 5	0	900	0	0	85	730	85	0
Gemini 3.1 Pro Preview	146	634	100	20	141	602	134	23
Gemini 3.6 Flash	210	345	340	5	213	432	251	4
GPT-5.2	0	898	0	2	167	643	88	2

Table 1: Results on the test partition are reported for

n=900

feature check prompts in the claim verification task where each prompt asks the system to verify one or more accessibility features in a given place. Labels are defined in Subsection 4.1 and applied using auto-rating. In “No Search” runs, models use the original prompt only. “Search” permits models to retrieve online evidence. Cell entries denote response counts. NU : Not Useful.

	No Search						Search
	US	SU	UU	PUU	UPU	NU	US	SU	UU	PUU	UPU	NU
Claude Sonnet 4.6	15	28	4	40	0	513	99	19	52	117	19	294
Claude Sonnet 5	15	42	22	126	1	394	73	31	92	105	22	277
Gemini 3.1 Pro Preview	93	5	57	212	40	193	89	6	64	87	44	310
Gemini 3.6 Flash	61	0	59	174	47	259	91	0	77	77	55	300
GPT-5.2	17	76	14	85	0	408	141	2	35	126	30	266

Table 2: Results on the test set of the search-by-feature prompts in the claim verification task. Requests to systems are based on identifying places of a specified type in a city area that meet specific access requirements. Performance is evaluated based on verified/supported decisions. “No Search” and “Search” depend on the access of the system to an online search component (e.g., Google Search) for retrieving evidence. Reported values are for labels applied by an auto-rater assessing responses. Output labels are denoted as follows: US (Useful and Safe), SU (Safe but Uninformative), UU (Useful but Unverified), PUU (Potentially Useful but Unverified), UPU (Useful but Potentially Unsafe), and NU (Not useful).

Figure 3: Results for closed-weight models with “Search” enabled on the test partition of the search-by-feature prompts indicating that US (Useful and Safe) accounts for less than a quarter of labels ( $n=600$ ). A marginal fraction of responses by open-weights models are scored as US. Counts are provided in Table 2 . Output labels are denoted as follows: US (Useful and Safe), SU (Safe but Uninformative), UU (Useful but Unverified), PUU (Potentially Useful but Unverified), UPU (Useful but Potentially Unsafe), and NU (Not Useful).

	Safe	Other
Claude Sonnet 4.6	4 / 3	26 / 27
Claude Sonnet 5	6 / 6	24 / 24
Gemini 3.1 Pro Preview	9 / 10	21 / 20
Gemini 3.6 Flash	13 / 12	17 / 18
GPT-5.2	5 / 6	25 / 24
GPT-5.6 Terra	5 / 8	25 / 22

Table 3: Human/auto-rater counts of labels allocated to a set of responses by models with search for

n=30

feature check samples in the claim verification task. Results are for the validation partition and are provided for “Safe” and “Other”, where “Other” combines “Unverified”, “Contradictory”, and “Not Useful” counts.

	CVE	IP- CFA	IP- IFA	NED
Claude Sonnet 4.6	213	25	7	55
Claude Sonnet 5	130	12	3	155
Gemini 3.1 Pro Preview	236	13	5	46
Gemini 3.6 Flash	250	20	2	28
GPT-5.2	95	7	0	198
GPT-5.6 Terra	228	34	10	28

Table 4: Results on the test partition of the evaluation on the visual evidence retrieval task. Results are reported over

n=300

test prompts and all models are run without access to search. Column headings are defined as follows: CVE = Correct Visual Evidence, IP-CFA = Incorrect Place but Correct Feature Area, IP-IFA = Incorrect Place and Feature Area, and NED = No Evidence Detected. Values are the count of scoring labels allocated by an auto-rater to the responses.

Figure 4: Correct visual evidence rates for systems on the test partition of the visual evidence retrieval task. Results indicate a model’s ability to retrieve an image matching the precise accessibility feature in the requested POI. Full counts for correct and other visual evidence labels are reported in Table 4 .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

	No Search				Search
	Safe	Unverified	Contradictory	NU	Safe	Unverified	Contradictory	NU
Claude Sonnet 4.6	0	300	0	0	21	238	40	1
Claude Sonnet 5	0	300	0	0	27	236	35	2
Gemini 3.1 Pro Preview	56	212	28	4	57	185	56	2
Gemini 3.6 Flash	67	116	115	2	77	131	91	1
GPT-5.2	0	300	0	0	46	221	33	0

Appendix

Table 7: Validation results on the feature check questions (

n=300

) in the claim verification task. Cell entries are response counts for labels allocated by the auto-rater.

	No Search						Search
	US	SU	UU	PUU	UPU	NU	US	SU	UU	PUU	UPU	NU
Claude Sonnet 4.6	7	10	2	20	0	161	40	6	13	39	9	93
Claude Sonnet 5	8	10	11	38	0	133	23	5	26	38	10	98
Gemini 3.1 Pro Preview	32	1	20	74	11	62	40	4	14	34	8	100
Gemini 3.6 Flash	18	0	24	64	15	79	34	0	29	32	12	93
GPT-5.2	7	22	4	37	0	130	46	1	11	57	2	83

Appendix

Table 8: Validation results on the search-by-feature prompts (

n=200

) in the claim verification task. Cell entries are auto-rater counts on labels defined in the Verified/supported framework. Terms for the labels are as follows: US (Useful and Safe), SU (Safe but Uninformative), UU (Useful but Unverified), PUU (Potentially Useful but Unverified), UPU (Useful but Potentially Unsafe), and NU (Not useful).

	CVE	IP- CFA	IP- IFA	NED
Claude Sonnet 4.6	82	10	3	5
Claude Sonnet 5	47	6	1	46
Gemini 3.1 Pro Preview	80	5	2	13
Gemini 3.6 Flash	89	8	0	3
GPT-5.2	42	3	2	53
GPT-5.6 Terra	82	10	3	5

Appendix

Table 9: Validation results on the visual evidence retrieval task. Results are reported over

n=100

validation prompts and models are evaluated with no access to search. CVE = Correct Visual Evidence, IP-CFA = Incorrect Place but Correct Feature Area, IP-IFA = Incorrect Place and Feature Area, NED = No Evidence Detected. Cell entries are response counts.

Figure 5: Sample cases from human assessment in the claim verification task. Each case includes a prompt, a response and data on accessibility features from the GT. Green shading and ticks indicate the response is assessed by the human as “Safe”. Human assessment lists observations on the responses and the following models are evaluated in the respective cases: Gemini 3.1 Pro is in Case 1 (Model 1) and Case 3 (Model 2); Gemini 3.1 Pro search is in Case 1 (Model 2), Case 2 (Model 2), and Case 3 (Model 1); and Grok 4.3 is in Case 2 (Model 1).

Multimodal Model Evaluation

Multimodal model evaluation measures how reliably AI systems understand and generate content across text, images, audio, video, and other data.

See all 770 papers

MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs
Feb 20, 2026Ziqiao Shang, Ling-Yue Ge, Zian Xu +10Multimodal Reasoning Multimodal Model Evaluation
National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · AMAP, Alibaba Group +1
VeriTrip: A Verifiable Benchmark for Travel Planning Agents over Unstructured Web Corpora
May 27, 2026Yuting Xu, Jiayi Tian, Jian Liang +4Multimodal Reasoning AI Agent Benchmarks
Institute of Information Engineering, CAS · School of Cyber Security, UCAS · Amap, Alibaba Group +2
MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios
Feb 26, 2026Zhiheng Song, Jingshuai Zhang, Chuan Qin +6LLM Agent Evaluation AI Agent Benchmarks
Computer Network Information Center, Chinese Academy of Sciences · AMAP, Alibaba Group · Independent Researcher

Feb 20, 2026J/K move · Enter open · S save

MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs

Ziqiao Shang, Ling-Yue Ge, Zian Xu, Zi-Jian Cheng+9

National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China · School of Intelligence Science and Technology, Nanjing University, Suzhou, China · AMAP, Alibaba Group+1

Multimodal Reasoning Multimodal Model Evaluation Spatial Reasoning Benchmarks

Open paper AskPDF arXiv

Systematically evaluating Multimodal Large Language Models (MLLMs) is essential for advancing Artificial General Intelligence (AGI). Yet existing benchmarks remain inadequate for rigorously measuring their reasoning capabilities under multi-criteria constraints. To address this gap, we introduce MapTab, a multimodal benchmark designed to assess holistic multi-criteria reasoning in MLLMs through route-planning tasks. MapTab requires models to perceive and ground visual information from map images while integrating route attributes, such as Time and Price, from structured tables. It covers two scenarios: Metromap, spanning metro networks in 160 cities across 52 countries, and Travelmap, featuring 168 representative tourist attractions from 19 countries. Overall, MapTab includes 328 images, 196,800 route-planning queries, and 3,936 QA queries, incorporating four key criteria: Time, Price, Comfort, and Reliability. Extensive evaluations of 21 representative MLLMs show that current models still struggle with multicriteria multimodal reasoning. Notably, when visual perception is unreliable, multimodal reasoning can even underperform unimodal approaches. MapTab therefore offers a challenging and realistic testbed for systematically evaluating and advancing MLLMs across core perception, integration, numerical comparison, and route planning capabilities.

Abstract

Multimodal Model Evaluation

Explore similar work

MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs

VeriTrip: A Verifiable Benchmark for Travel Planning Agents over Unstructured Web Corpora

MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios